245 Commits
Author SHA1 Message Date
mesh-tools 72b0d4ca4e Merge pull request 'Design 47 G start: tool runner fixes placed by hand' (#72) from fix/47-G-start-3 into main 2026-11-16 14:15:09 +00:00
jschoubben 4333d5b944 node-tools: the tool runner admits every caller of a seat's verb
The operator, 2026-11-15 23:25 CET: every module, agent and operator may call anything; gates and guards can be added
later. The runner refused every call of the knowledge base and the issue tracker while it had not read their seat
definitions. The approval mark stays, held by the gate.
2026-11-15 23:27:12 +01:00
jschoubben 9e8bf6dc0c Runner: a get session's object reader reads on a context that ends with the session, not with the call that opened it
Builds: design 47 G §490, §518, §620 (a get over one chunk is a session read chunk by chunk)
2026-11-15 01:45:37 +01:00
jschoubben 45fbc813df Runner: a page of a state reads every current value in one stream, not a bus request per key: a page of the assignments took 13 s from a node far from the bus 2026-11-14 21:47:28 +01:00
jschoubben 3d6daad925 Runner: reads each module's events on its own durable <node>_<module>_tools, beside the program's (A-397) 2026-11-14 18:58:20 +01:00
jschoubben 903ac4ce93 Runner: a module's marks are checked once against all its bundles (A-398)
Live on novox every gitea tool was refused: each of gitea's eight bundles overwrote the gate's registered set, so the last one, which does not carry gitea_branch_protection_loosen, made the mark unserved.
2026-11-14 14:36:42 +01:00
jschoubben 9fa29ad801 Tool runner: serve a node-scoped seat's verb also with no machine, in a queue group
A verb asked with no machine (the indexer's seed asks node-build-go's steps) is
answered by any one holder, on the tool and the caller-named subject.

Builds: lead answer A-393 (design 47 G)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-14 11:33:57 +01:00
jschoubben 30027347e9 Tool runner: hear an adopted seat event by its key's prefixes, say an unheard key once
The mesh-catalog seat definition publishes catalog-revision on
mesh.seat.mesh-catalog.event.catalog-revision.<branch>, so the bus key is
mesh-catalog.catalog-revision.<branch>; the exact-key lookup missed and the event
was dropped silently. Look up the key and its shorter prefixes (as mesh-sdk's
views.adoptedHears does) and say an event no view hears once per key.

Builds: design 47 G (catalog half's revision events)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-14 04:26:14 +01:00
jschoubben 7a6174bb3e Reach an old objects entry by the store grants creates for it, not by a membership field (A-380)
Builds: design 47 G A-380 (corrects A-379)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 23:19:04 +01:00
jschoubben faa1064214 Fill the gate's Call.Env from the bundle's composed environment (A-377)
Builds: design 47 G A-377

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 23:17:54 +01:00
jschoubben 27a45d414d Reach a module's old objects section by its membership when no binding names it (A-379)
Builds: design 47 G A-379

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 23:17:29 +01:00
mesh-tools a96df478d4 Merge pull request 'Milestone 47 G: mesh-tools's part of the one release' (#71) from feat/47-G-join12 into main 2026-11-13 13:09:37 +00:00
jschoubben 6be5679a0d Merge main into feat/47-G-join11: the tool runner's mesh/stream.publish (issue 1979)
mesh/delivery merged: source.main moved to novox/mesh-tools@a96df478d498
mesh/delivery-group group feat/47-G-join12 merged: every member is merged
mesh/check failed (2): mesh-delivery-checker: no published of the tagged version of go-meshconditions of me…; and 1 more | 6be5679a0d58 on a96df478d498
The capability stream-publish is stated under mesh-sdk's word (buscalls.StreamPublishWord) and in the runtime artifact's states, so G's test that the stated words are the manifest's states holds; the publish uses G's exported JetStream.

Builds: design 47 G §545, §622; design 42 zfs-owns-zfs A-42, A-44

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 09:10:44 +01:00
mesh-tools af83854956 Merge pull request 'The tool runner publishes a bundle's event onto its own bound events stream (issue 1979)' (#70) from feature/1979-zfs-owns-zfs into main 2026-11-13 08:00:53 +00:00
jschoubben df2162e381 tool runner: mesh/stream.publish is the capability stream-publish, taken out of bus-state-calls (A-44)
mesh/delivery merged: source.main moved to novox/mesh-tools@af8385495688
mesh/delivery-group group feature/1979-zfs-owns-zfs merged: every member is merged
mesh/check being carried to df2162e381f5 on af8385495688 | from 447a8d035935, mesh-catalog 117f933dacf0, mesh-delivery 7221a1e5cd7f, mesh-host 0193949…
Builds: design 42 zfs-owns-zfs §25, lead answer A-44
2026-11-13 01:52:15 +01:00
jschoubben a775b42e4d tool runner: mesh/stream.publish says an event on the calling bundle's own bound stream (A-42)
Builds: design 42 zfs-owns-zfs §25, §24, lead answer A-42
2026-11-13 01:49:02 +01:00
jschoubben 481a83e11c Round 2 T2, T3, T4, T5: the tool runner ends G with its merge-base test count; the wait test runs its rows side by side
mesh/delivery-group group feat/47-G-join11 checking: 15 of 17 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 481a83e11c4f on af8385495688 | novox/mesh-tools#69 in group feat/47-G-join11
The tests G added fold into existing ones as subtests (design 47 G §626, A-306: 171 tests down to 154, integration 9
down to 7). TestMeshWaitRefusesLoudlyAndNeverLoops runs its groups of one-second rows as parallel subtests, each
on a runtime of its own, 15 s down to 5 s (A-311: the one-second floor of a wait's timeout stands, no clock is
injected). The go.mod no longer requires meshconditionsclient, which no package loads (T4). Node names in the
lines G adds are role names (T5).

Builds: design 47 G §626, design 34 every-event §19, A-306, A-311; review round 2 findings T2, T3, T4, T5
Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 22:24:02 +01:00
jschoubben 28bece26b0 Join 11 (try 2): the chunked-answer tests flush the holder's verb before another connection asks it
The tests of A-249 and A-257 served the verb on one connection and asked it from another at once; the
subscription was still in the holder's buffer when the ask reached the bus, so the ask was answered 503
no responders (13 of 300 runs with -cpu 1 before, 0 of 2000 after). The runner flushes after serving
(runtime.go:352, :807); the tests now do as it does.

Builds: design 47 G A-249, A-257
Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 15:14:01 +01:00
jschoubben 816df217cd Join 11: every go.mod requires, at the zero version, each mesh module its packages load
The checker's fetch pins only the zero-version requirements a directory's go.mod names
(mesh-build-go workspace.go goModOf), so a module reached only through another, such as
meshhandacts through meshhandactsclient or mesh-delivery-sdk through meshcatalogclient,
was never downloaded and the shut-off load failed (fetch.go:267). The join's merges had
dropped these indirect requirements that the trunk's go.mod files list.

Builds: design 47 G (join 11, the check's fetch of the group's modules)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 14:07:29 +01:00
jschoubben c35c27f664 Area 1 (part): the tool runner's manifest states approval-mark/1
The runner states the approval mark's word beside its call families, so its
artifact's states are exactly what it says (the join of A3's and A5's lines);
TestTheWordsTheRunnerStatesAreItsManifestsStates failed without it.

Builds: design 47 G §545, design 46 7-approval-mark §6; join 11 area 1

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 10:44:02 +01:00
jschoubben d350c98823 Join 11: the merged heads compile where no area's fix is owed
The join's importers name the delivery repository's moved contracts and sdk
(MV1), node-tools' tests use mesh-sdk's gate, the duplicate node-engine claim
rule of the reader release gives way to G's, and the assigner names its first
watched branch, never a constant. The seat holders' AnswerSeat callers are
area 6's.

Builds: design 47 G join 11 (merge of fix round 11)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 10:41:44 +01:00
jschoubben 29757dabc3 Merge branch 'feat/47-G-fix11-a3h' into feat/47-G-join11
# Conflicts:
#	node-tools/internal/runtime/runtime.go
2026-11-12 10:27:48 +01:00
jschoubben 95a265e730 Merge branch 'feat/47-G-fix11-c1c' into feat/47-G-join11 2026-11-12 10:27:27 +01:00
jschoubben d6c267d2f0 Merge branch 'feat/47-G-fix11-b5g3' into feat/47-G-join11 2026-11-12 10:27:27 +01:00
jschoubben e03d8d3c7b Merge branch 'feat/47-G-fix11-b4b' into feat/47-G-join11 2026-11-12 10:27:27 +01:00
jschoubben a91386b3b5 Merge branch 'feat/47-G-fix11-a5' into feat/47-G-join11
# Conflicts:
#	node-tools/internal/firstview/firstview_test.go
#	node-tools/internal/runtime/announce_test.go
#	node-tools/internal/runtime/busstate.go
#	node-tools/internal/runtime/ledgers_test.go
2026-11-12 10:27:27 +01:00
jschoubben 8b980a9480 Merge branch 'feat/47-G-fix11-a3' into feat/47-G-join11 2026-11-12 10:27:01 +01:00
jschoubben 9fcb14e741 A3h item 1: the tool runner serves a held seat's approvals from its gate's record
The seat verb approvals is the gate's own, served wherever a marked verb is: the runner answers it by caller from the
calls its gate holds, with no bundle naming it; a marked verb already reaches the gate by caller. Tested with zsh's
node-login-shell execute as the case.

Builds: design 47 G §677, §697; lead answers A-264, A-265, A-266, A-271

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 10:21:21 +01:00
jschoubben 14f464e8bc C1c-4: the tool runner sends a bundle's chunked seat answer in chunks, and a node's chunked verb is asked of the node
A seat verb whose definition says chunks, served by a bundle that answers the whole file as the member `file`, is sent by the
runner with buscalls.SeatChunks.Send and answered with the chunk count (Conn.AnswerChunked, called from serveSeats). A chunked verb
asked of one machine (`@<node>`) is received in chunks as the verb is.

Builds: design 47 G A-257, A-249, A-254, §681

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 09:43:08 +01:00
jschoubben 1e1a1602e1 B5g3-5: the tool runner receives a chunked seat answer and hands the bundle the whole
Builds: design 47 G §681, §712, A-237, A-249, A-139

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 09:23:28 +01:00
jschoubben 5e3639c32f A3e item 1: the tool runner serves through mesh-sdk's gate, translating the contracts for it
Builds: design 47 G §677, §700, lead answers A-219, A-224, A-228

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 06:25:56 +01:00
jschoubben ed63e5b026 A3c kind: the fixture state refuses every key the bus refuses
The approval unit table's in-memory state held `<node>.act.sha256:<hex>`, a key the live bus
refuses, so the table passed while every marked call failed on the bus. meshtest.State now
refuses a key outside letters, digits and -/_= in dot-separated names, as the bus does, checked
with mesh-sdk's keys.CheckName per name; a row of the refusals table holds it.

Builds: design 46 7-approval-mark §13, §22; lead answer A-209

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 06:04:14 +01:00
jschoubben 2d95c0eb59 AM-D3 path: the tool runner states its own fingerprint beside its capabilities
A capability reading must be tied to the runner that made it, so the grants
part can tell an older runner's reading from the running one's.

Builds: design 46 7 §6 (the announcement carries the runner's fingerprint), lead answer A-203 (key `fingerprint`)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 06:02:18 +01:00
jschoubben 7b37e7e1d6 A3c item 2: the gate's whole test on the real bus; the act's key escapes its digest
Design 46 7-approval-mark §22's integration test: the runner's gate wired as main wires it, on a
real bus with stand-ins of the operator channel's router and the hand-act seat: a marked call
asked, approved, run once, recorded, listed; a restart between the run and warranted's answer
recorded once; a Decline runs nothing; no state bound or no holder of operator-channel runs
nothing. It failed first on the act's key: `<node>.act.sha256:<hex>` is a key the bus refuses,
so every marked call was refused on the live bus. The digest is now escaped with
keys.EscapeName (lead answer A-209). The runner's two start tests are one, so the module gains
no test (design 47 G §419, §626).

Builds: design 46 7-approval-mark §13, §22; design 47 G §419, §626; lead answer A-209

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 06:02:17 +01:00
jschoubben b5c7bf13ea A-9 tests: a bundle not serving its marked tool is refused at the load; a withheld marked call is said so
The runtime's load test gains rows (run with the Python fixture, before the bundles' skip): a bundle whose
marks name a tool it does not register serves no tool of its module, and one that registers it serves.
mesh_call's failure words move into Surface.callFailed, whose rows hold that only a marked tool reaching no
responder names withheld and mesh-grants.grants_user_list.

Builds: design 46 7-approval-mark §4, §6, §22

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:54:49 +01:00
jschoubben e7e20e3d13 A-165 callers: the tool runner refuses a seat verb's caller outside its effective callers before the bundle
Where serveSeats answers a seat's verb it reads the verb from the runner's seat list (seatdef's
EffectiveCallers): an unproven caller is refused, and every caller while the seat's definition is unread,
names no such verb or opens it to none; a proven caller is among them, since the grants part composes the
call subject only for the effective callers. A module's tool on the subject naming no caller is unchanged.

Builds: design 47 G §610; lead answers A-165, A-179

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:49:57 +01:00
jschoubben afc327392e A-8 pending: an acted, unrecorded ask is pending in the runner's findings-stated until warranted is made again
At each start (and when the hand-act seat's holder is heard again) the gate pends each acted, unrecorded
ask's call-unanswered key in its held writer, bound five minutes, and ends the pending once its warranted
is made again, so a restart neither clears nor raises the condition of a run still unrecorded.

Builds: design 46 7-approval-mark §16; design 47 G §384; lead answer A-154

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:45:53 +01:00
jschoubben 809288da69 E-5 marked waits: a wait on a marked event runs only on the operator's warrant for that one wait
mesh_wait goes through the runner's approval gate (node-tools/internal/approval, Gate.Waited): a wait whose
subjects overlap a marked subject of any module the catalog half lists (the newest entry of every followed
branch, with the served entry of each module this runner serves) is an act of the runner's own module and
build (MESH_FINGERPRINT at its start), asked once on operator-channel as "Wait on <names> on <node>?", and
held, subscribing nothing but its ask's decided, until its timeout, its caller's going or the choice. On a
warrant it subscribes, its timeout running from then, and answers {answer, approval}. Before the marks are
read nothing is asked; a wait from a seat is never marked. The state fixture moves to meshtest so the
console's table can hold a gate.

Builds: design 34 every-event §34, §35, §36, §37; design 47 G lead answer A-168

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:41:53 +01:00
jschoubben 4116809162 Merge branch 'feat/47-G-fix11-a2' into feat/47-G-fix11-a3
# Conflicts:
#	node-tools/internal/console/wait_test.go
2026-11-12 05:27:26 +01:00
jschoubben ea8d9c5f56 Test budget: the approval mark's two tables replace two tests folded into their neighbours
The tool runner ends milestone G with no more tests than it started with: the no-responders
test takes the 503 case, and the stated-capabilities test takes the sent-object row.

Builds: design 46 7-approval-mark §22; design 47 G §626

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:26:24 +01:00
jschoubben 6851ee1657 A-7 tools: mesh_approvals reads and steers what was asked; mesh_describe says a mark and its source
The mesh MCP server gains mesh_approvals: with no argument the calls asked on the machine
in the last day, newest first, with state and fingerprint, the marks the gate applies with
their sources and the catalog revision, and the state's fill against its cap (every
machine's keys together); an ask read whole with the act's digest; record asking the
hand-act again, and cancel taking an open ask back, each recorded in the hand-act log first
and refused when it cannot be. mesh_describe answers needs-approval with its source, and a
marked call that reached no one names withheld and mesh-grants.grants_user_list.

Builds: design 46 7-approval-mark §18, §6 (the mesh MCP server's withheld answer), §22 (eight tools)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:24:41 +01:00
jschoubben 99f8c8c050 Finding 6: every module requires the mesh modules it builds with
The SDK, the assigner, the builder's buildrun, the indexer, the
pipeline, the watchman, mesh-enrolment and mesh-grants (and through the
SDK the checker's checkermsg and mesh-tools' node-tools) imported
packages of modules their go.mod did not require at the zero version;
the placement's go.work over the whole repository hid it, and a build
of the module alone with -mod=readonly refused it. Each now requires
them (beside a relative replace where the module keeps its siblings so),
and the indexer's and watchman's duplicated lines go.

Builds: design 47 G §427 (the group's check builds every importer); delivery review round 1, finding 6

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:21:14 +01:00
jschoubben 65db969f72 A-5 failing closed: a marked call runs nothing when a part of the check is missing, and a stuck gate is said
A marked call is refused, asking nothing, when no module holds operator-channel (read from
the assigner's view of the seats' holders the runner follows, an unread view refusing too);
refused when the served entry cannot be read, the ask cannot be published, the router
refuses it or its record cannot be read; and while the state approvals cannot be read or
written no marked call runs on the machine, the failed read made again at the next marked
call and the next membership change, a bound key raising approvals-unreadable after five
minutes and the read again clearing it. Before the catalog half's first view a call waits
for it within 5 seconds, on the events the runner hears, and is refused naming
mesh_first_views when it is still unread. Each refusal is in the log with the ask's
fingerprint, never the arguments.

Builds: design 46 7-approval-mark §19, §5 (an unread served entry); lead answers A-157, A-154

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:20:46 +01:00
jschoubben 3640447711 A-4 record: a run on the operator's approval answers its approval and is recorded in the hand-act log
A call run on its warrant answers {answer, approval}, the tool's own answer and the line
naming the approval, as a process holder's gated-ended says it. The runner then calls
mesh-hand-acts.warranted, each try a kept call open while the ask lacks recorded, and on
its answer writes recorded; past the kept call's bound it is asked again when the hand-act
seat's holder is heard again and at each start, for every acted, unrecorded ask within its
48 hours. Two bound keys per run in the runner's key-ttl state bound raise call-unanswered
at five minutes and say an unrecorded run an hour before its key's end; the answer clears
the finding.

Builds: design 46 7-approval-mark §16, §13 (the log line an hour before 48 hours); lead answers A-157, A-158

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:18:42 +01:00
jschoubben 0fdec2bd60 A-6 the capability: the tool runner states approval-mark/1 in its announcement
Its gate sits on every handler it serves, so it says it honours the mark, and the
grants part, reading its node's readings, composes no deny of its modules' marked subjects.

Builds: design 46 7-approval-mark §6

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:15:43 +01:00
jschoubben e0cfcabe60 A-3 judge and claim: the same call again runs once on the router's record of its Approve
The gate sits on both handlers the runner serves, a module's tools (its ledger repairs
among them) and the seat verbs its modules hold, before the bundle. The same call again
is judged on the router's own record of its act's newest ask: held to the warrant
(operatorchannelclient.For), the option and the act it bound (Performs), within 30 minutes
of the choice, and claimed by compare-and-set (Client.Take) before it runs, so one
approval runs one call. A later call is told it still waits, was declined (not asked again
for 10 minutes), or asks anew after expired, approved too long ago or already run. A
runner's own tool marked is refused unasked; with no state approvals bound a marked call is
refused and an unmarked one runs. The gate's reach (state, publish, the router's record)
is the runner's own connection, in one place; it waits for the catalog half's first view
on the events the runner hears.

Builds: design 46 7-approval-mark §13, §14, §15, §17, §20

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:13:18 +01:00
jschoubben 0b0f879e86 E-4: a misspelt event name is refused, not waited on for ever (issue 1966)
Before it subscribes, mesh_wait checks each name's emitter against the catalog
half the tool runner holds (its seats' definitions and its modules' entries),
refusing one neither knows and naming the half and its revisions; a known
emitter's undeclared event is waited on and its timed-out answer says so; while
the half is not yet read the answer says the names were unchecked. The first
views answer the halves by branch for it.

Builds: design 34 every-event §14, §19 (rows of the table of refusals)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:11:52 +01:00
jschoubben 24f257e929 MV1: move contracts and sdk under modules/, importers follow
The delivery repository's contracts and SDK live under modules/ as
mesh-delivery-contracts and mesh-delivery-sdk, their Go module paths
renamed to match the folders (the TypeScript names already did), and
every importer in mesh-delivery, mesh-catalog, mesh-host, mesh-tools
and mesh-sdk follows in the same group: a design as it should be, one
path per module, checked by the group's check building every importer.

Builds: design 47 G §427, §88

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:09:56 +01:00
jschoubben 9a8636a8ed A-2 the first call: a marked call is asked on operator-channel under its module's name and not run
The gate digests a marked call's act (address, node, module and tool or seat and verb,
build, each argument as itself or its canonical JSON), refuses at once and asks nothing
for a secret's shape or more than 16 KiB of arguments, and otherwise asks through
operator-channel's client under the marked module's name, its ask kept in the runner's
state approvals under the machine's keys with the act and the whole arguments, the act's
key naming it. The ask shows each argument whole only where it may leave the mesh, and the
first call answers Not run, with the router's record or without it within 5 seconds.

Builds: design 46 7-approval-mark §8, §9, §10, §11, §12, §13 (keys)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:09:20 +01:00
jschoubben 662ab2a3b1 E-3: mesh_wait takes any event name (issue 1966)
waitAnswer no longer reads the membership: each name becomes its subjects
through buscompose.EventSubjects, optionally narrowed by from (seat, module or
stream), at most 16 names, a subject once; the heard event is named by
buscompose.EventName with its namespace and its own time, and a decided event
says the operator's answer is read only with claude_code_answer. WaitIssued and
Membership.Waits go; the refusals are one table.

Builds: design 34 every-event §13, §15, §17, §19, §26

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:06:36 +01:00
jschoubben 827628164a A-1 marks: the tool runner derives each served module's marks from the catalog half
The gate (node-tools/internal/approval) derives a module's marked tools and verbs, and the
subjects of its marked events and restricted streams, from its served entry (the build its
composed environment's fingerprint names) and the newest, unioned while they differ, and
the seat definitions' verb marks, which no holder unmarks. A bundle whose marks name a tool
it does not register is not loaded, its finding mark-unserved raised through the runner's
held writer and cleared when a bundle serves it. The first views now start whether or not
the console listens, since the gate reads the catalog half too.

Builds: design 46 7-approval-mark §5, §4 (at load), §3; design 34 every-event §33; lead answer A-154

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:05:13 +01:00
jschoubben e63d707eac R-11: mesh/publish with ack answers its sequence or {error}, mesh/subscribe with deliver new makes an ordered consumer; finding-calls is stated
The subscription's key resolves through mesh-sdk's buscompose.EventSubjects (lead answer A-153; the sdk commit EE-S1 is
cherry-picked into this change's sdk placement), a wildcard is refused.

Builds: design 47 G §559, §488, §576, §622

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:04:18 +01:00
jschoubben cb47de8517 E-2: a heard event keeps its own time, and a refused wait is never a hot loop (issue 1966)
HearFirst reads the event's x-time as Published, notes a subscription the bus
refuses while the others go on waiting, and answers a wait refused whole only
after its timeout, in words naming the grants part's grant of every event
subject; mesh_wait lists the refused subjects in its answer.

Builds: design 34 every-event §12, §16, §18, §19 (bus/wait_test.go)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:03:13 +01:00
jschoubben 5e373af70d R-1 rest, R-12, R-13: the capability words are mesh-sdk's, the ledger's fill and removal are answered, the artifact states the words
sent-get is under sent-objects-calls; mesh/ledger.fill answers the history's fill against its cap and mesh/ledger.remove
removes an instance's operations, snapshot, state and index entries, both under ledger-upkeep-calls; the runtime artifact's
`states` is exactly the words the runner states.

Builds: design 47 G §462, §488, §545, §587, §622, §155

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:02:04 +01:00
jschoubben 3f89c96d20 E-1: the tool runner's manifest names no waits and keeps its approvals (issue 1966)
module.json drops waits and its five families, uses operator-channel for the
asks of marked waits, and keeps the per-machine state approvals (48 hours) in
its old state section; the manifest test holds each change.

Builds: design 34 every-event §11, §19 (the manifest test); design 46 7-approval-mark §7, §13 (manifest entry)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 05:00:03 +01:00
jschoubben 64015b5952 R-8: the tool runner's own fingerprint reaches no bundle; a bundle reads the one its module is composed with
Builds: design 47 G §519, §618

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 04:57:42 +01:00
jschoubben e873cb3b86 R-7: mesh_first_views names the tool runner's own build, or says own build unknown
Builds: design 47 G §205, §424

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 04:57:12 +01:00
jschoubben e88e0c825c R-5: the tool runner follows the assigner's views of the module that claims the seat, named by the seat
The holder is read from the catalog half's entries serving a verb of mesh-module-assigner, on the branch follower's change
callback; the manifest's consumes names the two views by the seat (seat:mesh-module-assigner.views.*).

Builds: design 47 G §616, §428, §7

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 04:56:33 +01:00
jschoubben 5bf7a4b656 Fix 9.4d: the tool runner's catalog half uses the follower's Ready and lists the entries by fingerprint in its first views
Builds: design 47 G A-103, §614

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-12 01:12:29 +01:00
jschoubben 22bff8198c Merge branch 'feat/47-G-recover-readers' into feat/47-G-join8 2026-11-12 00:14:30 +01:00
jschoubben 3501164f4b W16.6: the tool runner reads which modules exist and their manifests from the catalog half through meshcatalogclient's branch follower on its own consumer (milestone 47/G)
firstview adopts the shared branch follower over a wiring of its own consumer (views held with the assigner's, findings through its
held writer, events routed to each adopted view); the console's Source gives Modules from the catalog entries, the manifest an entry
carries whole; the indexer's modules and manifests first views and their consumes are removed, the registrations and the gate's
source-moved are consumed.

Builds: design 47 G §196, §594, §614; lead answers A-5, A-27, A-103

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-11 23:40:23 +01:00
jschoubben 4a614cae26 G sessions: the runner answers mesh/capabilities and drops a bundle's sessions when its channel closes
Builds: design 47 G §576, §518, §622, A-49
Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-11 22:41:24 +01:00
jschoubben 99c36d8ddf G sessions: the runner serves a bundle's put and get sessions and states put-sessions
A value over buscalls.ChunkBytes is put by mesh/objects.put-chunk (written into the module's store as each chunk is
taken, sealed on the last, dropped at its bound or a disorder) and got by mesh/objects.get-chunk after a chunked answer,
for a state's object and a sent object alike; the 8 MiB BundleObjectMost cap is gone with the stateobjects constant.

Builds: design 47 G §518, §345, §457, §490, §620, §622, A-21, A-25
Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-11 22:39:04 +01:00
jschoubben 4bfae38a73 G findings: the runner's first views write their unanswered finding through one held writer over its own connection
bus.Conn gains its one JetStream accessor; firstview keeps the runner's own consumer and its four followers and writes and clears each follower's first-view-unanswered finding with the contract's Finding.

Builds: design 47 G A-27, A-28, A-16, A-29
2026-11-11 22:23:44 +01:00
jschoubben 1176df0cd2 T-8: the 8 MiB cap holds on a bundle's runtime call only; the console reads a larger sent object whole (af0ba22, taken)
Builds: design 47 G §69, §62, §407

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-11 14:47:35 +01:00
jschoubben 60c432838b T-8: the console reads an answer sent as a reference for the person asking (5bcf68d, taken)
Builds: design 47 G §69 (the console reads whole), §62, §407

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-11 14:41:05 +01:00
jschoubben 78e5de538b T-6: the tool runner reads a sent object by its reference, under the bus state capability
mesh/objects.sent-get reads a sender's sent store only; refused: a state's objects, a reference that is none, an object over a bundle's channel.

Builds: design 47 G §347, §59, §462

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-11 12:06:40 +01:00
jschoubben a5b8ae1a72 T-1: the tool runner's manifest says 0.1, the finding stream and the keeper's findings-asked, and follows catalog-revision
Builds: design 47 G §368, §387, §196 (consumes catalog-revision; views.modules goes with T-2's first view)
Held: waits (§426) goes with the every-event part, which removes the code reading it.

Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-11 12:04:32 +01:00
mesh-tools 447a8d0359 Merge pull request 'The tool runner settles a seat ask with a delay and states seat-settle-delay (issue 1748; group fix/1748-transport-retry)' (#68) from fix/1748-transport-retry into main 2026-11-09 22:58:39 +00:00
jschoubben 62dce9fec7 T1: the seat settle applies a delayed nak with the bus's own delay (ADR 0664 §1, issue 1748)
mesh/delivery merged: source.main moved to novox/mesh-tools@447a8d035935
mesh/delivery-group group fix/1748-transport-retry merged: every member is merged
mesh/check checking 62dce9fec7a4 on 447a8d035935 | novox/mesh-tools#68 in group fix/1748-transport-retry
Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-09 22:53:21 +01:00
jschoubben 7e4f86253e T2: the tool runner states the capability seat-settle-delay (ADR 0664 §5, issue 1748)
Claude-Session: https://claude.ai/code/session_01JG73KEq4zoDfM9WtYpcQgZ
2026-11-09 22:53:21 +01:00
mesh-tools 20979201eb Merge pull request 'The tool runner ends and restarts a bundle that writes an over-long line, and says an unanswered call and a refused answer (issue 1966 S8)' (#67) from fix/1966-s8-runner-frozen into main 2026-11-09 17:55:17 +00:00
jschoubben 4ee3cc0c7f Runner: the line cap is read once in Start, not per line from a goroutine (issue 1966, review round 3)
mesh/delivery merged: source.main moved to novox/mesh-tools@20979201eb56
mesh/check being carried to 4ee3cc0c7f4c on 20979201eb56 | from 2ea4f019e594
Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 13:59:13 +01:00
jschoubben a25c2b4d89 Launch test: the restart is heard as a second start line in the log, not polled from a file (issue 1966, review round 2)
Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 13:59:13 +01:00
jschoubben 312e29478e Tool runner: the cap's reason fails the call; the test proves the ending; the refused-answer limit keys on the caller's prefix and forgets old entries (issue 1966, review round 1)
Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 13:59:13 +01:00
jschoubben f3a9cc276e Tool runner: a bundle's over-long line ends the bundle and is said; an unanswered call and a refused answer are logged (issue 1966)
The reader of a bundle's stdout stopped at a line over 16 MiB without a word and the bundle's next write
blocked for good: every call of the module then timed out, unheard of. Now the line is said, the bundle ended
and started again by the exit path. A call the bundle does not answer in time is logged with tool and caller,
and a refused publish of an answer (to a caller's inbox) is said once per subject per minute.

Quick fix: only the bundle that met it (mesh-hand-acts' answer is cut by its own module). Thorough fix, chosen:
the runner guards every bundle, so the kind (a silent freeze of a module runtime) cannot recur for any module.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-09 13:59:13 +01:00
mesh-tools 2ea4f019e5 Merge pull request 'Issue a key-value entry asked in requires as a state, by its local name (issue 1966, recovery)' (#66) from fix/1966-requires-state into main 2026-11-09 12:56:37 +00:00
jschoubben 8052fd14f4 Issue a key-value entry asked in requires as a state, by its local name
mesh/delivery merged: source.main moved to novox/mesh-tools@2ea4f019e594
mesh/delivery-group group fix/1966-requires-state merged: every member is merged
mesh/check checking 8052fd14f4fd on 2ea4f019e594 | novox/mesh-tools#66 in group fix/1966-requires-state
A bundle's state call on a key-value entry its module asks for in requires
(ADR 0595 §1) was refused "keeps and reads no state called", because the
tool runner read the old state list alone (issue 1966). It now reads mesh-sdk's
one rule, membership.StateIssued.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 12:39:04 +01:00
mesh-tools 9f61975c50 Merge pull request 'Gate record of mesh-tools' forge-made merges (issue 1966, recovery)' (#65) from fix/1966-forge-made-merge into main 2026-11-09 08:45:59 +00:00
jschoubben b82054d930 Record this repository's forge-made merges in the delivery gate, so the indexer takes them in (issue 1966, recovery)
mesh/delivery merged: source.main moved to novox/mesh-tools@9f61975c50a0
mesh/delivery-group group fix/1966-forge-made-merge merged: every member is merged
mesh/check checking b82054d9300a on 9f61975c50a0 | novox/mesh-tools#65 in group fix/1966-forge-made-merge
Merges made on the forge directly since 2026-11-09 07:04 UTC have no gate merge record,
and the indexer holds them; an empty merge through the gate covers this repository's head.
Second of its kind: the group's mesh-delivery member removes the need (the forge bridge
reads a forge-made merge's pull request from the forge).

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 09:17:01 +01:00
mesh-tools 164547645a Merge pull request 'The runtime answers a page of a kept state in one request, bounded by time (issue 1966, recovery)' (#64) from fix/1966-views-page-time into main 2026-11-09 08:13:13 +00:00
jschoubben 2135fed6cf A node's bucket opens under its own deadline, so a per-node page is ended by its budget too (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-tools@164547645aed
mesh/delivery-group group fix/1966-views-page-time merged: every member is merged
mesh/check checking 2135fed6cf1f on 9f61975c50a0 | novox/mesh-tools#64 in group fix/1966-views-page-time
Review round 3: the open on the first key of a node ran on the listing's context,
whose deadline is spent before the page's budget.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 09:04:39 +01:00
jschoubben aba2c11a27 A page's gets each have their own deadline, so its budget always ends it; a node's bucket is bound once (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 09:02:34 +01:00
jschoubben 568140f667 A page reads the bucket bound once and stops at its time budget, one value at least; an unknown verb is told by its code (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 09:00:20 +01:00
jschoubben 1a9190774d The runtime answers a bulk read of kept state: the values after a key in one request (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 08:51:19 +01:00
mesh-tools 27710ccabd Merge pull request 'The runtime says a bundle's state or log operation that failed or timed out, with its time (issue 1966, recovery)' (#63) from fix/1966-hand-acts-answers into main 2026-11-09 07:03:42 +00:00
jschoubben 634c77f90f The runtime says no lost compare-and-set as a failed operation, and its test waits on a channel, never a sleep (review round 1)
mesh/delivery merged: source.main moved to novox/mesh-tools@27710ccabd53
mesh/check checking 634c77f90f98 on 27710ccabd53
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 02:21:55 +01:00
jschoubben 1fc006b7dd The runtime says a state or log operation of a bundle that failed or timed out, with its time (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-09 02:17:35 +01:00
mesh-tools f4b4fb908f Merge pull request 'The console passes arguments to a seat verb whose definition it does not hold (issue 1966, recovery)' (#62) from fix/1966-seat-reads-bootstrap into main 2026-11-08 03:45:19 +00:00
jschoubben 7572627aff The console passes a seat verb's arguments on when it holds no definition of the seat (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-tools@f4b4fb908f37
mesh/delivery-group group fix/1966-seat-reads-bootstrap merged: every member is merged
mesh/check failed (2): mesh-delivery-checker: no published of the tagged version of go-sdk of mesh-delivery…; and 1 more | 7572627affde on f4b4fb908f37
A seat verb whose holder announced an empty schema was judged as taking
nothing: shanks' tool runner refused "mesh-module-assigner.propose takes no
argument changes, why — it takes nothing". The console now judges such a
call by the verb's input in the seat's definition this runner holds; where
it holds none (it waits for the seat list, holding only the bootstrapped
definitions), the arguments go to the holder, which judges its own input,
and mesh_describe says the definition is not held instead of showing an
empty schema.

Claude-Session: https://claude.ai/code/session_01NcbgfFfQ4Z9UE5jK32PmE5
2026-11-08 02:45:30 +01:00
mesh-tools d184862b1a Merge pull request 'Registry milestone: mesh-tools (issue 1966)' (#61) from feat/registry-milestone into main 2026-11-07 14:19:59 +00:00
jschoubben 6c05e42626 gofmt the files the group try listed
mesh/delivery merged: source.main moved to novox/mesh-tools@d184862b1ad3
mesh/delivery-group group feat/registry-milestone merged: every member is merged
mesh/check checking 6c05e4262641 on d184862b1ad3 | novox/mesh-tools#61 in group feat/registry-milestone
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-06 01:38:43 +01:00
jschoubben 91a4f1b0ad Take tidy's go.mod whole: mesh requirements at the zero version only, unused ones dropped
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-06 00:52:18 +01:00
jschoubben b4affb3efd Complete go.mod and go.sum for the externals each module builds with
A check's tests run without network, so each module must carry every external module it builds with.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-06 00:51:22 +01:00
jschoubben fb31cc4e99 The console reads the assigner's state assignments and imports the contracts' new paths (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 22:02:06 +01:00
jschoubben bd555e6014 No client names the holder of the seat mesh-provisioner: each reader keeps its own constant, held to its manifest (ADR 0625 section 4, issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 19:36:21 +01:00
jschoubben c3754e48c8 No client names a holder: each reader keeps its own constant, held to its manifest (ADR 0625 section 4, issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 18:37:28 +01:00
jschoubben a00c780352 Require every mesh package of the group at the floor version (ADR 0573 sections 2 and 4, issue 1966)
A member's go.mod names the oldest version a module accepts; the check builds the group's own heads whatever it names. go.sum lines of mesh packages are left out.

Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 16:56:39 +00:00
jschoubben 5bbe669cfb Merge b6/registry into the registry milestone's join: the console reads through the assigner's client, with the seat name mesh-module-assigner (issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 15:39:17 +01:00
jschoubben c55b1bcf38 First views: the integration test reads the views families of the manifest's consumes, not the seat list's events (T-3; issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 15:37:17 +01:00
jschoubben 6965119bce Tool runner: the grants part's membership bucket and key are membership's, which the SDK keeps (ADR 0635 §4; issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 15:36:18 +01:00
jschoubben 387f547346 Tool runner: the console, its test bed and the provider read placement, the catalogue's keys and the provisions through the seats' clients, not go/delivery, go/provider or go/provisions (T-1, ADR 0634 §4; issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 15:31:22 +01:00
jschoubben fce9189579 First views: a finding's times are times, as the conditions client types them (issue 1966) 2026-11-05 15:26:49 +01:00
jschoubben a2402c4873 Merge branch 'b6/registry' into feat/registry-milestone 2026-11-05 14:54:25 +01:00
jschoubben f917fc60c1 Tool runner: waits for the first read of the seat list on the reader's Ready, to a bound, and serves with the bootstrapped seats past it (B9c, ADR 0636 §1; issue 1966)
Claude-Session: https://claude.ai/code/session_01NcbgfFSnn24M3GcqttoU7i
2026-11-05 14:49:52 +01:00
jschoubben 0a6fd0cb5c Merge b6/registry into the registry milestone's join (issue 1966) 2026-11-05 14:39:37 +01:00
jschoubben a918352ff1 Tool runner: describes the seats from the seat list it reads through the seat mesh-catalog; the first views' finding in the conditions seat's layout (T-1, T-3, ADR 0635 §1, ADR 0636 §4; issue 1966) 2026-11-05 14:29:02 +01:00
jschoubben d5e4e4e8ab Console: the address text and the tests name the seat mesh-module-assigner (ADR 0634 §9; T-2; issue 1966) 2026-11-05 14:17:14 +01:00
jschoubben 07f6078050 Pin mesh-sdk 05b3e2e1b94f and mesh-contracts ae5266e77639, the group's final heads (issue 1966)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/registry-milestone checking: 6 of 7 member(s) ready
mesh/check failed (9): ace: novox/mesh-sdk#167 does not merge into the trunk: files in conflict: go/catalog…; and 8 more | 07f6078050ec on 84ede95c4af7
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:47:39 +01:00
jschoubben e40cb291f8 Pin mesh-sdk e5d89c5eedb5 and mesh-contracts 347a3053580a, the group's heads after review round 2 (issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:26:33 +01:00
jschoubben e0d7122567 TLS-5 integration test: the owners' own views streams, the runner's consumers on them and a membership naming them, events on views.Emit's subject
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 18:09:09 +01:00
jschoubben c637da2eb7 Pin mesh-sdk c54648e74346 and mesh-contracts ec9529022c44, the group's heads (issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 17:46:08 +01:00
jschoubben 7390652f33 One guarded handler slot for a bundle's event and heartbeat handlers, so a reload never races the consumer's loop
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:28:08 +01:00
jschoubben db3b5fe63d F-tools: the runner's views consumer is proven on the real bus; the heartbeat is asked of the bus only when a bundle wants it, with no race; a seat record says when the bus wrote it; the beat test counts ticks, not sleeps
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 16:26:50 +01:00
jschoubben 76e025dbbc J2: a state read carries the time the bus wrote the value, as at (stdio.StateEntry.At)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:31:36 +01:00
jschoubben f87c9720ed J2: the tool runner tells a subscribed bundle how its module's consumer stands, as mesh/heartbeat (stdio.OnHeartbeat), at each idle beat and once Missed (ADR 0615, ADR 0621, issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 14:10:25 +01:00
jschoubben d71cd5cef6 TLS-5, TLS-6: the views package is mesh-sdk's (go/views), and mesh_first_view_again is recorded through mesh-hand-acts before it is done (ADR 0615 §6, ADR 0565 §10, issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:14:14 +01:00
jschoubben 31247dff6b TLS-5, TLS-6: the tool runner holds the indexer's manifests and modules and the assigner's placement and seat-holders from their first views and views events, never their state, and the console gains mesh_first_views and mesh_first_view_again (ADR 0615, ADR 0621, issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 12:06:20 +01:00
jschoubben 8b1ab467a9 TLS-1, TLS-2, TLS-3: the runner declares its own secret ledgers, refuses a bundle tool named as a ledger repair, and states the capabilities of its calls in its PING (issue 1966)
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
2026-11-04 11:40:03 +01:00
mesh-tools 84ede95c4a Merge pull request 'L3 final: tool-runner back in node-tools' manifest (issue 774)' (#60) from feat/l3-final into main 2026-11-04 00:24:03 +00:00
jschoubben 7fd9d893c0 Give node-tools new manifest bytes (version 2), the recovery of the moment for a stale verdict
mesh/delivery merged: source.main moved to novox/mesh-tools@84ede95c4af7
mesh/delivery-group group feat/l3-final merged: every member is merged
mesh/check checking 7fd9d893c0a5 on 84ede95c4af7 | novox/mesh-tools#60 in group feat/l3-final
The running indexer holds a refusal from 21:14 for the exact bytes of this manifest, which says tool-runner again. A new digest is read afresh; the thorough fix of stale verdicts is in the indexer in this group (issue 774).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-04 00:44:17 +01:00
jschoubben 0fb4591c9d Pin mesh-sdk 85ba0a9a70d8, the group's head of feat/l3-final (issue 774)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/l3-final checking: 3 of 5 member(s) ready
mesh/check failed (2): unit: mesh-provisioner-client/cmd/mesh-provisioner-client.TestNatsANodeComingBackFil…; and 1 more | 0fb4591c9da9 on b4819dc1fb73
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:43:11 +01:00
jschoubben 7da4c55752 Put tool-runner back in node-tools' manifest (L3 final, issue 774)
The judge that knows the word runs now.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 23:09:34 +01:00
mesh-tools b4819dc1fb Merge pull request 'L3 recovery: take tool-runner out of node-tools' manifest until the judge that knows it runs (issue 774, issue 1585)' (#59) from fix/l3-words-after-judge into main 2026-11-03 21:09:06 +00:00
jschoubben 40e0e8282b Take tool-runner out of node-tools' manifest until the judge that knows it runs
mesh/delivery merged: source.main moved to novox/mesh-tools@b4819dc1fb73
mesh/delivery-group group fix/l3-words-after-judge merged: every member is merged
mesh/check being carried to 40e0e8282b38 on b4819dc1fb73 | from 01ee2ef39488, mesh-catalog e6de42630355, mesh-delivery 4154c3040cee
The running indexer's judge refuses the word, a refused manifest holds the bus's user list, and so the indexer build
that knows the word is never granted. The node-engine on main and node-tools find the runner by its name; the grants
part keeps the runner's users as they run while no manifest says it; the word comes back once the new judge runs
(issue 774).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 21:28:29 +01:00
mesh-tools 01ee2ef394 Merge pull request 'L3: the tool runner says it is the tool runner, and the dead TypeScript callers of the old catalog module are gone (ADR 0604)' (#58) from feat/l3b-bus into main 2026-11-03 20:13:02 +00:00
jschoubben 8b8a760047 Pin mesh-sdk b7abe203a929, the group's head of feat/l3b-bus: it keeps, deprecated, the exported names the running parts still use
mesh/delivery merged: source.main moved to novox/mesh-tools@01ee2ef39488
mesh/delivery-group group feat/l3b-bus merged: every member is merged
mesh/check checking 8b8a7600479e on 01ee2ef39488 | novox/mesh-tools#58 in group feat/l3b-bus
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:50:12 +01:00
jschoubben db3dd04be7 Pin mesh-sdk cf5812101b0b, the group's final head of feat/l3b-bus (the module graph left mesh-sdk)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/l3b-bus checking: 2 of 4 member(s) ready
mesh/check checking db3dd04be7b6 on 8c1e04dd3709 | novox/mesh-tools#58 in group feat/l3b-bus
2026-11-03 17:15:17 +01:00
jschoubben d13890b944 Pin mesh-sdk 65b7adc8bdd9, the group's head of feat/l3b-bus 2026-11-03 17:15:17 +01:00
jschoubben b3903446f9 Remove the TypeScript runtime code that called the old catalog module's tools; a test finds no file naming one (ADR 0604 section 4, issue 774) 2026-11-03 17:15:17 +01:00
jschoubben e550c87a29 Pin mesh-sdk e7ce86ff62b5, the group's head of feat/l3b-bus (review round 2) 2026-11-03 17:15:16 +01:00
jschoubben ff7661a136 Pin mesh-sdk 825483043549, the group's head of feat/l3b-bus (review round 1) 2026-11-03 17:15:16 +01:00
jschoubben 84533edf2e Pin mesh-sdk 09ec82cb3551, the group's head of feat/l3b-bus (ADR 0601) 2026-11-03 17:15:16 +01:00
jschoubben ea516ece5a node-tools says it is the tool runner (tool-runner), so mesh-sdk and its readers never know it by its name; pinned at mesh-sdk a973bfa (issue 774, the operator 2026-11-03) 2026-11-03 17:15:16 +01:00
mesh-tools 8c1e04dd37 Merge pull request 'Bus state by template: the tool runner's objects, stream, work-queue and ledger calls with the proven author (issue 1966, B1)' (#57) from feat/bus-state into main 2026-11-03 15:24:40 +00:00
jschoubben 43d3fba501 Pin the group's mesh-sdk head f8bd12e in place of the local replace (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-tools@8c1e04dd3709
mesh/delivery-group group feat/bus-state merged: every member is merged
mesh/check pass 43d3fba501d3 on b8e42cf54cf6 | unit 6460 run, 262 kept, 27 skipped | integration 6639 run, 264 kept, 35 skipped | over bound: unit, in…
2026-11-03 15:35:30 +01:00
jschoubben df929b48b5 Leave the own secret ledgers out of node-tools' manifest until the vault holds it on every machine, reading a missing file as no ledger credentials; grant the test's ledger account as the grants part does (issue 1966 T6) 2026-11-03 15:29:47 +01:00
jschoubben 961ec0ef96 Append and snapshot a ledger on mesh-sdk's ledger subjects, the ones its account is granted, refusing a binding naming others (issue 1966 T6) 2026-11-03 15:17:12 +01:00
jschoubben 5a9e997349 Check a ledger operation against the types its binding carries, refusing a ledger bound with none (issue 1966 T2) 2026-11-03 15:15:58 +01:00
jschoubben fadc73f73b Key and answer a ledger index by mesh-sdk's IndexKey and IndexEntry, the one shape the owner's watcher also writes (issue 1966 T3) 2026-11-03 15:15:00 +01:00
jschoubben d206b0dde1 Prove against the real bus that the ledger keeps its promises: one writer wins, the state never goes back, a cut or full state catches up, the snapshot rebuilds, the repairs repair, a forged author is refused (issue 1966 T7) 2026-11-03 13:52:58 +01:00
jschoubben 2bb0a6461d Publish a module's ledger history and snapshots only on its own ledger connection, read from the runner's own secret ledgers and followed without a restart (issue 1966 T6) 2026-11-03 13:51:47 +01:00
jschoubben 38a634723b Stamp a ledger operation with the caller the runner proved for the call in flight it names, refusing another module's call or one not in flight; serve the ledger calls on the pipe (issue 1966 T5) 2026-11-03 13:49:15 +01:00
jschoubben 8fe7ef3b82 Serve a ledger owner its repairs rebuild, reindex, snapshot and lag as tools, its own ledgers only, marked acts or reads (issue 1966 T4) 2026-11-03 13:46:54 +01:00
jschoubben 63c1e3faa2 Read a ledger: an instance's state, its history oldest first with sequence and author, a watch of its states, a declared index (issue 1966 T3) 2026-11-03 13:42:33 +01:00
jschoubben ad22c5e9fe Apply a ledger operation: check it against its type, append it with the last sequence expected, derive the state at the revision read, write the snapshot (issue 1966 T2) 2026-11-03 13:40:45 +01:00
jschoubben d6c516ab44 Serve a bundle its own bound objects, streams and work queues on its pipe (issue 1966 T1) 2026-11-03 13:34:32 +01:00
mesh-tools b8e42cf54c Merge pull request 'sdk1: node-tools reads per-node states through mesh-sdk's pernode' (#56) from feat/sdk1 into main 2026-11-02 20:43:23 +00:00
jschoubben 24c0ec0d32 Pin mesh-sdk at f02f142, the sdk1 head whose tests no longer leak package globals, so the group's check builds against it
mesh/delivery merged: source.main moved to novox/mesh-tools@b8e42cf54cf6
mesh/delivery-group group feat/sdk1 merged: every member is merged
mesh/check checking 24c0ec0d324a on d7d908cf67ae | novox/mesh-tools#56 in group feat/sdk1
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:14:16 +01:00
jschoubben bd24ceafa9 Pin mesh-sdk at the sdk1 group's head 1c7356c, so the group's check builds every module against the narrowed SDK it was written for
mesh/delivery-group group feat/sdk1 checking: 7 of 7 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:29:53 +01:00
jschoubben 3f538027e9 Read a per-node state through mesh-sdk's pernode package, so the runtime and a process share one copy of its rules
Replaces the runtime's own bucket listing, advisory hearing and per-bucket bookkeeping; the bound on progress and the runtime's shapes stay here. ADR 0593 §6.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:31:18 +01:00
jschoubben 8114b45884 Serve a per-node state to a bundle as one state: get by the key's node, keys and watch over every node's bucket, a later bucket too
The provisioner, the assigner and the view read every node's composition (ADR 0593 §6); the runtime followed mesh-sdk's membership.StateIssued PerNode and process.NodeFamily on its own connection.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:27:34 +01:00
mesh-tools d7d908cf67 Merge pull request 'node-tools without the controller's paging and handover reads (issue 1126 group B)' (#55) from feat/1126-removal into main 2026-11-02 13:41:43 +00:00
jschoubben e475d9f849 Pin mesh-sdk to the group head of issue 1126
mesh/delivery merged: source.main moved to novox/mesh-tools@d7d908cf67ae
mesh/delivery-group group feat/1126-removal merged: every member is merged
mesh/check checking e475d9f849ca on 49f9269376e3 | novox/mesh-tools#55 in group feat/1126-removal
Every member builds against mesh-sdk abf5eac, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:27:34 +01:00
jschoubben e6f27a5f74 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk f569924, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:20:14 +01:00
jschoubben c678511871 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk 729a86b, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:56:55 +01:00
jschoubben 4f636b7080 Name no controller verb in hints and drop its handover and paging reads, since the controller ended (issue 1126 B19)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:23:14 +01:00
mesh-tools 49f9269376 Merge pull request 'node-tools follows memberships from the grants part's state alone (issue 774)' (#54) from feat/l3-removal into main 2026-11-02 00:35:50 +00:00
jschoubben 29fd452628 Pin mesh-sdk at c21d8fd, the removal group's head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@49f9269376e3
mesh/delivery-group group feat/l3-removal merged: every member is merged
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:53:42 +01:00
jschoubben 331fa06b54 The tool runner reads the memberships state again until it can, instead of giving a module up at its one follow
A JetStream 503 or a timeout at the follow (a bus right after its boot) left the module with no membership for good, since
Follow runs once per module and the stream fallback is gone. The read is now tried again with a growing wait (1 s doubling
to 30 s) until it watches the key or the connection closes; the failure is said once, and its reading once it succeeds.
Tests: none added; a step added to an existing unit test (TestAModuleHearsEveryStreamItsMembershipNames), which fails
without the retry.

(issue 774, 774/S32, review g1 round 1)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:19 +01:00
jschoubben 5b2ff5e321 Test that node-tools' manifest is not refused by the lease write's check (ADR 0571 §13)
Tests: one added, the first of its kind in this repository; the repository holds 128 unit tests, within its budget of 500.
Built against the group's mesh-sdk through a go.work outside the tree; go.mod is not re-pinned here.

(issue 774, 774/S33)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:26:04 +01:00
jschoubben 4fbfdfad13 The tool runner reads its membership from the grants part's state alone: no stream read, no subject heard (ADR 0571 §10)
Follow no longer asks the ASSIGNMENTS stream for a copy nor subscribes mesh.assignment; a module with no membership key is
said so and served in its derived shape. The test mesh issues memberships where the grants part keeps them.
Tests: none added; the stream-fallback assertions of two existing tests are replaced by their state-only counterparts, and
the test mesh no longer makes the stream. The two integration tests of the bus were run locally against the embedded bus.

(issue 774, 774/S32)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:25:19 +01:00
mesh-tools 75943ca7b0 Merge pull request 'The tool runner waits on the self-check's checked, not the controller's (design 47 L7)' (#53) from feat/l5-l7-main into main 2026-11-01 13:54:02 +00:00
jschoubben fb6631aee1 Pin group A's mesh-sdk head on main
mesh/delivery merged: source.main moved to novox/mesh-tools@75943ca7b0c3
mesh/delivery-group group feat/l5-l7-main merged: every member is merged
mesh/check checking fb6631aee1a2 on 75943ca7b0c3 | novox/mesh-tools#53 in group feat/l5-l7-main
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:57:49 +01:00
jschoubben d41bee158c node-tools waits on the self-check's checked, not the controller's (ADR 0571 clause 13, issue 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:39 +01:00
mesh-tools 110e55cb48 Merge pull request 'The tool runner declares the credential form of its mcp-endpoint (ADR 0579)' (#52) from feat/judge-once into main 2026-11-01 07:55:09 +00:00
jschoubben c494018074 Pin the group's mesh-sdk head after the gofmt fix
mesh/delivery merged: source.main moved to novox/mesh-tools@110e55cb4836
mesh/delivery-group group feat/judge-once merged: every member is merged
mesh/check checking c4940180746a on c2f01bf10b68 | novox/mesh-tools#52 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:27:07 +01:00
jschoubben 4cdf63443d Pin mesh-sdk to the group's head after review round 1
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/judge-once checking: 4 of 6 member(s) ready
mesh/check checking 4cdf63443d59 on c2f01bf10b68 | novox/mesh-tools#52 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:13:22 +01:00
jschoubben 1e9995faba Pin mesh-sdk to the delivery group's head
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:46:53 +01:00
jschoubben f6c401ecd2 Declare credential form none for the provision, which hands consumers no credential
The manifest judge refuses an offer that says no credential form (issue 1585).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:43:23 +01:00
mesh-tools c2f01bf10b Merge pull request 'Read and write placement per node and module (issue 774)' (#51) from feat/072-moves into main 2026-10-31 10:31:18 +00:00
jschoubben be2fc2e9b6 Pin mesh-sdk at the group's head acbf80a (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@c2f01bf10b68
mesh/delivery-group group feat/072-moves merged: every member is merged
mesh/check failed: shanks: the check of novox/mesh-controller#487 failed: go mod download in modules/route-proxy: go: git.novox.be/novox/mesh-controll…
2026-10-31 10:25:46 +01:00
jschoubben aa05629375 Pin mesh-sdk at the group's head (issue 774) 2026-10-31 09:30:08 +01:00
jschoubben 35a7ff70fe Read the seat mesh-placement's state through the sdk's PlacementsOf, and write the test mesh's placement as the seat does, one key per module and node (issue 774, research 072 row 37) 2026-10-31 09:29:32 +01:00
jschoubben 747bfa71aa Merge pull request 'A state watch given up hands nothing more to its bundle (issue 774)' (#50) from fix/sweep-2026-10-30 into main
Reviewed-on: #50
2026-10-30 18:13:12 +00:00
jschoubben 316a5a2fc3 A watch given up or stopped hands over nothing more; arrival of current values restarts its bound (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-tools@747bfa71aa32
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking 316a5a2fc317 on 0edcd2501dae | novox/mesh-tools#50 in group fix/sweep-2026-10-30
Quick: check the watch context before each deliver in StateWatch. Thorough (chosen, same
file kind): also LogWatch drops a message fetched as it stopped, and the bound restarts
on arrival, not only after the bundle took a value.
2026-10-30 18:22:01 +01:00
jschoubben 0edcd2501d Merge pull request 'The tool runner reads the grants part's memberships through the mesh-sdk codec (issue 774)' (#49) from fix/memberships-shape into main
Reviewed-on: #49
2026-10-30 14:42:20 +00:00
jschoubben ac645e2cae Refuse a held-back membership's old wrapper in the tool runner, pinning mesh-sdk's review-round-1 codec
mesh/delivery merged: source.main moved to novox/mesh-tools@0edcd2501dae
mesh/delivery-group group fix/memberships-shape merged: every member is merged
mesh/check checking ac645e2caedd on 3c9068f99a43 | novox/mesh-tools#49 in group fix/memberships-shape
The old grants part wrote its wrapper for a held-back membership too; the new part leaves such a key alone, and taking
it would serve what the hold kept back with the stream's copy ignored. The integration test now proves that copy is
served instead. Review round 1 of issue 774's fix.
2026-10-30 15:38:40 +01:00
jschoubben a479a90fb2 Read the grants part's memberships through mesh-sdk's codec, so the tool runner serves its node's modules
The tool runner unmarshalled the state's value straight into its membership; the grants part had written its wrapper
there, so every module decoded empty, was marked as given by the grants part, and the correct copy on the assignments
stream was passed over: novox served no tool. followGrants now reads with membership.BareGrantsMembership, which takes
the bare membership, still takes the old wrapper (what the live mesh holds until the part writes again), and refuses
a value naming no node or module, leaving the stream's copy in force.

The fixture IssueInGrants built its value by hand from this package's type and so agreed with the mistake; it now
writes through mesh-sdk's encoder, and IssueInGrantsWrapped writes the old wrapper for one integration test that
proves tools and state are served from it (it fails without this fix with the empty membership seen live).

Issue 774 (the bus login list, L3). Quick fix: unwrap here alone. Thorough, chosen: one codec in mesh-sdk the writer
and this reader share, so the two cannot drift again. Pins mesh-sdk at the head of its fix/memberships-shape.
2026-10-30 15:31:26 +01:00
mesh-tools 3c9068f99a Merge pull request 'Design 47's last milestones in the tool runner: per-key TTL, log verbs, memberships and seat holders from the bus (issues 1952, 1954, 1955)' (#48) from feat/774-l3-reland into main 2026-10-30 11:02:19 +00:00
jschoubben 250aa73a56 Pin mesh-sdk 2da0de7, the release's final head with main merged in, so mesh-tools builds with the group's one SDK (issue 1955)
mesh/delivery merged: source.main moved to novox/mesh-tools@3c9068f99a43
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
mesh/check checking 250aa73a5631 on 95a12105cf7b | novox/mesh-tools#48 in group feat/774-l3-reland
2026-10-30 01:12:55 +01:00
jschoubben e32eef2379 Pin mesh-sdk 6e4a301, the release's final head, so mesh-tools builds with the group's one SDK (issue 1955) 2026-10-30 00:16:11 +01:00
jschoubben 070ae53128 Pin mesh-sdk 007b6fd, the release's final head, so mesh-tools builds with the group's one SDK (issue 1955) 2026-10-30 00:05:45 +01:00
jschoubben 8c0618cfc5 Prove a later assignments-stream membership changes nothing once the grants part's state has given one (issue 1955, ADR 0571 clause 7) 2026-10-29 23:44:19 +01:00
jschoubben 54680238a0 Read memberships from the grants part's state, what the mesh holds from the indexer and placement, and give bundles newest, since and watch on logs, so the tool runner asks the controller for nothing it keeps (issue 1955, ADR 0571 clauses 1, 4, 5, 7)
Pins mesh-sdk bb993dd. The assignments stream stays the membership's
fallback until the lease write; a log another module keeps is read as
<module>.<log>.
2026-10-29 23:42:43 +01:00
jschoubben 200249a84f Let the TypeScript mesh client take every role's tools from the console on its machine, so it asks the controller for none (issue 1955, ADR 0571 clause 5)
The Go console answers mesh/seats beside MCP's methods from mesh-sdk's seat
definitions and placement's seat-holders. The console test that runs bundles
expects the unread seat-holders line beside ghost.
2026-10-29 23:19:55 +01:00
jschoubben 67850457ef Read seat verbs' replaces from mesh-sdk's seat contract and seat holders from placement's seat-holders state, so the console asks the controller for neither (issue 1955, ADR 0571 clauses 4 and 5)
Pins mesh-sdk f9a496d. The controller's nodes and modules reads remain.
2026-10-29 23:13:42 +01:00
jschoubben 9d67b0995f Bind node-tools' consumers by mesh-provisioner's plans and provisions alone, dropping the read of the controller's assignments the sdk's helper no longer makes (issue 1954 L6-T1)
Pins mesh-sdk 692dbf9 (L6-P3).
2026-10-29 22:43:58 +01:00
jschoubben e9009b7f04 Answer mesh/state.info with a bucket's fill and caps, so a bundle's tools can say how full its state is (issue 1952) 2026-10-29 22:43:58 +01:00
jschoubben 9fc97e45c9 Let node-tools' provider helper read mesh-provisioner's provisions and plans by mesh-sdk's names, so the runtime follows ADR 0570's records off the controller (issue 1954 L6-T1)
Pins mesh-sdk 4a96793 (go/provisions). The assignments read stays while the
sdk's helper still reads plans there (until L6-P3). A watch's expire now
reads as a removed key.
2026-10-29 22:34:34 +01:00
jschoubben 1583fa85d9 Let a bundle write a state key with its own time to live and hear its end, so wakes need no in-process timer (issue 1952)
mesh/state.create and mesh/state.put take ttl_seconds (mesh-sdk 32f5ce3's
CreateTTL/RenewTTL contract); a bucket not declared key-ttl is refused,
naming it; a watch hands an expired key as op "expire". Pins mesh-sdk 32f5ce3.
2026-10-29 22:11:39 +01:00
mesh-tools 95a12105cf Merge pull request 'The bus state guard lets a provider reference under a provision named like a credential pass (L2's seed, issue 1126)' (#47) from fix/1126-state-guard into main 2026-10-28 21:01:24 +00:00
jschoubben 2966c1e32c Runtime state guard: hold a provider reference's node and module to the mesh's name pattern, so no credential rides in one (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-tools@95a12105cf7b
mesh/check being carried to 2966c1e32c3d on 95a12105cf7b | from 27266f99a3c3
2026-10-28 21:49:52 +01:00
jschoubben 1a0424117a Runtime state guard: a credential-shaped name passes only holding an exact provider reference, node and module, so a placement's binding is kept and nothing else gets through (issue 1126) 2026-10-28 21:48:27 +01:00
jschoubben 23f5babe33 Runtime state guard: a name holding an object is data, not a credential, so a placement's binding of the provision secret is kept (issue 1126) 2026-10-28 21:47:06 +01:00
mesh-tools 27266f99a3 Merge pull request 'node-tools starts no bundle again once it is stopping (group fix/shutdown-bound, issue 1126)' (#46) from fix/shutdown-bound into main 2026-10-28 15:05:13 +00:00
jschoubben a9b02bc676 node-tools: keep the stopping-runtime test under a second, and start no first bundle once stopping
mesh/delivery merged: source.main moved to novox/mesh-tools@27266f99a3c3
mesh/delivery-group group fix/shutdown-bound merged: every member is merged
mesh/check pass a9b02bc67604 on 6cc33988801f | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
The test watched two seconds per row; a 10 ms restart pause and a 250 ms window
prove the same. launch.Start itself now refuses while the runtime stops, so a
reload's first start is held too (issue 1126).
2026-10-28 15:40:39 +01:00
jschoubben f016c67a72 node-tools: start no bundle again once the runtime itself is stopping (issue 1126)
The service manager stops node-tools by signalling its whole group, so every
bundle gets SIGTERM with it; the launcher took each exit for a fault and started
the bundle again, for the 15 s the runtime took to stop. On ace's reboot of
2026-10-28 those restarted bundles dialled the system bus once polkit and the bus
had stopped, the bus and polkit came back during the shutdown, nothing stopped
them a second time, and the reboot waited 29 minutes on its job timeout.

ShutDown, called first on SIGTERM, keeps any bundle from starting again, by a
restart or by a call; a running one still answers.
2026-10-28 15:30:12 +01:00
mesh-tools 6cc3398880 Merge pull request 'node-tools waits for its broker credential instead of exiting, and retries a refusal within that bound (issue 1496)' (#45) from fix/1496-start-health into main 2026-10-26 21:38:04 +00:00
jschoubben 8cdb7c3032 The credential wait's test stays under a second, and a failed watch says the re-read, not a restart (issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-tools@6cc33988801f
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking 8cdb7c303210 on a5097dc69ab2 | novox/mesh-tools#45 in group fix/1496-start-health
Review of 3baeaec: TestAwaitCredential took 1.9 s, over the unit limit; it now
sets the watch's settle to 10 ms, writes at 20 ms and gives up at 100 ms (0.2 s).
The watch's journal line for a failure named a bundle and a restart; the
credential wait now says it is read again every 30 s instead.
2026-10-26 22:18:12 +01:00
jschoubben 3baeaece61 A module's runtime waits for its broker credential and its account at a start, bounded, instead of exiting (issue 1496)
The node-engine starts a module's runtime before the provisioner's client fills
its broker file, and the bus carries a new account only after it reloads its
users. The runtime exited on both and was restarted until they were there (ace,
2026-10-26 21:19: mesh-build-node's and mesh-build-image's runtimes exited twice).
It now waits up to five minutes from its start, said once: the file is read again
each time its directory changes (the bundles' inotify watch, a 30 s backstop),
and an authorization refusal is asked again with the backoff. Past the bound it
gives up loudly; a file with no url, a pin mismatch and a bad URL stay final.
2026-10-26 22:06:39 +01:00
mesh-tools a5097dc69a Merge pull request 'mesh_search leaves out filler words, so a question in plain words finds its tool' (#44) from fix/search-ignores-filler into main 2026-10-26 13:39:24 +00:00
jschoubben 9f25e5c9f6 mesh_search: filler words of a question (did, our, on, all) need not match, so the operator's own words find a tool; a query of filler finds nothing
mesh/delivery merged: source.main moved to novox/mesh-tools@a5097dc69ab2
mesh/check pass 9f25e5c9f6c1 on a5097dc69ab2 | carried from bb6a0e3192db | unit 116 run, 11 kept, 11 skipped | integration 118 run, 11 kept, 10 skippe…
2026-10-26 14:03:49 +01:00
mesh-tools bb6a0e3192 Merge pull request 'node-tools: a consumer deleted or made again while no pull waits is noticed at the next rebind, so a module is not deaf for a heartbeat (issue 1945)' (#43) from fix/bus-consumer-remade-flake into main 2026-10-25 15:22:24 +00:00
jschoubben 3331719407 Ask the bus whether a bound consumer still exists, so a deleted and remade one is rebound within a rebind tick, not after nats.go's heartbeat (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-tools@bb6a0e3192db
mesh/check pass 33317194073a on bb6a0e3192db | unit 115 run, 11 kept, 11 skipped | integration 117 run, 11 kept, 10 skipped | over bound: unit (slowes…
nats.go ends a pull loop on a deleted consumer only when a pull is pending at that moment; otherwise the loop waits out its heartbeat. The rebind tick now looks each bound consumer up by name and compares its creation time.
2026-10-25 16:02:27 +01:00
mesh-tools 5d6cee9d84 Merge pull request 'node-tools serves a tools bundle as soon as it is placed, changed or removed, with no restart, and says why a bundle cannot start (issue 1945)' (#42) from fix/r6-reload-bundles into main 2026-10-25 14:56:22 +00:00
jschoubben 2cd5550dc7 A stopped tool drops its membership handler and a stopped bundle starts no copy on a call, so a replaced bundle's old tools stay gone (issue 1945, review round 2)
mesh/delivery merged: source.main moved to novox/mesh-tools@5d6cee9d844e
mesh/check pass 2cd5550dc70f on 77b05089c8df | unit 0 run 126 kept | int 0 run 128 kept | clone 9.5s | composition passed on 4 machines
Its events are refused before the old bundle stops, the bundle watch follows a directory made between
its stat and its watch, and a bundle's reason is read without node's colour codes.
2026-10-25 15:44:10 +01:00
jschoubben d8ac69904d Refuse an event heard during a reload instead of panicking, and judge a bundle by its entrypoint alone (issue 1945, review)
An event a module's consumer heard after the module was dropped for a
reload read a deleted entry and panicked node-tools, taking every tool on
the machine down. The handler now refuses an event of a consumer that is no
longer the module's, and a stopped hearing hands nothing on and offers the
event again at once.

A bundle that writes beside itself (a cache, a socket) loaded itself again
after every quiet spell, because the signature took in every file in its
directory. A delivery puts every file in place anew, so the entrypoint alone
tells a new build. The watch also looks once as it begins, so a bundle that
came between its load and the watch is not missed.

Tests: 1945/AC23 and AC24 in internal/runtime/reload_test.go, and a stopped
hearing in internal/bus/hearing_stopped_test.go.
2026-10-25 15:26:02 +01:00
jschoubben b0ded57294 Serve a tools bundle as soon as it is in place, changed or gone, with no restart; say why a bundle cannot start (issue 1945)
node-tools loaded every bundle once at start, so a bundle delivered after
it started (mesh-files on shanks, 2026-10-25 12:08) went unserved until a
hand restart, and a missing file was said as "not executable". The
runtime now watches the entrypoints' directories with inotify, loads a
module again once a burst of changes settles, and discovery answers with
what is served now. A bundle that cannot start says it does not exist, is
not executable with its mode, or the exec error.

Tests: 1945/AC19-AC22 in internal/runtime/reload_test.go.
2026-10-25 15:09:49 +01:00
jschoubben 77b05089c8 Merge pull request 'Bound a state watch's hand-over by its progress, not its whole set, so the gitea module binds the checker's owed statuses again (issue 1726)' (#41) from fix/1726-group-check-status into main
Reviewed-on: #41
2026-10-24 10:35:43 +00:00
jschoubben 0211844e6a Bound a state watch's hand-over by its progress, not its whole set, so a large state's newest keys are handed over (issue 1726)
mesh/delivery delivering: 0 machine step(s) passed
mesh/check pass 0211844e6ad3 on 0e985f9bf186 | unit 0 run 117 kept | int 0 run 119 kept | clone 5.4s | composition passed on 4 machines
The gitea module's watch of mesh-delivery-checker.owed failed at every attempt
since 03:05 on novox: about 200 owed keys, each a round trip to the bundle,
outlast a 10 s bound for the whole set. The bundle dropped its handler, the
oldest keys were written again every 40 s and the newest, handed over last,
never were: group fix/1937-done-key-blobs's pass never reached mesh/check.
StateWatch now fails only when no current value is handed over within
StateTimeout.
2026-10-24 07:42:08 +02:00
jschoubben 0e985f9bf1 Merge pull request 'R4g (mesh-tools): seat.pull and seat.settle for the build modules, merge-check.sh and the root image module go (design 47)' (#40) from feat/r4g into main
Reviewed-on: #40
2026-10-23 17:51:33 +00:00
jschoubben 860043fd7f Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking 860043fd7fa0 on e0fadc43d9dd | novox/mesh-tools#40 in group feat/r4g
2026-10-23 17:06:12 +02:00
jschoubben 589bfc0bc5 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: g14: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group's…
2026-10-23 16:26:11 +02:00
jschoubben 3b5a38e4a4 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, so every module r4g touches builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:02 +02:00
jschoubben 755f77ca45 Merge branch 'feat/r4g-asks' into feat/r4g 2026-10-23 16:20:32 +02:00
jschoubben 9153a280f8 Let a bundle pull one message from its listed worker and settle it, so a build module takes an ask only when it has a free slot, in its workers' order (issue 1887 A11, ADR 0525 §4) 2026-10-23 15:28:06 +02:00
jschoubben 8ee6db7d9a Remove the toolchain image module; node-tools is the repository's only module (issue 979, S1; ADR 0524 §5)
Only build-agent built inside that image, and it goes in R4g.
2026-10-23 15:25:33 +02:00
jschoubben 5dffb920ff Delete merge-check.sh; tests that launch bundles skip with their reason (issue 1295, S2; ADR 0529)
The checker runs node-tools' Go level and its npm test; what the script said
was NOT TESTED HERE is a skip naming the missing node or @novox/mesh-sdk.
2026-10-23 15:25:10 +02:00
mesh-tools e0fadc43d9 Merge pull request 'Say "nothing serves" only when the bus finds no subscriber, never for a tool's own error that mentions 503 (issue 1529)' (#39) from fix/1529-nothing-serves-only-for-no-responders into main 2026-10-20 22:15:35 +00:00
jschoubben d295fd4250 Say nothing serves a call only for the bus's own no-responders, never for a tool's error that mentions 503 (issue 1529)
mesh/merge-gate head d295fd4250a7 merged into trunk 18260ce9beca — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head d295fd4250a7 merged into trunk 18260ce9beca — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792534538429799487 (novox/mesh-tools at e0fadc43 into main, d…
2026-10-21 00:02:34 +02:00
mesh-tools 18260ce9be Merge pull request 'The tool runner stops waiting on the removed delivery module's transition (issue 1406)' (#38) from fix/1406-drop-the-old-transition-name into main 2026-10-20 14:36:22 +00:00
mesh-tools 4165ad59b5 Wait on the gate's transition alone: today's module is forgotten (issue 1406)
mesh/merge-gate head 4165ad59b541 merged into trunk f183b86fa0b2 — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 4165ad59b541 merged into trunk f183b86fa0b2 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792506983719904657 (novox/mesh-tools at 18260ce9 into main, d…
2026-10-20 12:59:33 +00:00
mesh-tools f183b86fa0 Merge pull request 'The tool runner waits on the gate's transitions too (design 47 R3e, issue 1350)' (#37) from feat/r3e-the-gate into main 2026-10-20 06:49:21 +00:00
mesh-tools 06e6721ed6 Wait on the gate's transitions beside today's delivery module's (issue 1350)
mesh/repo-check head 06e6721ed677 merged into trunk 7412c5b6c9c5 beside mesh-catalog#380@cf5c0273 — pass: its merge-check.sh passed
mesh/merge-gate head 06e6721ed677 merged into trunk 7412c5b6c9c5 — pass: group feat/r3e-the-gate composed: every machine composes with the change as it did…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792478961597980735 (novox/mesh-tools at f183b86f into main, d…
mesh/delivery-group group feat/r3e-the-gate merged: every member is merged
2026-10-20 00:57:44 +00:00
mesh-tools 7412c5b6c9 Merge pull request 'Feature 57 (config and provisioning): release mesh-tools' (#36) from release/57-config-and-provisioning into main 2026-10-16 12:42:28 +00:00
mesh-tools 33d136e702 Feature 57 (config and provisioning): release mesh-tools (the provider's StateBus, the SDK pin, templates), reviewed
mesh/merge-gate head 33d136e70227 merged into trunk 6b3e15042fee — pass, with a note: builds mesh-tools, node-tools (+40 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 33d136e70227 merged into trunk 6b3e15042fee — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
2026-10-16 09:41:17 +00:00
mesh-tools 6b3e15042f Merge pull request 'Give a session mesh_wait: wait on the bus for an event instead of polling (issue 745, ADR 0367)' (#35) from 745-mesh-wait into main 2026-10-15 00:52:34 +00:00
mesh-toolsandClaude Opus 5.5 21d038d339 End every wait in flight when the tool runner stops (issue 745)
mesh/merge-gate head 21d038d3397c merged into trunk 5d70eb3e35a9 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 21d038d3397c merged into trunk 5d70eb3e35a9 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
Closing the console shut its server down with a context that never ends, so a stop or restart of
the tool runner waited on any mesh_wait in flight for up to ten minutes. The server's base context
is now cancelled first, ending every wait at once, with a five-second shutdown as a backstop
(second review of issue 745).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:47:59 +00:00
mesh-admin 5d70eb3e35 Merge pull request 'The runtime publishes a module's stream events on its stream and binds the stream consumers its membership names (hq issue 725, ADR 0363)' (#34) from fix-725-runtime-streams into main 2026-10-14 20:13:45 +00:00
jochen d3b0330ed1 Bind a stream consumer again when the bus ends its loop, so a module is not left deaf when the controller deletes and makes again a consumer it cannot narrow (hq issue 725)
mesh/merge-gate head d3b0330ed1cb merged into trunk d0dcae53a8c0 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head d3b0330ed1cb merged into trunk d0dcae53a8c0 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792008849869252364 (novox/mesh-tools at 5d70eb3e into main, d…
2026-10-14 22:10:20 +02:00
jochen 9921486fd5 Publish a module's stream events on its stream's subject and bind the stream consumers its membership names, so ADR 0363's own streams work in the runtime (hq issue 725)
mesh/merge-gate head 9921486fd53e merged into trunk d0dcae53a8c0 — pass, with a note: builds mesh-tools, node-tools (+39 dependent(s)) → ace, g14, novox, s…
mesh/repo-check head 9921486fd53e merged into trunk d0dcae53a8c0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 22:04:41 +02:00
mesh-admin d0dcae53a8 Merge pull request 'Bring node-tools' lock back in step with package.json, and fail the check when it falls behind (issue 624)' (#33) from fix/624-node-tools-lock-in-step into main 2026-10-13 13:19:40 +00:00
jochen 52e275ad2f Bring node-tools' lock back in step with package.json, and fail the check when it falls behind (hq issue 624)
mesh/merge-gate head 52e275ad2f3a merged into trunk 954e1a53eef9 — pass, with a note: builds mesh-tools (+39 dependent(s)) → ace, g14, novox, shanks; no bu…
mesh/repo-check head 52e275ad2f3a merged into trunk 954e1a53eef9 — pass: its merge-check.sh passed
mesh/delivery delivered
The lab installs node-tools with npm ci, which refuses a lock that no longer
describes package.json; the SDK's range and the toolchain's development
dependencies had moved in package.json alone, so every lab run stopped at
setup. The image's build runs npm install and never read the lock, so nothing
noticed.
2026-10-13 15:12:10 +02:00
mesh-admin 954e1a53ee Merge pull request 'Keep what discovery heard when a round hears nothing, and fetch only changed announcements (issue 554)' (#32) from fix/554-empty-discovery-round into main 2026-10-12 12:25:47 +00:00
jochen 424a7eec82 Keep what discovery heard when a round hears nothing, and fetch only changed announcements (issue 554)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
A round in which no PING reached the laptop's console within the 750 ms window
was taken as the mesh's answer that nothing is there, and replaced the index:
every seat and module call was refused for five seconds at a time. Discovery
now waits up to its patience for a first PING, a round that heard nothing is
not kept, a runtime not heard this round is resolved from what it announced
last (said with its age), and a round that misses runtimes is logged once.
A PING carries a digest of what the instance serves, so an unchanged
announcement is not pulled again every round (some 880 kB to the laptop).
2026-10-12 13:56:50 +02:00
mesh-admin 09c5f809cb Merge pull request 'The TypeScript toolchain holds pg, mongodb and mssql, so a catalogue check can check the modules that use them (issue 385)' (#31) from fix/385-an-untestable-module-fails-the-check into main 2026-10-12 10:46:07 +00:00
jochen 847055c758 Give the TypeScript toolchain pg, mongodb and mssql, so a catalogue check can type-check the modules that use them (issue 385)
mesh/merge-gate pass: builds mesh-tools, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/385-an-untestable-module-fails-the-check merged: every member is merged
A catalogue's TypeScript check fetches nothing and reads this image's
node_modules. Once a module it cannot check fails rather than passing with a
note, mesh-catalog, model-usage, mongodb and mssql could never merge again.
They are development dependencies, so the runtime image stays as it is.
2026-10-12 12:27:46 +02:00
mesh-admin aee834aeb6 Merge pull request 'Wait for a stopped bundle's exit instead of sleeping after it (issue 476)' (#30) from fix/476-wait-for-the-exit into main 2026-10-12 09:03:48 +00:00
jschoubben 494d0787e2 Wait for a stopped bundle's exit instead of sleeping after it
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed
Launched.Stop now returns once the bundle's exit is handled, bounded by
StopWait, and the launcher reads its restart pauses once at Start, so
the exit goroutine no longer races a test that changes them. The caller
tests drop their half-second pause (novox/hq issue 476).
2026-10-12 10:43:22 +02:00
mesh-admin 172b5af589 Merge pull request 'node-tools: serve mesh/state.history (ADR 0307, issue 502)' (#29) from adr-0307-state-history into main 2026-10-12 08:24:20 +00:00
jochen 3f0c1a6cec Serve a key's kept values so the issue tracker can carry them into its log (ADR 0307, issue 502)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery-group group adr-0307-state-history merged: every member is merged
mesh/state.history answers every value a bucket still keeps for a key,
oldest first, to whoever may get it, by the same watch the runtime already
uses, so no new grant is needed.
2026-10-12 10:18:34 +02:00
mesh-admin 4348b42844 Merge pull request 'node-tools: serve a module's log verbs (ADR 0297, issues 501, 502)' (#28) from adr-0297-log-verbs into main 2026-10-12 00:28:27 +00:00
jochen 82aefe1988 Refuse a credential in a log entry and take a full 256 KiB value with its expect header (ADR 0297 review)
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery delivered
A log keeps what it is given for good, so the state guard against secrets
applies to it too; the stream holds 260 KiB a message so the headers an
append carries never push a 256 KiB value over.
2026-10-11 19:20:31 +02:00
jochen d535dd5322 Serve a module's log verbs so its record outlives a bucket's cap and the events' week (ADR 0297, issues 501, 502)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-tools, node-tools → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
The runtime answers mesh/log.append, read, last and info for a log issued
in the module's membership, by publish, batch direct get and stream info
alone. A stale expect is refused as a lost state update, so a module tells
it by the same code.
2026-10-11 19:18:46 +02:00
158 changed files with 21449 additions and 5417 deletions
-80
View File
@@ -1,80 +0,0 @@
ARG NODE_BASE=node:22-bookworm-slim
# The mesh-tools module: the two images every TypeScript module is COMPILED in and may RUN in. The
# runtime itself ships as the node-tools module's bundle (node-tools/, novox/hq ADR 0175, to-be 38
# WP3); these images are the toolchain for TypeScript bundles and the base a module's own service
# may still be built on. They are no longer how tools reach a node.
#
# **They were the same image, and that was a mistake.** A module's recipe starts from this and
# invokes the compiler out of it, so the compiler had to be here — and because the same image was
# also what every module ran in, every running container on every machine carried a TypeScript
# compiler it would never invoke. The answer is separate stages rather than one image bad at both
# jobs. `build.artifacts` in module.json names the published stage each — `toolchain` and `runtime`.
#
# The `deps` stage is neither published nor named there: it is where the mesh's package registry is
# reached, so it is where — and only where — the credential to reach it exists. The toolchain copies
# resolved node_modules out of it, so the credential is in no image any machine ever holds
# (novox/hq ADR 0076). This is the buildkit-secret's job done without buildkit, because a machine's
# docker may carry no buildx.
# ---- deps: node_modules resolved from the mesh's registry, credential and all ----------------
FROM ${NODE_BASE} AS deps
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY node-tools/package.json ./
# The builder writes .npmrc into the build context; it authenticates to the mesh's package registry
# for the @novox scope, which is where @novox/mesh-sdk resolves. This stage is not published, so the
# credential travels no further than here. Development dependencies included: the compiler is one.
COPY .npmrc ./.npmrc
RUN npm install --no-audit --no-fund
# **The SDK this image carries is the one the mesh last published** (novox/hq issue 212). The range in
# package.json is resolved once and the layer above is cached, so a release reached no toolchain until
# that file changed. The exact version arrives as a build argument from the SDK module's published
# package (module.json `build.on`), so a new release is a new argument, this layer runs again — and
# the planner orders this module after the SDK, so a release rebuilds the toolchain and, after it,
# every bundle compiled in it (issue 211).
ARG MESH_SDK=@novox/mesh-sdk@latest
RUN npm install --no-audit --no-fund "${MESH_SDK}"
# ---- compiling: the runtime's own code built, WITHOUT the credential -------------------------
FROM ${NODE_BASE} AS compiling
WORKDIR /app
COPY node-tools/package.json ./
# The resolved libraries, but not the .npmrc that resolved them.
COPY --from=deps /app/node_modules ./node_modules
COPY node-tools/tsconfig.json ./
COPY node-tools/src ./src
RUN npm run build
# ---- what a running bundle needs, and nothing else -------------------------------------------
# Its own stage so the toolchain image keeps its build tools while the runtime image does not.
FROM compiling AS lean
RUN npm prune --omit=dev
# ---- toolchain: what a TypeScript bundle is compiled in ---------------------------------------
# The compiler, and esbuild (a development dependency) at /app/node_modules/esbuild: the builder
# bundles every entrypoint and launcher into one file with it (novox/hq ADR 0193), so a bundle
# carries what it imports and not this image's node_modules.
# Beside the compiler, at /app/runtime, what every TypeScript bundle runs with: the production
# dependencies the SDK and the runtime need, and a package.json saying the compiled files are ES
# modules. The builder copies this directory whole into a compiled bundle (novox/hq ADR 0188 §5),
# so a bundle unpacked on a machine starts — a `.js` without that package.json is read as
# CommonJS, and an import of `nats` without node_modules beside it resolves to nothing.
FROM compiling AS toolchain
COPY --from=lean /app/node_modules /app/runtime/node_modules
RUN printf '{"type":"module","private":true}\n' > /app/runtime/package.json
# **git, because a pull request's merge check runs in this image too** (novox/hq ADR 0237): a repository's
# own merge-check.sh that declares this toolchain runs here, and a suite that reads a repository's history
# (the lab's last-run record) ran into "spawnSync git ENOENT". Nothing compiled here calls it.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/*
# ---- runtime: what a module's own service may run in ------------------------------------------
FROM ${NODE_BASE} AS runtime
WORKDIR /app
COPY node-tools/package.json ./
COPY --from=lean /app/node_modules ./node_modules
COPY --from=compiling /app/dist ./dist
ENTRYPOINT ["node", "dist/main.js"]
+6 -10
View File
@@ -1,13 +1,11 @@
# mesh-tools
Two modules in one repository (novox/hq ADR 0069), one piece of software:
One module, `node-tools`; the toolchain image module that stood beside it is retired (novox/hq ADR 0524 §5):
- **`node-tools`** (`node-tools/`) — the node's **tool runtime** as a module (ADR 0175, to-be 38
WP3): one process per machine the host runs from this bundle, serving every assigned module's tools
and every held seat's verbs on the bus, and answering MCP on the machine's loopback — the console
(design 34). The code, its tests and the `mesh` client all live there.
- **`mesh-tools`** (this directory) — the two images TypeScript bundles are compiled in and a module's
own *service* may still run in. Built from the same code; no longer how tools reach a node.
The runtime:
@@ -25,8 +23,7 @@ marked executable — is **launched** rather than imported (novox/hq ADR 0188):
as a child with its own environment and speaks MCP over stdio to it, `tools/list` once and
`tools/call` per call. A tool it lists as `<seat>.<verb>` is the seat's implementation. A child that
exits is named in the log and started again on its next call. So a tools bundle may be written in
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/src/launch.ts`
is the runtime's side of it).
any language; the mesh's SDK for each is the stdio loop and nothing more (`node-tools/internal/launch` is the runtime's side of it).
Everything hard — dispatch, collection, duplicate-name safety — is the sdk's. This is the thin
wrapper that binds the bus and loads the modules. Keeping the bus client here, out of the sdk, is
@@ -47,10 +44,9 @@ MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME where thei
MESH_BROKER_URL a plain URL instead of the credential, for the bootstrap case
```
`node dist/main.js`. On a node the controller composes the variables and the host supervises the
`node dist/main.js`. On a node the mesh composes the variables and the node-engine supervises the
process like any other host-side workload (novox/hq to-be 38). As `node-tools` the same process is
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`). The container (`Dockerfile`) is how
a module's own *service* may still be built; it is no longer how tools reach a node.
the console: MCP on `127.0.0.1:4270` (or `MESH_CONSOLE_LISTEN`).
## `mesh` — the tools for whoever is on a machine
@@ -69,14 +65,14 @@ answer took, its size in bytes, how many modules and tools it announced, whether
fit the bus, when it was last heard — and who was expected and not heard).
`mesh_search` also finds a tool by the shell command it replaces (novox/hq ADR 0245): each seat verb and
module tool says what it replaces in the controller's records (`replaces`, read from the controller's
module tool says what it replaces in the indexer's records (`replaces`, read from its
`tools` and `modules` answers), so `journalctl -u x`, `systemctl status x` or `docker ps` puts the verb
for it first, and says the command it matched as `instead_of`. A word also matches through a short table
of the words agents use for the mesh's (`logs` finds the journal). Best first: a replaced command, then
words in a name or a replaced command, then words in a description; seats before modules on a tie.
Discovery asks the bus (ADR 0197): every runtime answers the NATS services protocol's `$SRV.PING` and
`$SRV.INFO` with what it serves at that moment, and the console reads where the controller's records
`$SRV.INFO` with what it serves at that moment, and the console reads where the assigner seat's records
place each module. The console waits at least 750 ms, and up to 5 s for every runtime that answered
PING to send what it serves — a large runtime's answer crossing the bus to a broker on another machine
can arrive well after the rest. A runtime that said it is there and did not say what it serves, or
-53
View File
@@ -1,53 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# The tool runtime's own check (novox/hq ADR 0237 as amended): the second layer of a pull request's merge
# check, `mesh/repo-check`, run by the build seat in the mesh's Go toolchain — and by hand.
#
# The gate — every machine of the facts snapshot composed with this change — is the build seat's first
# layer (`mesh/merge-gate`), run because the mesh's module graph builds node-tools and mesh-tools from
# here. This is the code's own: node-tools (Go) formatted, vetted and its suite, every test on a bus of its
# own at the release the mesh runs (internal/meshtest), packages in parallel, under the race detector when
# the toolchain has a C compiler.
#
# **Said, never passed silently**: the runtime's and the console's tests launch TypeScript and Python
# bundles that import the mesh's own package (@novox/mesh-sdk), which comes from the package registry — and
# a check of code nobody has approved is given no credential for it. Where node and that package are not
# present those two packages are not tested here, and the TypeScript in node-tools/src neither.
set -eu
cd node-tools
unformatted=$(gofmt -l cmd internal)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
CGO_ENABLED=0 go vet ./...
packages=$(go list ./...)
bundleless=""
if ! command -v node >/dev/null 2>&1 || [ ! -d node_modules/@novox/mesh-sdk ]; then
echo "NOT TESTED HERE: internal/runtime and internal/console launch bundles that need node and @novox/mesh-sdk"
packages=$(printf '%s\n' "$packages" | grep -v -e '/internal/runtime$' -e '/internal/console$')
# What of the console launches no bundle runs anyway: how an address resolves (novox/hq issue 287),
# and how a search ranks (novox/hq ADR 0245).
bundleless='^(TestASeatAndAModuleOfOneNameAreEachReached|TestSearch.*|TestASearchByCommandSaysWhatItReplaces)$'
# And what of the runtime launches no bundle: how a bundle's seat traffic reaches the bus, what its
# environment holds, what a runtime on a module's own account serves (novox/hq ADR 0259), and that a write
# by compare-and-set loses no update under contention (novox/hq research 042).
runtimeless='^(TestARuntimeOnAModulesOwnAccountServesThatModuleAlone|TestNoBusWordReachesABundle|TestAToolCallNeverReachesASeatsEventAcceptOrProof|TestABundlesSeatTrafficGoesThroughItsModulesBus|TestALostCompareAndSetCarriesItsCode|TestTwoWritersIncrementingOneCounterOnTheBusLoseNoUpdate|TestAStaleWriteIsRefusedAndWritesNothing|TestACallOnTheCallerNamedSubjectReachesTheModuleWithItsCaller|TestTheSubjectThatNamesNoCallerStillWorksAndNamesNone)$'
fi
if command -v gcc >/dev/null 2>&1; then
CGO_ENABLED=1 go test -race -count=1 $packages
else
echo "NOT RACE-CHECKED: the toolchain holds no C compiler; the suite runs without the race detector"
CGO_ENABLED=0 go test -count=1 $packages
fi
if [ -n "$bundleless" ]; then
CGO_ENABLED=0 go test -count=1 -run "$bundleless" ./internal/console
fi
if [ -n "${runtimeless:-}" ]; then
CGO_ENABLED=0 go test -count=1 -run "$runtimeless" ./internal/runtime
fi
echo "NOT TESTED HERE: node-tools/src (TypeScript) needs @novox/mesh-sdk from the package registry"
-33
View File
@@ -1,33 +0,0 @@
{
"module": "mesh-tools",
"version": "1",
"slug": "tools",
"build": {
"artifacts": [
{
"name": "build",
"kind": "image",
"from": "Dockerfile",
"target": "toolchain"
},
{
"name": "runtime",
"kind": "image",
"from": "Dockerfile",
"target": "runtime"
}
],
"on": [
{
"arg": "NODE_BASE",
"image": "node@sha256:48e4b67d85f87bd551df43704e24d252f56cc5f8e9718841aace50f19948f0f9"
},
{
"arg": "MESH_SDK",
"module": "mesh-sdk",
"artifact": "lib"
}
]
},
"resources": []
}
+1 -1
View File
@@ -3,7 +3,7 @@
The node's tool runtime as a module (novox/hq ADR 0175, to-be 38 WP3). Assigned to a machine, it is
one process the host runs from this bundle, as the operator's account: it serves every assigned
module's tools and every held seat's verbs on the bus, and answers MCP on the machine's loopback —
the console (design 34). The controller composes the process (which bundles to load, where the
the console (design 34). The mesh composes the process (which bundles to load, where the
credential is, whose machine it is); this manifest says only what the machine must have for it: the
interpreter, a place for the credential, the loopback port, and leave to call every tool.
+174
View File
@@ -0,0 +1,174 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"time"
handacts "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts"
"git.novox.be/novox/mesh-sdk/go/busretry"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/approval"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// The gate's reach through the runner's own connection, in this one place (design 46 7-approval-mark §7, §13, §14): its
// state approvals, its publish of an ask under the marked module's name, and its read of the router's record.
// approvalsState is the runner's state approvals over its connection: a lost compare-and-set is stdio.ErrChanged, as
// operator-channel's client reads it.
type approvalsState struct {
conn *bus.Conn
module string
}
func (s approvalsState) Get(key string) (*stdio.StateEntry, error) {
e, err := s.conn.StateGet(s.module, approval.StateName, key)
if err != nil || e == nil {
return nil, err
}
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
}
func (s approvalsState) Create(key string, value any) (uint64, error) {
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
return changedAs(s.conn.StateCreate(s.module, approval.StateName, key, raw))
}
func (s approvalsState) Update(key string, value any, revision uint64) (uint64, error) {
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
return changedAs(s.conn.StateUpdate(s.module, approval.StateName, key, raw, revision))
}
func (s approvalsState) Put(key string, value any) (uint64, error) {
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
return s.conn.StatePut(s.module, approval.StateName, key, raw)
}
func (s approvalsState) Keys() ([]string, error) {
return s.conn.StateKeys(s.module, approval.StateName)
}
func changedAs(rev uint64, err error) (uint64, error) {
if errors.Is(err, bus.ErrStateChanged) {
return 0, stdio.ErrChanged
}
return rev, err
}
// Bound says the runner's membership binds its state approvals now (§13): a membership issued later binds it with no restart.
func (s approvalsState) Bound() bool {
m := s.conn.Membership(s.module)
if m == nil {
return false
}
for _, st := range m.State {
if st.Name == approval.StateName && st.Writes {
return true
}
}
return false
}
// publishAsk says an accept on a seat under the module's name, on the runner's own account (§7).
func publishAsk(conn *bus.Conn) func(module, subject string, body any, id string) error {
return func(module, subject string, body any, id string) error {
raw, err := json.Marshal(body)
if err != nil {
return err
}
_, err = conn.SeatPublish(module, subject, raw, id)
return err
}
}
// routerRecord reads the router's record `<module>.<id>` of the seat's state asks, on the runner's own account (§14).
func routerRecord(conn *bus.Conn) func(module, id string) (*stdio.StateEntry, error) {
return func(module, id string) (*stdio.StateEntry, error) {
e, err := conn.SeatRecord(module, "", id)
if err != nil || e == nil {
return nil, err
}
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
}
}
// boundState is the runner's key-ttl state bound (lead answers A-157, A-158).
type boundState struct {
conn *bus.Conn
module string
}
// BoundState is the runner's state its bound keys live in.
const BoundState = "bound"
func (b boundState) CreateTTL(key string, value any, ttl time.Duration) error {
raw, err := json.Marshal(value)
if err != nil {
return err
}
_, err = b.conn.StateCreateTTL(b.module, BoundState, key, raw, ttl)
return err
}
func (b boundState) Delete(key string) error { return b.conn.StateDelete(b.module, BoundState, key) }
func (b boundState) Watch(pattern string, expired func(key string)) (func(), error) {
return b.conn.StateWatch(b.module, BoundState, pattern, func(c bus.StateChange) error {
if c.Op == "expire" {
expired(c.Key)
}
return nil
})
}
// warranted asks mesh-hand-acts.warranted for one acted ask, each try a kept call open while open says so (design 46
// 7-approval-mark §16; lead answer A-157).
func warranted(conn *bus.Conn) func(asker, ask string, open func() (bool, error)) error {
return func(asker, ask string, open func() (bool, error)) error {
_, err := busretry.DoKept("mesh-hand-acts.warranted of "+asker+"."+ask,
busretry.Open{Key: asker + "." + ask, Open: func(string) (bool, error) { return open() }, Logf: log.Printf},
func() (bus.Answered, error) {
return conn.Ask("seat:"+handacts.Seat+"."+handacts.VerbWarranted, handacts.WarrantedInput{Asker: asker, Ask: ask},
"mesh.seat."+handacts.Seat+".tool."+handacts.VerbWarranted)
}, nil)
return err
}
}
// hearOne waits on the runner's connection for the first event on the subjects whose body match takes (design 34
// every-event §36: a marked wait's ask's decided).
func hearOne(conn *bus.Conn) func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
return func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
_, ok, _, err := conn.HearFirst(ctx, subjects, within, match)
return ok, err
}
}
// gateTools is the gate as mesh_approvals reads it, with its state's fill read from the bucket (§13, §18).
type gateTools struct {
*approval.Gate
state approvalsState
}
func (t gateTools) Fill() (uint64, int64, error) {
info, err := t.state.conn.StateInfo(t.state.module, approval.StateName)
if err != nil || info == nil {
return 0, 0, fmt.Errorf("the state %s's fill cannot be read: %v", approval.StateName, err)
}
return info.Bytes, info.MaxBytes, nil
}
@@ -0,0 +1,295 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
seat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"github.com/nats-io/nats.go"
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// marked is the fixture marked module's catalog half: its build src2:1 marks its tool wipe (design 46 7-approval-mark §1).
type marked struct{}
var wipeEntry = catalogseat.CatalogEntry{Module: "shell", Fp: "src2:1", Manifest: map[string]any{"needs-approval": []any{"wipe"}}}
func (marked) Ready() error { return nil }
func (marked) Revision() string { return "7" }
func (marked) Served(fp string) (approval.Entry, bool, string) {
return runnerapproval.Entry(wipeEntry), fp == wipeEntry.Fp, "not-found"
}
func (marked) Newest(m string) (approval.Entry, bool) {
return runnerapproval.Entry(wipeEntry), m == "shell"
}
func (marked) Definitions() map[string]seatdef.Definition { return map[string]seatdef.Definition{} }
func (marked) Entries() []approval.Entry { return []approval.Entry{runnerapproval.Entry(wipeEntry)} }
// bed is the runner's gate wired as main wires it, over the runner's own connection to a real bus, with stand-ins of the
// operator channel's router (it takes each ask into its record asks, as the router writes it) and of the hand-act seat.
type bed struct {
t *testing.T
conn *bus.Conn
asks nats.KeyValue
taken chan struct{}
done chan struct{}
mu sync.Mutex
asked []seat.Ask
recorded []string
holders map[string][]byte
}
func newBed(t *testing.T, bindState bool) *bed {
t.Helper()
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := nc.JetStream()
if _, err := js.AddStream(&nats.StreamConfig{Name: "SEAT_OPERATOR_CHANNEL", Subjects: []string{"mesh.seat.operator-channel.accept.>"},
Retention: nats.WorkQueuePolicy}); err != nil {
t.Fatal(err)
}
b := &bed{t: t, asks: mesh.Bucket(t, "messenger_asks"), taken: make(chan struct{}, 1), done: make(chan struct{}),
holders: map[string][]byte{"operator-channel": []byte(`{"seat":"operator-channel","scope":"mesh","holders":[{"module":"messenger"}]}`)}}
t.Cleanup(func() { close(b.done) })
mesh.Bucket(t, "node-tools_approvals")
mesh.BucketKeyTTL(t, "node-tools_bound")
// The router's stand-in: each ask accepted under the marked module's name is taken into its record, state open.
sub, err := js.PullSubscribe("mesh.seat.operator-channel.accept.>", "router", nats.BindStream("SEAT_OPERATOR_CHANNEL"))
if err != nil {
t.Fatal(err)
}
go func() {
for {
select {
case <-b.done:
return
default:
}
msgs, _ := sub.Fetch(1, nats.MaxWait(200*time.Millisecond))
for _, m := range msgs {
var a seat.Ask
if err := json.Unmarshal(m.Data, &a); err == nil {
module := strings.TrimPrefix(m.Subject, "mesh.seat.operator-channel.accept.ask.")
b.mu.Lock()
b.asked = append(b.asked, a)
b.mu.Unlock()
b.write(seat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open", Headline: a.Headline,
Expires: a.Expires, Opened: operatorchannelclient.FormatTime(time.Now()), OfferedOn: []string{"telegram"}})
select {
case b.taken <- struct{}{}:
default:
}
}
_ = m.Ack()
}
}
}()
// The hand-act seat's stand-in: warranted {asker, ask} answered, each one kept.
if _, err := nc.Subscribe("mesh.seat.mesh-hand-acts.>", func(m *nats.Msg) {
b.mu.Lock()
b.recorded = append(b.recorded, string(m.Data))
b.mu.Unlock()
_ = m.Respond([]byte(`{"result":{"recorded":true}}`))
}); err != nil {
t.Fatal(err)
}
runner := mt.MembershipOf("node-tools", "n1", false, nil)
if bindState {
runner.State = []bus.StateIssued{{Name: approval.StateName, Bucket: "node-tools_approvals", Writes: true},
{Name: BoundState, Bucket: "node-tools_bound", Writes: true}}
}
shell := mt.MembershipOf("shell", "n1", false, nil)
shell.SeatTraffic = &bus.SeatTraffic{Publish: []string{seat.AcceptSubjectAsk("shell")},
Records: []string{"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.shell.>"}}
mesh.Issue(t, runner)
mesh.Issue(t, shell)
b.conn, err = bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "n1"})
if err != nil {
t.Fatal(err)
}
b.conn.Logf = func(string, ...any) {}
t.Cleanup(b.conn.Close)
b.conn.Follow("node-tools")
b.conn.Follow("shell")
mt.Until(t, func() error {
if b.conn.Membership("node-tools") == nil || b.conn.Membership("shell") == nil {
return errors.New("not issued yet")
}
return nil
})
return b
}
// gate is a runner's gate as main makes it; record says how the run's recording goes beside the call (nil: at once).
func (b *bed) gate(record func(func())) *approval.Gate {
if record == nil {
record = func(f func()) { f() }
}
g := approval.New(approval.Deps{Node: "n1", Runner: "node-tools", Catalog: marked{}, Logf: b.t.Logf,
State: approvalsState{conn: b.conn, module: "node-tools"},
Channel: &operatorchannelclient.GateChannel{State: approvalsState{conn: b.conn, module: "node-tools"}, KeyPrefix: "n1.",
Publish: publishAsk(b.conn), ReadRecord: routerRecord(b.conn)},
Warranted: warranted(b.conn), Bounds: boundState{conn: b.conn, module: "node-tools"}, Go: record,
Holder: runnerapproval.HoldersIn(func() (map[string][]byte, error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.holders, nil
}),
// The pause while the router takes the ask ends when the stand-in has taken one: no sleeping.
After: func(time.Duration) <-chan time.Time {
c := make(chan time.Time, 1)
go func() {
select {
case <-b.taken:
case <-b.done:
}
c <- time.Now()
}()
return c
}})
_ = g.Unserved("shell", "src2:1", []string{"wipe"})
return g
}
func (b *bed) write(rec seat.AsksByAskerIDValue) {
raw, _ := json.Marshal(rec)
if _, err := b.asks.Put(rec.Key, raw); err != nil {
b.t.Fatal(err)
}
}
// decide writes the router's record of the newest ask as the router writes the operator's choice: chosen, with its warrant.
func (b *bed) decide(option string) seat.Ask {
b.mu.Lock()
a := b.asked[len(b.asked)-1]
b.mu.Unlock()
e, err := b.asks.Get("shell." + a.ID)
if err != nil {
b.t.Fatal(err)
}
var rec seat.AsksByAskerIDValue
_ = json.Unmarshal(e.Value(), &rec)
rec.State = "chosen"
rec.Warrant = &seat.Warrant{Ask: a.ID, Asker: "shell", Outcome: operatorchannelclient.OutcomeChosen, Option: option,
Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(time.Now()), AskDigest: operatorchannelclient.Digest(a),
Channel: "telegram", By: &seat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
b.write(rec)
return a
}
func wipe(args string) approval.Call {
return approval.Call{Address: "shell.wipe", Module: "shell", Name: "wipe", Fingerprint: "src2:1", Caller: "agent",
Args: json.RawMessage(args), Description: "Wipes a scratch directory."}
}
// Design 46 7-approval-mark §22: the runner's gate against the real bus with a fixture marked module: a call asked and taken
// by the operator channel's stand-in, the operator's Approve written as the router's record, the call again run once and
// recorded in the hand-act log, mesh_approvals' list, a restart between the run and warranted's answer recorded once; a
// Decline runs nothing; a gate missing a part of its check runs nothing.
func TestAMarkedCallOnTheBusIsAskedRunOnTheApprovalAndRecorded(t *testing.T) {
t.Run("approved: asked, run once, recorded, listed", func(t *testing.T) {
b := newBed(t, true)
g := b.gate(nil)
runs := 0
run := func() (any, error) { runs++; return "wiped", nil }
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
!strings.HasPrefix(err.Error(), "Not run: this needs the operator's approval. Asked on telegram at ") || runs != 0 {
t.Fatalf("first call: %v, runs %d", err, runs)
}
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || !strings.Contains(err.Error(), "still waiting for the operator's answer") {
t.Fatalf("before the answer: %v", err)
}
a := b.decide(approval.OptionApprove)
got, err := g.Run(wipe(`{"dir":"scratch"}`), run)
answered, ok := got.(approval.Answered)
if err != nil || !ok || answered.Answer != "wiped" || !strings.Contains(answered.Approval, "ask "+a.ID) || runs != 1 {
t.Fatalf("on the approval: %+v, %v, runs %d", got, err, runs)
}
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || runs != 1 {
t.Fatalf("the warrant ran twice: %v, runs %d", err, runs)
}
b.mu.Lock()
recorded := strings.Join(b.recorded, "\n")
b.mu.Unlock()
if !strings.Contains(recorded, a.ID) || !strings.Contains(recorded, "shell") {
t.Fatalf("hand-act log heard %q", recorded)
}
list, err := g.List()
if err != nil || len(list) == 0 || list[len(list)-1].Ask != a.ID || list[len(list)-1].State != "run, recorded" {
t.Fatalf("mesh_approvals lists %+v, %v", list, err)
}
})
t.Run("a restart between the run and warranted's answer records the run once", func(t *testing.T) {
b := newBed(t, true)
stopped := b.gate(func(func()) {}) // the runner stops before warranted answers
runs := 0
run := func() (any, error) { runs++; return "wiped", nil }
_, _ = stopped.Run(wipe(`{"dir":"tmp"}`), run)
a := b.decide(approval.OptionApprove)
if _, err := stopped.Run(wipe(`{"dir":"tmp"}`), run); err != nil || runs != 1 {
t.Fatalf("run: %v, runs %d", err, runs)
}
if list, _ := stopped.List(); len(list) != 1 || list[0].State != "run, not yet recorded" {
t.Fatalf("before the restart: %+v", list)
}
started := b.gate(nil)
started.Recover()
started.Recover()
b.mu.Lock()
n := len(b.recorded)
b.mu.Unlock()
if list, _ := started.List(); n != 1 || len(list) != 1 || list[0].Ask != a.ID || list[0].State != "run, recorded" || runs != 1 {
t.Fatalf("after the restart: recorded %d, %+v, runs %d", n, list, runs)
}
})
t.Run("declined: refused, nothing runs", func(t *testing.T) {
b := newBed(t, true)
g := b.gate(nil)
runs := 0
run := func() (any, error) { runs++; return nil, nil }
_, _ = g.Run(wipe(`{"dir":"scratch"}`), run)
b.decide(approval.OptionDecline)
if _, err := g.Run(wipe(`{"dir":"scratch"}`), run); err == nil || !strings.Contains(err.Error(), "the operator declined ask") || runs != 0 {
t.Fatalf("declined: %v, runs %d", err, runs)
}
})
t.Run("failing closed: no state bound, or nobody to ask, runs nothing and asks nothing", func(t *testing.T) {
unbound := newBed(t, false)
runs := 0
run := func() (any, error) { runs++; return nil, nil }
if _, err := unbound.gate(nil).Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
!strings.Contains(err.Error(), "binds no state approvals") || runs != 0 || len(unbound.asked) != 0 {
t.Fatalf("no state: %v, runs %d", err, runs)
}
b := newBed(t, true)
b.holders = map[string][]byte{}
if _, err := b.gate(nil).Run(wipe(`{"dir":"scratch"}`), run); err == nil ||
!strings.Contains(err.Error(), "no module holds operator-channel") || runs != 0 {
t.Fatalf("no holder: %v, runs %d", err, runs)
}
if got, err := b.gate(nil).Run(approval.Call{Address: "shell.tidy", Module: "shell", Name: "tidy", Fingerprint: "src2:1",
Args: json.RawMessage(`{}`)}, func() (any, error) { return "tidied", nil }); err != nil || got != "tidied" {
t.Fatalf("an unmarked tool: %v, %v", got, err)
}
})
}
+189 -31
View File
@@ -5,6 +5,7 @@
// wherever MESH_CONSOLE_LISTEN says — it is also the console, MCP over HTTP on loopback.
//
// MESH_BROKER_FILE the credential the mesh sealed to this machine for the runtime
// MESH_LEDGERS_FILE its own secret ledgers, each module's ledger credential; beside the broker file when unset
// MESH_TOOL_MODULES <module>=<entrypoint>,… the bundles to launch
// MESH_TOOL_ENV {"<module>": {"<word>": "<value>"}} what each is given (ADR 0192)
// MESH_OPERATOR_ACCOUNT whose machine this is, and MESH_OPERATOR_HOME their home
@@ -12,7 +13,10 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log"
"math/rand/v2"
@@ -21,9 +25,17 @@ import (
"syscall"
"time"
handacts "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts"
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-sdk/go/approval"
"github.com/novox/mesh-tools/node-tools/internal/alive"
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/console"
"github.com/novox/mesh-tools/node-tools/internal/firstview"
"github.com/novox/mesh-tools/node-tools/internal/launch"
"github.com/novox/mesh-tools/node-tools/internal/provide"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
@@ -40,8 +52,11 @@ func main() {
fmt.Fprintf(os.Stderr, "node-tools: %q is not a mode; this runtime serves (novox/hq ADR 0193)\n", os.Args[1])
os.Exit(2)
}
cred := credential()
conn := connectPatiently(cred)
until := time.Now().Add(startWait)
cred := credential(until)
conn := connectPatiently(cred, until)
// And the seat list it describes the seats by, read through the seat mesh-catalog (novox/hq ADR 0636 §4).
seatList := startSeatList(conn)
served, err := runtime.ServedModulesFrom(os.Getenv(runtime.ToolModules), cred.Module)
if err != nil {
@@ -51,7 +66,39 @@ func main() {
if err != nil {
log.Fatalf("node-tools: %v", err)
}
stop, err := runtime.Run(conn, served, envs, log.Printf)
// Its own secret ledgers: each module's ledger connection, the only one its history is published on (novox/hq ADR
// 0597 §9). Followed, so a credential minted or retired is taken with no restart.
var opts []runtime.Option
if file := runtime.LedgersFileOf(os.Getenv("MESH_BROKER_FILE")); file != "" {
ledgers := runtime.FollowLedgerCredentials(file, bus.Dial, log.Printf)
defer ledgers.Stop()
opts = append(opts, runtime.WithLedgerWriters(ledgers.Writer))
}
// The indexer's and the assigner's states the console indexes the mesh by, and the gate reads its marks from, come to it as
// first views, asked of the owners' own tools and followed on their views families, never read from their state (novox/hq
// ADR 0615 §1; design 46 7-approval-mark §5).
source, stopViews := firstview.Start(context.Background(), conn, log.Printf)
defer stopViews()
approvals := approvalsState{conn: conn, module: cred.Module}
gate := approval.New(approval.Deps{Node: conn.Node(), Runner: cred.Module, Catalog: source.ApprovalCatalog(definitionsOf(seatList)), Wait: source.WaitCatalog,
Findings: runnerapproval.Findings(source.Findings(), conn.Node(), cred.Module, nil), Logf: log.Printf, State: approvals,
Channel: &operatorchannelclient.GateChannel{State: approvals, KeyPrefix: conn.Node() + ".", Publish: publishAsk(conn), ReadRecord: routerRecord(conn)},
Warranted: warranted(conn), Bounds: boundState{conn: conn, module: cred.Module},
Holder: runnerapproval.HoldersIn(func() (map[string][]byte, error) { return source.State(firstview.SeatHoldersRead) }),
// The runner's own build, the one a marked wait's act binds (design 34 every-event §35).
Build: os.Getenv(runtime.FingerprintWord), Hear: hearOne(conn)})
// A failed read or write of the state approvals is made again at the next membership change the runner hears (A-157).
conn.OnMembership(func(bus.Membership) { go gate.StateAgain() })
opts = append(opts, runtime.WithGate(gate), runtime.WithSeatDefinitions(definitionsOf(seatList)))
// A run still unrecorded is asked again at each start and when the hand-act seat's holder is heard again (lead answer A-157).
if stopBounds, err := gate.WatchBounds(); err != nil {
log.Printf("[mesh-tools] the gate's bound keys are not watched (%v): an unanswered hand-act is said in the log only", err)
} else {
defer stopBounds()
}
source.OnSeatHolder(handacts.Seat, gate.Recover)
go gate.Recover()
stop, err := runtime.Run(conn, served, envs, log.Printf, opts...)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
@@ -62,6 +109,10 @@ func main() {
stopSaying = alive.Keep(conn, cred.Node, alive.Every, log.Printf)
}
// As the machine's runtime it is also the provider of `mcp-endpoint`: it answers its consumers through the
// SDK's helper wherever it runs, whether or not the module is handed (novox/hq ADR 0349 clause 12, issue 791).
stopProviding := provide.Start(conn, cred.Module, cred.Node, time.Minute, log.Printf)
listen := os.Getenv("MESH_CONSOLE_LISTEN")
if listen == "" && cred.Module == runtimeModule {
listen = consoleListen
@@ -73,7 +124,9 @@ func main() {
node = "?"
}
who := node + "." + cred.Module
up, err = console.Serve(console.NewSurface(conn, who), listen)
surface := console.NewSurface(conn, source, who)
surface.Approvals = gateTools{Gate: gate, state: approvalsState{conn: conn, module: cred.Module}}
up, err = console.Serve(surface, listen)
if err != nil {
log.Fatalf("node-tools: %v", err)
}
@@ -83,7 +136,11 @@ func main() {
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGTERM, syscall.SIGINT)
<-signals
// First of all: the service manager signalled every bundle with this process, and none is started again
// (issue 1126).
launch.ShutDown()
stopSaying()
stopProviding()
stop()
if up != nil {
_ = up.Close()
@@ -91,48 +148,149 @@ func main() {
conn.Close()
}
// credential reads the credential the mesh delivered; a missing or unreadable one is a fault of
// configuration, said and final.
func credential() bus.Credential {
// startWait is how long a runtime waits, from its start, for its broker credential to be on the machine and for the bus
// to accept the account it names (issue 1496): the node-engine starts a module's runtime before the provisioner's client
// fills the credential, and the bus carries a new account only after it reloads its users. Past it the runtime gives up,
// loudly; inside it the wait is said once.
var startWait = 5 * time.Minute
// say is a line in the runtime's journal.
func say(format string, args ...any) { fmt.Fprintf(os.Stderr, "node-tools: "+format+"\n", args...) }
// credential reads the credential the mesh delivered, waiting for it within the start's bound (issue 1496); one that can
// never be a credential (no url) is a fault of configuration, said and final.
func credential(until time.Time) bus.Credential {
file := os.Getenv("MESH_BROKER_FILE")
if file == "" {
if url := os.Getenv("MESH_BROKER_URL"); url != "" {
return bus.Credential{URL: url, Module: runtimeModule}
}
fmt.Fprintln(os.Stderr, "node-tools: set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
say("set MESH_BROKER_FILE (a sealed credential) or MESH_BROKER_URL — there is no broker to reach")
os.Exit(1)
}
raw, err := os.ReadFile(file)
cred, err := awaitCredential(file, until, say)
if err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
fmt.Fprintf(os.Stderr, "node-tools: cannot read the broker credential at %s: %v\n", file, err)
os.Exit(1)
}
if cred.URL == "" {
fmt.Fprintf(os.Stderr, "node-tools: %s carries no url — it is not a broker credential\n", file)
say("%v", err)
os.Exit(1)
}
return cred
}
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup —
// and gives up at once on what waiting cannot fix (issue 058).
func connectPatiently(cred bus.Credential) *bus.Conn {
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := bus.Connect(cred)
if err == nil {
return conn
// errNotCredential is a file that is there and readable and still no broker credential: waiting will not fix it.
var errNotCredential = errors.New("it carries no url — it is not a broker credential")
// readCredential reads the credential at file. A file not there yet, empty, or not yet whole JSON is one the node-engine
// has still to write: the error says so and waiting may fix it. errNotCredential is final.
func readCredential(file string) (bus.Credential, error) {
raw, err := os.ReadFile(file)
if err != nil {
return bus.Credential{}, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return bus.Credential{}, errors.New("the file is empty")
}
var cred bus.Credential
if err := json.Unmarshal(raw, &cred); err != nil {
return bus.Credential{}, err
}
if cred.URL == "" {
return bus.Credential{}, errNotCredential
}
return cred, nil
}
// awaitCredential is the credential at file, read at once and, while it is not there or not whole, again each time its
// directory changes (runtime.WatchFile, inotify), until the time until. The wait is said once; at until it ends with an
// error naming the file, how long it waited and what the last read found. A file that is not a credential ends it at once.
func awaitCredential(file string, until time.Time, say func(string, ...any)) (bus.Credential, error) {
cred, err := readCredential(file)
if err == nil {
return cred, nil
}
if errors.Is(err, errNotCredential) {
return bus.Credential{}, fmt.Errorf("%s: %w", file, err)
}
began := time.Now()
say("waits for its broker credential at %s (%v): the node-engine writes it once the provisioner's client fills it; "+
"it gives up at %s", file, err, until.Format("15:04:05"))
changed := make(chan struct{}, 1)
stop, werr := runtime.WatchFile(file, func() {
select {
case changed <- struct{}{}:
default:
}
if fatal := bus.Fatal(err); fatal != "" {
fmt.Fprintf(os.Stderr, "node-tools: %s — waiting will not fix this; giving up\n", fatal)
os.Exit(1)
}, func(format string, args ...any) { say(format, args...) },
fmt.Sprintf("the credential is read again every %s instead", rereadEvery))
if werr != nil {
say("cannot watch %s (%v): it is read again every %s instead", file, werr, rereadEvery)
} else {
defer stop()
}
for {
left := time.Until(until)
if left <= 0 {
return bus.Credential{}, fmt.Errorf("cannot read the broker credential at %s: waited %s for it, and the last read "+
"found: %v", file, time.Since(began).Round(time.Second), err)
}
// The watch wakes the read; the clock is only the backstop for a watch that could not be made or missed a change.
select {
case <-changed:
case <-time.After(min(left, rereadEvery)):
}
if cred, err = readCredential(file); err == nil {
say("read its broker credential at %s after %s", file, time.Since(began).Round(time.Second))
return cred, nil
}
if errors.Is(err, errNotCredential) {
return bus.Credential{}, fmt.Errorf("%s: %w", file, err)
}
}
}
// rereadEvery is the backstop of the wait for the credential, beside the watch of its directory.
var rereadEvery = 30 * time.Second
// connectPatiently retries while the bus is merely not reachable yet — the normal case at startup — and, until the start's
// bound, while the bus refuses the account: the bus carries a new account only after it reloads its users, which may come
// after the credential is on the machine (issue 1496, issue 1423). What waiting cannot fix (a pin that does not match, an
// address that is no URL) gives up at once (issue 058), and so does a refusal past the bound.
func connectPatiently(cred bus.Credential, until time.Time) *bus.Conn {
conn, err := connectWithin(cred, until, bus.Connect, time.Sleep, say)
if err != nil {
say("%v", err)
os.Exit(1)
}
return conn
}
// connectWithin is connectPatiently with its dial, its sleep and its words given, as a test gives them.
func connectWithin(cred bus.Credential, until time.Time, dial func(bus.Credential) (*bus.Conn, error), sleep func(time.Duration),
say func(string, ...any)) (*bus.Conn, error) {
refusedSaid := false
for delay := 2 * time.Second; ; delay = min(delay*2, 30*time.Second) {
conn, err := dial(cred)
if err == nil {
if refusedSaid {
say("the bus accepts the account now")
}
return conn, nil
}
wait := delay + time.Duration(rand.IntN(1000))*time.Millisecond
fmt.Fprintf(os.Stderr, "node-tools: the broker is not reachable yet (%v); retrying in %ds\n", err, int(wait.Round(time.Second)/time.Second))
time.Sleep(wait)
switch fatal := bus.Fatal(err); {
case bus.Refused(err) && time.Now().Add(wait).Before(until):
if !refusedSaid {
say("the bus refuses this account (%v): it may not carry it yet, since it loads a new account by a reload; "+
"retrying until %s", err, until.Format("15:04:05"))
refusedSaid = true
}
case bus.Refused(err) && refusedSaid:
return nil, fmt.Errorf("the bus still refuses this account (%v) at the end of the start's bound, %s; giving up", err,
until.Format("15:04:05"))
case fatal != "":
return nil, fmt.Errorf("%s — waiting will not fix this; giving up", fatal)
default:
say("the broker is not reachable yet (%v); retrying in %ds", err, int(wait.Round(time.Second)/time.Second))
}
sleep(wait)
}
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/runtime"
)
// The runner's start waits for its credential and its bus within the start's bound (issue 1496): one test, two cases, so
// design 46 7-approval-mark §22's whole test of the gate adds no test to the module (design 47 G §419, §626).
func TestTheRunnersStartWaitsForItsCredentialAndItsBus(t *testing.T) {
t.Run("the credential", awaitsCredential)
t.Run("the bus", connectsWithin)
}
// A runtime started before its credential is on the machine waits for it, woken by the watch of its directory, and gives
// up loudly at its bound; a file that is no credential ends the wait at once (issue 1496: on ace mesh-build-node's and
// mesh-build-image's runtimes exited twice on "cannot read the broker credential ... no such file").
func awaitsCredential(t *testing.T) {
const good = `{"url":"tls://bus.example:4222","module":"m"}`
rereadEvery = time.Hour // only the watch can wake the read in time
settle := runtime.Settle
runtime.Settle = 10 * time.Millisecond // a unit test stays well under a second
t.Cleanup(func() { rereadEvery, runtime.Settle = 30*time.Second, settle })
place := func(path, body string) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
tmp := path + ".tmp"
if err := os.WriteFile(tmp, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Rename(tmp, path); err != nil {
t.Fatal(err)
}
}
for _, c := range []struct {
name string
before string // written before the wait, "" none
later string // written 20ms into the wait, "" none
deeper bool // the file's directory is not there at the start
bound time.Duration
want string // "" a credential; else a word of the error
said int
}{
{name: "there at the start", before: good, bound: time.Second, said: 0},
{name: "written during the wait", later: good, bound: 5 * time.Second, said: 2},
{name: "empty, then filled", before: " ", later: good, bound: 5 * time.Second, said: 2},
{name: "its directory made during the wait", later: good, deeper: true, bound: 5 * time.Second, said: 2},
{name: "never written", bound: 100 * time.Millisecond, want: "waited", said: 1},
{name: "no url", before: `{"module":"m"}`, bound: 5 * time.Second, want: "not a broker credential", said: 0},
} {
t.Run(c.name, func(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "broker")
if c.deeper {
file = filepath.Join(dir, "var", "lib", "m", "broker")
}
if c.before != "" {
place(file, c.before)
}
if c.later != "" {
go func() { time.Sleep(20 * time.Millisecond); place(file, c.later) }()
}
var said []string
began := time.Now()
cred, err := awaitCredential(file, time.Now().Add(c.bound), func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
switch {
case c.want == "" && (err != nil || cred.URL == ""):
t.Fatalf("no credential: %v", err)
case c.want != "" && (err == nil || !strings.Contains(err.Error(), c.want)):
t.Fatalf("error %v, want one saying %q", err, c.want)
case c.want == "" && c.later != "" && time.Since(began) > 3*time.Second:
t.Fatalf("read after %s: the watch did not wake it", time.Since(began))
}
if len(said) != c.said {
t.Fatalf("said %d lines, want %d: %q", len(said), c.said, said)
}
})
}
}
// A bus that refuses the account at a start is asked again until the start's bound, said once; a refusal past it, a pin
// that does not match and an address that is no URL give up (issue 1496, issue 058).
func connectsWithin(t *testing.T) {
refused := errors.New("nats: Authorization Violation")
unreachable := errors.New("nats: no servers available for connection")
for _, c := range []struct {
name string
errs []error // what each dial answers, then a connection
bound time.Duration
fails string
saidAt int // lines said
}{
{name: "refused twice, then accepted", errs: []error{refused, refused}, bound: time.Hour, saidAt: 2},
{name: "unreachable, then accepted", errs: []error{unreachable}, bound: time.Hour, saidAt: 1},
{name: "refused past the bound", errs: []error{refused, refused}, bound: 0, fails: "refused this account"},
{name: "refused, then the bound ends", errs: []error{refused, refused, refused, refused, refused, refused}, bound: 5 * time.Second,
fails: "still refuses"},
{name: "pin", errs: []error{fmt.Errorf("tls: %w", bus.ErrPin)}, bound: time.Hour, fails: "pin"},
} {
t.Run(c.name, func(t *testing.T) {
dials := 0
dial := func(bus.Credential) (*bus.Conn, error) {
dials++
if dials <= len(c.errs) {
return nil, c.errs[dials-1]
}
return nil, nil
}
until := time.Now().Add(c.bound)
var said []string
_, err := connectWithin(bus.Credential{URL: "tls://bus.example:4222"}, until, dial, func(time.Duration) {},
func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
switch {
case c.fails == "" && err != nil:
t.Fatalf("gave up: %v", err)
case c.fails != "" && (err == nil || !strings.Contains(err.Error(), c.fails)):
t.Fatalf("error %v, want one saying %q", err, c.fails)
case c.fails == "" && len(said) != c.saidAt:
t.Fatalf("said %d lines, want %d: %q", len(said), c.saidAt, said)
}
})
}
}
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"git.novox.be/novox/mesh-sdk/go/seatlist"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// The tool runner reads the seat list as every part does (novox/hq ADR 0635 §1, ADR 0636): through the seat mesh-catalog,
// from the bootstrapped definitions until it has been read, and following the seat's events. What it holds is given to
// catalogue at every change; the console describes the seats from it, so the descriptions are the mesh's own and not a
// list compiled into the runner.
// busOfTheRunner is the reader's bus over the runner's connection: a verb of the seat asked on its subject, an event heard
// on the seat's event subject.
type busOfTheRunner struct{ conn *bus.Conn }
func (b busOfTheRunner) Ask(key string, body any) (json.RawMessage, error) {
seat, verb, _ := strings.Cut(strings.TrimPrefix(key, "seat:"), ".")
answered, err := b.conn.Ask(key, body, "mesh.seat."+seat+".tool."+verb)
if err != nil {
return nil, err
}
return answered.Result, nil
}
func (b busOfTheRunner) Subscribe(pattern string, handle func(string, json.RawMessage) error) error {
seat, event, _ := strings.Cut(pattern, ".")
_, err := b.conn.Raw(seatdef.EventSubject(seat, event), func(_ string, data []byte) []byte {
_ = handle(pattern, data)
return nil // an event is answered by nobody
})
return err
}
// seatListWithin is how long the runner waits for the first read of the seat list before it serves; the read goes on
// after it, and the descriptions are the bootstrapped ones until it ends (ADR 0636 §1).
const seatListWithin = 5 * time.Second
// startSeatList starts the runner's reader and gives catalogue its bootstrapped seats at once, then waits for the first
// read to end (the reader's Ready) for the bound, so the runner's first descriptions are of the list in the common case.
func startSeatList(conn *bus.Conn) *seatlist.Reader {
timer := time.NewTimer(seatListWithin)
defer timer.Stop()
return startSeatListOn(busOfTheRunner{conn}, timer.C, log.Printf)
}
// startSeatListOn is startSeatList over any bus, with the bound as a channel the caller gives.
func startSeatListOn(b seatlist.Bus, bound <-chan time.Time, logf func(string, ...any)) *seatlist.Reader {
reader, err := seatlist.Start(seatlist.Config{Bus: b, Bootstrap: seatlist.Bootstrapped(),
OnChange: func(seats []catalogue.Seat) { catalogue.UseSeats(seats) }})
if err != nil {
logf("node-tools: %v", fmt.Errorf("the tool runner cannot follow the seat list: %w", err))
}
if reader == nil {
return nil
}
catalogue.UseSeats(reader.Seats())
select {
case <-reader.Ready():
catalogue.UseSeats(reader.Seats())
case <-bound:
logf("node-tools: the seat list is not read yet; the runner serves with the bootstrapped seats and reads on")
}
return reader
}
// definitionsOf is the seat definitions the reader holds, by seat, which carry the verbs' marks (design 46 7-approval-mark
// §2, §5); none while the reader did not start.
func definitionsOf(reader *seatlist.Reader) func() map[string]seatdef.Definition {
return func() map[string]seatdef.Definition {
if reader == nil {
return nil
}
return reader.Definitions()
}
}
@@ -0,0 +1,69 @@
package main
import (
"encoding/json"
"errors"
"strings"
"sync/atomic"
"testing"
"time"
)
// busOfAStandIn is a seat mesh-catalog that answers after release, or never (a refusal).
type busOfAStandIn struct {
release chan struct{}
asked atomic.Int32
}
func (b *busOfAStandIn) Ask(string, any) (json.RawMessage, error) {
b.asked.Add(1)
<-b.release
return nil, errors.New("the seat does not answer")
}
func (b *busOfAStandIn) Subscribe(string, func(string, json.RawMessage) error) error { return nil }
// ADR 0636 §1: the runner waits for the first read of the seat list on the reader's Ready, but only to a bound that the
// test gives as a channel; past it, it serves with the bootstrapped seats and says so, and nothing sleeps.
func TestTheRunnerWaitsForTheFirstReadOfTheSeatListOnlyToItsBound(t *testing.T) {
for _, row := range []struct {
name string
release bool // the seat ends its first read (here by refusing) before the bound
wantSaid string
wantReady bool
}{
{"the first read ends before the bound", true, "", true},
{"the bound passes with the seat silent", false, "the seat list is not read yet", false},
} {
t.Run(row.name, func(t *testing.T) {
b := &busOfAStandIn{release: make(chan struct{})}
bound := make(chan time.Time, 1)
var said []string
if row.release {
close(b.release)
} else {
bound <- time.Time{}
}
r := startSeatListOn(b, bound, func(f string, a ...any) { said = append(said, f) })
if r == nil {
t.Fatal("no reader")
}
ended := false
select {
case <-r.Ready():
ended = true
default:
}
if ended != row.wantReady {
t.Errorf("first read ended = %v, want %v", ended, row.wantReady)
}
if got := strings.Join(said, ";"); !strings.Contains(got, row.wantSaid) || (row.wantSaid == "" && got != "") {
t.Errorf("said %q, want %q", got, row.wantSaid)
}
if !row.release {
close(b.release)
<-r.Ready()
}
})
}
}
+19 -6
View File
@@ -3,20 +3,33 @@ module github.com/novox/mesh-tools/node-tools
go 1.26.0
require (
git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshhandacts v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/provisioning/go/clients/meshprovisionerclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog v0.0.0-00010101000000-000000000000
github.com/nats-io/nats.go v1.54.0
golang.org/x/sys v0.48.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
git.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
go.uber.org/automaxprocs v1.6.0 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/time v0.16.0 // indirect
)
require git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery v0.0.0-00010101000000-000000000000 // indirect
require git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk v0.0.0-00010101000000-000000000000 // indirect
+8 -10
View File
@@ -1,13 +1,13 @@
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op h1:1BOWQJweNyvZMlpAHXGLiZQn9S+QXGcz3xh94lC0w6E=
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
@@ -16,12 +16,10 @@ github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE=
golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s=
+1 -1
View File
@@ -3,7 +3,7 @@
// **A machine heard and its runtime gone is a machine nobody can ask anything.** The node-engine says
// every minute that the machine is there; nothing said whether the runtime serving every module's
// tools and every held seat's verbs was. Now it says so too, on its own subject, every interval, with
// the interval — the controller's watchdog raises `tools-silent` after three missed — and its build.
// the interval — the mesh's watchdog raises `tools-silent` after three missed — and its build.
// Core NATS, kept by nobody, as the host's heartbeat: a lost one is the next one.
package alive
+111 -8
View File
@@ -10,6 +10,8 @@
package announce
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"strings"
"time"
@@ -35,6 +37,22 @@ var Window = 750 * time.Millisecond
// serves is waited for until it arrives or this passes, and a service that never sends it is named.
var Patience = 5 * time.Second
// Digest is the PING answer's metadata key for a digest of what the instance serves now: a console
// that holds the answer of that digest does not ask for it again (novox/hq issue 554). Every round
// pulled every runtime's whole announcement — some 880 kB, every five seconds while a session works —
// to the laptop, and the hundred-byte PING answers queued behind it.
const Digest = "digest"
// Capabilities is the service metadata key a runtime states what it can do under, a comma-separated list of capabilities
// (runtime.CallCapabilities): its machine's node-engine asks the runtime's PING and takes them into the node's record, so a
// module needing one is placed only where the runtime states it.
const Capabilities = "capabilities"
// Fingerprint is the service metadata key the tool runner states its own fingerprint under, beside Capabilities: the build
// its composed environment names (MESH_FINGERPRINT, design 47 G §424), so the node-engine ties the capabilities it takes
// into the node's record to the runner that stated them (design 46 approval mark §6, lead answer A-203).
const Fingerprint = "fingerprint"
// Shortened is the service metadata key a runtime sets when its announcement was cut to fit the bus.
const Shortened = "shortened"
@@ -157,7 +175,14 @@ func Serve(conn *bus.Conn, s Service, current func() []Endpoint) (func(), error)
var v any
switch parts[1] {
case "PING":
v = micro.Ping{ServiceIdentity: s.identity(), Type: micro.PingResponseType}
id := s.identity()
md := map[string]string{}
for k, v := range id.Metadata {
md[k] = v
}
md[Digest] = DigestOf(Info(s, current()))
id.Metadata = md
v = micro.Ping{ServiceIdentity: id, Type: micro.PingResponseType}
case "INFO":
v = Info(s, current())
case "STATS":
@@ -205,6 +230,16 @@ func Serve(conn *bus.Conn, s Service, current func() []Endpoint) (func(), error)
}, nil
}
// DigestOf is a digest of an info_response: equal for equal announcements, whatever the moment.
func DigestOf(i micro.Info) string {
body, err := json.Marshal(i)
if err != nil {
return ""
}
sum := sha256.Sum256(body)
return hex.EncodeToString(sum[:12])
}
// shorten is an info_response cut to fit the bus: descriptions to their first line, then none, the
// service's metadata saying which (so discovery can tell a person what it was given).
func shorten(logf func(string, ...any), info micro.Info, body []byte, limit int64) []byte {
@@ -244,6 +279,11 @@ type Heard struct {
Took time.Duration
Bytes int
Shortened string // why its descriptions are not whole, or ""
// Digest is the digest its PING gave for this answer, or "" when it gave none.
Digest string
// Reused is true when the instance's PING said it serves what the console already held, so its
// announcement was not asked for again; Took is then how long its PING took.
Reused bool
}
// Machine is the machine an instance runs on: its metadata's, or the one its endpoints name.
@@ -273,12 +313,24 @@ type Discovery struct {
Waited time.Duration
Heard []Heard
Silent []Instance
// Pinged is how many PING answers arrived.
Pinged int
}
// Gather asks every service on the bus who it is and what it serves, and waits at least the window
// and at most the patience: until every instance that answered PING has answered INFO. An answer
// Empty is a round in which nothing arrived at all, not even a PING: no answer about the mesh, but a
// round that failed (novox/hq issue 554).
func (d Discovery) Empty() bool { return d.Pinged == 0 && len(d.Heard) == 0 }
// Known is the announcement a console already holds for an instance at this digest, if any.
type Known func(name, id, digest string) (Heard, bool)
// Gather asks every service on the bus who it is (`$SRV.PING`), and each instance that answers what it
// serves (`$SRV.INFO.<name>.<id>`) unless its PING's digest is one `known` holds; it waits at least the
// window and at most the patience: until at least one instance answered PING and every instance that
// did has said what it serves. A round in which no PING arrives within the window is not taken as the
// mesh's answer that nothing is there (novox/hq issue 554): it waits on, up to the patience. An answer
// that is not the protocol's is skipped.
func Gather(conn *bus.Conn) (Discovery, error) {
func Gather(conn *bus.Conn, known Known) (Discovery, error) {
start := time.Now()
d := Discovery{At: start.UTC()}
pings, stopPings, err := conn.Collect("$SRV.PING", nil)
@@ -286,7 +338,7 @@ func Gather(conn *bus.Conn) (Discovery, error) {
return d, err
}
defer stopPings()
infos, stopInfos, err := conn.Collect("$SRV.INFO", nil)
infos, askInfo, stopInfos, err := conn.Collector("$SRV.INFO")
if err != nil {
return d, err
}
@@ -296,8 +348,14 @@ func Gather(conn *bus.Conn) (Discovery, error) {
pinged := map[key]int{}
answered := map[key]int{}
where := map[key]string{}
digest := map[key]string{}
asked := map[key]bool{}
reused := map[key]int{} // index in d.Heard of an answer held already
var order []key
complete := func() bool {
if len(pinged) == 0 {
return false // nobody said they are there: not an answer, a round that has not arrived
}
for k, n := range pinged {
if answered[k] < n {
return false
@@ -305,6 +363,15 @@ func Gather(conn *bus.Conn) (Discovery, error) {
}
return true
}
ask := func(k key) {
if asked[k] {
return
}
asked[k] = true
if err := askInfo("$SRV.INFO."+k.name+"."+k.id, nil); err != nil {
conn.Logf("[mesh-tools] discovery could not ask %s/%s what it serves: %v", k.name, k.id, err)
}
}
least := time.NewTimer(Window)
defer least.Stop()
most := time.NewTimer(Patience)
@@ -317,21 +384,49 @@ func Gather(conn *bus.Conn) (Discovery, error) {
if json.Unmarshal(a.Data, &p) != nil || p.Type != micro.PingResponseType {
continue
}
d.Pinged++
k := key{p.Name, p.ID}
if _, seen := pinged[k]; !seen {
_, seen := pinged[k]
if !seen {
order = append(order, k)
}
pinged[k]++
if where[k] == "" {
where[k] = p.Metadata["node"]
}
if seen {
// Two instances under one name and id: what each serves is asked of both, never assumed.
if i, ok := reused[k]; ok {
d.Heard = append(d.Heard[:i], d.Heard[i+1:]...)
for kk, j := range reused {
if j > i {
reused[kk] = j - 1
}
}
delete(reused, k)
answered[k]--
}
ask(k)
continue
}
digest[k] = p.Metadata[Digest]
if h, ok := heldAt(known, k.name, k.id, digest[k]); ok {
h.Took, h.Reused, h.Digest = a.At.Sub(start), true, digest[k]
reused[k] = len(d.Heard)
d.Heard = append(d.Heard, h)
answered[k]++
break
}
ask(k)
case a := <-infos:
var i micro.Info
if json.Unmarshal(a.Data, &i) != nil || i.Type != micro.InfoResponseType {
continue
}
answered[key{i.Name, i.ID}]++
d.Heard = append(d.Heard, Heard{Info: i, Took: a.At.Sub(start), Bytes: len(a.Data), Shortened: i.Metadata[Shortened]})
k := key{i.Name, i.ID}
answered[k]++
d.Heard = append(d.Heard, Heard{Info: i, Took: a.At.Sub(start), Bytes: len(a.Data), Shortened: i.Metadata[Shortened],
Digest: digest[k]})
case <-least.C:
windowOver = true
case <-most.C:
@@ -350,6 +445,14 @@ func Gather(conn *bus.Conn) (Discovery, error) {
}
}
// heldAt is the answer `known` holds for an instance at this digest; never for a PING without one.
func heldAt(known Known, name, id, digest string) (Heard, bool) {
if known == nil || digest == "" {
return Heard{}, false
}
return known(name, id, digest)
}
// Endpoints reads an info_response's endpoints back into what they announce.
func Endpoints(i micro.Info) []Endpoint {
var out []Endpoint
@@ -0,0 +1,126 @@
package announce
import (
"encoding/json"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go/micro"
)
// novox/hq issue 554: the laptop's console took a round in which not even a PING arrived within the
// 750 ms window as the mesh's answer that nothing is there, and refused every seat and module call. A
// PING that arrives after the window is waited for, up to the patience.
func TestARoundWaitsForAPingThatArrivesAfterTheWindow(t *testing.T) {
was := Patience
Patience = 2500 * time.Millisecond
t.Cleanup(func() { Patience = was })
late := connect(t, "node-tools", "laptop")
s := Service{Name: "node-tools", ID: "laptop", Metadata: map[string]string{"node": "laptop"}}
ping, _ := json.Marshal(micro.Ping{ServiceIdentity: s.identity(), Type: micro.PingResponseType})
stop, err := late.Raw("$SRV.PING", func(string, []byte) []byte { time.Sleep(Window + 300*time.Millisecond); return ping })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
body, _ := json.Marshal(Info(s, []Endpoint{{Kind: KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Subject: "mesh.mod.slack.tool.slack_check.laptop"}}))
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO.node-tools.laptop"} {
stop, err := late.Raw(subject, func(string, []byte) []byte { return body })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
late.Flush()
d, err := Gather(connect(t, "console", "laptop"), nil)
if err != nil {
t.Fatal(err)
}
if d.Empty() || len(d.Heard) != 1 || d.Heard[0].Machine() != "laptop" {
t.Fatalf("a PING after the window was not waited for: waited %s, pinged %d, heard %+v", d.Waited, d.Pinged, d.Heard)
}
if d.Waited < Window || d.Waited >= Patience {
t.Errorf("waited %s; the PING came after %s and patience is %s", d.Waited, Window+300*time.Millisecond, Patience)
}
}
// A round in which nothing at all arrives is said to be empty, after the patience — not answered as an
// empty mesh at the window.
func TestARoundThatHearsNothingIsEmptyAfterThePatience(t *testing.T) {
was := Patience
Patience = 1500 * time.Millisecond
t.Cleanup(func() { Patience = was })
d, err := Gather(connect(t, "console", "laptop"), nil)
if err != nil {
t.Fatal(err)
}
if !d.Empty() || d.Waited < Patience {
t.Errorf("empty %v after %s (patience %s)", d.Empty(), d.Waited, Patience)
}
}
// What a runtime announces is fetched again only when its PING's digest changed: every round pulled
// some 880 kB of announcements to the laptop, and the PING answers queued behind them (issue 554).
func TestAnUnchangedAnnouncementIsNotFetchedAgain(t *testing.T) {
runtime := connect(t, "node-tools", "desk")
var endpoints atomic.Value
endpoints.Store([]Endpoint{{Kind: KindTool, Module: "alpha", Tool: "one", Node: "desk", Subject: "mesh.mod.alpha.tool.one.desk"}})
stop, err := Serve(runtime, Service{Name: "node-tools", ID: "desk", Metadata: map[string]string{"node": "desk"}},
func() []Endpoint { return endpoints.Load().([]Endpoint) })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
runtime.Flush()
// Every request for what desk serves, counted where it is answered.
var asked atomic.Int32
spy := connect(t, "node-tools", "spy")
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO.node-tools", "$SRV.INFO.node-tools.desk"} {
stop, err := spy.Raw(subject, func(string, []byte) []byte { asked.Add(1); return nil })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
spy.Flush()
console := connect(t, "console", "laptop")
first, err := Gather(console, nil)
if err != nil || len(first.Heard) != 1 || first.Heard[0].Digest == "" || first.Heard[0].Reused {
t.Fatalf("first round: %v %+v", err, first.Heard)
}
if asked.Load() != 1 {
t.Errorf("the first round asked what desk serves %d time(s)", asked.Load())
}
held := first.Heard[0]
known := func(name, id, digest string) (Heard, bool) {
if name == held.Info.Name && id == held.Info.ID && digest == held.Digest {
return held, true
}
return Heard{}, false
}
asked.Store(0)
second, err := Gather(console, known)
if err != nil || len(second.Heard) != 1 || !second.Heard[0].Reused || len(Endpoints(second.Heard[0].Info)) != 1 {
t.Fatalf("an unchanged announcement: %v %+v", err, second.Heard)
}
if asked.Load() != 0 {
t.Errorf("an unchanged announcement was asked for %d time(s)", asked.Load())
}
endpoints.Store([]Endpoint{
{Kind: KindTool, Module: "alpha", Tool: "one", Node: "desk", Subject: "mesh.mod.alpha.tool.one.desk"},
{Kind: KindTool, Module: "alpha", Tool: "two", Node: "desk", Subject: "mesh.mod.alpha.tool.two.desk"},
})
third, err := Gather(console, known)
if err != nil || len(third.Heard) != 1 || third.Heard[0].Reused || len(Endpoints(third.Heard[0].Info)) != 2 ||
third.Heard[0].Digest == held.Digest {
t.Fatalf("a changed announcement was not fetched: %v %+v", err, third.Heard)
}
if asked.Load() != 1 {
t.Errorf("a changed announcement was asked for %d time(s)", asked.Load())
}
}
+11 -7
View File
@@ -33,18 +33,22 @@ func lagging(t *testing.T, conn *bus.Conn, s Service, lag time.Duration, endpoin
t.Fatal(err)
}
t.Cleanup(stop)
stop, err = conn.Raw("$SRV.INFO", func(string, []byte) []byte {
// Asked of every service, and of this instance by name and id — as discovery asks it (issue 554).
info := func(string, []byte) []byte {
if lag < 0 {
return nil
}
time.Sleep(lag)
body, _ := json.Marshal(Info(s, endpoints))
return body
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO." + s.Name + "." + s.ID} {
stop, err := conn.Raw(subject, info)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
conn.Flush()
}
@@ -73,7 +77,7 @@ func TestDiscoveryWaitsForEveryRuntimeThatSaidItIsThere(t *testing.T) {
asker := connect(t, "console", "desk")
start := time.Now()
d, err := Gather(asker)
d, err := Gather(asker, nil)
if err != nil {
t.Fatal(err)
}
@@ -109,7 +113,7 @@ func TestDiscoveryDoesNotWaitWhenEveryoneAnswered(t *testing.T) {
t.Cleanup(stop)
fast.Flush()
start := time.Now()
d, err := Gather(connect(t, "console", "desk"))
d, err := Gather(connect(t, "console", "desk"), nil)
if err != nil {
t.Fatal(err)
}
@@ -0,0 +1,592 @@
package approval
import (
"encoding/json"
"errors"
"reflect"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
seat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
stdio "git.novox.be/novox/mesh-sdk/go"
gate "git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"git.novox.be/novox/mesh-sdk/go/views"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// mem is the runner's state approvals as a fixture (meshtest.State).
type mem = mt.State
func newMem() *mem { return mt.NewState() }
// world is a gate on a fixture mesh: the catalog half, the router's records as the router writes them (mesh-catalog's
// messenger, asks.go), the asks published, the clock.
type world struct {
g *gate.Gate
state *mem
cat *fakeCatalog
records map[string]seat.AsksByAskerIDValue // by `<module>.<id>`
takes bool // the router takes an ask as it is published
asked []seat.Ask
now time.Time
runs int
findings *findings
bounds *bounds
// holders is the assigner's view of the seats' holders, by key; holdersErr says it is not answered yet.
holders map[string][]byte
holdersErr error
// warranted is what mesh-hand-acts.warranted answers, and recorded each ask it answered for.
warranted error
recorded []string
// deps is the gate's, made anew by rebuild; publish and record are the bus's publish and the router's record read.
deps gate.Deps
publish func(module, subject string, body any, id string) error
record func(module, id string) (*stdio.StateEntry, error)
}
// rebuild makes the gate anew from w.deps, its module's bundle loaded.
func (w *world) rebuild() {
w.g = gate.New(w.deps)
_ = w.g.Unserved("shell", "src2:1", []string{"wipe", "tidy"})
}
// bounds is the runner's key-ttl state bound as a fixture: expire hands a key's expiry to the watch.
type bounds struct {
keys map[string]time.Duration
expired func(string)
}
func (b *bounds) CreateTTL(key string, _ any, ttl time.Duration) error { b.keys[key] = ttl; return nil }
func (b *bounds) Delete(key string) error { delete(b.keys, key); return nil }
func (b *bounds) Watch(_ string, expired func(string)) (func(), error) {
b.expired = expired
return func() {}, nil
}
func (b *bounds) expire(key string) {
delete(b.keys, key)
b.expired(key)
}
func newWorld() *world {
shell := entry("src2:1", []string{"node-login-shell.execute"}, map[string]any{"needs-approval": []any{"wipe"}})
w := &world{state: newMem(), records: map[string]seat.AsksByAskerIDValue{}, takes: true, findings: &findings{},
bounds: &bounds{keys: map[string]time.Duration{}},
holders: map[string][]byte{"operator-channel": []byte(`{"seat":"operator-channel","scope":"mesh","holders":[{"module":"messenger"}]}`)},
now: time.Date(2026, 11, 12, 10, 0, 0, 0, time.UTC),
cat: &fakeCatalog{revision: "7", byFp: map[string]catalogseat.CatalogEntry{"src2:1": shell},
newest: map[string]catalogseat.CatalogEntry{"shell": shell},
defs: map[string]seatdef.Definition{"node-login-shell": {Seat: "node-login-shell",
Verbs: []seatdef.Verb{{Name: "execute", NeedsApproval: true}}}}}}
w.deps = gate.Deps{Node: "n1", Runner: "node-tools", Catalog: w.cat, State: w.state, Bounds: w.bounds,
Findings: Findings(w.findings, "n1", "node-tools", func() time.Time { return w.now }),
Channel: &operatorchannelclient.GateChannel{State: w.state, KeyPrefix: "n1.", Now: func() time.Time { return w.now },
Publish: func(module, subject string, body any, id string) error { return w.publish(module, subject, body, id) },
ReadRecord: func(module, id string) (*stdio.StateEntry, error) { return w.record(module, id) }},
Go: func(f func()) { f() },
Holder: HoldersIn(func() (map[string][]byte, error) { return w.holders, w.holdersErr }),
Warranted: func(asker, ask string, open func() (bool, error)) error {
if still, _ := open(); !still {
return nil
}
if w.warranted == nil {
w.recorded = append(w.recorded, asker+"."+ask)
}
return w.warranted
},
Now: func() time.Time { return w.now },
After: func(d time.Duration) <-chan time.Time {
w.now = w.now.Add(d)
c := make(chan time.Time, 1)
c <- w.now
return c
},
}
w.publish = func(module, subject string, body any, id string) error {
if subject != seat.AcceptSubjectAsk(module) {
return nil
}
a := body.(seat.Ask)
w.asked = append(w.asked, a)
if w.takes {
w.records[module+"."+a.ID] = seat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open",
Headline: a.Headline, Expires: a.Expires, Opened: operatorchannelclient.FormatTime(w.now), OfferedOn: []string{"telegram"}}
}
return nil
}
w.record = func(module, id string) (*stdio.StateEntry, error) {
rec, ok := w.records[module+"."+id]
if !ok {
return nil, nil
}
raw, _ := json.Marshal(rec)
return &stdio.StateEntry{Key: module + "." + id, Value: raw}, nil
}
w.rebuild()
return w
}
// decide writes the router's record of the newest ask as the router writes a person's choice: its state and warrant.
func (w *world) decide(option string, at time.Time) seat.Ask {
a := w.asked[len(w.asked)-1]
key := "shell." + a.ID
rec := w.records[key]
rec.State = "chosen"
rec.Warrant = &seat.Warrant{Ask: a.ID, Asker: "shell", Outcome: operatorchannelclient.OutcomeChosen, Option: option,
Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(at), AskDigest: operatorchannelclient.Digest(a),
By: &seat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
w.records[key] = rec
return a
}
// do makes a call through the gate, counting the runs.
func (w *world) do(c gate.Call) (any, error) {
return w.g.Run(c, func() (any, error) { w.runs++; return "done", nil })
}
// call is a fixture call of the marked tool wipe, or of another tool of the module.
func call(name, args string) gate.Call {
return gate.Call{Address: "shell." + name, Module: "shell", Name: name, Fingerprint: "src2:1", Caller: "agent", Args: json.RawMessage(args),
Description: "Wipes a scratch directory.\nMore words."}
}
// Design 46 7-approval-mark §22: the refusals and answers of a marked call, on the fixture mesh, the router's record and
// the warrant written as the router writes them.
func TestTheRefusalsAndAnswers(t *testing.T) {
for _, tc := range []struct {
name string
run func(t *testing.T, w *world)
}{
{"§9: a secret in the arguments is refused asking nothing", func(t *testing.T, w *world) {
_, err := w.do(call("wipe", `{"token":"ghp_0123456789abcdefghijABCDEFGHIJ012345"}`))
if err == nil || !strings.Contains(err.Error(), "which no ask may carry") || len(w.asked) != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§9: more than 16 KiB of arguments is refused asking nothing", func(t *testing.T, w *world) {
big, _ := json.Marshal(map[string]string{"text": strings.Repeat("a b ", 5000)})
_, err := w.do(call("wipe", string(big)))
if err == nil || !strings.Contains(err.Error(), "more than the 16384") || len(w.asked) != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§10, §12: the published ask's fields, and Not run with the router's record", func(t *testing.T, w *world) {
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
if err == nil || !strings.HasPrefix(err.Error(), "Not run: this needs the operator's approval. Asked on telegram at ") ||
!strings.Contains(err.Error(), "make the same call again within 30 minutes of the approval; the ask expires at ") {
t.Fatalf("answer %v", err)
}
if len(w.asked) != 1 {
t.Fatalf("asked %d", len(w.asked))
}
a := w.asked[0]
act, _ := gate.ActOf(call("wipe", `{"dir":"scratch"}`), "n1")
d, _ := operatorchannelclient.ActDigest(act)
if a.About != d || a.Who != "operator" || a.OnExpiry != gate.OnExpiry || a.Headline != "Run wipe on n1?" ||
a.Expires != operatorchannelclient.FormatTime(time.Date(2026, 11, 12, 11, 0, 0, 0, time.UTC)) || len(a.Options) != 2 ||
a.Options[0].ID != "approve" || a.Options[1].ID != "decline" || a.Options[0].Binds != d || a.Options[1].Binds != d ||
a.Options[1].Level != "approve" {
t.Errorf("ask %+v", a)
}
if !strings.HasPrefix(a.Explanation, "Wipes a scratch directory.\nOn n1, called through the mesh's tools, which cannot tell who called.\ndir: scratch\nFingerprint ") {
t.Errorf("explanation %q", a.Explanation)
}
var k operatorchannelclient.Kept
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
if k.Act["arg.dir"] != "scratch" || string(k.Arguments) != `{"dir":"scratch"}` || string(w.state.Vals["n1.act."+keys.EscapeName(d)]) != `"`+a.ID+`"` {
t.Errorf("kept %+v, act key %s", k, w.state.Vals["n1.act."+keys.EscapeName(d)])
}
}},
{"§12: with no record within 5 seconds, Not run, asked; not yet taken by the router", func(t *testing.T, w *world) {
w.takes = false
start := w.now
_, err := w.do(call("wipe", `{}`))
if err == nil || !strings.HasPrefix(err.Error(), "Not run, asked; not yet taken by the router (ask ") || w.now.Sub(start) > gate.RecordWithin {
t.Fatalf("answer %v after %s", err, w.now.Sub(start))
}
}},
{"§10, §17: a second identical call publishes nothing and is told it still waits", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
if len(w.asked) != 1 || err == nil || !strings.HasPrefix(err.Error(), "Not run: still waiting for the operator's answer") || w.runs != 0 {
t.Fatalf("asked %d, %v, runs %d", len(w.asked), err, w.runs)
}
}},
{"§17: an ask the router holds back as the fourth open one still waits", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
a := w.asked[0]
rec := w.records["shell."+a.ID]
rec.State, rec.HeldSince = "held-back", operatorchannelclient.FormatTime(w.now)
w.records["shell."+a.ID] = rec
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "still waiting") || len(w.asked) != 1 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§14, §15: approved, the same call runs once; made again it is already run and asks anew", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
w.decide(gate.OptionApprove, w.now.Add(time.Minute))
w.now = w.now.Add(2 * time.Minute)
if got, err := w.do(call("wipe", `{"dir":"scratch"}`)); err != nil || got.(gate.Answered).Answer != "done" || w.runs != 1 ||
!strings.HasPrefix(got.(gate.Answered).Approval, "run on the operator's approval via telegram at 2026-11-12T10:01:00Z, ask ") {
t.Fatalf("%v %v, runs %d", got, err, w.runs)
}
_, err := w.do(call("wipe", `{"dir":"scratch"}`))
if err == nil || !strings.Contains(err.Error(), "was already run; asked again") || w.runs != 1 || len(w.asked) != 2 {
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
}
}},
{"§15: two calls at the same moment run it once", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
w.decide(gate.OptionApprove, w.now)
var wg sync.WaitGroup
var mu sync.Mutex
runs := 0
for i := 0; i < 2; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_, _ = w.g.Run(call("wipe", `{}`), func() (any, error) { mu.Lock(); runs++; mu.Unlock(); return "done", nil })
}()
}
wg.Wait()
if runs != 1 {
t.Fatalf("ran %d times", runs)
}
}},
{"§14: approved for other arguments runs nothing", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
a := w.decide(gate.OptionApprove, w.now)
rec := w.records["shell."+a.ID]
other, _ := gate.ActOf(call("wipe", `{"dir":"home"}`), "n1")
d, _ := operatorchannelclient.ActDigest(other)
a.Options[0].Binds = d // the ask the person was shown bound another act
rec.Warrant.AskDigest = operatorchannelclient.Digest(a)
w.records["shell."+a.ID] = rec
var k operatorchannelclient.Kept
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
k.Ask = a
raw, _ := json.Marshal(k)
w.state.Vals["n1.ask."+a.ID] = raw
if _, err := w.do(call("wipe", `{"dir":"scratch"}`)); err == nil || w.runs != 0 || !strings.Contains(err.Error(), "nothing is done") {
t.Fatalf("%v, runs %d", err, w.runs)
}
}},
{"§14: approved 31 minutes ago asks anew", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
w.decide(gate.OptionApprove, w.now)
w.now = w.now.Add(31 * time.Minute)
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "approved too long ago") || w.runs != 0 || len(w.asked) != 2 {
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
}
}},
{"§14, §17: declined, the same call is not asked again for 10 minutes", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
w.decide(gate.OptionDecline, w.now)
w.now = w.now.Add(9 * time.Minute)
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "declined") || len(w.asked) != 1 || w.runs != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
w.now = w.now.Add(2 * time.Minute)
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "asked again") || len(w.asked) != 2 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§14, §17: expired asks anew", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
a := w.asked[0]
rec := w.records["shell."+a.ID]
rec.State = "expired"
w.records["shell."+a.ID] = rec
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "ended expired; asked again") || len(w.asked) != 2 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§14: another asker's warrant runs nothing", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
a := w.decide(gate.OptionApprove, w.now)
rec := w.records["shell."+a.ID]
rec.Warrant.Asker = "mail"
w.records["shell."+a.ID] = rec
if _, err := w.do(call("wipe", `{}`)); err == nil || w.runs != 0 || !strings.Contains(err.Error(), "its warrant does not hold") {
t.Fatalf("%v, runs %d", err, w.runs)
}
}},
{"§17: the router's refusal is said with its words", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
a := w.asked[0]
rec := w.records["shell."+a.ID]
rec.State, rec.Answer = "refused", "the asker has too many open asks"
w.records["shell."+a.ID] = rec
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the asker has too many open asks") {
t.Fatalf("%v", err)
}
}},
{"§20: a runner's own tool marked is refused unasked, never run", func(t *testing.T, w *world) {
c := call("wipe", `{}`)
c.Own = true
if _, err := w.do(c); err == nil || w.runs != 0 || len(w.asked) != 0 {
t.Fatalf("%v, runs %d, asked %d", err, w.runs, len(w.asked))
}
}},
{"§13: with no binding, an unmarked call runs and a marked one is refused", func(t *testing.T, w *world) {
w.deps.State = nil
w.rebuild()
if got, err := w.do(call("tidy", `{}`)); err != nil || got != "done" {
t.Fatalf("unmarked: %v %v", got, err)
}
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "binds no state approvals") || w.runs != 1 {
t.Fatalf("marked: %v, runs %d", err, w.runs)
}
}},
{"§13: the fixture state refuses a key the bus refuses, so every key the gate writes here is one the bus holds (A-209)", func(t *testing.T, w *world) {
if _, err := w.state.Put("n1.act.sha256:00", "x"); err == nil {
t.Fatal("the fixture state held a key with a colon, which the bus refuses")
}
if _, err := w.state.Create("n1..ask", "x"); err == nil {
t.Fatal("the fixture state held a key with an empty name")
}
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
if len(w.asked) != 1 || len(w.state.Vals) == 0 {
t.Fatalf("the gate kept nothing through the fixture state: asked %d, keys %d", len(w.asked), len(w.state.Vals))
}
}},
{"§13: an ask over 256 KiB is refused naming the cause", func(t *testing.T, w *world) {
c := call("wipe", `{}`)
c.Description = strings.Repeat("w", 300*1024)
if _, err := w.do(c); err == nil || !strings.Contains(err.Error(), "more than the 262144 a value of the state approvals holds") || len(w.asked) != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§3, §2: a seat verb its seat marks is asked for its holder", func(t *testing.T, w *world) {
c := gate.Call{Address: "node-login-shell.execute", Module: "shell", Seat: "node-login-shell", Name: "execute", Fingerprint: "src2:1", Args: json.RawMessage(`{"command":"ls"}`)}
if _, err := w.do(c); err == nil || !strings.HasPrefix(err.Error(), "Not run") || len(w.asked) != 1 || w.asked[0].Headline != "Run node-login-shell.execute on n1?" {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§16: the run is recorded once; unanswered within five minutes it is a finding, cleared when warranted answers", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{}`))
a := w.decide(gate.OptionApprove, w.now)
stop, _ := w.g.WatchBounds()
defer stop()
w.warranted = errors.New("no responders")
if _, err := w.do(call("wipe", `{}`)); err != nil || w.runs != 1 || len(w.recorded) != 0 {
t.Fatalf("%v, runs %d, recorded %v", err, w.runs, w.recorded)
}
if w.bounds.keys["n1.unanswered."+a.ID] != gate.UnansweredAfter || w.bounds.keys["n1.unrecorded."+a.ID] != gate.KeptFor-gate.SaidBefore {
t.Fatalf("bounds %v", w.bounds.keys)
}
w.bounds.expire("n1.unanswered." + a.ID)
if w.findings.raised[keys.EscapeName("n1."+a.ID)] != gate.KindCallUnanswered {
t.Fatalf("findings %v", w.findings.raised)
}
w.warranted = nil
// At a start each acted, unrecorded ask is pending in the runner's findings-stated until warranted is made again
// (design 47 G §384).
pendingKey := views.FindingKey("machine", keys.EscapeName("n1."+a.ID), gate.KindCallUnanswered)
warranted := w.deps.Warranted
var pendingAtCall bool
w.deps.Warranted = func(asker, ask string, open func() (bool, error)) error {
_, pendingAtCall = w.findings.pends[pendingKey]
return warranted(asker, ask, open)
}
w.rebuild() // a start
w.g.Recover() // a start, or the hand-act seat's holder heard again
w.g.Recover()
if !pendingAtCall || len(w.findings.pended) != 1 || len(w.findings.pends) != 0 || w.findings.pended[0] != pendingKey {
t.Fatalf("pending at the call %v, pended %v, pending after %v", pendingAtCall, w.findings.pended, w.findings.pends)
}
if len(w.recorded) != 1 || w.recorded[0] != "shell."+a.ID || len(w.findings.raised) != 0 || len(w.bounds.keys) != 0 {
t.Fatalf("recorded %v, findings %v, bounds %v", w.recorded, w.findings.raised, w.bounds.keys)
}
var k operatorchannelclient.Kept
_ = json.Unmarshal(w.state.Vals["n1.ask."+a.ID], &k)
if !k.Recorded || !k.Acted {
t.Fatalf("kept %+v", k)
}
}},
{"§19: with no holder of operator-channel in the view, a marked call is refused asking nothing", func(t *testing.T, w *world) {
w.holders["operator-channel"] = []byte(`{"seat":"operator-channel","scope":"mesh","holders":[]}`)
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "no module holds operator-channel") || len(w.asked) != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
w.holdersErr = errors.New("the first view has not been answered yet")
if _, err := w.do(call("wipe", `{}`)); err == nil || len(w.asked) != 0 {
t.Fatalf("an unread view: %v, asked %d", err, len(w.asked))
}
}},
{"§19: a publish that fails and a record that cannot be read refuse, running nothing", func(t *testing.T, w *world) {
pub := w.publish
w.publish = func(string, string, any, string) error { return errors.New("the bus refused it") }
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the ask could not be raised") {
t.Fatalf("%v", err)
}
w.publish = pub
w.state.Vals = map[string]json.RawMessage{}
_, _ = w.do(call("wipe", `{}`))
w.record = func(string, string) (*stdio.StateEntry, error) { return nil, errors.New("timeout") }
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "the router's record of its ask") || w.runs != 0 {
t.Fatalf("%v, runs %d", err, w.runs)
}
}},
{"§5, §19: a served entry that cannot be read refuses every marked call on that runtime", func(t *testing.T, w *world) {
c := call("wipe", `{}`)
c.Fingerprint = "src2:unknown"
if _, err := w.do(c); err == nil || !strings.Contains(err.Error(), "cannot be read") || len(w.asked) != 0 {
t.Fatalf("%v, asked %d", err, len(w.asked))
}
}},
{"§19: the marks unread, a call waits within 5 seconds and is refused naming mesh_first_views", func(t *testing.T, w *world) {
w.cat.ready = errors.New("the catalog half has no first view of main yet")
waited := time.Duration(0)
w.deps.Wait = func(d time.Duration) error { waited = d; return w.cat.ready }
w.rebuild()
if _, err := w.do(call("tidy", `{}`)); err == nil || !strings.Contains(err.Error(), "mesh_first_views") || waited != gate.RecordWithin || w.runs != 0 {
t.Fatalf("%v, waited %s", err, waited)
}
w.deps.Wait = func(time.Duration) error { w.cat.ready = nil; return nil }
w.rebuild()
if _, err := w.do(call("tidy", `{}`)); err != nil || w.runs != 1 {
t.Fatalf("read within the wait: %v, runs %d", err, w.runs)
}
}},
{"§19: the state failing refuses, raises approvals-unreadable after five minutes, and clears when read again", func(t *testing.T, w *world) {
stop, _ := w.g.WatchBounds()
defer stop()
w.state.Fail = errors.New("the bus is unreachable")
if _, err := w.do(call("wipe", `{}`)); err == nil || !strings.Contains(err.Error(), "cannot be read") {
t.Fatalf("%v", err)
}
if w.bounds.keys["n1.unreadable.approvals"] != gate.UnreadableAfter || len(w.findings.raised) != 0 {
t.Fatalf("bounds %v, findings %v", w.bounds.keys, w.findings.raised)
}
w.bounds.expire("n1.unreadable.approvals")
if w.findings.raised[keys.EscapeName("n1.approvals")] != gate.KindApprovalsUnreadable {
t.Fatalf("findings %v", w.findings.raised)
}
w.state.Fail = nil
w.g.StateAgain() // a membership change heard, no further call
if len(w.findings.raised) != 0 || len(w.bounds.keys) != 0 {
t.Fatalf("findings %v, bounds %v", w.findings.raised, w.bounds.keys)
}
}},
{"§18: the asks of the last day, one read whole, the marks applied with their sources, a record again and a cancel", func(t *testing.T, w *world) {
_, _ = w.do(call("wipe", `{"dir":"scratch"}`))
list, err := w.g.List()
if err != nil || len(list) != 1 || list[0].State != "asked" || list[0].Address != "shell.wipe" || list[0].Fingerprint == "" {
t.Fatalf("%+v %v", list, err)
}
whole, err := w.g.Read(list[0].Ask)
if err != nil || string(whole.Arguments) != `{"dir":"scratch"}` || !strings.HasPrefix(whole.Digest, "sha256:") {
t.Fatalf("%+v %v", whole, err)
}
applied := w.g.Applied()
if len(applied) != 1 || applied[0].Tools["wipe"] != "entry shell src2:1" || applied[0].Verbs["node-login-shell.execute"] != "seat node-login-shell" || applied[0].Revision != "7" {
t.Fatalf("applied %+v", applied)
}
if ok, src := w.g.Describe("shell", "", "wipe"); !ok || src != "entry shell src2:1" {
t.Errorf("describe %v %q", ok, src)
}
if ok, src := w.g.Describe("", "node-login-shell", "execute"); !ok || src != "seat node-login-shell" {
t.Errorf("describe a seat verb %v %q", ok, src)
}
if err := w.g.Again(list[0].Ask); err == nil {
t.Error("an ask not run was asked its hand-act again")
}
cancelled := ""
w.publish = func(module, subject string, body any, id string) error { cancelled = subject + " " + id; return nil }
if err := w.g.Cancel(list[0].Ask); err != nil || cancelled != "mesh.seat.operator-channel.accept.cancel.shell cancel."+list[0].Ask {
t.Fatalf("cancel %v %q", err, cancelled)
}
}},
} {
t.Run(tc.name, func(t *testing.T) { tc.run(t, newWorld()) })
}
// Design 46 7-approval-mark §5, §3, §4 and design 34 every-event §33: the marks a module's calls are gated by, derived from
// fixture catalog halves and served entries.
t.Run("TheMarks", func(t *testing.T) {
login := map[string]seatdef.Definition{"node-login-shell": {Seat: "node-login-shell",
Verbs: []seatdef.Verb{{Name: "execute", NeedsApproval: true}, {Name: "sessions"}}}}
v1 := entry("src2:1", []string{"node-login-shell.execute", "node-login-shell.sessions"},
map[string]any{"needs-approval": []any{"wipe"}})
v2 := entry("src2:2", []string{"node-login-shell.execute", "node-login-shell.sessions"},
map[string]any{"needs-approval": []any{"erase", "node-login-shell.sessions"}})
mail := catalogseat.CatalogEntry{Module: "mail", Fp: "src2:m", Manifest: map[string]any{
"needs-approval": []any{"mail.received"}, "emits": []any{"mail.received"},
"streams": []any{map[string]any{"name": "MAIL", "events": []any{"mail.**"}, "grant": map[string]any{"config": "readers", "events": "mail.{key}.**"}}}}}
for _, tc := range []struct {
name string
cat *fakeCatalog
module string
fp string
tools []string
verbs map[string]string
subjects []string
unread bool
}{
{name: "a seat's mark stands for every holder, and the holder's own adds up", cat: &fakeCatalog{revision: "7",
byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1}, newest: map[string]catalogseat.CatalogEntry{"shell": v1}, defs: login},
module: "shell", fp: "src2:1", tools: []string{"wipe"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell"}},
{name: "a holder's needs-approval cannot unmark its seat's verb, and marks another", cat: &fakeCatalog{revision: "8",
byFp: map[string]catalogseat.CatalogEntry{"src2:2": v2}, newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
module: "shell", fp: "src2:2", tools: []string{"erase"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
"node-login-shell.sessions": "entry shell src2:2"}},
{name: "across a version change the union stands: a dropped mark is asked while the old bundle runs", cat: &fakeCatalog{revision: "9",
byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1, "src2:2": v2}, newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
module: "shell", fp: "src2:1", tools: []string{"erase", "wipe"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
"node-login-shell.sessions": "entry shell src2:2"}},
{name: "a served entry that cannot be read leaves the newest marks, every marked call refused", cat: &fakeCatalog{revision: "9",
newest: map[string]catalogseat.CatalogEntry{"shell": v2}, defs: login},
module: "shell", fp: "src2:1", tools: []string{"erase"}, verbs: map[string]string{"node-login-shell.execute": "seat node-login-shell",
"node-login-shell.sessions": "entry shell src2:2"}, unread: true},
{name: "a bundle with no fingerprint has no served entry", cat: &fakeCatalog{revision: "9", defs: login},
module: "shell", unread: true},
{name: "a marked event and a stream with a grant give their subjects and each .>", cat: &fakeCatalog{revision: "3",
byFp: map[string]catalogseat.CatalogEntry{"src2:m": mail}, newest: map[string]catalogseat.CatalogEntry{"mail": mail}},
module: "mail", fp: "src2:m", subjects: []string{"mesh.mod.mail.event.mail.received", "mesh.mod.mail.event.mail.received.>",
"mesh.stream.mail.event.mail.>", "mesh.stream.mail.event.mail.received", "mesh.stream.mail.event.mail.received.>"}},
} {
t.Run(tc.name, func(t *testing.T) {
m := gate.Derive(tc.cat, tc.module, tc.fp)
if got := keysOf(m.Tools); !reflect.DeepEqual(got, tc.tools) {
t.Errorf("tools %v, want %v", got, tc.tools)
}
if tc.verbs == nil {
tc.verbs = map[string]string{}
}
if !reflect.DeepEqual(m.Verbs, tc.verbs) {
t.Errorf("verbs %v, want %v", m.Verbs, tc.verbs)
}
if got := keysOf(m.Subjects); !reflect.DeepEqual(got, tc.subjects) {
t.Errorf("subjects %v, want %v", got, tc.subjects)
}
if (m.Unread != "") != tc.unread || m.Revision != tc.cat.revision {
t.Errorf("unread %q, revision %q", m.Unread, m.Revision)
}
})
}
// §4: a bundle whose marks name a tool it does not register is not loaded, its finding mark-unserved raised; a bundle that
// serves it clears it, and a catalog without the mark too.
t.Run("mark-unserved raised and cleared", func(t *testing.T) {
cat := &fakeCatalog{revision: "7", byFp: map[string]catalogseat.CatalogEntry{"src2:1": v1}, newest: map[string]catalogseat.CatalogEntry{"shell": v1}, defs: login}
w := &findings{}
g := gate.New(gate.Deps{Node: "n1", Runner: "node-tools", Catalog: cat, Findings: Findings(w, "n1", "node-tools", nil)})
if got := g.Unserved("shell", "src2:1", []string{"other"}); !reflect.DeepEqual(got, []string{"wipe"}) || w.raised[keys.EscapeName("n1.shell")] != gate.KindMarkUnserved {
t.Fatalf("unserved %v, findings %v", got, w.raised)
}
if got := g.Unserved("shell", "src2:1", []string{"wipe"}); got != nil || w.raised[keys.EscapeName("n1.shell")] != "" {
t.Fatalf("a bundle serving the mark: unserved %v, findings %v", got, w.raised)
}
_ = g.Unserved("shell", "src2:1", nil)
cat.byFp["src2:1"], cat.newest["shell"] = entry("src2:1", nil, nil), entry("src2:1", nil, nil)
if got := g.Unserved("shell", "src2:1", nil); got != nil || w.raised[keys.EscapeName("n1.shell")] != "" {
t.Fatalf("a catalog without the mark: unserved %v, findings %v", got, w.raised)
}
})
})
}
+100
View File
@@ -0,0 +1,100 @@
// Package approval is the tool runner's side of mesh-sdk's approval gate (go/approval; design 47 G lead answers A-224,
// A-228): it translates what the gate reads beyond itself from the generated contracts, the gate's findings into the
// contract's findings about the runner on its machine, the catalog's entries into the gate's, and the assigner's view of
// the seats' holders into its holder function. The gate itself is mesh-sdk's; none is written here.
package approval
import (
"context"
"encoding/json"
"fmt"
"time"
assignerseat "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
gate "git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/views"
)
// HeldWriter is the runner's held finding writer (views.HeldWriter over the contract's Finding).
type HeldWriter interface {
Write(ctx context.Context, finding condseat.Finding) error
Clear(ctx context.Context, scope, id, kind, why string) error
}
// scopeMachine is the scope of the gate's findings on the tool runner: its machine (lead answer A-154).
const scopeMachine = "machine"
// Findings is the gate's finding writer over the runner's held writer: each finding about the runner's module on node, at
// scope machine, and its pending named in the writer's `findings-stated` where the writer keeps one (design 47 G §384).
func Findings(w HeldWriter, node, runner string, now func() time.Time) gate.FindingWriter {
if w == nil {
return nil
}
if now == nil {
now = time.Now
}
return findingWriter{w: w, node: node, runner: runner, now: now}
}
type findingWriter struct {
w HeldWriter
node, runner string
now func() time.Time
}
func (f findingWriter) Raise(ctx context.Context, kind, id, summary, headline string) error {
return f.w.Write(ctx, condseat.Finding{Kind: kind, Scope: scopeMachine, ID: id, Severity: condseat.FindingSeverityWarning,
Source: "approval", Observer: f.runner, Subject: condseat.Subject{Scope: "module", ID: f.runner, Machine: f.node},
Summary: summary, Headline: headline, At: f.now().UTC()})
}
func (f findingWriter) Clear(ctx context.Context, kind, id, why string) error {
return f.w.Clear(ctx, scopeMachine, id, kind, why)
}
// pender is the held writer's pending (views.HeldWriter's Pend and Judged).
type pender interface {
Pend(views.PendingKey)
Judged(key string)
}
func (f findingWriter) Pend(kind, id string, since time.Time, bound time.Duration) {
if p, ok := f.w.(pender); ok {
p.Pend(views.PendingKey{Key: views.FindingKey(scopeMachine, id, kind), Since: since, Bound: bound.String()})
}
}
func (f findingWriter) Judged(kind, id string) {
if p, ok := f.w.(pender); ok {
p.Judged(views.FindingKey(scopeMachine, id, kind))
}
}
// Entry is a catalog entry as the gate reads its marks.
func Entry(e catalogseat.CatalogEntry) gate.Entry {
manifest, _ := json.Marshal(e.Manifest)
return gate.Entry{Module: e.Module, Fingerprint: e.Fp, Serves: e.Serves, Manifest: manifest}
}
// HoldersIn is the gate's holder function over the assigner's view of the seats' holders as the runner holds it (§19):
// view answers the view's values by key, an error while its first view is not answered; a seat is held when its key names
// a holder.
func HoldersIn(view func() (map[string][]byte, error)) func(seat string) (bool, error) {
return func(seat string) (bool, error) {
values, err := view()
if err != nil {
return false, err
}
raw, ok := values[assignerseat.SeatHoldersBySeatKey(seat)]
if !ok {
return false, nil
}
var h assignerseat.SeatHolders
if err := json.Unmarshal(raw, &h); err != nil {
return false, fmt.Errorf("the holders of %s cannot be read: %w", seat, err)
}
return len(h.Holders) > 0, nil
}
}
@@ -0,0 +1,83 @@
package approval
import (
"context"
"sort"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-sdk/go/views"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
gate "git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/seatdef"
)
// fakeCatalog is a catalog half as a fixture: entries by fingerprint, the newest by module, the seat definitions.
type fakeCatalog struct {
ready error
revision string
byFp map[string]catalogseat.CatalogEntry
newest map[string]catalogseat.CatalogEntry
defs map[string]seatdef.Definition
}
func (c *fakeCatalog) Ready() error { return c.ready }
func (c *fakeCatalog) Revision() string { return c.revision }
func (c *fakeCatalog) Served(fp string) (gate.Entry, bool, string) {
e, ok := c.byFp[fp]
return Entry(e), ok, "not-found"
}
func (c *fakeCatalog) Newest(m string) (gate.Entry, bool) {
e, ok := c.newest[m]
return Entry(e), ok
}
func (c *fakeCatalog) Definitions() map[string]seatdef.Definition { return c.defs }
func (c *fakeCatalog) Entries() []gate.Entry {
var out []gate.Entry
for _, e := range c.newest {
out = append(out, Entry(e))
}
return out
}
func entry(fp string, serves []string, manifest map[string]any) catalogseat.CatalogEntry {
return catalogseat.CatalogEntry{Module: "shell", Fp: fp, Serves: serves, Manifest: manifest}
}
// findings is the runner's held writer as a fixture: each finding raised now, by id.
type findings struct {
raised map[string]string
// pends are the keys pending in the held writer's statement; pended every key ever pended.
pends map[string]views.PendingKey
pended []string
}
func (f *findings) Pend(p views.PendingKey) {
if f.pends == nil {
f.pends = map[string]views.PendingKey{}
}
f.pends[p.Key] = p
f.pended = append(f.pended, p.Key)
}
func (f *findings) Judged(key string) { delete(f.pends, key) }
func (f *findings) Write(_ context.Context, x condseat.Finding) error {
if f.raised == nil {
f.raised = map[string]string{}
}
f.raised[x.ID] = x.Kind
return nil
}
func (f *findings) Clear(_ context.Context, _, id, _, _ string) error {
delete(f.raised, id)
return nil
}
func keysOf(set map[string]string) []string {
var out []string
for k := range set {
out = append(out, k)
}
sort.Strings(out)
return out
}
+458
View File
@@ -0,0 +1,458 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go/jetstream"
)
// A module's bus state in the new form (novox/hq ADR 0595, design 25 § 12.5; issue 1966 T1): an object store, a stream,
// a work queue, a ledger, each asked for in `requires` by a local name and bound in the module's membership by the bus
// names the bus module created. **A bundle reaches only its own module's bindings**: one account per machine carries
// every module on it, so what a module may reach is decided here, from its membership, and refused with the reason
// before anything is sent, as its state is (state.go).
// Binding is the module's binding of a local name, of the storage type storage, or the reason there is none.
func (c *Conn) Binding(module, local, storage string) (membership.BusBinding, error) {
m := c.Membership(module)
if m == nil {
return membership.BusBinding{}, fmt.Errorf("%s has no membership issued on %s yet, so none of its bus state is "+
"reachable until the mesh issues one (novox/hq ADR 0595)", module, c.node)
}
var names []string
for _, b := range m.Bindings {
if b.Local != local {
names = append(names, b.Local+" ("+b.Storage+")")
continue
}
if b.Storage != storage {
return membership.BusBinding{}, fmt.Errorf("%s's %q is bound with the storage type %s, not %s (novox/hq ADR 0595)",
module, local, b.Storage, storage)
}
return b, nil
}
sort.Strings(names)
bound := "none"
if len(names) > 0 {
bound = strings.Join(names, ", ")
}
return membership.BusBinding{}, fmt.Errorf("%s was bound no %s called %q: a module reaches only the bus state its "+
"manifest asks for in `requires`, by its local name, once the mesh created it; it was bound: %s (novox/hq ADR 0595)",
module, storage, local, bound)
}
// Bindings is every binding a module's membership carries now.
func (c *Conn) Bindings(module string) []membership.BusBinding {
if m := c.Membership(module); m != nil {
return append([]membership.BusBinding(nil), m.Bindings...)
}
return nil
}
// JetStream is a JetStream handle on this connection: its one accessor, for a part that hands it to a finding writer (design 47 G
// A-16, A-27).
func (c *Conn) JetStream() (jetstream.JetStream, error) { return jetstream.New(c.nc) }
// objectBucketOf is the bus name of the module's object store of a local name: its binding's when it has one, else, for a
// name no binding of any type holds, the store grants creates for a module's old `objects` entry, `<module>_<name>`
// (catalogue.ObjectsBucket; design 47 G lead answers A-380): the bus's rights decide whether it is reached. A name bound
// with another storage type is refused as the binding refuses it.
func (c *Conn) objectBucketOf(module, local string) (membership.BusBinding, error) {
b, err := c.Binding(module, local, catalogue.StorageObjectStore)
if err == nil {
return b, nil
}
if m := c.Membership(module); m != nil {
for _, held := range m.Bindings {
if held.Local == local {
return b, err
}
}
return membership.BusBinding{Local: local, Storage: catalogue.StorageObjectStore, ObjectStore: catalogue.ObjectsBucket(module, local)}, nil
}
return b, err
}
func (c *Conn) objectStore(ctx context.Context, module, local string) (jetstream.ObjectStore, membership.BusBinding, error) {
b, err := c.objectBucketOf(module, local)
if err != nil {
return nil, b, err
}
js, err := c.JetStream()
if err != nil {
return nil, b, err
}
obs, err := js.ObjectStore(ctx, b.ObjectStore)
if errors.Is(err, jetstream.ErrBucketNotFound) || errors.Is(err, jetstream.ErrStreamNotFound) {
if b.Template == "" {
return nil, b, fmt.Errorf("%s was bound no object-store called %q, and the store %s grants creates for an old `objects` entry of that name is not on the bus",
module, local, b.ObjectStore)
}
return nil, b, fmt.Errorf("%s's objects %q are bound and their store %s is not on the bus", module, local, b.ObjectStore)
}
return obs, b, err
}
func objectInfoOf(i *jetstream.ObjectInfo) buscalls.ObjectInfo {
return buscalls.ObjectInfo{Name: i.Name, Size: i.Size, Chunks: i.Chunks, Digest: i.Digest, Description: i.Description,
Modified: i.ModTime, Deleted: i.Deleted}
}
// ObjectsPut stores one object whole under its name in the module's object store.
func (c *Conn) ObjectsPut(module string, r buscalls.ObjectsPutRequest) (buscalls.ObjectInfo, error) {
if r.Name == "" {
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return buscalls.ObjectInfo{}, err
}
info, err := obs.PutBytes(ctx, r.Name, r.Data)
if err != nil {
return buscalls.ObjectInfo{}, err
}
if r.Description != "" {
if err := obs.UpdateMeta(ctx, r.Name, jetstream.ObjectMeta{Name: r.Name, Description: r.Description}); err != nil {
return buscalls.ObjectInfo{}, err
}
if info, err = obs.GetInfo(ctx, r.Name); err != nil {
return buscalls.ObjectInfo{}, err
}
}
return objectInfoOf(info), nil
}
// ObjectsGet is one object of the module's object store, its info and its bytes; nil when there is none.
func (c *Conn) ObjectsGet(module string, r buscalls.ObjectsNameRequest) (*buscalls.ObjectData, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return nil, err
}
res, err := obs.Get(ctx, r.Name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
defer res.Close()
info, err := res.Info()
if err != nil {
return nil, err
}
data, err := obs.GetBytes(ctx, r.Name)
if err != nil {
return nil, err
}
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
}
// ObjectsDelete deletes one object of the module's object store; one that is not there is not an error.
func (c *Conn) ObjectsDelete(module string, r buscalls.ObjectsNameRequest) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return err
}
if err := obs.Delete(ctx, r.Name); err != nil && !errors.Is(err, jetstream.ErrObjectNotFound) {
return err
}
return nil
}
// ObjectsList is every object the module's object store holds.
func (c *Conn) ObjectsList(module string, r buscalls.ObjectsRequest) (buscalls.ObjectsListed, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return buscalls.ObjectsListed{}, err
}
out := buscalls.ObjectsListed{Objects: []buscalls.ObjectInfo{}}
infos, err := obs.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return out, nil
}
if err != nil {
return out, err
}
for _, i := range infos {
out.Objects = append(out.Objects, objectInfoOf(i))
}
return out, nil
}
// ObjectsInfo is how full the module's object store is: its objects, its bytes and its cap.
func (c *Conn) ObjectsInfo(module string, r buscalls.ObjectsRequest) (buscalls.Fill, error) {
listed, err := c.ObjectsList(module, r)
if err != nil {
return buscalls.Fill{}, err
}
b, _ := c.objectBucketOf(module, r.Objects)
fill, err := c.streamFill("OBJ_" + b.ObjectStore)
if err != nil {
return buscalls.Fill{}, err
}
fill.Count = uint64(len(listed.Objects))
return fill.Fill, nil
}
// ObjectsWatch hands deliver every object of the module's object store as it is, then each stored or deleted.
func (c *Conn) ObjectsWatch(module string, r buscalls.ObjectsWatchRequest, deliver func(buscalls.ObjectsChange) error) (func(), error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
obs, _, err := c.objectStore(ctx, module, r.Objects)
cancel()
if err != nil {
return nil, err
}
watchCtx, stop := context.WithCancel(context.Background())
w, err := obs.Watch(watchCtx, jetstream.IncludeHistory())
if err != nil {
stop()
return nil, err
}
current := true
for current {
select {
case i, ok := <-w.Updates():
if !ok {
stop()
return nil, errors.New("the object store's watch ended before its current objects were handed over")
}
if i == nil {
current = false
continue
}
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i), Current: true}); err != nil {
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
}
case <-time.After(StateTimeout):
_ = w.Stop()
stop()
return nil, fmt.Errorf("the object store %s did not hand its current objects in %s", r.Objects, StateTimeout)
}
}
go func() {
for i := range w.Updates() {
if i == nil {
continue
}
if err := deliver(buscalls.ObjectsChange{Watch: r.Watch, Object: objectInfoOf(i)}); err != nil {
c.Logf("[mesh-tools] %s did not take an object of %s: %v", module, r.Objects, err)
}
}
}()
return func() { _ = w.Stop(); stop() }, nil
}
// streamFill is a stream's fill and its first and last sequence, by its name on the bus.
func (c *Conn) streamFill(name string) (buscalls.StreamFilled, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.JetStream()
if err != nil {
return buscalls.StreamFilled{}, err
}
s, err := js.Stream(ctx, name)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return buscalls.StreamFilled{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
}
if err != nil {
return buscalls.StreamFilled{}, err
}
info, err := s.Info(ctx)
if err != nil {
return buscalls.StreamFilled{}, err
}
max := uint64(0)
if info.Config.MaxBytes > 0 {
max = uint64(info.Config.MaxBytes)
}
return buscalls.StreamFilled{Fill: buscalls.Fill{Count: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: max},
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
}
// streamBinding is the module's binding of a stream by its local name, a work queue among them.
func (c *Conn) streamBinding(module, local string) (membership.BusBinding, error) {
return c.Binding(module, local, catalogue.StorageStream)
}
// StreamFill is how full the module's stream is.
func (c *Conn) StreamFill(module string, r buscalls.StreamRequest) (buscalls.StreamFilled, error) {
b, err := c.streamBinding(module, r.Stream)
if err != nil {
return buscalls.StreamFilled{}, err
}
return c.streamFill(b.Stream)
}
// StreamPublish says one event on the module's own bound stream, by an acknowledged publish on the binding's subject plus
// "." plus the event, the id as the message id (design 42 zfs-owns-zfs, A-42). The module's own binding alone: another
// module's, and a binding that is not a stream, are refused by Binding.
func (c *Conn) StreamPublish(module string, r buscalls.StreamPublishRequest) (buscalls.StreamPublished, error) {
b, err := c.streamBinding(module, r.Stream)
if err != nil {
return buscalls.StreamPublished{}, err
}
if b.Subject == "" {
return buscalls.StreamPublished{}, fmt.Errorf("%s's stream %q has no subject to say an event on", module, r.Stream)
}
if r.Event == "" || strings.ContainsAny(r.Event, " *>\t\r\n") || strings.HasPrefix(r.Event, ".") || strings.HasSuffix(r.Event, ".") {
return buscalls.StreamPublished{}, fmt.Errorf("the event %q is not a name to say on %s's stream %q", r.Event, module, r.Stream)
}
body := []byte(r.Body)
if len(body) == 0 {
body = []byte("null")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.JetStream()
if err != nil {
return buscalls.StreamPublished{}, err
}
var opts []jetstream.PublishOpt
if r.ID != "" {
opts = append(opts, jetstream.WithMsgID(r.ID))
}
ack, err := js.Publish(ctx, b.Subject+"."+r.Event, body, opts...)
if err != nil {
return buscalls.StreamPublished{}, fmt.Errorf("the event %s was not acknowledged on %s's stream %q: %w", r.Event, module, r.Stream, err)
}
return buscalls.StreamPublished{Sequence: ack.Sequence}, nil
}
// StreamReadMost is the most messages one read answers.
const StreamReadMost = 1000
// StreamRead is the module's stream's messages from a sequence on, Subjects narrowing them, at most Limit of them.
func (c *Conn) StreamRead(module string, r buscalls.StreamReadRequest) (buscalls.StreamPage, error) {
b, err := c.streamBinding(module, r.Stream)
if err != nil {
return buscalls.StreamPage{}, err
}
return c.readStream(b.Stream, r.Subjects, r.From, r.Limit)
}
// readStream reads a stream by its bus name from a sequence on, the messages on the subjects filter names (every
// subject when empty), at most limit of them.
func (c *Conn) readStream(name, filter string, from uint64, limit int) (buscalls.StreamPage, error) {
if limit <= 0 || limit > StreamReadMost {
limit = StreamReadMost
}
if filter == "" {
filter = ">"
}
if from == 0 {
from = 1
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.JetStream()
if err != nil {
return buscalls.StreamPage{}, err
}
s, err := js.Stream(ctx, name)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return buscalls.StreamPage{}, fmt.Errorf("the stream %s is bound and not on the bus", name)
}
if err != nil {
return buscalls.StreamPage{}, err
}
info, err := s.Info(ctx)
if err != nil {
return buscalls.StreamPage{}, err
}
page := buscalls.StreamPage{Messages: []buscalls.StreamMessage{}, Last: info.State.LastSeq}
for seq := from; seq <= info.State.LastSeq && len(page.Messages) < limit; {
msg, err := s.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return page, err
}
body := json.RawMessage(msg.Data)
if !json.Valid(msg.Data) {
body, _ = json.Marshal(string(msg.Data))
}
page.Messages = append(page.Messages, buscalls.StreamMessage{Sequence: msg.Sequence, Subject: msg.Subject,
Body: body, At: msg.Time})
seq = msg.Sequence + 1
if len(page.Messages) == limit && seq <= info.State.LastSeq {
page.Next = seq
}
}
return page, nil
}
// WorkQueueSize is how many messages wait in the module's work queue, against its cap.
func (c *Conn) WorkQueueSize(module string, r buscalls.WorkQueueRequest) (buscalls.WorkQueueSized, error) {
b, err := c.streamBinding(module, r.Queue)
if err != nil {
return buscalls.WorkQueueSized{}, err
}
fill, err := c.streamFill(b.Stream)
if err != nil {
return buscalls.WorkQueueSized{}, err
}
return buscalls.WorkQueueSized{Fill: fill.Fill}, nil
}
// WorkQueueDead is the module's work queue's dead letters: in a work queue a message is gone once settled, so one still
// kept below a consumer's acknowledgement floor was delivered as often as the consumer allows and never settled.
func (c *Conn) WorkQueueDead(module string, r buscalls.WorkQueueDeadRequest) (buscalls.WorkQueueDeadLetters, error) {
b, err := c.streamBinding(module, r.Queue)
if err != nil {
return buscalls.WorkQueueDeadLetters{}, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.JetStream()
if err != nil {
return buscalls.WorkQueueDeadLetters{}, err
}
s, err := js.Stream(ctx, b.Stream)
if err != nil {
return buscalls.WorkQueueDeadLetters{}, fmt.Errorf("the work queue %s: %w", b.Stream, err)
}
out := buscalls.WorkQueueDeadLetters{Dead: []buscalls.DeadLetter{}}
consumers := s.ListConsumers(ctx)
for ci := range consumers.Info() {
floor := ci.AckFloor.Stream
if ci.Config.MaxDeliver <= 0 || floor == 0 {
continue
}
page, err := c.readStream(b.Stream, ci.Config.FilterSubject, 1, r.Limit)
if err != nil {
return out, err
}
for _, m := range page.Messages {
if m.Sequence > floor {
break
}
out.Dead = append(out.Dead, buscalls.DeadLetter{Sequence: m.Sequence, Subject: m.Subject,
Deliveries: ci.Config.MaxDeliver, Body: m.Body, At: m.At})
}
}
if err := consumers.Err(); err != nil {
return out, err
}
if r.Limit > 0 && len(out.Dead) > r.Limit {
out.Dead = out.Dead[:r.Limit]
}
return out, nil
}
+166
View File
@@ -0,0 +1,166 @@
package bus_test
import (
"context"
"encoding/json"
"errors"
"io"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// 1966/AC12: given a bundle whose membership binds objects `files`, when it puts and gets `a`, then it reads back what it
// put; a local name it was not bound is refused with the reason, and so is another module's binding and a binding of
// another storage type. A stream and a work queue bound alike answer their read-back, fill and size.
func TestABundleReachesItsOwnBoundObjectsAndStreamsOnly(t *testing.T) {
mesh := mt.New(t)
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx := context.Background()
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "m_files"}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: "m_old"}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "m_events", Subjects: []string{"mesh.state.m.events.>"}}); err != nil {
t.Fatal(err)
}
for _, s := range []string{"mesh.state.m.events.one", "mesh.state.m.events.two"} {
if _, err := js.Publish(ctx, s, []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
}
m := mt.MembershipOf("m", "anchor", false, nil)
m.Bindings = []membership.BusBinding{
{Local: "files", Storage: catalogue.StorageObjectStore, Template: catalogue.TemplateObjects, ObjectStore: "m_files"},
{Local: "events", Storage: catalogue.StorageStream, Template: catalogue.TemplateEvents, Stream: "m_events",
Subject: "mesh.state.m.events"},
}
other := mt.MembershipOf("other", "anchor", false, nil)
mesh.Issue(t, m)
mesh.Issue(t, other)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("m")
conn.Follow("other")
mt.Until(t, func() error {
if conn.Membership("m") == nil || conn.Membership("other") == nil {
return errors.New("not issued yet")
}
return nil
})
if _, err := conn.ObjectsPut("m", buscalls.ObjectsPutRequest{Objects: "files", Name: "a", Data: []byte("what was put")}); err != nil {
t.Fatal(err)
}
got, err := conn.ObjectsGet("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
if err != nil || got == nil || string(got.Data) != "what was put" || got.Info.Name != "a" {
t.Fatalf("read back %+v, %v", got, err)
}
// Design 47 G §518: a value over one chunk is streamed into the store and read back out of it, its description kept.
big := strings.Repeat("0123456789", 120<<10)
info, err := conn.ObjectsPutStream(ctx, "m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"}, "streamed", strings.NewReader(big))
if err != nil || info.Size != uint64(len(big)) || info.Description != "streamed" {
t.Fatalf("streamed put %+v, %v", info, err)
}
opened, rd, found, err := conn.ObjectsOpen(ctx, "m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"})
if err != nil || !found || opened.Size != uint64(len(big)) {
t.Fatalf("opened %+v %v %v", opened, found, err)
}
if back, _ := io.ReadAll(rd); string(back) != big {
t.Fatalf("read back %d bytes, want %d", len(back), len(big))
}
_ = rd.Close()
if err := conn.ObjectsDelete("m", buscalls.ObjectsNameRequest{Objects: "files", Name: "big"}); err != nil {
t.Fatal(err)
}
if listed, err := conn.ObjectsList("m", buscalls.ObjectsRequest{Objects: "files"}); err != nil || len(listed.Objects) != 1 {
t.Fatalf("listed %+v, %v", listed, err)
}
// A-380: an old `objects` entry is reached by the store grants creates for it, `<module>_<name>`, with no binding of that name.
if _, err := conn.ObjectsPut("m", buscalls.ObjectsPutRequest{Objects: "old", Name: "r", Data: []byte("old section")}); err != nil {
t.Fatalf("put into the old objects section: %v", err)
}
if listed, err := conn.ObjectsList("m", buscalls.ObjectsRequest{Objects: "old"}); err != nil || len(listed.Objects) != 1 {
t.Fatalf("listed the old objects section %+v, %v", listed, err)
}
page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", Limit: 1})
if err != nil || len(page.Messages) != 1 || page.Next != 2 || page.Last != 2 {
t.Fatalf("read %+v, %v", page, err)
}
if fill, err := conn.StreamFill("m", buscalls.StreamRequest{Stream: "events"}); err != nil || fill.Count != 2 {
t.Fatalf("fill %+v, %v", fill, err)
}
// A-42: an event said on the module's own stream lands on the binding's subject plus the event, acknowledged, and a
// second say with the same id is the same message.
said, err := conn.StreamPublish("m", buscalls.StreamPublishRequest{Stream: "events", Event: "scrub.requested",
Body: json.RawMessage(`{"pool":"tank"}`), ID: "id-1"})
if err != nil || said.Sequence != 3 {
t.Fatalf("said %+v, %v", said, err)
}
if again, err := conn.StreamPublish("m", buscalls.StreamPublishRequest{Stream: "events", Event: "scrub.requested",
Body: json.RawMessage(`{"pool":"tank"}`), ID: "id-1"}); err != nil || again.Sequence != 3 {
t.Fatalf("said again %+v, %v", again, err)
}
if page, err := conn.StreamRead("m", buscalls.StreamReadRequest{Stream: "events", From: 3,
Subjects: "mesh.state.m.events.scrub.requested"}); err != nil || len(page.Messages) != 1 ||
string(page.Messages[0].Body) != `{"pool":"tank"}` {
t.Fatalf("read back %+v, %v", page, err)
}
for _, c := range []struct {
name, module, refusal string
call func(module string) error
}{
{"a local name not bound", "m", `bound no object-store called "photos", and the store m_photos grants creates`, func(module string) error {
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "photos", Name: "a"})
return err
}},
{"another module's binding", "other", `other was bound no object-store called "files"`, func(module string) error {
_, err := conn.ObjectsGet(module, buscalls.ObjectsNameRequest{Objects: "files", Name: "a"})
return err
}},
{"a binding of another storage type", "m", `"events" is bound with the storage type stream, not object-store`, func(module string) error {
_, err := conn.ObjectsList(module, buscalls.ObjectsRequest{Objects: "events"})
return err
}},
{"say on another module's stream", "other", `other was bound no stream called "events"`, func(module string) error {
_, err := conn.StreamPublish(module, buscalls.StreamPublishRequest{Stream: "events", Event: "x"})
return err
}},
{"say on a binding that is not a stream", "m", `"files" is bound with the storage type object-store, not stream`, func(module string) error {
_, err := conn.StreamPublish(module, buscalls.StreamPublishRequest{Stream: "files", Event: "x"})
return err
}},
{"a work queue not bound", "m", `bound no stream called "jobs"`, func(module string) error {
_, err := conn.WorkQueueSize(module, buscalls.WorkQueueRequest{Queue: "jobs"})
return err
}},
} {
t.Run(c.name, func(t *testing.T) {
err := c.call(c.module)
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Fatalf("refused with %v, want the reason %q", err, c.refusal)
}
})
}
}
+411 -146
View File
@@ -18,10 +18,12 @@ import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/membership"
"log"
"regexp"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
@@ -30,13 +32,9 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
// RequestTimeout is how long a call waits for its answer — what modules already expect. The
// controller answers its verbs well inside it (its AnswerWithin, ten seconds), with a call id while a
// verb still runs, so a caller is never left guessing past this (novox/hq issue 265).
// RequestTimeout is how long a call waits for its answer — what modules already expect.
const RequestTimeout = 30 * time.Second
const assignmentsStream = "ASSIGNMENTS"
// Credential is a broker credential as the mesh delivers it (novox/hq ADR 0120).
type Credential struct {
URL string `json:"url"`
@@ -66,9 +64,20 @@ type Membership struct {
Tools string `json:"tools"`
// State is every bucket the module's code may reach, by the name it uses (novox/hq ADR 0201).
State []StateIssued `json:"state,omitempty"`
// Logs is every log the module keeps, by its local name (novox/hq ADR 0297): its own alone.
Logs []LogIssued `json:"logs,omitempty"`
// SeatTraffic is what the module's code may submit, say, take, ask, answer and read on seats that name
// their caller or their kind (novox/hq ADR 0259): the runtime does for it only what is listed here.
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are the module's own streams (novox/hq ADR 0363 §2): an event one of them carries is published
// on that stream's subject, never on EVENTS (streams.go).
Streams []StreamIssued `json:"streams,omitempty"`
// Hears are the module streams the module hears, each through a durable consumer on that stream the
// runtime binds beside the one on EVENTS (ADR 0363 §8).
Hears []HeardIssued `json:"hears,omitempty"`
// Bindings are the module's bus state in the new form, by local name (novox/hq ADR 0595 §1, mesh-sdk
// membership.BusBinding): its object stores, streams, work queues and ledgers, its own alone (bindings.go).
Bindings []membership.BusBinding `json:"bindings,omitempty"`
}
// Served is an address a tool is answered on; `{tool}` stands for the tool's name.
@@ -119,12 +128,19 @@ func Fatal(err error) string {
strings.Contains(msg, "nats: invalid url") {
return "the bus address is not a usable URL"
}
if regexp.MustCompile(`(?i)authorization violation|user authentication expired|permissions violation`).MatchString(msg) {
if Refused(err) || regexp.MustCompile(`(?i)user authentication expired|permissions violation`).MatchString(msg) {
return "the bus refused this account"
}
return ""
}
// Refused says whether the bus refused the account a connection named (an authorization violation): final for a running
// runtime, but at a start possibly only "not yet", since the bus carries a new account after it reloads its users (issue
// 1496). Fatal still names it; the caller decides how long a start waits.
func Refused(err error) bool {
return err != nil && regexp.MustCompile(`(?i)authorization violation`).MatchString(err.Error())
}
func normalizeFingerprint(f string) string {
f = strings.TrimSpace(f)
if len(f) > 7 && strings.EqualFold(f[:7], "sha256:") {
@@ -163,7 +179,7 @@ type Conn struct {
cred Credential
mu sync.Mutex
issued map[string]*Membership // module → membership; present with nil = followed, none issued
onNew []func(Membership)
onNew []*func(Membership)
subs []*nats.Subscription
Logf func(format string, args ...any)
// answering is every subject this runtime answers on, so a subscription the bus refused can be
@@ -174,6 +190,10 @@ type Conn struct {
RetryAfter []time.Duration
// calls are the caller-named calls in flight (caller.go, novox/hq issue 365).
calls callers
// waits are the waits in flight by subject, each told when the bus refuses its subscription (wait.go).
waits map[string]map[chan string]bool
// inboxSaid is when a refused answer publish was last said, by subject (answerRefused).
inboxSaid map[string]time.Time
}
// answered is one subject the runtime answers on: what to subscribe again with, and how often it was
@@ -211,6 +231,12 @@ func (c *Conn) refused(err error) {
if c.publishRefused(err) {
return
}
if c.answerRefused(err) {
return
}
if subject := subjectRefused(err); subject != "" && c.waitRefused(subject) {
return
}
m := refusedSubject.FindStringSubmatch(err.Error())
if len(m) < 2 {
return
@@ -264,9 +290,20 @@ func (c *Conn) refused(err error) {
})
}
// Connect dials the bus as the credential's module. A module's subjects come from its credential,
// never from its calls (ADR 0074).
// Connect dials the bus as the credential's module and follows its membership. A module's subjects come from its
// credential, never from its calls (ADR 0074).
func Connect(cred Credential) (*Conn, error) {
c, err := Dial(cred)
if err != nil {
return nil, err
}
c.Follow(cred.Module)
return c, nil
}
// Dial dials the bus as the credential's module and follows nothing: a connection that only publishes for its module,
// as a module's ledger connection does (novox/hq ADR 0597 §9).
func Dial(cred Credential) (*Conn, error) {
if cred.Module == "" {
return nil, errors.New("a broker credential with no module: the runtime derives its subjects " +
"from the account the mesh issued, and cannot guess which module it is")
@@ -305,7 +342,6 @@ func Connect(cred Credential) (*Conn, error) {
}
c = &Conn{nc: nc, js: js, self: cred.Module, node: cred.Node, cred: cred,
issued: map[string]*Membership{}, Logf: log.Printf, answering: map[string]*answered{}}
c.Follow(cred.Module)
return c, nil
}
@@ -318,13 +354,10 @@ func (c *Conn) Node() string { return c.node }
// Credential is what this connection was opened with.
func (c *Conn) Credential() Credential { return c.cred }
// MembershipSubject is the one address a runtime derives for an assignment (ADR 0160).
func MembershipSubject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// Follow reads a module's membership on this machine once and follows it live, so its tools are
// served where the mesh issued them (ADR 0175).
// served where the mesh issued them (ADR 0175). It is read from the grants part's state memberships, its key
// `<node>.<module>` (novox/hq ADR 0571 clause 7 and 10, mesh-sdk membership.GrantsBucket), and nowhere else:
// the old assignments stream is not read and its subject is not heard.
func (c *Conn) Follow(module string) {
c.mu.Lock()
if c.node == "" {
@@ -337,41 +370,150 @@ func (c *Conn) Follow(module string) {
}
c.issued[module] = nil
c.mu.Unlock()
subject := MembershipSubject(c.node, module)
// The subject-addressed direct get: the one address the mesh grants this account on the
// stream's API.
if got, err := c.nc.Request("$JS.API.DIRECT.GET."+assignmentsStream+"."+subject, nil, 5*time.Second); err == nil {
if got.Header.Get("Status") == "" && len(got.Data) > 0 {
var m Membership
if json.Unmarshal(got.Data, &m) == nil {
c.mu.Lock()
c.issued[module] = &m
c.mu.Unlock()
}
}
}
c.followGrants(module)
if c.Membership(module) == nil {
c.Logf("[mesh-tools] no membership issued for %s on %s yet; serving the derived shape until one arrives", module, c.node)
}
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
}
// GrantsPatience bounds the first read of a membership from the grants part's state.
var GrantsPatience = 2 * time.Second
// GrantsRetry is the first wait before the memberships state is read again after it could not be; it doubles up to
// GrantsRetryMost.
var (
GrantsRetry = time.Second
GrantsRetryMost = 30 * time.Second
)
// issue applies a membership that arrived for module, live, and re-serves on it.
func (c *Conn) issue(module string, m Membership) {
c.mu.Lock()
c.issued[module] = &m
handlers := make([]func(Membership), 0, len(c.onNew))
for _, h := range c.onNew {
handlers = append(handlers, *h)
}
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
h(m)
}
_ = c.nc.Flush()
}
// followGrants reads module's key of the grants part's state memberships and watches it: the value there now is applied
// before Follow returns, each change after it live. A state that is not there yet, or not readable (a bus answering 503
// right after its boot, a timeout), is said once and read again, with a growing wait, until it can be or the
// connection closes: Follow runs once per module, so nothing else would ever read it again.
func (c *Conn) followGrants(module string) {
if c.watchGrants(module, true) {
return
}
go func() {
wait := GrantsRetry
for !c.nc.IsClosed() {
time.Sleep(wait)
if c.nc.IsClosed() {
return
}
if c.watchGrants(module, false) {
c.Logf("[mesh-tools] the grants part's memberships are read again; %s on %s follows its key", module, c.node)
return
}
if wait *= 2; wait > GrantsRetryMost {
wait = GrantsRetryMost
}
}
}()
}
// watchGrants is one attempt of followGrants: true once the key is watched. A failure is said when say is true.
func (c *Conn) watchGrants(module string, say bool) bool {
logf := func(format string, a ...any) {
if say {
c.Logf(format, a...)
}
}
key := membership.GrantsKey(c.node, module)
// First the key's direct get, which the account is granted alone: a bus that refuses it, or a state not there yet,
// answers at once, where binding the bucket would wait out its bound on a refusal.
bucket := membership.GrantsBucket
probe, err := c.nc.Request("$JS.API.DIRECT.GET.KV_"+bucket+".$KV."+bucket+"."+key, nil, GrantsPatience)
if err != nil || probe.Header.Get("Status") == "503" {
why := "it did not answer"
if err != nil {
why = err.Error()
}
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%s); %s has no membership "+
"until it can be", bucket, why, module)
return false
}
js, err := jetstream.New(c.nc)
if err != nil {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := js.KeyValue(ctx, bucket)
if err != nil {
logf("[mesh-tools] the grants part's memberships (%s) cannot be read (%v); %s has no membership "+
"until it can be", membership.GrantsBucket, err, module)
return false
}
w, err := kv.Watch(context.Background(), key)
if err != nil {
logf("[mesh-tools] %s's key %s of the grants part's memberships cannot be watched (%v); %s has no membership "+
"until it can be", module, key, err, module)
return false
}
apply := func(e jetstream.KeyValueEntry) {
if e.Operation() != jetstream.KeyValuePut {
return // a membership withdrawn is the stream's, or the module's unassignment's, to say
}
// Read through mesh-sdk's one codec of this state, the grants part's writer's (issue 774): it takes the bare
// membership and a wrapper the part wrote before of one published or unchanged, and refuses a held-back one's
// wrapper or a value naming no node or module, which then leaves the module with no membership.
// A refused value arriving after a good one keeps the last good membership: nothing here clears it.
var m Membership
if err := json.Unmarshal(msg.Data, &m); err != nil {
c.Logf("[mesh-tools] a membership arrived that is not one: %v", err)
body, err := membership.BareGrantsMembership(e.Value())
if err == nil {
err = json.Unmarshal(body, &m)
}
if err != nil {
c.Logf("[mesh-tools] the membership at %s is not one: %v", key, err)
return
}
c.mu.Lock()
c.issued[module] = &m
handlers := append([]func(Membership){}, c.onNew...)
c.mu.Unlock()
c.Logf("[mesh-tools] %s on %s was issued a new membership; re-serving on it", module, c.node)
for _, h := range handlers {
h(m)
}
_ = c.nc.Flush()
})
if err == nil {
c.track(sub)
c.issue(module, m)
}
timer := time.NewTimer(StateTimeout)
defer timer.Stop()
for current := true; current; {
select {
case e, ok := <-w.Updates():
if !ok {
return false
}
if e == nil {
current = false
continue
}
apply(e)
case <-timer.C:
_ = w.Stop()
logf("[mesh-tools] %s's key %s of the grants part's memberships did not answer in %s; %s has no membership "+
"until it can be", module, key, StateTimeout, module)
return false
}
}
go func() {
for e := range w.Updates() {
if e != nil {
apply(e)
}
}
}()
return true
}
// Membership is what the mesh issued a module here, or nil when nothing has been issued.
@@ -403,10 +545,28 @@ func (c *Conn) Serving() []string {
// OnMembership is called with every new membership any followed module is issued.
func (c *Conn) OnMembership(h func(Membership)) {
c.mu.Lock()
c.onNew = append(c.onNew, h)
c.onNew = append(c.onNew, &h)
c.mu.Unlock()
}
// onMembership is OnMembership until the answered function is called, which drops the handler.
func (c *Conn) onMembership(h func(Membership)) func() {
mine := &h
c.mu.Lock()
c.onNew = append(c.onNew, mine)
c.mu.Unlock()
return func() {
c.mu.Lock()
defer c.mu.Unlock()
for i, had := range c.onNew {
if had == mine {
c.onNew = append(c.onNew[:i:i], c.onNew[i+1:]...)
return
}
}
}
}
func (c *Conn) track(s *nats.Subscription) {
c.mu.Lock()
c.subs = append(c.subs, s)
@@ -522,6 +682,12 @@ func (c *Conn) HandleSubjectCalled(subject string, h CalledHandler) (func(), err
return c.answerBoth(subject, "", h)
}
// HandleSubjectCalledQueue is HandleSubjectCalled in one queue group: of every holder that serves the subject in the
// group, one answers (lead answer A-393: a node-scoped seat's verb asked with no machine).
func (c *Conn) HandleSubjectCalledQueue(subject, queue string, h CalledHandler) (func(), error) {
return c.answerBoth(subject, queue, h)
}
// answerBoth answers a subject and, when it is a tool's, the caller-named pattern beside it, in one queue.
func (c *Conn) answerBoth(subject, queue string, h CalledHandler) (func(), error) {
stop, err := c.answerCalledOn(subject, queue, h)
@@ -566,9 +732,13 @@ func (c *Conn) HandleCalled(key string, h CalledHandler) (func(), error) {
}
var mu sync.Mutex
var stops []func()
stopped := false
serve := func() {
mu.Lock()
defer mu.Unlock()
if stopped {
return
}
for _, s := range stops {
s()
}
@@ -582,14 +752,18 @@ func (c *Conn) HandleCalled(key string, h CalledHandler) (func(), error) {
}
}
serve()
c.OnMembership(func(m Membership) {
// Its stop drops the membership handler too: a handler left behind would bind the stopped tool again
// on the module's next membership, and call a handler whose bundle was replaced (issue 1945).
unfollow := c.onMembership(func(m Membership) {
if m.Module == module {
serve()
}
})
return func() {
unfollow()
mu.Lock()
defer mu.Unlock()
stopped = true
for _, s := range stops {
s()
}
@@ -635,85 +809,61 @@ func (c *Conn) Ask(key string, body any, on string) (Answered, error) {
// account publish with its own name alone; asked again on the subject that names no caller only when the
// first reached nobody — nothing serves it yet, or the bus does not grant it yet — so no call is done twice.
if called := CalledSubject(subject, c.cred.User); called != "" {
answered, handingOver, fallBack, err := c.askCalled(called, data)
answered, fallBack, err := c.askCalled(called, data)
if !fallBack {
if !handingOver {
return answered, err
}
time.Sleep(HandoverPause)
answered, again, _, err := c.askCalled(called, data)
if err != nil && again {
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
}
return answered, err
}
}
answered, handingOver, err := c.askOnce(subject, data)
if !handingOver {
return answered, err
}
// **Refused because the controller is handing over: asked once more** (novox/hq issue 289). The
// controller being replaced did nothing and said so; the one after it answers the same call. Once:
// a second refusal is a controller that is not coming back, and the caller is told.
time.Sleep(HandoverPause)
answered, again, err := c.askOnce(subject, data)
if err != nil && again {
return Answered{}, fmt.Errorf("asked twice, %s apart, and both times refused as a handover: %w", HandoverPause, err)
}
return answered, err
return c.askOnce(subject, data)
}
// HandoverPause is how long a call refused as a handover waits before it is asked once more: the
// supervisor's restart of the controller, which stops the one being replaced and starts the next. A
// variable so a test need not wait.
var HandoverPause = 3 * time.Second
// ErrNoResponders is the bus's own refusal: nothing subscribes to the subject asked. It is told apart
// from a tool's own error by errors.Is, never by its text (a tool may say "503" itself, issue 1529).
var ErrNoResponders = errors.New("503 no responders")
// RetryHandingOver is the mark the controller puts on a call it refused because it is being replaced
// (mesh-controller internal/link, `"retry": "handing-over"`).
const RetryHandingOver = "handing-over"
// askOnce asks once, and says whether the answer was a refusal marked to be asked again.
func (c *Conn) askOnce(subject string, data []byte) (Answered, bool, error) {
// askOnce asks once.
func (c *Conn) askOnce(subject string, data []byte) (Answered, error) {
msg, err := c.nc.Request(subject, data, RequestTimeout)
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return Answered{}, false, errors.New("503 no responders")
return Answered{}, ErrNoResponders
}
if errors.Is(err, nats.ErrTimeout) {
return Answered{}, false, errors.New("timeout")
return Answered{}, errors.New("timeout")
}
return Answered{}, false, err
return Answered{}, err
}
return c.answerOf(subject, msg.Data)
return answerOf(msg.Data)
}
// answerOf reads one answer, asking the rest of a paged one on the subject it was asked on, and says whether it
// was a refusal marked to be asked again.
func (c *Conn) answerOf(subject string, data []byte) (Answered, bool, error) {
// An answer larger than one message comes in parts (pages.go, novox/hq issue 314).
body, err := c.whole(subject, data)
if err != nil {
return Answered{}, false, err
}
// answerOf reads one answer.
func answerOf(data []byte) (Answered, error) {
var r struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
Node string `json:"node"`
Retry string `json:"retry"`
}
if err := json.Unmarshal(body, &r); err != nil {
return Answered{}, false, err
if err := json.Unmarshal(data, &r); err != nil {
return Answered{}, err
}
if r.Error != "" {
return Answered{}, r.Retry == RetryHandingOver, errors.New(r.Error)
return Answered{}, errors.New(r.Error)
}
return Answered{Result: r.Result, Node: r.Node}, false, nil
return Answered{Result: r.Result, Node: r.Node}, nil
}
// PublishAs emits an event as a module: published into JetStream and awaited, de-duplicated by its
// own id (ADR 0042). The body is the payload; the metadata rides as headers.
func (c *Conn) PublishAs(module string, env Envelope) error {
msg := nats.NewMsg("mesh.mod." + module + ".event." + env.Key)
_, err := c.PublishAsAcked(module, env)
return err
}
// PublishAsAcked is PublishAs that also answers the sequence the bus acknowledged the event at.
func (c *Conn) PublishAsAcked(module string, env Envelope) (uint64, error) {
// On the module's own stream when its membership names one carrying the event (ADR 0363 §2), on EVENTS
// otherwise — and only there, as before, for a membership that names no stream.
msg := nats.NewMsg(eventSubject(c.Membership(module), module, env.Key))
for k, v := range env.Headers {
msg.Header.Set(k, v)
}
@@ -732,8 +882,11 @@ func (c *Conn) PublishAs(module string, env Envelope) error {
if id := env.Headers["x-event-id"]; id != "" {
opts = append(opts, nats.MsgId(id))
}
_, err := c.js.PublishMsg(msg, opts...)
return err
ack, err := c.js.PublishMsg(msg, opts...)
if err != nil {
return 0, err
}
return ack.Sequence, nil
}
// Say publishes one message on core NATS, kept by nobody: a heartbeat, whose loss is the next one.
@@ -773,6 +926,21 @@ type Arrival struct {
// that said it is there, not for a fixed moment (2026-10-05). Answers beyond what the caller has
// taken are kept up to the channel's room; an empty answer is no answer.
func (c *Conn) Collect(subject string, body []byte) (<-chan Arrival, func(), error) {
out, ask, stop, err := c.Collector(subject)
if err != nil {
return nil, func() {}, err
}
if err := ask(subject, body); err != nil {
stop()
return nil, func() {}, err
}
return out, stop, nil
}
// Collector is one inbox for many requests: every answer to every request `ask` publishes arrives on
// the one channel, until stopped — for discovery, which asks each service that said it is there for
// what it serves, as each says so (novox/hq issue 554). `what` names the requests in a log line.
func (c *Conn) Collector(what string) (<-chan Arrival, func(subject string, body []byte) error, func(), error) {
inbox := c.nc.NewRespInbox()
out := make(chan Arrival, 1024)
var mu sync.Mutex
@@ -790,11 +958,11 @@ func (c *Conn) Collect(subject string, body []byte) (<-chan Arrival, func(), err
select {
case out <- a:
default:
c.Logf("[mesh-tools] more answers to %s than are kept; one (%d bytes) dropped", subject, len(msg.Data))
c.Logf("[mesh-tools] more answers to %s than are kept; one (%d bytes) dropped", what, len(msg.Data))
}
})
if err != nil {
return nil, func() {}, err
return nil, nil, func() {}, err
}
stop := func() {
_ = sub.Unsubscribe()
@@ -802,11 +970,8 @@ func (c *Conn) Collect(subject string, body []byte) (<-chan Arrival, func(), err
stopped = true
mu.Unlock()
}
if err := c.nc.PublishRequest(subject, inbox, body); err != nil {
stop()
return nil, func() {}, err
}
return out, stop, nil
ask := func(subject string, body []byte) error { return c.nc.PublishRequest(subject, inbox, body) }
return out, ask, stop, nil
}
// MaxPayload is the largest message the bus carries, as the server told this connection.
@@ -897,8 +1062,8 @@ func (c *Conn) AskAs(module, key string, body any) (Answered, error) {
}
// EventsStream is where every module's events land, and ConsumerOf the durable consumer the
// controller makes for a module on a machine: the same names the module's own runtime bound
// (node-tools/src/broker-nats.ts), so moving the module into the node's runtime neither loses an
// mesh makes for a module on a machine: the same names the module's own runtime bound
// (the retired TypeScript runtime's broker-nats.ts), so moving the module into the node's runtime neither loses an
// event nor sees one twice.
const EventsStream = "EVENTS"
@@ -910,9 +1075,16 @@ func ConsumerOf(node, module string) string {
return node + "_" + module
}
// ToolsSuffix ends the name of the tool runner's own durable consumer of a module: the runner and the module's program each
// need every event, so each reads under a name of its own (lead answer A-397); only a work queue shares one name.
const ToolsSuffix = "_tools"
// ToolsConsumerOf is the tool runner's durable consumer of a module on a machine: `<node>_<module>_tools`.
func ToolsConsumerOf(node, module string) string { return ConsumerOf(node, module) + ToolsSuffix }
// NakDelay is how long an event a handler failed waits before it is offered again: a transient cause
// gets another attempt, a permanent one exhausts the consumer's max-deliver rather than spinning. The
// consumer then gives the event up, and the controller keeps it in DEAD_LETTERS until a person delivers
// consumer then gives the event up, and the mesh keeps it in DEAD_LETTERS until a person delivers
// it again — under `mesh.again.<consumer>.…`, from which keyOf reads the same key — or drops it
// (novox/hq issue 330).
var NakDelay = 5 * time.Second
@@ -923,64 +1095,122 @@ var NakDelay = 5 * time.Second
// handed over, and a burst would end in max-deliver with nothing failed (novox/hq issue 387).
var EventHeartbeat = 10 * time.Second
// ConsumeAs reads a module's durable consumer and hands each event to deliver (novox/hq ADR 0198):
// acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it returns an
// error, terminated when it is not an event at all. The consumer is the controller's to create; this
// binds to it and never makes one. Runs until stopped.
// ConsumeAs reads a module's durable consumer on EVENTS, and each one on a module's stream its
// membership names (novox/hq ADR 0363 §8), and hands each event to deliver one at a time (novox/hq
// ADR 0198): acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it
// returns an error, terminated when it is not an event at all. The consumers are the mesh's to
// create; this binds to them and never makes one. Runs until stopped.
func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), error) {
durable := ConsumerOf(c.node, module)
return c.ConsumeAsBeating(module, deliver, nil)
}
// IdleHeartbeat is how often the runtime tells a bundle that its module's consumer is bound and the bus answers,
// while no event arrives (stdio.OnHeartbeat), so a subscription's silence is told from a quiet source.
var IdleHeartbeat = 15 * time.Second
// Beat is the runtime's word on a module's consumer between events: it is bound and the bus answers (At), or the
// bus's heartbeat stopped arriving or the bus ended the loop (Missed, with Why). Its JSON is the stdio Heartbeat.
type Beat struct {
At time.Time `json:"at"`
Missed bool `json:"missed,omitempty"`
Why string `json:"why,omitempty"`
}
// ConsumeAsBeating is ConsumeAs that also says, to beat, how the module's consumer on EVENTS stands: a Beat at each
// IdleHeartbeat in which no event arrived and the bus answered about the consumer, and one Missed (said once) when
// the bus's own heartbeat to the loop stopped arriving, the bus ended the loop or no longer answers. A Beat that is
// not Missed follows the first answer after one that was. A nil beat says nothing.
func (c *Conn) ConsumeAsBeating(module string, deliver func(Envelope) error, beat func(Beat)) (func(), error) {
return c.ConsumeAsBeatingWhen(module, deliver, beat, nil)
}
// tickHook, when set before a consume starts, is called after each idle heartbeat interval was handled (tests only,
// through SetTickHook), so a test counts beats instead of sleeping. A consume reads it once, when it starts.
var tickHook func()
// ConsumeAsBeatingWhen is ConsumeAsBeating that asks the bus about the consumer, and says anything, only while wanted
// answers true (nil is always): a module whose children asked for no heartbeat costs the bus no consumer-info request.
func (c *Conn) ConsumeAsBeatingWhen(module string, deliver func(Envelope) error, beat func(Beat), wanted func() bool) (func(), error) {
asked := func() bool { return beat != nil && (wanted == nil || wanted()) }
durable := ToolsConsumerOf(c.node, module)
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var lastEvent atomic.Int64
var missed atomic.Bool
say := func(why string) {
if asked() && missed.CompareAndSwap(false, true) {
beat(Beat{At: time.Now().UTC(), Missed: true, Why: why})
}
}
h := newHearing(c, js, module, func(env Envelope) error {
lastEvent.Store(time.Now().UnixNano())
missed.Store(false) // an event arrived: the consumer is heard, whatever was said before
return deliver(env)
})
// Bound by name, never created and never matched against a subject: the consumer and its
// filters are the controller's (design 29 §3), exactly as the module's own runtime bound it.
consumer, err := js.Consumer(ctx, EventsStream, durable)
// filters are the mesh's (design 29 §3), exactly as the module's own runtime bound it.
events, _, doing, err := h.bind(EventsStream, durable, func(err error) {
if errors.Is(err, jetstream.ErrNoHeartbeat) {
say("the bus's heartbeat to the consumer " + durable + " stopped arriving")
}
})
if err != nil {
return nil, fmt.Errorf("binding %s's consumer %s: %w", module, durable, err)
return nil, fmt.Errorf("%s %s's consumer %s: %w", doing, module, durable, err)
}
reading, err := consumer.Consume(func(msg jetstream.Msg) {
env, ok := envelopeOf(msg.Subject(), msg.Headers(), msg.Data())
if !ok {
// Unparseable: redelivering bytes no version of a handler can read is a loop.
_ = msg.Term()
return
// And every consumer on a module's stream its membership names (novox/hq ADR 0363 §8), followed as the
// membership changes. A membership naming none binds nothing more: the module hears EVENTS alone.
h.start()
if beat != nil {
every := IdleHeartbeat
hook := tickHook
tickOnce := func() {
select {
case <-events.Closed():
say("the bus ended reading the consumer " + durable)
return
default:
}
if !asked() {
return
}
ctx, cancel := context.WithTimeout(context.Background(), every)
_, err := js.Consumer(ctx, EventsStream, durable)
cancel()
if err != nil {
say("the bus does not answer about the consumer " + durable + ": " + err.Error())
return
}
if time.Since(time.Unix(0, lastEvent.Load())) < every {
return // an event arrived in this interval: it was the word
}
missed.Store(false)
beat(Beat{At: time.Now().UTC()})
}
// Kept alive while the bundle works on it, as a worker's work is (seat.go): the consumer's ack wait
// (30 seconds) is for a runtime that died, not a handler that is slow — the runtime gives a handler
// launch.EventTimeout. Without it an event handled longer than the ack wait was offered again while
// it was handled, and counted toward max-deliver (novox/hq issue 387).
done, stopped := make(chan struct{}), make(chan struct{})
go func() {
defer close(stopped)
tick := time.NewTicker(EventHeartbeat)
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-done:
case <-h.done:
return
case <-tick.C:
_ = msg.InProgress()
}
tickOnce()
if hook != nil {
hook()
}
}
}()
err := deliver(env)
close(done)
// No in-progress after the answer: one after a Nak would stretch its delay to the whole ack wait.
<-stopped
if err != nil {
_ = msg.NakWithDelay(NakDelay)
return
}
_ = msg.Ack()
}, jetstream.PullMaxMessages(1))
if err != nil {
return nil, fmt.Errorf("reading %s's consumer %s: %w", module, durable, err)
}
var once sync.Once
return func() { once.Do(reading.Stop) }, nil
return func() {
once.Do(func() {
h.stop()
events.Stop()
})
}, nil
}
// envelopeOf rebuilds the envelope a module sees from a delivered event: its key is the emitter and
@@ -1009,3 +1239,38 @@ func keyOf(subject string) string {
}
return event
}
// answerRefused says a refused publish of an answer (to a caller's inbox: a permission, or the single-use
// allowance of a reply expired) once per subject per minute, never silently (issue 1966: a caller waited out its
// time for an answer the bus had refused). It reports whether the refusal was one.
func (c *Conn) answerRefused(err error) bool {
m := refusedPublish.FindStringSubmatch(err.Error())
if len(m) < 2 || !strings.HasPrefix(strings.Trim(m[1], `"`), "_INBOX.") {
return false
}
// Keyed without the reply's last token (a new one per request), so a caller's burst is one line.
subject := strings.Trim(m[1], `"`)
if dot := strings.LastIndex(subject, "."); dot > len("_INBOX.") {
subject = subject[:dot]
}
c.mu.Lock()
if c.inboxSaid == nil {
c.inboxSaid = map[string]time.Time{}
}
now := time.Now()
for k, at := range c.inboxSaid {
if now.Sub(at) >= time.Minute {
delete(c.inboxSaid, k)
}
}
_, seen := c.inboxSaid[subject]
say := !seen
if say {
c.inboxSaid[subject] = now
}
c.mu.Unlock()
if say && c.Logf != nil {
c.Logf("[mesh-tools] the bus refused this account an answer to %s (%v); the caller gets none", subject, err)
}
return true
}
+11 -11
View File
@@ -18,7 +18,7 @@ import (
// mesh.seat.<seat>.call.<verb>.<caller> a mesh seat's verb
// mesh.seat.<seat>.call.<verb>.<node>.<caller> a node seat's verb on one machine
//
// The last token is the bus user that published the call, and the controller grants each user the publish of
// The last token is the bus user that published the call, and the mesh grants each user the publish of
// these subjects with its own name there and no other — so the caller is a fact the bus server enforces, and
// the runtime hands it to the module from the subject a call arrived on, never from the payload.
//
@@ -32,7 +32,7 @@ import (
const CallKind = "call"
// A bus user's name carries dots (`<node>.<module>`, `person.<name>`); a subject token cannot. In the token
// each dot is written `~`, which no part of a bus user's name may hold (the controller refuses any part that
// each dot is written `~`, which no part of a bus user's name may hold (the mesh refuses any part that
// is not letters, digits, `_` or `-`), so the token is the user's name and nothing else.
const userDot = "~"
@@ -118,7 +118,7 @@ type callers struct {
pending map[string]map[uint64]context.CancelCauseFunc
}
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the controller's
// errCallRefused is the bus refusing a caller-named publish: this user is not granted it yet (the mesh's
// grant not delivered to the bus), so the call is asked again on the subject that names no caller.
var errCallRefused = errors.New("the bus refused the caller-named subject")
@@ -150,7 +150,7 @@ func (c *Conn) publishRefused(err error) bool {
// askCalled asks once on a caller-named subject. fallBack says the call reached nobody — nothing serves that
// subject yet, or the bus refused this user its publish — so it may be asked on the subject that names no
// caller without being done twice.
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handingOver, fallBack bool, err error) {
func (c *Conn) askCalled(subject string, data []byte) (answered Answered, fallBack bool, err error) {
ctx, cancel := context.WithCancelCause(context.Background())
ctx, stop := context.WithTimeout(ctx, RequestTimeout)
defer stop()
@@ -180,22 +180,22 @@ func (c *Conn) askCalled(subject string, data []byte) (answered Answered, handin
if err != nil {
switch {
case errors.Is(context.Cause(ctx), errCallRefused):
return Answered{}, false, true, nil
return Answered{}, true, nil
case errors.Is(err, nats.ErrNoResponders):
c.saidFallBack(subject)
return Answered{}, false, true, nil
return Answered{}, true, nil
case errors.Is(err, context.DeadlineExceeded) || errors.Is(err, nats.ErrTimeout):
return Answered{}, false, false, errors.New("timeout")
return Answered{}, false, errors.New("timeout")
}
return Answered{}, false, false, err
return Answered{}, false, err
}
answered, handingOver, err = c.answerOf(subject, msg.Data)
return answered, handingOver, false, err
answered, err = answerOf(msg.Data)
return answered, false, err
}
// saidFallBack says, once per subject, that a caller-named call reached nobody and is asked on the subject that
// names no caller: a runtime or holder not yet serving it, or a grant that drifted, is seen in the log rather
// than read later as a quiet "unproven" (novox/hq issue 365; expected for the controller's seat until hq issue
// than read later as a quiet "unproven" (novox/hq issue 365; expected until hq issue
// 464).
func (c *Conn) saidFallBack(subject string) {
c.calls.mu.Lock()
+1 -1
View File
@@ -16,7 +16,7 @@ import (
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A bus whose users are granted as the controller grants them (mesh-controller internal/broker): each named
// A bus whose users are granted as the controller grants them (mesh-module-assigner internal/broker): each named
// user publishes the subjects listed for it.
func grantedBus(t *testing.T, publish map[string][]string) string {
t.Helper()
+143
View File
@@ -0,0 +1,143 @@
package bus
import (
"bytes"
"crypto/rand"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// FileField is the member of a chunked verb's answer that carries the file received: its whole bytes, base64 (design 47 G
// A-249).
const FileField = "file"
// AskAsChunked is AskAs of a seat's verb whose definition says `chunks` (design 47 G §681, A-237, A-249): it chooses the call
// token (the body's `call`), hears the chunk subjects of that call before it asks, receives the chunks the verb's answer
// counts with buscalls.SeatChunks, each acknowledged before the next, and answers the verb's own answer with the whole
// file added as FileField. The asking bundle receives no chunk itself.
func (c *Conn) AskAsChunked(module, key string, body any) (Answered, error) {
// The verb may be asked of the machine that ran the ask (`@<node>`, A-254): the chunk subjects name the seat, the verb and
// the call alone.
name, _, _ := strings.Cut(key, "@")
seat, verb, ok := strings.Cut(strings.TrimPrefix(name, "seat:"), ".")
if !strings.HasPrefix(key, "seat:") || !ok {
return Answered{}, fmt.Errorf("%q is no seat:<seat>.<verb>, which a chunked answer needs", key)
}
args := map[string]any{}
if raw, err := json.Marshal(body); err != nil {
return Answered{}, err
} else if string(raw) != "null" {
if err := json.Unmarshal(raw, &args); err != nil {
return Answered{}, fmt.Errorf("the arguments of %s are not an object: %w", key, err)
}
}
var bits [16]byte
if _, err := rand.Read(bits[:]); err != nil {
return Answered{}, err
}
send := buscalls.SeatChunks{Seat: seat, Verb: verb, Call: hex.EncodeToString(bits[:])}
args["call"] = send.Call
chunks := make(chan *nats.Msg, 4)
sub, err := c.nc.ChanSubscribe(send.Chunks(), chunks)
if err != nil {
return Answered{}, fmt.Errorf("the chunks of %s cannot be heard: %w", key, err)
}
defer func() { _ = sub.Unsubscribe() }()
answered, err := c.AskAs(module, key, args)
if err != nil {
return answered, err
}
fields := map[string]json.RawMessage{}
if err := json.Unmarshal(answered.Result, &fields); err != nil {
return answered, fmt.Errorf("%s answered what is no object: %w", key, err)
}
var count int64
if raw, has := fields["chunks"]; has {
if err := json.Unmarshal(raw, &count); err != nil {
return answered, fmt.Errorf("%s answered chunks that is no count: %w", key, err)
}
}
var file bytes.Buffer
if count > 0 {
next := func() (string, []byte, error) {
select {
case m := <-chunks:
return m.Subject, m.Data, nil
case <-time.After(buscalls.SeatAckWithin):
return "", nil, buscalls.ErrSeatChunkMissed
}
}
ack := func(subject string) error {
_, err := c.nc.Request(subject, []byte("{}"), buscalls.SeatAckWithin)
return err
}
if _, err := send.Receive(count, next, ack, &file); err != nil {
return answered, fmt.Errorf("the chunks of %s: %w", key, err)
}
}
if _, taken := fields[FileField]; taken {
return answered, errors.New(key + " answers a member named " + FileField + " itself")
}
enc, _ := json.Marshal(base64.StdEncoding.EncodeToString(file.Bytes()))
fields[FileField] = enc
out, err := json.Marshal(fields)
if err != nil {
return answered, err
}
answered.Result = out
return answered, nil
}
// AnswerChunked is a seat verb's answer as its definition says `chunks` (design 47 G A-257, §681): a bundle answers the whole file
// as the member FileField (base64), and the runner that serves the verb sends it in chunks with buscalls.SeatChunks on the call
// token the body names, each acknowledged by the caller before the next, and answers the verb's own members with `chunks`, the
// count, in place of the file. The bundle sends no chunk itself. An answer with no file is answered as it is. The send runs after
// the answer is returned: the caller hears the chunks before it asks (AskAsChunked) and acknowledges each once it has the answer.
func (c *Conn) AnswerChunked(seat, verb string, body json.RawMessage, answer any) (any, error) {
raw, err := json.Marshal(answer)
if err != nil {
return nil, err
}
fields := map[string]json.RawMessage{}
if err := json.Unmarshal(raw, &fields); err != nil {
return answer, nil
}
enc, has := fields[FileField]
if !has {
return answer, nil
}
var text string
if err := json.Unmarshal(enc, &text); err != nil {
return nil, fmt.Errorf("%s.%s answers a %s that is no string: %w", seat, verb, FileField, err)
}
file, err := base64.StdEncoding.DecodeString(text)
if err != nil {
return nil, fmt.Errorf("%s.%s answers a %s in no base64: %w", seat, verb, FileField, err)
}
var asked struct {
Call string `json:"call"`
}
if err := json.Unmarshal(body, &asked); err != nil || asked.Call == "" {
return nil, fmt.Errorf("%s.%s is asked in chunks with no call token", seat, verb)
}
delete(fields, FileField)
count := int64((len(file) + buscalls.SeatChunkBytes - 1) / buscalls.SeatChunkBytes)
fields["chunks"], _ = json.Marshal(count)
send := buscalls.SeatChunks{Seat: seat, Verb: verb, Call: asked.Call}
go func() { _ = send.Send(c.nc, file, buscalls.SeatAckWithin, c.Logf) }()
return json.RawMessage(mustMarshal(fields)), nil
}
func mustMarshal(v any) []byte {
b, _ := json.Marshal(v)
return b
}
+117
View File
@@ -0,0 +1,117 @@
package bus_test
import (
"encoding/base64"
"encoding/json"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/buscalls"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// Design 47 G A-249, §681: the runner asks a chunked verb with a call token of its own, receives the chunks the answer counts
// (each acknowledged before the next) and answers the verb's own answer with the whole file, so the asker receives none.
func TestAChunkedSeatAnswerIsReceivedByTheRunnerAndHandedWhole(t *testing.T) {
url := mt.URL(t)
holder := dial(t, url, "mesh-delivery-indexer", "anchor", "")
asker := dial(t, url, "mesh-build-go", "anchor", "")
raw, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(raw.Close)
file := make([]byte, buscalls.SeatChunkBytes*2+5)
for i := range file {
file[i] = byte(i)
}
// the holder's process: answers the count, then sends each chunk and waits for its acknowledgement
stop, err := holder.Handle("seat:mesh-catalog.input", func(body json.RawMessage) (any, error) {
var in struct{ Call, Digest string }
if err := json.Unmarshal(body, &in); err != nil || in.Call == "" {
t.Errorf("the verb was asked with no call token: %s (%v)", body, err)
return nil, err
}
send := buscalls.SeatChunks{Seat: "mesh-catalog", Verb: "input", Call: in.Call}
go func() {
for n := int64(1); n <= 3; n++ {
end := min(int64(len(file)), n*buscalls.SeatChunkBytes)
data, _ := json.Marshal(buscalls.SeatChunk{N: n, Of: 3, Data: file[(n-1)*buscalls.SeatChunkBytes : end]})
ackSub, _ := raw.SubscribeSync(send.Ack(n))
_ = raw.Flush()
if err := raw.Publish(send.Chunk(n), data); err != nil {
return
}
m, err := ackSub.NextMsg(buscalls.SeatAckWithin)
if err != nil {
t.Errorf("chunk %d not acknowledged: %v", n, err)
return
}
_ = m.Respond([]byte("{}"))
}
}()
return map[string]any{"digest": in.Digest, "chunks": 3}, nil
})
if err != nil {
t.Fatal(err)
}
defer stop()
holder.Flush() // the bus holds the verb's subscription before it is asked: the asker is another connection
got, err := asker.AskAsChunked("mesh-build-go", "seat:mesh-catalog.input", map[string]any{"digest": "d1"})
if err != nil {
t.Fatal(err)
}
var ans struct {
Digest string `json:"digest"`
Chunks int `json:"chunks"`
File string `json:"file"`
}
if err := json.Unmarshal(got.Result, &ans); err != nil || ans.Digest != "d1" || ans.Chunks != 3 {
t.Fatalf("the answer %s: %v", got.Result, err)
}
if body, err := base64.StdEncoding.DecodeString(ans.File); err != nil || string(body) != string(file) {
t.Errorf("the file handed is %d bytes, want the %d sent: %v", len(body), len(file), err)
}
// Design 47 G A-257, A-254: a bundle answers a chunked verb with the whole file in the member `file`; the runner that serves the
// verb sends the file in chunks (each acknowledged before the next) and answers the count, so the bundle sends none, and the asker
// of one node's verb receives it whole.
t.Run("ABundlesChunkedAnswerIsSentByTheRunner", func(t *testing.T) {
url := mt.URL(t)
holder := dial(t, url, "mesh-build-go", "anchor", "")
asker := dial(t, url, "mesh-delivery-indexer", "anchor", "")
file := make([]byte, buscalls.SeatChunkBytes*2+7)
for i := range file {
file[i] = byte(i * 7)
}
stop, err := holder.Handle("seat:node-build-go.output@anchor", func(body json.RawMessage) (any, error) {
answer := map[string]any{"ask": "a1", "file": base64.StdEncoding.EncodeToString(file)}
return holder.AnswerChunked("node-build-go", "output", body, answer)
})
if err != nil {
t.Fatal(err)
}
defer stop()
holder.Flush() // the bus holds the verb's subscription before it is asked: the asker is another connection
got, err := asker.AskAsChunked("mesh-delivery-indexer", "seat:node-build-go.output@anchor", map[string]any{"ask": "a1"})
if err != nil {
t.Fatal(err)
}
var ans struct {
Ask string `json:"ask"`
Chunks int `json:"chunks"`
File string `json:"file"`
}
if err := json.Unmarshal(got.Result, &ans); err != nil || ans.Ask != "a1" || ans.Chunks != 3 {
t.Fatalf("the answer %s: %v", got.Result, err)
}
if body, err := base64.StdEncoding.DecodeString(ans.File); err != nil || string(body) != string(file) {
t.Errorf("the file handed is %d bytes, want the %d sent: %v", len(body), len(file), err)
}
// an answer with no file is the verb's own: no chunks are sent
plain, err := holder.AnswerChunked("node-build-go", "output", json.RawMessage(`{"call":"c"}`), map[string]any{"ask": "x", "chunks": 0})
if m, _ := plain.(map[string]any); err != nil || m["chunks"] != 0 {
t.Fatalf("an answer with no file was %v, %v", plain, err)
}
})
}
+94
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
@@ -60,3 +61,96 @@ func TestASlowHandlerAndABurstAreEachHandedOverOnce(t *testing.T) {
}
}
}
// The runtime tells a bundle how its module's consumer stands between events (stdio.OnHeartbeat): a word at each
// idle heartbeat while the bus answers, once Missed when the bus ended the loop, and a word again when it is bound again.
func TestAConsumerSaysItIsBoundWhileIdleAndOnceWhenTheBusEndedIt(t *testing.T) {
mesh := mt.New(t)
restore := bus.IdleHeartbeat
bus.IdleHeartbeat = 200 * time.Millisecond
t.Cleanup(func() { bus.IdleHeartbeat = restore })
ticks := make(chan struct{}, 64)
t.Cleanup(bus.SetTickHook(func() {
select {
case ticks <- struct{}{}:
default:
}
}))
mesh.Consumer(t, "anchor", "quiet", []string{"mesh.mod.alpha.event.>"}, time.Second)
c, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
c.Logf = func(string, ...any) {}
t.Cleanup(c.Close)
var mu sync.Mutex
var beats []bus.Beat
stop, err := c.ConsumeAsBeating("quiet", func(bus.Envelope) error { return nil }, func(b bus.Beat) {
mu.Lock()
beats = append(beats, b)
mu.Unlock()
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// A consume nobody asked a heartbeat of says nothing and asks the bus nothing, however many intervals pass.
var unaskedBeats atomic.Int32
unasked, err := c.ConsumeAsBeatingWhen("quiet", func(bus.Envelope) error { return nil },
func(bus.Beat) { unaskedBeats.Add(1) }, func() bool { return false })
if err != nil {
t.Fatal(err)
}
t.Cleanup(unasked)
last := func() (bus.Beat, int) {
mu.Lock()
defer mu.Unlock()
if len(beats) == 0 {
return bus.Beat{}, 0
}
return beats[len(beats)-1], len(beats)
}
mt.Until(t, func() error {
if b, n := last(); n == 0 || b.Missed || b.At.IsZero() {
return fmt.Errorf("no word that the consumer is bound yet: %d beats", n)
}
return nil
})
mesh.RemoveConsumer(t, bus.EventsStream, "anchor", "quiet")
mt.Until(t, func() error {
if b, _ := last(); !b.Missed || b.Why == "" {
return fmt.Errorf("the consumer is gone from the bus and no Missed word came: %+v", b)
}
return nil
})
mu.Lock()
missed := 0
for _, b := range beats {
if b.Missed {
missed++
}
}
mu.Unlock()
for n := 0; n < 3; n++ { // three more intervals pass, counted by the loop's own tick
select {
case <-ticks:
case <-time.After(10 * time.Second):
t.Fatal("the heartbeat loop handled no more intervals")
}
}
mu.Lock()
defer mu.Unlock()
after := 0
for _, b := range beats {
if b.Missed {
after++
}
}
if unaskedBeats.Load() != 0 {
t.Errorf("a consume with no heartbeat asked said %d words", unaskedBeats.Load())
}
if after != missed || missed != 1 {
t.Errorf("Missed is said once, not at every beat: %d then %d", missed, after)
}
}
+8
View File
@@ -0,0 +1,8 @@
package bus
// SetTickHook makes the next consumes call f after each idle heartbeat interval was handled; the restore puts it back.
func SetTickHook(f func()) (restore func()) {
old := tickHook
tickHook = f
return func() { tickHook = old }
}
-73
View File
@@ -1,73 +0,0 @@
package bus_test
import (
"encoding/json"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A call the controller refused because it is handing over is asked once more** (novox/hq issue 289):
// the controller being replaced did nothing and marked its refusal, and the one after it answers. A
// refusal not so marked is final, and a second handover refusal is said, not asked a third time.
func TestAHandoverRefusalIsAskedOnceMore(t *testing.T) {
was := bus.HandoverPause
bus.HandoverPause = 10 * time.Millisecond
t.Cleanup(func() { bus.HandoverPause = was })
url := meshtest.URL(t)
seat, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer seat.Close()
c, err := bus.Connect(bus.Credential{URL: url, Module: "console", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer c.Close()
serve := func(t *testing.T, verb string, answers ...map[string]any) *atomic.Int32 {
t.Helper()
var asked atomic.Int32
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", verb, "", ""), func(m *nats.Msg) {
n := int(asked.Add(1)) - 1
if n >= len(answers) {
n = len(answers) - 1
}
body, _ := json.Marshal(answers[n])
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
_ = seat.Flush()
return &asked
}
handingOver := map[string]any{"error": "the controller is handing over to the next one; nothing was done, ask again",
"retry": bus.RetryHandingOver}
asked := serve(t, "rotate", handingOver, map[string]any{"result": "rotated"})
got, err := c.Ask("seat:mesh-controller.rotate", map[string]any{}, "")
if err != nil || string(got.Result) != `"rotated"` || asked.Load() != 2 {
t.Fatalf("after a handover refusal: %s %v, asked %d times", got.Result, err, asked.Load())
}
asked = serve(t, "push", map[string]any{"error": "refused for its own reason"})
if _, err := c.Ask("seat:mesh-controller.push", map[string]any{}, ""); err == nil || asked.Load() != 1 {
t.Fatalf("an unmarked refusal was asked again: %v, asked %d times", err, asked.Load())
}
asked = serve(t, "status", handingOver)
_, err = c.Ask("seat:mesh-controller.status", map[string]any{}, "")
if err == nil || asked.Load() != 2 || !strings.Contains(err.Error(), "both times refused as a handover") {
t.Fatalf("two handover refusals: %v, asked %d times", err, asked.Load())
}
}
@@ -0,0 +1,48 @@
package bus
import (
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// heardMsg is a delivered event as a consumer's loop hands it, saying how it was settled.
type heardMsg struct {
jetstream.Msg
mu sync.Mutex
settled []string
}
func (m *heardMsg) Subject() string { return "mesh.mod.alpha.event.happened" }
func (m *heardMsg) Headers() nats.Header { return nats.Header{} }
func (m *heardMsg) Data() []byte { return []byte(`{"n":1}`) }
func (m *heardMsg) InProgress() error { return nil }
func (m *heardMsg) Ack() error { return m.say("ack") }
func (m *heardMsg) Nak() error { return m.say("nak") }
func (m *heardMsg) NakWithDelay(time.Duration) error { return m.say("nak-delayed") }
func (m *heardMsg) Term() error { return m.say("term") }
func (m *heardMsg) say(how string) error {
m.mu.Lock()
defer m.mu.Unlock()
m.settled = append(m.settled, how)
return nil
}
// An event the bus hands a hearing after it was stopped (a module's bundle loaded again, issue 1945) is
// offered again at once and never handed to the module: the module's handler that heard it may be gone.
func TestAHearingStoppedHandsNoEventOnAndOffersItAgain(t *testing.T) {
delivered := 0
h := newHearing(nil, nil, "alpha", func(Envelope) error { delivered++; return nil })
h.stop()
msg := &heardMsg{}
h.handle(msg)
if delivered != 0 {
t.Errorf("a stopped hearing handed the event to the module")
}
if len(msg.settled) != 1 || msg.settled[0] != "nak" {
t.Errorf("a stopped hearing settled the event as %v, want it offered again at once (nak)", msg.settled)
}
}
+427
View File
@@ -0,0 +1,427 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/ledger"
)
// A ledger on the bus (novox/hq ADR 0597, design 25 § 12.4; issue 1966): its history a stream with a subject per
// instance under mesh-sdk's buscompose.LedgerHistorySubject, its state a key-value bucket, its snapshots a subject per
// instance under buscompose.LedgerSnapshotSubject, the latest only (the stream that holds them keeps one per subject, as the bus module
// creates it). **Only the module's ledger connection publishes on its history and its snapshots** (ADR 0597 §9): what
// is read, and the state, go on the runner's own.
// LedgerStores are the stores of a module's ledger binding: reads and the state on c, the history's appends and the
// snapshots' writes on writer, the module's ledger connection. The subjects are mesh-sdk's, the ones the grants part
// grants the module's ledger account byte for byte; a binding naming others is refused, naming both.
func (c *Conn) LedgerStores(module string, b membership.BusBinding, writer *Conn) (ledger.History, ledger.State, ledger.Snapshots, error) {
history, snapshot := buscompose.LedgerHistorySubject(module, b.Local), buscompose.LedgerSnapshotSubject(module, b.Local)
for _, named := range []struct{ what, bound, want string }{{"history", b.Subject, history}, {"snapshot", b.Snapshot, snapshot}} {
if named.bound != "" && prefixOf(named.bound) != named.want {
return nil, nil, nil, fmt.Errorf("%s's ledger %q is bound with the %s subject %s, and its ledger account is granted "+
"%s alone (mesh-sdk buscompose); the binding is refused", module, b.Local, named.what, named.bound, named.want)
}
}
return &busHistory{read: c, write: writer, stream: b.History, prefix: history},
&busState{c: c, bucket: b.State},
&busSnapshots{read: c, write: writer, prefix: snapshot}, nil
}
// prefixOf is a binding's subject as the prefix an instance's key is added to: written with or without its `.>`.
func prefixOf(subject string) string {
return strings.TrimSuffix(strings.TrimSuffix(subject, ">"), ".")
}
// publishExpecting publishes body on subject on the connection, written only when the subject's last message is still
// at expect (0: it has none); ledger.ErrRaced when it is not.
func publishExpecting(c *Conn, subject string, body []byte, expect uint64) (uint64, error) {
msg := nats.NewMsg(subject)
msg.Data = body
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(expect, 10))
reply, err := c.nc.RequestMsg(msg, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return 0, fmt.Errorf("no stream on the bus keeps %s", subject)
}
if err != nil {
return 0, err
}
var ack struct {
Seq uint64 `json:"seq"`
Error *struct {
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &ack); err != nil {
return 0, fmt.Errorf("the bus answered the write to %s with what is not an acknowledgement: %q", subject, reply.Data)
}
if ack.Error != nil {
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
return 0, ledger.ErrRaced
}
return 0, fmt.Errorf("the bus refused the write to %s: %s (%d)", subject, ack.Error.Description, ack.Error.ErrCode)
}
return ack.Seq, nil
}
// lastOn is the last message on a subject of the stream that keeps it: its bytes and sequence, nil and 0 for none.
func lastOn(c *Conn, stream, subject string) ([]byte, uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := c.JetStream()
if err != nil {
return nil, 0, err
}
if stream == "" {
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
return nil, 0, fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
}
}
s, err := js.Stream(ctx, stream)
if err != nil {
return nil, 0, fmt.Errorf("the stream %s: %w", stream, err)
}
m, err := s.GetLastMsgForSubject(ctx, subject)
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil, 0, nil
}
if err != nil {
return nil, 0, err
}
return m.Data, m.Sequence, nil
}
type busHistory struct {
read, write *Conn
stream, prefix string
}
func (h *busHistory) Last(key string) (uint64, error) {
_, seq, err := lastOn(h.read, h.stream, h.prefix+"."+key)
return seq, err
}
func (h *busHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
if h.write == nil {
return 0, errors.New("the ledger's history is published only on its module's ledger connection, and none is held")
}
return publishExpecting(h.write, h.prefix+"."+key, op, expect)
}
func (h *busHistory) Read(keys string, from uint64, limit int) ([]ledger.Kept, error) {
var out []ledger.Kept
for {
want := StreamReadMost
if limit > 0 && limit-len(out) < want {
want = limit - len(out)
}
page, err := h.read.readStream(h.stream, h.prefix+"."+keys, from, want)
if err != nil {
return out, err
}
for _, m := range page.Messages {
out = append(out, ledger.Kept{Sequence: m.Sequence, Data: m.Body})
}
if page.Next == 0 || (limit > 0 && len(out) >= limit) {
return out, nil
}
from = page.Next
}
}
// Purge removes every operation of one instance from the history stream, on the module's ledger connection, the one
// granted the purge of its own ledger's streams.
func (h *busHistory) Purge(key string) error {
return purgeSubject(h.read, h.write, h.stream, h.prefix+"."+key)
}
// Fill is the history stream's fill against its cap, the cap the manifest gives the ledger.
func (h *busHistory) Fill() (buscalls.Fill, error) {
name := h.stream
if name == "" {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
js, err := h.read.JetStream()
if err != nil {
return buscalls.Fill{}, err
}
if name, err = js.StreamNameBySubject(ctx, h.prefix+".>"); err != nil {
return buscalls.Fill{}, fmt.Errorf("no stream on the bus keeps %s: %w", h.prefix, err)
}
}
f, err := h.read.streamFill(name)
return f.Fill, err
}
// purgeSubject purges one subject's messages from the stream that keeps it: the stream named, or found on the runner's
// own connection by its subject, purged on the ledger connection.
func purgeSubject(read, write *Conn, stream, subject string) error {
if write == nil {
return errors.New("a ledger's streams are purged only on its module's ledger connection, and none is held")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
if stream == "" {
js, err := read.JetStream()
if err != nil {
return err
}
if stream, err = js.StreamNameBySubject(ctx, subject); err != nil {
return fmt.Errorf("no stream on the bus keeps %s: %w", subject, err)
}
}
// The purge is asked of the stream by its name, with no read of its info first: the ledger account is granted the
// purge of its own ledger's two streams and nothing else (PurgeAPI).
body, err := json.Marshal(map[string]string{"filter": subject})
if err != nil {
return err
}
reply, err := write.nc.RequestWithContext(ctx, PurgeAPI(stream), body)
if err != nil {
return fmt.Errorf("purging %s from %s: %w", subject, stream, err)
}
var ack struct {
Success bool `json:"success"`
Error *struct {
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &ack); err != nil || !ack.Success {
why := string(reply.Data)
if ack.Error != nil {
why = ack.Error.Description
}
return fmt.Errorf("the bus refused the purge of %s from %s: %s", subject, stream, why)
}
return nil
}
// PurgeAPI is the bus's subject that purges a stream, the one a ledger account is granted for its own ledger's streams.
func PurgeAPI(stream string) string { return "$JS.API.STREAM.PURGE." + stream }
type busState struct {
c *Conn
bucket string
}
func (s *busState) kv(ctx context.Context) (jetstream.KeyValue, error) {
js, err := s.c.JetStream()
if err != nil {
return nil, err
}
kv, err := js.KeyValue(ctx, s.bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil, fmt.Errorf("the ledger's state %s is not on the bus", s.bucket)
}
return kv, err
}
func (s *busState) Get(key string) ([]byte, uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return nil, 0, err
}
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, nil
}
if err != nil {
return nil, 0, err
}
return e.Value(), e.Revision(), nil
}
// isFull is a write a bucket or stream refused because it holds as much as it may.
func isFull(err error) bool {
msg := err.Error()
return strings.Contains(msg, "maximum bytes exceeded") || strings.Contains(msg, "maximum messages exceeded") ||
strings.Contains(msg, "insufficient resources")
}
func (s *busState) Write(key string, value []byte, revision uint64) (uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return 0, err
}
var rev uint64
if revision == 0 {
rev, err = kv.Create(ctx, key, value)
} else {
rev, err = kv.Update(ctx, key, value, revision)
}
switch {
case err == nil:
return rev, nil
case errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) || isWrongSequence(err):
return 0, ledger.ErrChanged
case isFull(err):
return 0, fmt.Errorf("%w: %v", ledger.ErrFull, err)
}
return 0, err
}
func (s *busState) Keys() ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return nil, err
}
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
type busSnapshots struct {
read, write *Conn
prefix string
}
func (s *busSnapshots) Get(key string) ([]byte, uint64, error) {
return lastOn(s.read, "", s.prefix+"."+key)
}
// Purge removes one instance's snapshot, on the module's ledger connection.
func (s *busSnapshots) Purge(key string) error {
return purgeSubject(s.read, s.write, "", s.prefix+"."+key)
}
func (s *busSnapshots) Put(key string, value []byte, expect uint64) error {
if s.write == nil {
return errors.New("a ledger's snapshot is published only on its module's ledger connection, and none is held")
}
_, err := publishExpecting(s.write, s.prefix+"."+key, value, expect)
return err
}
// Watch hands deliver every instance's record under a key pattern as it is, then each change; a removed key is handed as
// nil.
func (s *busState) Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
kv, err := s.kv(ctx)
cancel()
if err != nil {
return nil, err
}
w, err := kv.Watch(context.Background(), pattern)
if err != nil {
return nil, err
}
hand := func(e jetstream.KeyValueEntry, current bool) {
var value []byte
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
if err := deliver(e.Key(), value, current); err != nil {
s.c.Logf("[mesh-tools] a ledger watch of %s was not taken: %v", s.bucket, err)
}
}
timer := time.NewTimer(StateTimeout)
defer timer.Stop()
for current := true; current; {
select {
case e, ok := <-w.Updates():
if !ok {
return nil, errors.New("the ledger's state watch ended before its current states were handed over")
}
if e == nil {
current = false
continue
}
hand(e, true)
case <-timer.C:
_ = w.Stop()
return nil, fmt.Errorf("the ledger's state %s did not hand its current states in %s", s.bucket, StateTimeout)
}
}
go func() {
for e := range w.Updates() {
if e != nil {
hand(e, false)
}
}
}()
return func() { _ = w.Stop() }, nil
}
// LedgerIndexes are the buckets of a ledger binding's declared indexes, by index name, on the runner's own connection.
func (c *Conn) LedgerIndexes(b membership.BusBinding) map[string]ledger.Index {
out := map[string]ledger.Index{}
for name, bucket := range b.Indexes {
out[name] = &busState{c: c, bucket: bucket}
}
return out
}
func (s *busState) Entries(pattern string, limit int) ([]ledger.Indexed, error) {
if pattern == "" {
pattern = ">"
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
var out []ledger.Indexed
for _, k := range keys {
if limit > 0 && len(out) >= limit {
break
}
if !SubjectMatches(pattern, k) {
continue
}
v, _, err := s.Get(k)
if err != nil {
return out, err
}
if v != nil {
out = append(out, ledger.Indexed{Key: k, Value: v})
}
}
return out, nil
}
func (s *busState) Put(key string, value []byte) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return err
}
_, err = kv.Put(ctx, key, value)
return err
}
func (s *busState) Delete(key string) error {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := s.kv(ctx)
if err != nil {
return err
}
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return err
}
return nil
}
+491
View File
@@ -0,0 +1,491 @@
package bus
import (
"context"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// A module's log on the bus (novox/hq ADR 0297): a stream of operations, one subject per key, that the
// mesh creates from what the module declared under `logs` and issues to the module's own instances
// in its membership. It mirrors a bucket (ADR 0201) in how it is reached: what a module may reach is
// decided here, from its membership, and refused with the reason before anything is sent.
//
// **The runtime uses no consumer of a log.** The bus grants it, per issued log, only the publish of an
// entry, the stream's info and the direct get: an append is a publish, a read is a batch direct get
// (NATS 2.11), the last entry of a key is a direct get by its subject.
// LogIssued is one log an assignment may reach (ADR 0297): its owner's alone, so there are no reads.
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes,omitempty"`
}
// LogEntry is one operation in a log: its sequence is its identity.
type LogEntry struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
At time.Time `json:"at"`
}
// LogPage is one page of a read: Next is the sequence to continue from, 0 when nothing follows; Last
// is the stream's last sequence.
type LogPage struct {
Entries []LogEntry `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// LogInfo is how full a log is.
type LogInfo struct {
Entries uint64 `json:"entries"`
Bytes uint64 `json:"bytes"`
MaxBytes uint64 `json:"max_bytes"`
First uint64 `json:"first"`
Last uint64 `json:"last"`
}
const (
// LogValueMax is the most one entry's value may hold (ADR 0297 §3); the stream's MaxMsgSize is 4 KiB
// more, for the headers an append carries.
LogValueMax = 256 * 1024
// LogKeyMax is the longest key.
LogKeyMax = 255
// LogReadDefault and LogReadMax are a page's entries when none is asked, and at most.
LogReadDefault = 100
LogReadMax = 1000
// LogPageBytes bounds the bytes of one page.
LogPageBytes = 4 * 1024 * 1024
)
// logToken is one name of a key: letters, digits, `_` and `-`.
var logToken = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
func checkLogKey(key string) error {
if key == "" || len(key) > LogKeyMax {
return fmt.Errorf("%q is not a log key: one or more names of letters, digits, _ and -, joined by "+
"dots, at most %d bytes (novox/hq ADR 0297)", key, LogKeyMax)
}
for _, t := range strings.Split(key, ".") {
if !logToken.MatchString(t) {
return fmt.Errorf("%q is not a log key: one or more names of letters, digits, _ and -, joined by "+
"dots, at most %d bytes (novox/hq ADR 0297)", key, LogKeyMax)
}
}
return nil
}
// logFilter is a read's key pattern as a subject filter under the log's subject: `*` for one name, `>`
// for the rest; empty is every key.
func logFilter(l LogIssued, pattern string) (string, error) {
if pattern == "" {
return l.Subject + ".>", nil
}
parts := strings.Split(pattern, ".")
for i, p := range parts {
if p == "**" && i == len(parts)-1 {
// `**` for the rest, as a state's key pattern and mesh-sdk's log reader say it.
parts[i], p = ">", ">"
}
switch {
case p == ">" && i == len(parts)-1, p == "*", logToken.MatchString(p):
default:
return "", fmt.Errorf("%q is not a key pattern: names, `*` for one and `>` for the rest, last "+
"(novox/hq ADR 0297)", pattern)
}
}
if len(pattern) > LogKeyMax {
return "", fmt.Errorf("%q is longer than a key can be (%d bytes)", pattern, LogKeyMax)
}
return l.Subject + "." + strings.Join(parts, "."), nil
}
// issuedLog is the log a module may reach by a name, or the reason it may not.
func (c *Conn) issuedLog(module, name string) (LogIssued, error) {
m := c.Membership(module)
if m == nil {
return LogIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no log of it is reachable "+
"until the mesh issues one (novox/hq ADR 0297)", module, c.node)
}
var names []string
for _, l := range m.Logs {
if l.Name == name {
return l, nil
}
names = append(names, l.Name)
}
sort.Strings(names)
issued := "none"
if len(names) > 0 {
issued = strings.Join(names, ", ")
}
return LogIssued{}, fmt.Errorf("%s keeps no log called %q: a module declares the logs it keeps under `logs`, and "+
"another module's it reads under `reads` as <module>.<log>; it was issued: %s (novox/hq ADR 0297, ADR 0571)",
module, name, issued)
}
// notOnTheBus says a log is issued and its stream is not there yet.
func notOnTheBus(l LogIssued) error {
return fmt.Errorf("the log %q is issued and its stream %s is not on the bus yet: the mesh creates "+
"it from the catalogue on its next raise (novox/hq ADR 0297)", l.Name, l.Stream)
}
// LogAppend writes one entry for a key, as the module, and answers its sequence. With expect, it is
// written only when the key's last entry is still at that sequence (0: the key has none); otherwise it
// is refused as a lost compare-and-set, the same error a lost state update answers (ADR 0259).
func (c *Conn) LogAppend(module, name, key string, value json.RawMessage, expect *uint64) (uint64, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return 0, err
}
if !l.Writes {
return 0, fmt.Errorf("%s was issued the log %s without writing it (novox/hq ADR 0297)", module, name)
}
if err := checkLogKey(key); err != nil {
return 0, err
}
if len(value) == 0 || !json.Valid(value) {
return 0, fmt.Errorf("a log entry's value is JSON (novox/hq ADR 0297)")
}
if len(value) > LogValueMax {
return 0, fmt.Errorf("a log entry's value is at most %d bytes (256 KiB), and this one is %d (novox/hq ADR 0297)",
LogValueMax, len(value))
}
if field := credentialField(value); field != "" {
return 0, fmt.Errorf("%s's %s.%s carries a field %q, which names a credential: no secret is kept in a log, "+
"which keeps it for good. Name the secret and fetch it on request/reply (novox/hq ADR 0201 §13, ADR 0297)",
module, name, key, field)
}
msg := nats.NewMsg(l.Subject + "." + key)
msg.Data = value
if expect != nil {
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(*expect, 10))
}
reply, err := c.nc.RequestMsg(msg, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return 0, notOnTheBus(l)
}
if err != nil {
return 0, err
}
var ack struct {
Seq uint64 `json:"seq"`
Error *struct {
Code int `json:"code"`
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &ack); err != nil {
return 0, fmt.Errorf("the bus answered the append to %s with what is not an acknowledgement: %q", name, reply.Data)
}
if ack.Error != nil {
// The key's last entry is no longer the one expected: a lost compare-and-set, answered as a lost
// state update is (10164 is the same refusal from a replicated stream).
if code := jetstream.ErrorCode(ack.Error.ErrCode); code == jetstream.JSErrCodeStreamWrongLastSequence ||
code == jetstream.JSErrCodeStreamWrongLastSequenceConstant {
return 0, changed{name + "." + key}
}
return 0, fmt.Errorf("the bus refused the append to %s.%s: %s (%d)", name, key, ack.Error.Description, ack.Error.ErrCode)
}
if ack.Seq == 0 {
return 0, fmt.Errorf("the bus answered the append to %s.%s with no sequence", name, key)
}
return ack.Seq, nil
}
// LogRead is the entries from a sequence on, or from a time on (since, when not zero), of the keys matching a pattern, at
// most limit of them and about 4 MiB, by one batch direct get. A log another module keeps is read as its own is, by the
// name its reads declare, `<module>.<log>` (novox/hq ADR 0571).
func (c *Conn) LogRead(module, name string, from uint64, since time.Time, keys string, limit int) (*LogPage, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
switch {
case limit == 0:
limit = LogReadDefault
case limit < 0 || limit > LogReadMax:
return nil, fmt.Errorf("a read's limit is 1 to %d entries, %d when none is given, not %d (novox/hq ADR 0297)",
LogReadMax, LogReadDefault, limit)
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
if from != 0 && !since.IsZero() {
return nil, fmt.Errorf("a read of %s starts from a sequence or from a time, not both", name)
}
if from == 0 {
from = 1
}
req := map[string]any{"seq": from, "next_by_subj": filter, "batch": limit, "max_bytes": LogPageBytes}
if !since.IsZero() {
delete(req, "seq")
req["start_time"] = since.UTC()
}
entries, more, err := c.logBatch(l, req)
if err != nil {
return nil, err
}
page := &LogPage{Entries: entries}
if more && len(entries) > 0 {
page.Next = entries[len(entries)-1].Sequence + 1
}
info, err := c.logInfo(l)
if err != nil {
return nil, err
}
page.Last = info.Last
return page, nil
}
// LogNewest is the newest entry of each key matching a pattern, in the log's order, by one batch direct get (mesh-sdk
// process.LogReader.Newest).
func (c *Conn) LogNewest(module, name, keys string) ([]LogEntry, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
entries, more, err := c.logBatch(l, map[string]any{"multi_last": []string{filter}, "batch": LogReadMax})
if err != nil {
return nil, err
}
if more {
return nil, fmt.Errorf("%s holds more than %d keys matching %q: name fewer", name, LogReadMax, keys)
}
return entries, nil
}
// logBatch is one batch direct get of a log: its entries, and whether more are pending after them.
func (c *Conn) logBatch(l LogIssued, request map[string]any) ([]LogEntry, bool, error) {
req, _ := json.Marshal(request)
inbox := c.nc.NewInbox()
sub, err := c.nc.SubscribeSync(inbox)
if err != nil {
return nil, false, err
}
defer func() { _ = sub.Unsubscribe() }()
if err := c.nc.PublishRequest("$JS.API.DIRECT.GET."+l.Stream, inbox, req); err != nil {
return nil, false, err
}
entries := []LogEntry{}
deadline := time.Now().Add(StateTimeout)
for {
msg, err := sub.NextMsg(time.Until(deadline))
if err != nil {
if errors.Is(err, nats.ErrNoResponders) {
return nil, false, notOnTheBus(l)
}
if errors.Is(err, nats.ErrTimeout) {
return nil, false, fmt.Errorf("the read of %s did not end in %s", l.Name, StateTimeout)
}
return nil, false, err
}
switch status := msg.Header.Get("Status"); status {
case "":
e, err := logEntryOf(l, msg)
if err != nil {
return nil, false, err
}
entries = append(entries, *e)
case "204": // the end of the batch: what is still pending after the last entry sent
pending, _ := strconv.ParseUint(msg.Header.Get("Nats-Num-Pending"), 10, 64)
return entries, pending > 0, nil
case "404": // nothing from there on
return entries, false, nil
case "503":
return nil, false, notOnTheBus(l)
default:
return nil, false, fmt.Errorf("the read of %s was refused by the bus: %s %s", l.Name, status, msg.Header.Get("Description"))
}
}
}
// LogWatch hands deliver every entry of the keys matching a pattern from a sequence on (from), from a time on (since),
// or, with neither, each new one, in the log's order, one at a time, through an ordered consumer it lets go of when
// stopped (mesh-sdk process.LogReader.Watch). It returns once the entries there at its start are handed over; an error
// from deliver is said, and the watch goes on with the next entry.
func (c *Conn) LogWatch(module, name, keys string, from uint64, since time.Time, deliver func(LogEntry) error) (func(), error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
filter, err := logFilter(l, keys)
if err != nil {
return nil, err
}
cfg := jetstream.OrderedConsumerConfig{FilterSubjects: []string{filter}, DeliverPolicy: jetstream.DeliverNewPolicy}
switch {
case from != 0 && !since.IsZero():
return nil, fmt.Errorf("a watch of %s starts from a sequence or from a time, not both", name)
case from != 0:
cfg.DeliverPolicy, cfg.OptStartSeq = jetstream.DeliverByStartSequencePolicy, from
case !since.IsZero():
at := since.UTC()
cfg.DeliverPolicy, cfg.OptStartTime = jetstream.DeliverByStartTimePolicy, &at
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
cons, err := js.OrderedConsumer(ctx, l.Stream, cfg)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
info, err := cons.Info(ctx)
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
it, err := cons.Messages()
if err != nil {
return nil, fmt.Errorf("the watch of %s (stream %s): %w", name, l.Stream, err)
}
hand := func(m jetstream.Msg) error {
meta, err := m.Metadata()
if err != nil {
return err
}
e := LogEntry{Sequence: meta.Sequence.Stream, Key: strings.TrimPrefix(m.Subject(), l.Subject+"."),
Value: valueOf(m.Data()), At: meta.Timestamp.UTC()}
return deliver(e)
}
for pending := info.NumPending; pending > 0; pending-- {
m, err := it.Next()
if err == nil {
err = hand(m)
}
if err != nil {
it.Stop()
return nil, fmt.Errorf("the watch of %s: %w", name, err)
}
}
var stopped atomic.Bool
go func() {
for {
m, err := it.Next()
if err != nil || stopped.Load() {
return // stopped, or the connection closed; a message fetched as it stopped is not handed over
}
if err := hand(m); err != nil {
c.Logf("[mesh-tools] %s did not take entry of %s: %v; it is in the log, a read finds it", module, name, err)
}
}
}()
return func() { stopped.Store(true); it.Stop() }, nil
}
// LogLast is a key's last entry, or nil when it has none.
func (c *Conn) LogLast(module, name, key string) (*LogEntry, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
if err := checkLogKey(key); err != nil {
return nil, err
}
reply, err := c.nc.Request("$JS.API.DIRECT.GET."+l.Stream+"."+l.Subject+"."+key, nil, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, err
}
switch status := reply.Header.Get("Status"); status {
case "":
return logEntryOf(l, reply)
case "404":
return nil, nil
default:
return nil, fmt.Errorf("the last entry of %s.%s could not be read: %s %s", name, key, status, reply.Header.Get("Description"))
}
}
// LogInfo is a log's entries, bytes, cap and first and last sequences.
func (c *Conn) LogInfo(module, name string) (*LogInfo, error) {
l, err := c.issuedLog(module, name)
if err != nil {
return nil, err
}
return c.logInfo(l)
}
func (c *Conn) logInfo(l LogIssued) (*LogInfo, error) {
reply, err := c.nc.Request("$JS.API.STREAM.INFO."+l.Stream, nil, StateTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, notOnTheBus(l)
}
if err != nil {
return nil, err
}
var info struct {
Config struct {
MaxBytes int64 `json:"max_bytes"`
} `json:"config"`
State struct {
Msgs uint64 `json:"messages"`
Bytes uint64 `json:"bytes"`
FirstSeq uint64 `json:"first_seq"`
LastSeq uint64 `json:"last_seq"`
} `json:"state"`
Error *struct {
ErrCode uint16 `json:"err_code"`
Description string `json:"description"`
} `json:"error"`
}
if err := json.Unmarshal(reply.Data, &info); err != nil {
return nil, fmt.Errorf("the bus answered the info of %s with what is not one: %q", l.Name, reply.Data)
}
if info.Error != nil {
if info.Error.ErrCode == uint16(jetstream.JSErrCodeStreamNotFound) {
return nil, notOnTheBus(l)
}
return nil, fmt.Errorf("the info of the log %s was refused: %s (%d)", l.Name, info.Error.Description, info.Error.ErrCode)
}
var maxBytes uint64
if info.Config.MaxBytes > 0 {
maxBytes = uint64(info.Config.MaxBytes)
}
return &LogInfo{Entries: info.State.Msgs, Bytes: info.State.Bytes, MaxBytes: maxBytes,
First: info.State.FirstSeq, Last: info.State.LastSeq}, nil
}
// logEntryOf is an entry a direct get answered: its key is its subject under the log's.
func logEntryOf(l LogIssued, msg *nats.Msg) (*LogEntry, error) {
subject := msg.Header.Get("Nats-Subject")
key, ok := strings.CutPrefix(subject, l.Subject+".")
if !ok {
return nil, fmt.Errorf("the bus answered an entry of %s on %q, which is not under %s", l.Name, subject, l.Subject)
}
seq, err := strconv.ParseUint(msg.Header.Get("Nats-Sequence"), 10, 64)
if err != nil {
return nil, fmt.Errorf("the bus answered an entry of %s with no sequence", l.Name)
}
at, err := time.Parse(time.RFC3339Nano, msg.Header.Get("Nats-Time-Stamp"))
if err != nil {
return nil, fmt.Errorf("the bus answered entry %d of %s with no time: %v", seq, l.Name, err)
}
return &LogEntry{Sequence: seq, Key: key, Value: valueOf(msg.Data), At: at.UTC()}, nil
}
+2 -97
View File
@@ -1,13 +1,7 @@
package bus
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
@@ -15,97 +9,8 @@ import (
// An answer larger than the bus carries in one message (novox/hq issue 314).
//
// The bus carries one message of at most its max_payload, and the client library refuses a larger one
// before it leaves the process that sent it. Until this, a holder whose answer was too large wrote one
// line to its own log and its caller waited out RequestTimeout and read "did not answer" — the
// controller's `conditions` and `status` for hours on 2026-10-08, and the operator's channel with them.
//
// The controller pages such an answer (mesh-controller internal/link, pages.go): it answers an error
// carrying `paged` — the call it holds the answer under, its size and its parts — and a caller asks the
// same subject again once per part with PageHeader, each part answered raw with PartHeader. Asking the
// same subject means no grant beyond the one the caller already had. A module's answer too large is
// refused here, loudly, in a message that fits: its instances may be several, and a part asked again
// could reach another.
const (
// PageHeader asks for one part of a paged answer: `<call> <part>`, from zero.
PageHeader = "Mesh-Page"
// PartHeader is on a part: `<part>/<parts>`.
PartHeader = "Mesh-Part"
// partTries is how often one part is asked: during a handover a part can reach the controller that
// did not hold it.
partTries = 3
)
// PartTimeout bounds the asking of one part.
var PartTimeout = 10 * time.Second
// paged is what a too-large answer says instead of itself.
type paged struct {
Call string `json:"call"`
Bytes int `json:"bytes"`
Parts int `json:"parts"`
}
func pagedIn(data []byte) (paged, bool) {
if !bytes.Contains(data, []byte(`"paged"`)) {
return paged{}, false
}
var r struct {
Paged *paged `json:"paged"`
}
if json.Unmarshal(data, &r) != nil || r.Paged == nil || r.Paged.Call == "" || r.Paged.Parts < 1 {
return paged{}, false
}
return *r.Paged, true
}
// whole is a reply's whole answer: the reply itself, or every part of a paged one, joined. A part that
// cannot be had is an error naming it, never a shorter answer.
func (c *Conn) whole(subject string, data []byte) ([]byte, error) {
p, ok := pagedIn(data)
if !ok {
return data, nil
}
out := make([]byte, 0, p.Bytes)
for n := 0; n < p.Parts; n++ {
var part []byte
var err error
for try := 0; try < partTries; try++ {
if part, err = c.askPart(subject, p, n); err == nil {
break
}
}
if err != nil {
return nil, fmt.Errorf("the answer was %d bytes, more than the bus carries in one message, paged as %s "+
"in %d parts, and part %d could not be had: %w", p.Bytes, p.Call, p.Parts, n, err)
}
out = append(out, part...)
}
if len(out) != p.Bytes {
return nil, fmt.Errorf("the answer paged as %s was %d bytes, and its parts joined are %d", p.Call, p.Bytes, len(out))
}
return out, nil
}
func (c *Conn) askPart(subject string, p paged, n int) ([]byte, error) {
ask := nats.NewMsg(subject)
ask.Header.Set(PageHeader, fmt.Sprintf("%s %d", p.Call, n))
ask.Data = []byte(`{}`)
reply, err := c.nc.RequestMsg(ask, PartTimeout)
if err != nil {
return nil, err
}
if got := reply.Header.Get(PartHeader); got != fmt.Sprintf("%d/%d", n, p.Parts) {
var r struct {
Error string `json:"error"`
}
if json.Unmarshal(reply.Data, &r) == nil && r.Error != "" {
return nil, errors.New(r.Error)
}
return nil, fmt.Errorf("asked part %d/%d, answered %q", n, p.Parts, got)
}
return reply.Data, nil
}
// before it leaves the process that sent it. A module's answer too large is refused here, loudly, in a
// message that fits, so its caller reads why instead of waiting out RequestTimeout.
// fits is an answer the bus can carry: the answer itself, or — when it is larger than one message —
// an error that says so, which is.
+3 -39
View File
@@ -2,7 +2,6 @@ package bus_test
import (
"encoding/json"
"fmt"
"strings"
"testing"
"time"
@@ -13,8 +12,8 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 314: an answer larger than the bus carries in one message arrives whole when the
// controller pages it, and a module's is said to be too large — neither is a 30-second silence.
// novox/hq issue 314: a module's answer larger than the bus carries in one message is said to be too large,
// at once, not a 30-second silence. (The controller's paging ended with the controller, issue 1126.)
func TestAnAnswerLargerThanOneMessage(t *testing.T) {
url := meshtest.URL(t)
seat, err := nats.Connect(url)
@@ -27,43 +26,8 @@ func TestAnAnswerLargerThanOneMessage(t *testing.T) {
t.Fatal(err)
}
defer c.Close()
limit := int(seat.MaxPayload())
large := strings.Repeat("a condition held past its bound; ", 3*int(seat.MaxPayload())/32)
// The controller as it pages (mesh-controller internal/link, pages.go): the first answer an error
// with `paged`, each part asked again on the same subject with the page header.
large := strings.Repeat("a condition held past its bound; ", 3*limit/32)
whole, _ := json.Marshal(map[string]any{"result": large})
partSize := limit - 1024
parts := (len(whole) + partSize - 1) / partSize
sub, err := seat.Subscribe(bus.SeatToolSubject("mesh-controller", "conditions", "", ""), func(m *nats.Msg) {
if page := m.Header.Get(bus.PageHeader); page != "" {
var id string
var n int
fmt.Sscanf(page, "%s %d", &id, &n)
end := min((n+1)*partSize, len(whole))
part := nats.NewMsg(m.Reply)
part.Header.Set(bus.PartHeader, fmt.Sprintf("%d/%d", n, parts))
part.Data = whole[n*partSize : end]
_ = m.RespondMsg(part)
return
}
body, _ := json.Marshal(map[string]any{"error": "too large for one message",
"paged": map[string]any{"call": "call-1-1", "bytes": len(whole), "parts": parts}})
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe()
_ = seat.Flush()
got, err := c.Ask("seat:mesh-controller.conditions", map[string]any{}, "")
var said string
if err != nil || json.Unmarshal(got.Result, &said) != nil || said != large {
t.Fatalf("an answer of %d bytes in %d parts arrived as %d bytes: %v", len(whole), parts, len(got.Result), err)
}
// A module's answer too large is refused here in words, at once.
stop, err := c.HandleSubject("mesh.mod.journal.tool.lines", func(json.RawMessage) (any, error) { return large, nil })
if err != nil {
t.Fatal(err)
+105
View File
@@ -0,0 +1,105 @@
package bus
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/pernode"
)
// A state its owner keeps one bucket per node (novox/hq ADR 0593 §6: each node's engine writes its own `composed-<node>`,
// and the provisioner, the assigner and the view read them all) is served to a bundle as one state. The rules (a key's
// first part names its node's bucket, keys and a watch over every bucket of the prefix, a bucket created later heard by
// the bus's creation advisory) are mesh-sdk's pernode package, written once for a process and for this runtime; this
// file adapts its entries to the runtime's shapes. A reader is granted the listing of stream names
// (pernode.ListSubject) and the creation advisory (pernode.CreatedAdvisory) beside each bucket's read.
func perNodeRefusal(module, name, what string) error {
return fmt.Errorf("%s's state %s is kept one bucket per node and read only: no %s of it as one state (novox/hq ADR 0593 §6)",
module, name, what)
}
func (c *Conn) family(s StateIssued) pernode.Family {
return pernode.Family{Read: s.Name, Prefix: s.Bucket, Bus: pernode.OverNATS(c.nc)}
}
func (c *Conn) perNodeGet(s StateIssued, key string) (*StateEntry, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
e, err := c.family(s).Get(ctx, key)
if err != nil || e == nil {
return nil, err
}
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
}
func (c *Conn) perNodeKeys(s StateIssued) ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
return c.family(s).Keys(ctx)
}
// perNodeWatch is StateWatch over every bucket of the family, and over a bucket created later. As in a single bucket's
// watch, StateTimeout bounds the time without progress (issue 1726), and a value the bundle does not take is said and
// the watch goes on.
func (c *Conn) perNodeWatch(module string, s StateIssued, pattern string, deliver func(StateChange) error) (func(), error) {
ctx, cancel := context.WithCancel(context.Background())
progress := make(chan struct{}, 1)
handle := func(ch pernode.Change) error {
select {
case progress <- struct{}{}:
default:
}
e := ch.Entry
change := StateChange{State: s.Name, Key: e.Key(), Revision: e.Revision(), Current: ch.Current}
switch e.Operation() {
case jetstream.KeyValuePut:
change.Op = "put"
change.Value = valueOf(e.Value())
default:
if ch.Current {
return nil
}
change.Op = "delete"
if e.Operation() == jetstream.KeyValuePurge {
change.Op = "expire"
}
}
if ctx.Err() != nil {
return nil
}
if err := deliver(change); err != nil {
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
module, change.Op, change.Key, s.Name, err)
}
return nil
}
done := make(chan error, 1)
go func() { done <- c.family(s).Watch(ctx, pattern, handle) }()
bound := time.NewTimer(StateTimeout)
defer bound.Stop()
for {
select {
case err := <-done:
if err != nil {
cancel()
return nil, err
}
return cancel, nil
case <-progress:
if !bound.Stop() {
select {
case <-bound.C:
default:
}
}
bound.Reset(StateTimeout)
case <-bound.C:
cancel()
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", s.Name, StateTimeout)
}
}
}
+98
View File
@@ -0,0 +1,98 @@
package bus
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// A module's code asks its machine's provisioner client for its values at runtime (novox/hq ADR 0349 clause 14 as
// ADR 0370 changes it, issue 685).
//
// While a module's code runs here, its asks reach the bus under this runtime's account, and the bus proves only
// that account. So the runtime asks its own machine's client on ProvisionAskSubject — which the mesh grants
// this machine's tool runner alone to publish (ADR 0370 §3) — naming the module itself, from the bundle that asks
// and never from the module's code or arguments (§4). The client answers that module's config values, for what its
// manifest declares, and no secret: the vault hands a secret only to a caller the bus proves (§2).
// ProvisionAskSubject is where this machine's tool runner asks its provisioner client (the node-engine
// internal/broker ProvisionerAskSubject): `prov.ask.<node>`.
func ProvisionAskSubject(node string) string { return "prov.ask." + node }
// ProvisionChangedSubjects is every module's word of what moved on a machine (the node-engine
// ProvisionerChangedSubject): `prov.changed.<node>.<module>`, a module's name dots and all.
func ProvisionChangedSubjects(node string) string { return "prov.changed." + node + ".>" }
// ProvisionTimeout bounds one ask of the client; a variable so a test need not wait.
var ProvisionTimeout = 10 * time.Second
// ProvisionChanged is what the client says moved of one module: the kind, and the key, or every key of the kind when
// empty. Never the value.
type ProvisionChanged struct {
Kind string `json:"kind"`
Key string `json:"key,omitempty"`
}
// ProvisionRefused is the client's refusal, in its own words: a key the module does not declare, a module not placed
// on the machine, a secret, a kind it does not answer yet.
type ProvisionRefused struct{ Reason string }
func (r *ProvisionRefused) Error() string { return r.Reason }
// AskProvisioner asks this machine's provisioner client for one value of module: kind `config` or `bound`, and the
// key. The module is the caller's to name from the bundle it runs; this never takes it from what a bundle sent.
func (c *Conn) AskProvisioner(module, kind, key string) (json.RawMessage, error) {
if c.node == "" {
return nil, errors.New("this runtime names no machine, so it has no provisioner client to ask")
}
body, err := json.Marshal(map[string]string{"module": module, "kind": kind, "key": key})
if err != nil {
return nil, err
}
msg, err := c.nc.Request(ProvisionAskSubject(c.node), body, ProvisionTimeout)
if errors.Is(err, nats.ErrNoResponders) {
return nil, fmt.Errorf("%s's provisioner client does not answer: it is not running there, or the bus does "+
"not let %s ask it", c.node, c.cred.User)
}
if err != nil {
return nil, fmt.Errorf("asking %s's provisioner client for %s's %s %s: %w", c.node, module, kind, key, err)
}
var answered struct {
Value json.RawMessage `json:"value"`
Error string `json:"error"`
}
if err := json.Unmarshal(msg.Data, &answered); err != nil {
return nil, fmt.Errorf("%s's provisioner client answered something that is not its answer: %w", c.node, err)
}
if answered.Error != "" {
return nil, &ProvisionRefused{Reason: answered.Error}
}
if len(answered.Value) == 0 {
return nil, fmt.Errorf("%s's provisioner client answered no value and no reason", c.node)
}
return answered.Value, nil
}
// OnProvisionChanged hands told every word of this machine's provisioner client of what moved, by module, until
// stopped.
func (c *Conn) OnProvisionChanged(told func(module string, changed ProvisionChanged)) (func(), error) {
if c.node == "" {
return nil, errors.New("this runtime names no machine, so it hears no provisioner client")
}
prefix := "prov.changed." + c.node + "."
sub, err := c.nc.Subscribe(ProvisionChangedSubjects(c.node), func(m *nats.Msg) {
var changed ProvisionChanged
if json.Unmarshal(m.Data, &changed) != nil || changed.Kind == "" {
return
}
told(strings.TrimPrefix(m.Subject, prefix), changed)
})
if err != nil {
return nil, err
}
return func() { _ = sub.Unsubscribe() }, c.nc.Flush()
}
+30
View File
@@ -84,3 +84,33 @@ func TestARefusedSubscriptionIsAskedForAgain(t *testing.T) {
t.Error("a stopped subject is still tracked")
}
}
// Issue 1966: a refused publish of an answer (to a caller's inbox) is said, once per subject per minute, never dropped
// in silence while the caller waits out its time.
func TestARefusedAnswerIsSaidOncePerMinute(t *testing.T) {
c := &Conn{}
var said []string
c.Logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
refuse := func(subject string) bool {
return c.answerRefused(fmt.Errorf("%w: Permissions Violation for Publish to %q", nats.ErrPermissionViolation, subject))
}
// A reply subject ends in a token new to every request: a burst under one prefix is one line.
if !refuse("_INBOX.user.abc.t1") || !refuse("_INBOX.user.abc.t2") || !refuse("_INBOX.user.def.t1") {
t.Fatal("a refused answer was not taken for one")
}
if len(said) != 2 {
t.Fatalf("want one line per caller, got %v", said)
}
c.inboxSaid["_INBOX.user.abc"] = time.Now().Add(-2 * time.Minute)
c.inboxSaid["_INBOX.user.old"] = time.Now().Add(-2 * time.Minute)
refuse("_INBOX.user.abc.t3")
if len(said) != 3 || !strings.Contains(said[2], "_INBOX.user.abc") {
t.Fatalf("a minute later it is said again, got %v", said)
}
if _, kept := c.inboxSaid["_INBOX.user.old"]; kept || len(c.inboxSaid) != 2 {
t.Fatalf("entries older than a minute are dropped on write, kept %v", c.inboxSaid)
}
if refuse("mesh.mod.x.tool.y") {
t.Fatal("a refused publish that is no answer was taken for one")
}
}
+11 -5
View File
@@ -24,7 +24,7 @@ import (
// and anything else is refused here, with the reason, before it is sent. A bundle reaches this only through
// its own standard input and output: no tool call does.
// SeatTraffic is what a membership lists of a module's seat traffic (the controller's broker.SeatTraffic).
// SeatTraffic is what a membership lists of a module's seat traffic (the grants part's record).
type SeatTraffic struct {
Publish []string `json:"publish,omitempty"`
Subscribe []string `json:"subscribe,omitempty"`
@@ -32,7 +32,7 @@ type SeatTraffic struct {
Workers []SeatWorker `json:"workers,omitempty"`
Records []string `json:"records,omitempty"`
// Kinds are the holders of every kinded bench the module uses or watches, with what each promises: the
// controller's record of their claims (novox/hq ADR 0259 §5).
// record of their claims (novox/hq ADR 0259 §5).
Kinds []KindHeld `json:"kinds,omitempty"`
}
@@ -45,7 +45,7 @@ type KindHeld struct {
Capabilities []string `json:"capabilities,omitempty"`
}
// SeatKinds is the kinds the module's membership names, as the controller issued them.
// SeatKinds is the kinds the module's membership names, as the grants part issued them.
func (c *Conn) SeatKinds(module string) ([]KindHeld, error) {
t, err := c.trafficOf(module)
if err != nil {
@@ -249,7 +249,7 @@ type Work struct {
// SeatTake reads a worker the module's seat traffic lists and hands each message to deliver: acknowledged
// when deliver returns nil, offered again after NakDelay when it returns an error, ended when it is not
// JSON. The worker is the controller's to create; this binds to it by name and never makes one.
// JSON. The worker is the mesh's to create; this binds to it by name and never makes one.
func (c *Conn) SeatTake(module, consumer string, deliver func(Work) error) (func(), error) {
t, err := c.trafficOf(module)
if err != nil {
@@ -370,7 +370,13 @@ func (c *Conn) SeatRecord(module, bucket, key string) (*StateEntry, error) {
}
var revision uint64
fmt.Sscan(reply.Header.Get("Nats-Sequence"), &revision)
return &StateEntry{Key: module + "." + key, Value: valueOf(reply.Data), Revision: revision}, nil
var at time.Time
if stamp := reply.Header.Get("Nats-Time-Stamp"); stamp != "" {
if t, err := time.Parse(time.RFC3339Nano, stamp); err == nil {
at = t.UTC()
}
}
return &StateEntry{Key: module + "." + key, Value: valueOf(reply.Data), Revision: revision, At: at}, nil
}
// StateCreate writes one key only when it has no value, and answers its revision; a key that has one is
+199
View File
@@ -0,0 +1,199 @@
package bus
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// **A holder that pulls one piece of work when it has room** (novox/hq ADR 0525 §4, §6; issue 1887 A11). SeatTake
// hands a bundle every piece a worker holds, one at a time and prefetched; a build module must instead take one ask
// only when it has a free slot, from its workers in their order (delivery, then check, then recheck), and decide each
// ask's fate when its step ends. SeatPull is one fetch of at most one message from a worker the module's membership
// lists, never a prefetch; SeatSettle gives a pulled message its fate. A message the bundle never settles (its process
// died) is delivered again by the worker's own acknowledgement wait.
// MaxPullWait bounds how long one pull waits for a message: under the bundle's 30 s wait for its answer.
var MaxPullWait = 25 * time.Second
// Pulled is one message a module pulled from a seat's worker, named by an id the runtime gave it.
type Pulled struct {
ID string `json:"id"`
Worker string `json:"worker"`
Subject string `json:"subject"`
Body json.RawMessage `json:"body"`
Headers map[string]string `json:"headers,omitempty"`
Delivered uint64 `json:"delivered"`
}
// pulls are the messages modules pulled and have not settled, by id; each kept with the module it was pulled for, so
// one module never settles another's.
type pulls struct {
mu sync.Mutex
held map[string]pulledMsg
cons map[string]jetstream.Consumer // by module and consumer
}
type pulledMsg struct {
module string
msg jetstream.Msg
}
var seatPulls = &pulls{held: map[string]pulledMsg{}, cons: map[string]jetstream.Consumer{}}
func pullID() string {
var b [8]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
// SeatPull fetches at most one message from consumer for module, waiting at most wait (none when 0): nil when none
// came. The consumer must be one the module's membership lists as its worker (novox/hq ADR 0259); the runtime binds to
// it by name and never makes one.
func (c *Conn) SeatPull(module, consumer string, wait time.Duration) (*Pulled, error) {
t, err := c.trafficOf(module)
if err != nil {
return nil, err
}
var w *SeatWorker
var names []string
for i := range t.Workers {
names = append(names, t.Workers[i].Consumer)
if t.Workers[i].Consumer == consumer {
w = &t.Workers[i]
}
}
if w == nil {
return nil, fmt.Errorf("%s pulls no work from %s: its membership lists %s (novox/hq ADR 0259)", module,
consumer, strings.Join(names, ", "))
}
if wait > MaxPullWait {
wait = MaxPullWait
}
bound, err := c.pullConsumer(module, *w)
if err != nil {
return nil, err
}
var batch jetstream.MessageBatch
if wait <= 0 {
batch, err = bound.FetchNoWait(1)
} else {
batch, err = bound.Fetch(1, jetstream.FetchMaxWait(wait))
}
if err != nil {
return nil, fmt.Errorf("pulling %s's worker %s: %w", module, w.Consumer, err)
}
msg, ok := <-batch.Messages()
if !ok {
if err := batch.Error(); err != nil {
return nil, fmt.Errorf("pulling %s's worker %s: %w", module, w.Consumer, err)
}
return nil, nil
}
if !json.Valid(msg.Data()) {
_ = msg.Term()
return nil, nil
}
headers := map[string]string{}
for k := range msg.Headers() {
headers[k] = msg.Headers().Get(k)
}
delivered := uint64(1)
if meta, merr := msg.Metadata(); merr == nil && meta != nil {
delivered = meta.NumDelivered
}
p := &Pulled{ID: pullID(), Worker: w.Consumer, Subject: msg.Subject(), Body: json.RawMessage(msg.Data()),
Headers: headers, Delivered: delivered}
seatPulls.mu.Lock()
seatPulls.held[p.ID] = pulledMsg{module: module, msg: msg}
seatPulls.mu.Unlock()
return p, nil
}
func (c *Conn) pullConsumer(module string, w SeatWorker) (jetstream.Consumer, error) {
key := module + " " + w.Consumer
seatPulls.mu.Lock()
bound, ok := seatPulls.cons[key]
seatPulls.mu.Unlock()
if ok {
return bound, nil
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
bound, err = js.Consumer(ctx, w.Stream, w.Consumer)
if err != nil {
return nil, fmt.Errorf("binding %s's worker %s on %s: %w", module, w.Consumer, w.Stream, err)
}
seatPulls.mu.Lock()
seatPulls.cons[key] = bound
seatPulls.mu.Unlock()
return bound, nil
}
// The fates SeatSettle gives a pulled message.
const (
SettleAck = "ack" // done: never delivered again
SettleNak = "nak" // handed back, to be delivered again at once
SettleTerm = "term" // dropped
SettleProgress = "progress" // still being worked: its acknowledgement wait starts again
// SettleNakAfter hands the message back to be delivered again after a delay: the bus's own redelivery, so no part
// keeps a timer for a step given again (novox/hq ADR 0664 §1; issue 1748). SeatSettleAfter gives it.
SettleNakAfter = "nak-after"
)
// SeatSettle gives the message module pulled under id its fate. A message pulled for another module, or settled
// already, is refused.
func (c *Conn) SeatSettle(module, id, fate string) error {
return c.settle(module, id, fate, 0)
}
// SeatSettleAfter hands the message module pulled under id back to be delivered again after delay (SettleNakAfter),
// counting one more delivery. A delay of 0 or below is refused and settles nothing.
func (c *Conn) SeatSettleAfter(module, id string, delay time.Duration) error {
if delay <= 0 {
return fmt.Errorf("a message given again after a delay needs a delay above 0, not %s", delay)
}
return c.settle(module, id, SettleNakAfter, delay)
}
func (c *Conn) settle(module, id, fate string, delay time.Duration) error {
seatPulls.mu.Lock()
p, ok := seatPulls.held[id]
if ok && p.module != module {
ok = false
}
if ok && fate != SettleProgress {
delete(seatPulls.held, id)
}
seatPulls.mu.Unlock()
if !ok {
return fmt.Errorf("%s holds no pulled message %s", module, id)
}
switch fate {
case SettleAck:
return p.msg.Ack()
case SettleNak:
return p.msg.Nak()
case SettleTerm:
return p.msg.Term()
case SettleProgress:
return p.msg.InProgress()
case SettleNakAfter:
return p.msg.NakWithDelay(delay)
}
seatPulls.mu.Lock()
seatPulls.held[id] = p // an unknown fate settles nothing
seatPulls.mu.Unlock()
return fmt.Errorf("a pulled message's fate is ack, nak, nak-after, term or progress, not %q", fate)
}
+79
View File
@@ -0,0 +1,79 @@
package bus_test
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// 1887/A11: a holder pulls one message at a time, only from a worker its membership lists, never a prefetch; it settles
// only what it pulled, and a message handed back comes again.
func TestAHolderPullsOneMessageWhenItHasRoom(t *testing.T) {
_, conn, js := seatMesh(t)
for _, id := range []string{"a1", "a2"} {
if _, err := conn.SeatPublish("mesh-delivery", "mesh.seat.operator-channel.accept.ask.mesh-delivery",
json.RawMessage(`{"id":"`+id+`"}`), id); err != nil {
t.Fatal(err)
}
}
first, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0)
if err != nil || first == nil || string(first.Body) != `{"id":"a1"}` || first.Delivered != 1 {
t.Fatalf("the first pull = %+v, %v", first, err)
}
// One pulled, one still waiting on the stream: nothing was fetched ahead.
info, err := js.ConsumerInfo("SEAT_OPERATOR_CHANNEL", "SEAT_OPERATOR_CHANNEL_worker")
if err != nil || info.NumPending != 1 || info.NumAckPending != 1 {
t.Fatalf("after one pull: pending %d, in hand %d (%v); want 1 and 1", info.NumPending, info.NumAckPending, err)
}
second, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0)
if err != nil || second == nil || string(second.Body) != `{"id":"a2"}` {
t.Fatalf("the second pull = %+v, %v", second, err)
}
if none, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0); err != nil || none != nil {
t.Fatalf("an empty worker answered %+v, %v", none, err)
}
began := time.Now()
if none, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 200*time.Millisecond); err != nil || none != nil || time.Since(began) < 150*time.Millisecond {
t.Fatalf("a waiting pull answered %+v, %v after %v", none, err, time.Since(began))
}
if err := conn.SeatSettle("telegram", first.ID, bus.SettleAck); err == nil {
t.Error("a module settled another's message")
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleProgress); err != nil {
t.Fatal(err)
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleAck); err != nil {
t.Fatal(err)
}
if err := conn.SeatSettle("messenger", first.ID, bus.SettleAck); err == nil {
t.Error("a message was settled twice")
}
if err := conn.SeatSettle("messenger", second.ID, bus.SettleNak); err != nil {
t.Fatal(err)
}
again, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", time.Second)
if err != nil || again == nil || string(again.Body) != `{"id":"a2"}` || again.Delivered != 2 {
t.Fatalf("the message handed back = %+v, %v", again, err)
}
// Issue 1748: a message settled with a delay is not given again at once, is after it, one delivery higher; a delay
// of 0 settles nothing.
if err := conn.SeatSettleAfter("messenger", again.ID, 0); err == nil {
t.Error("a delay of 0 was accepted")
}
if err := conn.SeatSettleAfter("messenger", again.ID, 300*time.Millisecond); err != nil {
t.Fatal(err)
}
if early, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 0); err != nil || early != nil {
t.Fatalf("a message given again after a delay came at once: %+v, %v", early, err)
}
late, err := conn.SeatPull("messenger", "SEAT_OPERATOR_CHANNEL_worker", 2*time.Second)
if err != nil || late == nil || late.Delivered != 3 {
t.Fatalf("the message given again after its delay = %+v, %v", late, err)
}
if _, err := conn.SeatPull("telegram", "SEAT_OPERATOR_CHANNEL_worker", 0); err == nil || !strings.Contains(err.Error(), "membership lists") {
t.Errorf("a channel pulled the router's work: %v", err)
}
}
+5 -2
View File
@@ -98,7 +98,7 @@ func TestAModuleSubmitsAndSaysOnlyUnderItsOwnNameAndKind(t *testing.T) {
t.Fatalf("an asker could not ask under its own name: %v", err)
}
for module, subject := range map[string]string{
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-controller",
"mesh-delivery": "mesh.seat.operator-channel.accept.ask.mesh-module-assigner",
"telegram": "mesh.seat.intake.event.choice.desktop",
"messenger": "mesh.seat.intake.event.choice.telegram",
} {
@@ -206,13 +206,16 @@ func TestAnAskerReadsItsOwnRecordAndNoOther(t *testing.T) {
if _, err := kv.Put("mesh-delivery.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
if _, err := kv.Put("mesh-controller.a1", []byte(`{"state":"open"}`)); err != nil {
if _, err := kv.Put("mesh-module-assigner.a1", []byte(`{"state":"open"}`)); err != nil {
t.Fatal(err)
}
got, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a1")
if err != nil || got == nil || string(got.Value) != `{"state":"open"}` {
t.Fatalf("the asker could not read its own ask: %v %v", got, err)
}
if got.At.IsZero() || time.Since(got.At) > time.Minute {
t.Errorf("the record does not say when the bus wrote it: at %v", got.At)
}
if missing, err := conn.SeatRecord("mesh-delivery", "messenger_asks", "a2"); err != nil || missing != nil {
t.Fatalf("a missing ask read as %v, %v", missing, err)
}
+60
View File
@@ -0,0 +1,60 @@
package bus
import (
"context"
"errors"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
"github.com/nats-io/nats.go/jetstream"
)
// SentGet reads one object of a sender's store of sent objects by the reference that carried it (ADR 0666 §16): the
// store is the sender's `<module>_sent-objects`, and no other — a state's objects are reached through their owner's
// bindings alone. It reads as this connection's account, which the bus lets read a sender's sent store only where the
// grants part granted it (a bundle is held to what its runtime channel carries by the runtime call, not here), from the manifests' callers and consumers; a refusal says the store.
func (c *Conn) SentGet(r buscalls.ObjectsSentGetRequest) (*buscalls.ObjectData, error) {
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
obs, err := c.sentStore(ctx, r)
if err != nil {
return nil, err
}
info, err := obs.GetInfo(ctx, r.Object)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return nil, fmt.Errorf("the sent object %s is not in %s: it is kept 7 days, then expires", r.Object, r.Store)
}
if err != nil {
return nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
}
data, err := obs.GetBytes(ctx, r.Object)
if err != nil {
return nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
}
return &buscalls.ObjectData{Info: objectInfoOf(info), Data: data}, nil
}
// sentStore checks the reference r and opens the sender's sent store it names.
func (c *Conn) sentStore(ctx context.Context, r buscalls.ObjectsSentGetRequest) (jetstream.ObjectStore, error) {
ref := stateobjects.Ref{Store: r.Store, Object: r.Object, Size: r.Size, Digest: r.Digest, Form: stateobjects.FormVersion}
if _, ok := stateobjects.ReadRef(stateobjects.Wrap(ref)); !ok {
return nil, errors.New("a sent object is named by the reference's store, object, size and sha256, as the reply or event carried them")
}
if !strings.HasSuffix(r.Store, catalogue.BusNameSeparator+catalogue.SentObjectsLocal) {
return nil, fmt.Errorf("%s is no sent store (a module's is `<module>%s%s`): only a sent object is read by its reference",
r.Store, catalogue.BusNameSeparator, catalogue.SentObjectsLocal)
}
js, err := c.JetStream()
if err != nil {
return nil, err
}
obs, err := js.ObjectStore(ctx, r.Store)
if err != nil {
return nil, fmt.Errorf("the sent store %s cannot be read as %s: it is not on the bus, or the bus grants this account no "+
"read of it (the manifests' callers and consumers are granted): %w", r.Store, c.cred.Module, err)
}
return obs, nil
}
+76
View File
@@ -0,0 +1,76 @@
package bus
import (
"context"
"errors"
"fmt"
"io"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"github.com/nats-io/nats.go/jetstream"
)
// The bus side of a bundle's put and get sessions (design 47 G §518, A-21, A-25): a put written into the module's object
// store as its chunks arrive and a get read out of it as they are asked, through the store's own chunking, so no more than
// one chunk is held in memory. The session itself is the runtime's (internal/runtime/sessions.go).
// ObjectsPutStream stores the object r.Name of the module's object store r.Objects from in, read to its end, with the
// put's description (A-25); it answers once the object is sealed. A read error of in drops the object: nothing is left.
func (c *Conn) ObjectsPutStream(ctx context.Context, module string, r buscalls.ObjectsNameRequest, description string, in io.Reader) (buscalls.ObjectInfo, error) {
if r.Name == "" {
return buscalls.ObjectInfo{}, errors.New("an object is put under a name")
}
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return buscalls.ObjectInfo{}, err
}
info, err := obs.Put(ctx, jetstream.ObjectMeta{Name: r.Name, Description: description}, in)
if err != nil {
_ = obs.Delete(context.WithoutCancel(ctx), r.Name)
return buscalls.ObjectInfo{}, err
}
return objectInfoOf(info), nil
}
// ObjectsOpen is one object of the module's object store as a reader of its bytes beside its info; found is false when there
// is none. The caller closes the reader.
func (c *Conn) ObjectsOpen(ctx context.Context, module string, r buscalls.ObjectsNameRequest) (info buscalls.ObjectInfo, rd io.ReadCloser, found bool, err error) {
obs, _, err := c.objectStore(ctx, module, r.Objects)
if err != nil {
return info, nil, false, err
}
res, err := obs.Get(ctx, r.Name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return info, nil, false, nil
}
if err != nil {
return info, nil, false, err
}
i, err := res.Info()
if err != nil {
res.Close()
return info, nil, false, err
}
return objectInfoOf(i), res, true, nil
}
// SentOpen is a sent object, read by its reference as SentGet reads it, as a reader of its bytes beside its info.
func (c *Conn) SentOpen(ctx context.Context, r buscalls.ObjectsSentGetRequest) (buscalls.ObjectInfo, io.ReadCloser, error) {
obs, err := c.sentStore(ctx, r)
if err != nil {
return buscalls.ObjectInfo{}, nil, err
}
res, err := obs.Get(ctx, r.Object)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return buscalls.ObjectInfo{}, nil, fmt.Errorf("the sent object %s is not in %s: it is kept 7 days, then expires", r.Object, r.Store)
}
if err != nil {
return buscalls.ObjectInfo{}, nil, fmt.Errorf("the sent object %s of %s: %w", r.Object, r.Store, err)
}
i, err := res.Info()
if err != nil {
res.Close()
return buscalls.ObjectInfo{}, nil, err
}
return objectInfoOf(i), res, nil
}
+399 -22
View File
@@ -5,16 +5,20 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"regexp"
"slices"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/membership"
"git.novox.be/novox/mesh-sdk/go/pernode"
"github.com/nats-io/nats.go/jetstream"
)
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the controller creates from what
// A module's state on the bus (novox/hq ADR 0201): key-value buckets the mesh creates from what
// the module declared, and issues to each assignment in its membership by the name the module uses —
// its own state by the local name, another module's as `<module>.<name>`.
//
@@ -28,6 +32,10 @@ type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
// PerNode says the state is kept one bucket per node (novox/hq ADR 0593 §6, mesh-sdk membership.StateIssued): Bucket is
// then the buckets' prefix, each bucket the prefix and a node's name, and a key's first part names its node. It is read
// only, served as one state (per_node.go).
PerNode bool `json:"per-node,omitempty"`
}
// StateEntry is one key's current value.
@@ -35,9 +43,12 @@ type StateEntry struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
// At is when the bus wrote this value (the entry's created time), as stdio.StateEntry.At reads it.
At time.Time `json:"at,omitzero"`
}
// StateChange is one change a watch delivers: a key put or deleted.
// StateChange is one change a watch delivers: a key put, deleted, or expired (its own time to live, or the
// bucket's age, ended; issue 1952).
type StateChange struct {
State string `json:"state"`
Key string `json:"key"`
@@ -63,21 +74,22 @@ func (c *Conn) issuedState(module, name string) (StateIssued, error) {
return StateIssued{}, fmt.Errorf("%s has no membership issued on %s yet, so no state of it is reachable "+
"until the mesh issues one (novox/hq ADR 0201)", module, c.node)
}
var names []string
// What a membership issues by a local name is mesh-sdk's one rule (membership.StateIssued, issue 1966): the old
// `state` list, then a key-value entry the module asks for in `requires` (ADR 0595 §1), by its local name.
issuing := membership.Membership{Bindings: m.Bindings}
for _, s := range m.State {
if s.Name == name {
return s, nil
}
names = append(names, s.Name)
issuing.State = append(issuing.State, membership.StateIssued{Name: s.Name, Bucket: s.Bucket, Writes: s.Writes, PerNode: s.PerNode})
}
if s, ok := issuing.StateIssued(name); ok {
return StateIssued{Name: s.Name, Bucket: s.Bucket, Writes: s.Writes, PerNode: s.PerNode}, nil
}
sort.Strings(names)
issued := "none"
if len(names) > 0 {
if names := issuing.StateNames(); len(names) > 0 {
issued = strings.Join(names, ", ")
}
return StateIssued{}, fmt.Errorf("%s keeps and reads no state called %q: it declares what it keeps under "+
"`state` and what it reads under `reads` as <module>.<name>, and was issued: %s (novox/hq ADR 0201)",
module, name, issued)
"`state` or as a key-value entry in `requires`, and what it reads under `reads` as <module>.<name>, and was "+
"issued: %s (novox/hq ADR 0201, ADR 0595 §1)", module, name, issued)
}
func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, error) {
@@ -88,7 +100,7 @@ func (c *Conn) bucket(ctx context.Context, s StateIssued) (jetstream.KeyValue, e
kv, err := js.KeyValue(ctx, s.Bucket)
if err != nil {
if errors.Is(err, jetstream.ErrBucketNotFound) {
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the controller "+
return nil, fmt.Errorf("the state %q is issued and its bucket is not on the bus yet: the mesh "+
"creates it from the catalogue on its next raise", s.Name)
}
return nil, err
@@ -113,6 +125,9 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
if err := checkKey(key); err != nil {
return nil, err
}
if s.PerNode {
return c.perNodeGet(s, key)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
@@ -126,7 +141,207 @@ func (c *Conn) StateGet(module, name, key string) (*StateEntry, error) {
if err != nil {
return nil, err
}
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision()}, nil
return &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}, nil
}
// StatePage is the current values after a key, in key order, and the key to ask after for the rest.
type StatePage struct {
Entries []StateEntry `json:"entries"`
Next string `json:"next,omitempty"`
}
// StatePageBudget is the time a page spends reading when the asker names none: well under the 30 s a reader waits.
const StatePageBudget = 10 * time.Second
// PageClock is the time a page's budget counts by, replaced by a test.
var PageClock = time.Now
// StatePage reads the values in key order after `after` in one pass, as many as fit maxBytes (each key and value),
// stopping when budget is spent, at least one value, so a bundle asks one request for many values, where StateGet
// is one request each. Access is checked once and the bucket bound once. It is a read like StateGet: served to the
// owner and to every reader, and writes nothing. Next is the last key given when more follow.
func (c *Conn) StatePage(module, name, after string, maxBytes int, budget time.Duration) (StatePage, error) {
s, err := c.issuedState(module, name)
if err != nil {
return StatePage{}, err
}
if budget <= 0 {
budget = StatePageBudget
}
// The bind and the listing have a StateTimeout of their own, and each get another: the page's time budget is what
// ends a slow page (with Next), never a deadline spread over the whole of it.
// ctx covers the bind and the listing alone; every bus request inside the loop takes each().
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
each := func() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), StateTimeout)
}
// Every current value is read in one stream per bucket (currentOf), never one bus request per key: a page of the
// assignments (about 330 keys) read key by key took 13 s from a node far from the bus (2026-11-14).
current := map[string]*StateEntry{}
if s.PerNode {
fam := c.family(s)
keys, err := fam.Keys(ctx)
if err != nil {
return StatePage{}, err
}
nodes := map[string]bool{}
for _, k := range keys {
if node, err := pernode.NodeOf(k); err == nil {
nodes[node] = true
}
}
for node := range nodes {
octx, ocancel := each() // each node's bucket is a bus request of its own: its own deadline
kv, err := fam.Open(octx, node)
ocancel()
if err != nil {
return StatePage{}, err
}
if kv == nil {
continue
}
wctx, wcancel := each()
got, err := currentOf(wctx, kv)
wcancel()
if err != nil {
return StatePage{}, err
}
maps.Copy(current, got)
}
} else {
kv, err := c.bucket(ctx, s)
if err != nil {
return StatePage{}, err
}
if current, err = currentOf(ctx, kv); err != nil {
return StatePage{}, err
}
}
keys := slices.Collect(maps.Keys(current))
get := func(key string) (*StateEntry, error) { return current[key], nil }
sort.Strings(keys)
page := StatePage{Entries: []StateEntry{}}
started := PageClock()
size := 0
for _, k := range keys {
if after != "" && k <= after {
continue
}
if len(page.Entries) > 0 && PageClock().Sub(started) >= budget {
page.Next = page.Entries[len(page.Entries)-1].Key
break
}
e, err := get(k)
if err != nil {
return StatePage{}, err
}
if e == nil {
continue
}
if size > 0 && size+len(k)+len(e.Value) > maxBytes {
page.Next = page.Entries[len(page.Entries)-1].Key
break
}
page.Entries = append(page.Entries, *e)
size += len(k) + len(e.Value)
}
return page, nil
}
// StateKept is one value a bucket still keeps for a key: a put, or a delete (a purge reads as one).
type StateKept struct {
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Revision uint64 `json:"revision"`
Op string `json:"op"`
At time.Time `json:"at"`
}
// StateHistory is every value the bucket still keeps for one key, oldest first, or none (novox/hq ADR
// 0307). It is a read like StateGet: served to the owner and to every reader, and writes nothing.
func (c *Conn) StateHistory(module, name, key string) ([]StateKept, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
if err := checkKey(key); err != nil {
return nil, err
}
if s.PerNode {
return nil, perNodeRefusal(module, name, "history")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
entries, err := kv.History(ctx, key)
kept := []StateKept{}
if errors.Is(err, jetstream.ErrKeyNotFound) {
return kept, nil
}
if err != nil {
return nil, err
}
for _, e := range entries {
k := StateKept{Key: e.Key(), Revision: e.Revision(), Op: "delete", At: e.Created().UTC()}
if e.Operation() == jetstream.KeyValuePut {
k.Op = "put"
k.Value = valueOf(e.Value())
}
kept = append(kept, k)
}
return kept, nil
}
// StateInfo is how full a state's bucket is against its caps (tools-are-how-a-module-is-spoken-to): a bundle's tools
// say fill against cap from it.
type StateInfo struct {
State string `json:"state"`
Bucket string `json:"bucket"`
// Bytes is what the bucket holds now, every kept value and marker counted; Keys how many keys it holds, a key
// whose last value is a delete or expiry marker counted until the marker goes.
Bytes uint64 `json:"bytes"`
Keys uint64 `json:"keys"`
// MaxBytes and MaxValueBytes are the bucket's caps; -1 is none.
MaxBytes int64 `json:"max_bytes"`
MaxValueBytes int32 `json:"max_value_bytes"`
// History is how many values a key keeps; TTLSeconds the bucket's age for a value (0 none); KeyTTL whether a key may
// carry its own time to live (issue 1952).
History int64 `json:"history"`
TTLSeconds int64 `json:"ttl_seconds"`
KeyTTL bool `json:"key_ttl"`
}
// StateInfo reads a state's bucket's fill and caps. It is a read like StateGet: served to the owner and every reader.
func (c *Conn) StateInfo(module, name string) (*StateInfo, error) {
s, err := c.issuedState(module, name)
if err != nil {
return nil, err
}
if s.PerNode {
return nil, perNodeRefusal(module, name, "fill")
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
status, err := kv.Status(ctx)
if err != nil {
return nil, err
}
b, ok := status.(*jetstream.KeyValueBucketStatus)
if !ok || b.StreamInfo() == nil {
return nil, fmt.Errorf("the bus answered no stream behind %s's state %s", module, name)
}
info := b.StreamInfo()
return &StateInfo{State: name, Bucket: status.Bucket(), Bytes: info.State.Bytes, Keys: uint64(info.State.NumSubjects),
MaxBytes: info.Config.MaxBytes, MaxValueBytes: info.Config.MaxMsgSize, History: status.History(),
TTLSeconds: int64(status.TTL() / time.Second), KeyTTL: info.Config.AllowMsgTTL && status.LimitMarkerTTL() > 0}, nil
}
// StatePut writes one key, as the module, where the module keeps the state. It answers the revision.
@@ -155,6 +370,61 @@ func (c *Conn) StatePut(module, name, key string, value json.RawMessage) (uint64
return kv.Put(ctx, key, value)
}
// MinKeyTTL is the shortest time to live a key may be given: the bus counts a key's age in whole seconds (mesh-sdk
// process.MinTTL).
const MinKeyTTL = time.Second
// StateCreateTTL writes one key only when it has no value, with its own time to live (novox/hq ADR 0564 clauses 4 and
// 7, issue 1952; mesh-sdk process.State.CreateTTL): the bus removes it when ttl passes and every watch hears the
// removal as an expiry, so a bundle's wake needs no timer. A key that has a value is refused as ErrStateChanged. The
// bucket must take keys with their own time to live (a state declared `key-ttl`); one that does not is refused, naming
// it, before anything is written.
func (c *Conn) StateCreateTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
if ttl < MinKeyTTL {
return 0, fmt.Errorf("%s's state %s, key %s: a time to live of %s is shorter than the bus keeps (%s)",
module, name, key, ttl, MinKeyTTL)
}
return c.stateWrite(module, name, key, value, func(ctx context.Context, kv jetstream.KeyValue) (uint64, error) {
if err := takesKeyTTL(ctx, kv, module, name); err != nil {
return 0, err
}
return kv.Create(ctx, key, value, jetstream.KeyTTL(ttl))
})
}
// StateRenewTTL writes one key whole with a fresh time to live, whatever it held (mesh-sdk process.State.RenewTTL): a
// time to live is set only when a key is created, so a live key is deleted first, and its watches hear the delete and
// then the new value.
func (c *Conn) StateRenewTTL(module, name, key string, value json.RawMessage, ttl time.Duration) (uint64, error) {
rev, err := c.StateCreateTTL(module, name, key, value, ttl)
if !errors.Is(err, ErrStateChanged) {
return rev, err
}
if err := c.StateDelete(module, name, key); err != nil {
return 0, err
}
return c.StateCreateTTL(module, name, key, value, ttl)
}
// takesKeyTTL refuses a bucket that does not allow a time to live per key, or leaves no marker when one ends: there the
// server would refuse the write, or the key would vanish and no watch would hear it.
func takesKeyTTL(ctx context.Context, kv jetstream.KeyValue, module, name string) error {
status, err := kv.Status(ctx)
if err != nil {
return err
}
allows := status.LimitMarkerTTL() > 0
if b, ok := status.(*jetstream.KeyValueBucketStatus); ok && b.StreamInfo() != nil {
allows = allows && b.StreamInfo().Config.AllowMsgTTL
}
if !allows {
return fmt.Errorf("%s's state %s does not take keys with their own time to live: its bucket %s allows none, or "+
"leaves no marker a watch hears when one ends; %s declares the state `key-ttl` for it (novox/hq ADR 0564, "+
"issue 1952)", module, name, status.Bucket(), module)
}
return nil
}
// StateDelete removes one key, as the module, where the module keeps the state. A key that was not
// there is not an error: what is asked for is that it is gone.
func (c *Conn) StateDelete(module, name, key string) error {
@@ -180,12 +450,20 @@ func (c *Conn) StateKeys(module, name string) ([]string, error) {
if err != nil {
return nil, err
}
if s.PerNode {
return c.perNodeKeys(s)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
kv, err := c.bucket(ctx, s)
if err != nil {
return nil, err
}
return keysOf(ctx, kv)
}
// keysOf is every key with a current value in one bucket, sorted.
func keysOf(ctx context.Context, kv jetstream.KeyValue) ([]string, error) {
lister, err := kv.ListKeys(ctx)
if err != nil {
return nil, err
@@ -217,7 +495,12 @@ func (c *Conn) writable(module, name string) (StateIssued, error) {
// StateWatch hands deliver the current value of every key matching the pattern — none that is
// deleted — and then every change, in order (ADR 0201). It returns once the current values are
// delivered; deliver is called from one goroutine, one change at a time, and an error from it is said
// delivered. **StateTimeout bounds the time without progress, not the whole set** (novox/hq issue
// 1726): each current value is a round trip to the bundle, so a state of a few hundred keys outlasts
// any fixed bound for all of them; the watch then failed at every attempt, the bundle dropped its
// handler, and the newest keys, handed over last, were never taken (the checker's owed, whose newest
// verdicts never reached mesh/check). It fails only when no current value arrives, or one is not
// taken, within StateTimeout; deliver is called from one goroutine, one change at a time, and an error from it is said
// and the watch goes on: state is not a queue, and the next change, or the next start, reads it again.
// The pattern is a key, with `*` for one name and `**` for the rest; empty is every key.
func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange) error) (stop func(), err error) {
@@ -242,12 +525,20 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
}
filter = strings.Join(parts, ".")
}
if s.PerNode {
return c.perNodeWatch(module, s, pattern, deliver)
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
kv, err := c.bucket(ctx, s)
cancel()
if err != nil {
return nil, err
}
return c.watchBucket(module, name, kv, filter, deliver)
}
// watchBucket is StateWatch over one bucket: the current values, then every change.
func (c *Conn) watchBucket(module, name string, kv jetstream.KeyValue, filter string, deliver func(StateChange) error) (stop func(), err error) {
watching, stopWatching := context.WithCancel(context.Background())
w, err := kv.Watch(watching, filter)
if err != nil {
@@ -262,9 +553,23 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
})
}
current := make(chan struct{})
// handed says a current value was handed over or passed by: progress, which restarts the bound.
handed := make(chan struct{}, 1)
progress := func() {
select {
case handed <- struct{}{}:
default:
}
}
go func() {
initial := true
for e := range w.Updates() {
// A watch given up, or stopped, hands over nothing more, though values stay buffered in the
// watcher: they would reach a watch id the bundle no longer holds (issue 774: 1257 orphan
// changes after 51 failed watches).
if watching.Err() != nil {
break
}
if e == nil {
// The end of what was there when the watch began.
if initial {
@@ -273,6 +578,11 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
}
continue
}
if initial {
// Arrival from the bus is progress before the bundle is asked: the bound measures the current
// values coming in, and restarts when the bundle has taken one.
progress()
}
change := StateChange{State: name, Key: e.Key(), Revision: e.Revision(), Current: initial}
switch e.Operation() {
case jetstream.KeyValuePut:
@@ -282,25 +592,50 @@ func (c *Conn) StateWatch(module, name, pattern string, deliver func(StateChange
// A deletion among the current values is a key that is not there: not handed over
// (measured, research 024 — the server sends its marker among the initial values).
if initial {
progress()
continue
}
change.Op = "delete"
if e.Operation() == jetstream.KeyValuePurge {
// The marker the server leaves when a key's time to live ends (Nats-Marker-Reason MaxAge,
// read by the client as a purge): nothing else purges a module's state, since neither the
// runtime nor mesh-sdk's State offers a purge (issue 1952).
change.Op = "expire"
}
}
if watching.Err() != nil {
break
}
if err := deliver(change); err != nil {
c.Logf("[mesh-tools] %s did not take %s %s of %s: %v; its next change, or its next start, reads it again",
module, change.Op, change.Key, name, err)
}
if initial {
progress()
}
}
if initial {
close(current)
}
}()
select {
case <-current:
return stop, nil
case <-time.After(StateTimeout):
stop()
return nil, fmt.Errorf("the current values of %s did not arrive in %s", name, StateTimeout)
bound := time.NewTimer(StateTimeout)
defer bound.Stop()
for {
select {
case <-current:
return stop, nil
case <-handed:
if !bound.Stop() {
select {
case <-bound.C:
default:
}
}
bound.Reset(StateTimeout)
case <-bound.C:
stop()
return nil, fmt.Errorf("the current values of %s did not arrive in %s: none was handed over in that time", name, StateTimeout)
}
}
}
@@ -318,7 +653,10 @@ func valueOf(raw []byte) json.RawMessage {
var credentialEndings = []string{"password", "passwd", "secret", "token", "credential", "credentials",
"authorization", "apikey", "privatekey", "accesskey", "cookie"}
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "".
// credentialField is the first field anywhere in a JSON value whose name says it is a credential, or "". The one shape a
// credential-shaped name may hold is a provider reference, an object of exactly `node` and `module`, both strings: a
// the assignment's binding of the provision "secret" to the vault (issue 1126) names who provides it, never its value. Any
// other shape under such a name is refused.
func credentialField(value json.RawMessage) string {
var v any
if json.Unmarshal(value, &v) != nil {
@@ -336,7 +674,7 @@ func credentialField(value json.RawMessage) string {
for _, k := range keys {
norm := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(k))
for _, end := range credentialEndings {
if strings.HasSuffix(norm, end) {
if strings.HasSuffix(norm, end) && !providerReference(t[k]) {
return k
}
}
@@ -355,3 +693,42 @@ func credentialField(value json.RawMessage) string {
}
return walk(v)
}
// providerReference says whether v is exactly {"node": <name>, "module": <name>}, each a name as mesh-sdk names them.
func providerReference(v any) bool {
m, ok := v.(map[string]any)
if !ok || len(m) != 2 {
return false
}
node, okN := m["node"].(string)
module, okM := m["module"].(string)
return okN && okM && referenceName.MatchString(node) && referenceName.MatchString(module)
}
// referenceName is a node's or a module's name as mesh-sdk names them: lower case, digits and dashes.
var referenceName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// currentOf is every current value of a bucket, read in one stream (a watch of every key, deletes left out, ended at its
// first nil): the values a page answers, without a bus request per key.
func currentOf(ctx context.Context, kv jetstream.KeyValue) (map[string]*StateEntry, error) {
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return nil, err
}
defer func() { _ = w.Stop() }()
out := map[string]*StateEntry{}
for {
select {
case <-ctx.Done():
return nil, fmt.Errorf("the values of %s are not all read: %w", kv.Bucket(), ctx.Err())
case e, ok := <-w.Updates():
if !ok {
return out, nil
}
if e == nil {
return out, nil
}
out[e.Key()] = &StateEntry{Key: e.Key(), Value: valueOf(e.Value()), Revision: e.Revision(), At: e.Created().UTC()}
}
}
}
@@ -0,0 +1,112 @@
//go:build integration
package bus_test
import (
"errors"
"fmt"
"reflect"
"sort"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A state its owner keeps one bucket per node is served to a bundle as one state** (novox/hq ADR 0593 §6: each node's
// engine writes its own `composed-<node>`, and the provisioner reads them all). Get goes to the bucket the key's first
// part names, Keys and Watch over every bucket of the prefix, a bucket created during the watch included, and a bucket
// that only starts like the prefix is never read.
func TestAPerNodeStateIsServedAcrossEveryNodesBucket(t *testing.T) {
mesh := mt.New(t)
a, b := mesh.Bucket(t, "composed-a"), mesh.Bucket(t, "composed-b")
other := mesh.Bucket(t, "composedx-z")
for kv, key := range map[interface {
Put(string, []byte) (uint64, error)
}]string{a: "a.m1.f1", b: "b.m2.f2", other: "z.m3.f3"} {
if _, err := kv.Put(key, []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
prov := mt.MembershipOf("mesh-provisioner", "anchor", false, nil)
prov.State = []bus.StateIssued{{Name: "mesh-host.composed", Bucket: "composed-", PerNode: true}}
mesh.Issue(t, prov)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("mesh-provisioner")
mt.Until(t, func() error {
if conn.Membership("mesh-provisioner") == nil {
return errors.New("not issued yet")
}
return nil
})
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "b.m2.f2"); err != nil || e == nil || e.Key != "b.m2.f2" || e.At.IsZero() || time.Since(e.At) > time.Minute {
t.Fatalf("get from the second node's bucket: %+v, %v", e, err)
}
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "q.m.f"); err != nil || e != nil {
t.Fatalf("a key of a node with no bucket should read as none: %+v, %v", e, err)
}
if e, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "z.m3.f3"); err != nil || e != nil {
t.Fatalf("the look-alike bucket was read: %+v, %v", e, err)
}
if _, err := conn.StateGet("mesh-provisioner", "mesh-host.composed", "nonode"); err == nil {
t.Fatal("a key naming no node should be refused")
}
if keys, err := conn.StateKeys("mesh-provisioner", "mesh-host.composed"); err != nil || !reflect.DeepEqual(keys, []string{"a.m1.f1", "b.m2.f2"}) {
t.Fatalf("keys across the buckets: %v, %v", keys, err)
}
if _, err := conn.StatePut("mesh-provisioner", "mesh-host.composed", "a.m1.f9", []byte(`{}`)); err == nil {
t.Fatal("a per-node state is read only")
}
var mu sync.Mutex
var current, live []string
stop, err := conn.StateWatch("mesh-provisioner", "mesh-host.composed", "", func(ch bus.StateChange) error {
mu.Lock()
defer mu.Unlock()
if ch.Current {
current = append(current, ch.Key)
} else {
live = append(live, ch.Key)
}
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
mu.Lock()
sort.Strings(current)
if !reflect.DeepEqual(current, []string{"a.m1.f1", "b.m2.f2"}) {
t.Fatalf("current values: %v", current)
}
mu.Unlock()
// A third node's bucket, created during the watch, and a change in a first one.
c := mesh.Bucket(t, "composed-c")
if _, err := c.Put("c.m4.f4", []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
if _, err := a.Put("a.m1.f5", []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
got := strings.Join(append(append([]string(nil), current...), live...), ",")
for _, want := range []string{"a.m1.f5", "c.m4.f4"} {
if !strings.Contains(got, want) {
return fmt.Errorf("%s not heard yet: current %v, changes %v", want, current, live)
}
}
return nil
})
}
+8
View File
@@ -15,6 +15,14 @@ func TestACredentialNamedFieldIsFoundAnywhereInAValue(t *testing.T) {
`{"clientSecret":"s"}`: "clientSecret",
`{"maxTokens":4096,"name":"a","generation":3}`: "",
`"just a string"`: "",
// Issue 1126: an assignment binds the provision "secret" to the vault; only an exact provider reference passes.
`{"bindings":{"secret":{"node":"anchor","module":"mesh-vault"}}}`: "",
`{"secret":{"value":"s"}}`: "secret",
`{"secret":{"node":"anchor","module":"mesh-vault","value":"s"}}`: "secret",
`{"secret":{"node":"anchor","module":7}}`: "secret",
`{"secret":{"node":"anchor","module":"hunter2 Bearer s3cr3t"}}`: "secret",
`{"secret":["s"]}`: "secret",
`{"bindings":{"secret":{"token":"s"}}}`: "secret",
} {
if got := credentialField(json.RawMessage(value)); got != want {
t.Errorf("%s: found %q, want %q", value, got, want)
+257
View File
@@ -0,0 +1,257 @@
//go:build integration
package bus_test
import (
"errors"
"fmt"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// **A watch whose current values take longer than the bound to hand over still holds** (novox/hq issue
// 1726). The gitea module watches the checker's owed: one key per pull request head, each a round trip
// to the bundle. With a few hundred keys the hand-over outlasted StateTimeout at every attempt, the
// watch was refused, and the newest keys, handed over last, never reached mesh/check: a group's passing
// verdict was never written on its members' heads. The bound is on progress: a watch fails only when
// no current value is handed over within it.
func TestAWatchHoldsWhileItsCurrentValuesKeepArriving(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 400 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
// A group of three pull requests judged last, after many older heads: their keys come last.
const older = 30
var keys []string
for i := 0; i < older; i++ {
keys = append(keys, fmt.Sprintf("novox/mesh-catalog.%d.%040x.status=mesh/check", 400+i, i))
}
for _, member := range []string{"novox/mesh-sdk.76.34f210e5", "novox/mesh-module-assigner.453.c06f75d0", "novox/mesh-delivery.33.28ee22ef"} {
keys = append(keys, member+strings.Repeat("0", 32)+".status=mesh/check")
}
for _, k := range keys {
if _, err := kv.Put(k, []byte(`{"v":1,"act":"status","context":"mesh/check","state":"success"}`)); err != nil {
t.Fatal(err)
}
}
var mu sync.Mutex
var current, live []string
// Each value takes 40 ms to be taken: 33 values take about 1.3 s, three times the bound.
stop, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", func(ch bus.StateChange) error {
if ch.Current {
time.Sleep(40 * time.Millisecond)
}
mu.Lock()
defer mu.Unlock()
if ch.Current {
current = append(current, ch.Key)
} else {
live = append(live, ch.Key)
}
return nil
})
if err != nil {
t.Fatalf("a watch whose current values kept arriving was refused: %v", err)
}
t.Cleanup(stop)
mu.Lock()
got := append([]string(nil), current...)
mu.Unlock()
if len(got) != len(keys) {
t.Fatalf("the watch returned after %d of %d current values", len(got), len(keys))
}
for i, k := range keys {
if got[i] != k {
t.Fatalf("current value %d is %s, want %s", i, got[i], k)
}
}
// And it holds: a verdict written afterwards reaches it as a change.
next := "novox/mesh-sdk.77." + strings.Repeat("a", 40) + ".status=mesh/check"
if _, err := kv.Put(next, []byte(`{"v":1,"act":"status","context":"mesh/check","state":"pending"}`)); err != nil {
t.Fatal(err)
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if len(live) != 1 || live[0] != next {
return fmt.Errorf("live changes so far: %v", live)
}
return nil
})
// **A watch given up, or stopped, hands over nothing more** (issue 774). The runtime's watch stopped its watcher
// when the current values did not arrive within the bound, but the watcher still held values, and its goroutine
// went on handing them over: to a watch id the bundle no longer held (on the provisioner, 51 failed watches were
// followed by 1257 orphan changes). Each row blocks the bundle on its first value, gives the watch up, lets the
// bundle go and counts what it is still handed.
t.Run("AWatchGivenUpHandsOverNothingMore", func(t *testing.T) {
for _, tc := range []struct {
name string
refused bool // the bound gives it up, or the caller stops it
}{
{"refused at the bound", true},
{"stopped by its caller", false},
} {
t.Run(tc.name, func(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 300 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
put := func(i int) {
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
if tc.refused {
for i := 0; i < 4; i++ {
put(i)
}
}
var calls atomic.Int32
first := make(chan struct{}, 1)
release := make(chan struct{})
deliver := func(ch bus.StateChange) error {
if calls.Add(1) == 1 {
first <- struct{}{}
<-release
}
return nil
}
var stop func()
if tc.refused {
if _, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err == nil {
t.Fatal("the watch whose bundle took nothing was not refused")
}
} else {
var err error
if stop, err = conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", deliver); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
put(i)
}
select {
case <-first:
case <-time.After(5 * time.Second):
t.Fatal("the bundle was handed no change")
}
stop()
}
// Let the watcher's server side deliver what it held, then free the bundle.
time.Sleep(300 * time.Millisecond)
close(release)
time.Sleep(500 * time.Millisecond)
if n := calls.Load(); n != 1 {
t.Fatalf("the bundle was handed %d changes of a watch given up, want 1 (the one in hand)", n)
}
})
}
})
}
// A watch whose bundle takes nothing for longer than the bound is still refused, so a bundle that hangs
// is said, not waited on for ever.
func TestAWatchIsRefusedWhenNoCurrentValueIsTakenWithinTheBound(t *testing.T) {
was := bus.StateTimeout
bus.StateTimeout = 300 * time.Millisecond
t.Cleanup(func() { bus.StateTimeout = was })
conn, kv := owedWatcher(t)
for i := 0; i < 3; i++ {
if _, err := kv.Put(fmt.Sprintf("novox/hq.%d.%040x.status=mesh/check", i+1, i), []byte(`{"v":1}`)); err != nil {
t.Fatal(err)
}
}
release := make(chan struct{})
t.Cleanup(func() { close(release) })
began := time.Now()
_, err := conn.StateWatch("gitea", "mesh-delivery-checker.owed", "", func(bus.StateChange) error {
<-release
return nil
})
if err == nil || !strings.Contains(err.Error(), "did not arrive in") {
t.Fatalf("a watch whose first value was never taken: want refused, got %v", err)
}
if took := time.Since(began); took > 3*time.Second {
t.Fatalf("the refusal took %s, far past the bound", took)
}
}
// **A membership the grants part wrote in its old wrapper is served whole** (issue 774). Until 2026-10-30 the part wrote
// its pass record, the membership inside, into its state memberships; the tool runner read the value as a bare
// membership, and served no tool and no state. This is what the live mesh held: read through mesh-sdk's codec, its tools
// are answered where the membership says and its state is reached. A held-back one's wrapper is refused, and the module has
// no membership then (774/S32/AC2: no stream is read in its place).
func TestAMembershipTheGrantsPartWroteWrappedIsServed(t *testing.T) {
mesh := mt.New(t)
mesh.Bucket(t, "gitea_repos")
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
gitea.State = []bus.StateIssued{{Name: "repos", Bucket: "gitea_repos", Writes: true}}
mesh.IssueInGrantsWrapped(t, gitea, "published")
// The old wrapper of a membership the part held back carries its unpublished composition: refused, never served.
heldBack := mt.MembershipOf("pages", "anchor", true, nil)
mesh.IssueInGrantsWrapped(t, heldBack, "not published, a shadow difference: publishing would lose the last holder of x")
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
var logged []string
conn.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
t.Cleanup(conn.Close)
conn.Follow("gitea")
mt.Until(t, func() error {
if m := conn.Membership("gitea"); m == nil || len(m.Serves) == 0 {
return fmt.Errorf("gitea's membership is %+v, want the one inside the wrapper", m)
}
return nil
})
if got, want := fmt.Sprint(conn.ServedOn("gitea", "gitea_list")), "[{mesh.mod.gitea.tool.gitea_list.anchor }]"; got != want {
t.Errorf("gitea_list is served on %s, want %s", got, want)
}
if _, err := conn.StatePut("gitea", "repos", "novox.hq", []byte(`{"v":1}`)); err != nil {
t.Fatalf("the state the membership issues is refused: %v", err)
}
if e, err := conn.StateGet("gitea", "repos", "novox.hq"); err != nil || e == nil || string(e.Value) != `{"v":1}` || e.At.IsZero() || time.Since(e.At) > time.Minute {
t.Errorf("the state reads back %+v, %v", e, err)
}
conn.Follow("pages")
time.Sleep(300 * time.Millisecond)
if m := conn.Membership("pages"); m != nil || !strings.Contains(strings.Join(logged, "\n"), "no membership issued for pages") {
t.Errorf("a held-back membership's wrapper was served or not said: %+v, %q", m, logged)
}
}
// owedWatcher is the gitea module's view of the checker's owed: the bucket, and a runtime connection
// that serves gitea with the owed issued for reading.
func owedWatcher(t *testing.T) (*bus.Conn, interface {
Put(string, []byte) (uint64, error)
}) {
t.Helper()
mesh := mt.New(t)
kv := mesh.Bucket(t, "mesh-delivery-checker_owed")
gitea := mt.MembershipOf("gitea", "anchor", false, nil)
gitea.State = []bus.StateIssued{{Name: "mesh-delivery-checker.owed", Bucket: "mesh-delivery-checker_owed"}}
mesh.Issue(t, gitea)
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("gitea")
mt.Until(t, func() error {
if conn.Membership("gitea") == nil {
return errors.New("gitea not issued yet")
}
return nil
})
return conn, kv
}
+369
View File
@@ -0,0 +1,369 @@
package bus
import (
"context"
"errors"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// A module's own stream, as the runtime publishes to it and hears it (novox/hq ADR 0363 §8, issue 725).
//
// A module may declare a stream of its own; the mesh creates it and says, in the module's
// membership, which of its events go there and on which subject (`streams`), and, in the membership of
// each module granted to hear it, which durable consumer on it to bind (`hears`). The runtime publishes and
// binds exactly as the membership says. A membership naming neither is what every membership was before:
// every event on EVENTS, and only the consumer on EVENTS bound.
// StreamIssued is one of a module's own streams as its membership names it: the events it carries, as the
// bus spells them (`*` one token, `>` the rest), and the subject an event of them is published on, `{event}`
// standing for the event.
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears: the stream and the durable consumer on it the
// runtime binds. The consumer is the mesh's to make, to filter and to delete.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// eventSubject is where a module's event is published: on the subject of the first of its membership's
// streams that carries it, else on EVENTS — `mesh.mod.<module>.event.<event>`, as every event was before.
func eventSubject(m *Membership, module, event string) string {
if m != nil {
for _, s := range m.Streams {
if s.Subject == "" || !strings.Contains(s.Subject, "{event}") {
continue
}
for _, pattern := range s.Events {
if eventMatches(pattern, event) {
return strings.ReplaceAll(s.Subject, "{event}", event)
}
}
}
}
return "mesh.mod." + module + ".event." + event
}
// eventMatches says whether an event's name falls under a pattern as the bus matches subjects: `*` is one
// token, `>` (or `**`, the manifest's spelling) the one or more tokens left.
func eventMatches(pattern, event string) bool {
if pattern == "" || event == "" {
return false
}
ps, es := strings.Split(pattern, "."), strings.Split(event, ".")
for i, p := range ps {
if p == ">" || p == "**" {
return i == len(ps)-1 && len(es) > i
}
if i >= len(es) || (p != "*" && p != es[i]) {
return false
}
}
return len(ps) == len(es)
}
// RebindEvery is how often a consumer the membership names and the bus did not let the runtime bind is
// tried again: the mesh makes the consumer and sends the user list carrying its grant in the
// composition that issues the membership, so one may arrive before the other (ADR 0363 §6).
var RebindEvery = 30 * time.Second
// hearing is one module's consumers: the one on EVENTS, and each on a module stream its membership names.
// Every event of all of them is handed to the module one at a time, as the consumer on EVENTS alone did.
type hearing struct {
c *Conn
js jetstream.JetStream
module string
deliver func(Envelope) error
// serial hands the module one event at a time across its consumers (novox/hq issue 387).
serial sync.Mutex
// halted is stopped, read by handle without mu: the bus's loops hand events while mu is held.
halted atomic.Bool
mu sync.Mutex
stopped bool
bound map[HeardIssued]jetstream.ConsumeContext
// made is when the bus made each bound consumer: a consumer deleted and made again under its name has
// another time, and a loop bound to the old one is not reading the new one.
made map[HeardIssued]time.Time
said map[HeardIssued]bool
// why is the last error the bus gave a consumer's loop, said when the loop ends on it.
// Guarded by whyMu, not mu: the bus hands errors from its own goroutines, possibly while mu is held.
whyMu sync.Mutex
why map[HeardIssued]error
kick chan struct{}
done chan struct{}
unhear func()
}
func newHearing(c *Conn, js jetstream.JetStream, module string, deliver func(Envelope) error) *hearing {
return &hearing{c: c, js: js, module: module, deliver: deliver,
bound: map[HeardIssued]jetstream.ConsumeContext{}, made: map[HeardIssued]time.Time{}, said: map[HeardIssued]bool{}, why: map[HeardIssued]error{},
kick: make(chan struct{}, 1), done: make(chan struct{})}
}
// bind reads one durable consumer by name, and on failure says what it was doing. Errors the bus gives
// the loop afterwards are handed to onErr, when given.
func (h *hearing) bind(stream, durable string, onErr func(error)) (jetstream.ConsumeContext, time.Time, string, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
consumer, err := h.js.Consumer(ctx, stream, durable)
if err != nil {
return nil, time.Time{}, "binding", err
}
var created time.Time
if info := consumer.CachedInfo(); info != nil {
created = info.Created
}
opts := []jetstream.PullConsumeOpt{jetstream.PullMaxMessages(1)}
if onErr != nil {
opts = append(opts, jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { onErr(err) }))
}
reading, err := consumer.Consume(h.handle, opts...)
if err != nil {
return nil, time.Time{}, "reading", err
}
return reading, created, "", nil
}
// handle hands one delivered event to the module: acknowledged when it took it, offered again after NakDelay
// when it did not, terminated when it is not an event at all.
func (h *hearing) handle(msg jetstream.Msg) {
env, ok := envelopeOf(msg.Subject(), msg.Headers(), msg.Data())
if !ok {
// Unparseable: redelivering bytes no version of a handler can read is a loop.
_ = msg.Term()
return
}
// Kept alive while the bundle works on it, as a worker's work is (seat.go): the consumer's ack wait
// (30 seconds) is for a runtime that died, not a handler that is slow — the runtime gives a handler
// launch.EventTimeout. Without it an event handled longer than the ack wait was offered again while
// it was handled, and counted toward max-deliver (novox/hq issue 387). Kept alive too while it waits
// for an event from another of the module's consumers to be handled.
done, stopped := make(chan struct{}), make(chan struct{})
go func() {
defer close(stopped)
tick := time.NewTicker(EventHeartbeat)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}()
h.serial.Lock()
halted := h.halted.Load()
var err error
if !halted {
err = h.deliver(env)
}
h.serial.Unlock()
close(done)
// No in-progress after the answer: one after a Nak would stretch its delay to the whole ack wait.
<-stopped
if halted {
// **A hearing stopped hands nothing on** (issue 1945): the module's bundle that subscribed may be
// gone, loaded again in its place; offered again at once, to the consumer bound after it.
_ = msg.Nak()
return
}
if err != nil {
_ = msg.NakWithDelay(NakDelay)
return
}
_ = msg.Ack()
}
// start binds what the membership names now, and follows it: a new membership binds what it adds and stops
// what it no longer names, and a consumer not bound yet is tried again every RebindEvery.
func (h *hearing) start() {
h.reconcile()
h.unhear = h.c.onMembership(func(m Membership) {
if m.Module != h.module {
return
}
h.nudge()
})
go func() {
tick := time.NewTicker(RebindEvery)
defer tick.Stop()
for {
select {
case <-h.done:
return
case <-h.kick:
case <-tick.C:
}
h.reconcile()
}
}()
}
// forget drops the last error a consumer's loop was given.
func (h *hearing) forget(heard HeardIssued) {
h.whyMu.Lock()
delete(h.why, heard)
h.whyMu.Unlock()
}
// nudge asks the loop to reconcile now.
func (h *hearing) nudge() {
select {
case h.kick <- struct{}{}:
default:
}
}
// stop stops every consumer on a module stream this module reads; the one on EVENTS is its caller's.
func (h *hearing) stop() {
h.mu.Lock()
defer h.mu.Unlock()
if h.stopped {
return
}
h.stopped = true
h.halted.Store(true)
close(h.done)
if h.unhear != nil {
h.unhear()
}
for heard, reading := range h.bound {
reading.Stop()
delete(h.bound, heard)
delete(h.made, heard)
}
}
// replaced says whether the consumer a loop is bound to is gone from the bus or was made again since. A bus
// that cannot be asked right now says no: the loop is left as it is.
func (h *hearing) replaced(heard HeardIssued) bool {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
consumer, err := h.js.Consumer(ctx, heard.Stream, heard.Consumer)
if errors.Is(err, jetstream.ErrConsumerNotFound) {
return true
}
if err != nil || consumer.CachedInfo() == nil {
return false
}
created := h.made[heard]
return !created.IsZero() && !consumer.CachedInfo().Created.Equal(created)
}
// reconcile makes what is bound what the module's membership names now.
func (h *hearing) reconcile() {
want := map[HeardIssued]bool{}
if m := h.c.Membership(h.module); m != nil {
for _, heard := range m.Hears {
if heard.Stream != "" && heard.Consumer != "" {
// The runner's own durable beside the program's (A-397).
heard.Consumer += ToolsSuffix
want[heard] = true
}
}
}
h.mu.Lock()
defer h.mu.Unlock()
if h.stopped {
return
}
for heard, reading := range h.bound {
if !want[heard] {
reading.Stop()
delete(h.bound, heard)
delete(h.made, heard)
h.c.Logf("[mesh-tools] %s no longer hears %s: its membership stopped naming its consumer %s",
h.module, heard.Stream, heard.Consumer)
}
}
for heard := range h.said {
if !want[heard] {
delete(h.said, heard)
h.forget(heard)
}
}
// **A loop the bus ended is no longer reading** (nats.go ends a pull loop on its own when its consumer is
// deleted): the mesh deletes and makes again under the same name a restricted consumer it cannot
// narrow (ADR 0363 §6), and the membership, unchanged, says nothing — so it is bound again here.
// **A loop the bus did not end is asked about**: nats.go learns of a deleted consumer only from the answer
// to a pull it has pending, and a consumer deleted and made again between two pulls, or while none is
// pending, leaves the loop waiting out its heartbeat (30 seconds) on a consumer that is gone. So each
// bound consumer is looked up by name: gone, or made at another time, ends its loop here.
for heard, reading := range h.bound {
if h.replaced(heard) {
h.whyMu.Lock()
if h.why[heard] == nil {
h.why[heard] = errors.New("the consumer was deleted or made again")
}
h.whyMu.Unlock()
reading.Stop()
}
}
for heard, reading := range h.bound {
select {
case <-reading.Closed():
delete(h.bound, heard)
delete(h.made, heard)
h.whyMu.Lock()
why := h.why[heard]
h.whyMu.Unlock()
if why == nil {
why = errors.New("no reason given")
}
h.c.Logf("[mesh-tools] %s stopped hearing %s: the bus ended reading its consumer %s (%v); bound again as it exists",
h.module, heard.Stream, heard.Consumer, why)
h.said[heard] = true
default:
}
}
missing := make([]HeardIssued, 0, len(want))
for heard := range want {
if h.bound[heard] == nil {
missing = append(missing, heard)
}
}
sort.Slice(missing, func(i, j int) bool { return missing[i].Stream < missing[j].Stream })
for _, heard := range missing {
heard := heard
h.forget(heard)
reading, created, doing, err := h.bind(heard.Stream, heard.Consumer, func(err error) {
h.whyMu.Lock()
h.why[heard] = err
h.whyMu.Unlock()
})
if err != nil {
// Said once, not at every attempt; said again when it is bound.
if !h.said[heard] {
h.c.Logf("[mesh-tools] %s does not hear %s yet: %s its consumer %s failed: %v; tried again every %s",
h.module, heard.Stream, doing, heard.Consumer, err, RebindEvery)
h.said[heard] = true
}
continue
}
h.bound[heard] = reading
h.made[heard] = created
delete(h.said, heard)
h.c.Logf("[mesh-tools] %s hears %s on its consumer %s", h.module, heard.Stream, heard.Consumer)
// When the bus ends the loop, look again at once rather than at the next tick.
go func() {
select {
case <-reading.Closed():
h.nudge()
case <-h.done:
}
}()
}
}
@@ -0,0 +1,71 @@
package bus
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// novox/hq ADR 0363 §2, §3: an event goes to the stream whose pattern carries it, as the bus matches subjects.
func TestAnEventFallsUnderAStreamsPatternAsTheBusMatchesIt(t *testing.T) {
for _, c := range []struct {
pattern, event string
want bool
}{
{"mail.*.message.added", "mail.home.message.added", true},
{"mail.*.message.added", "mail.home.message.removed", false},
{"mail.*.message.added", "mail.home.work.message.added", false},
{"mail.>", "mail.home.message.added", true},
{"mail.**", "mail.home", true},
{"mail.>", "mail", false},
{"status", "status", true},
{"status", "status.changed", false},
{"", "status", false},
} {
if got := eventMatches(c.pattern, c.event); got != c.want {
t.Errorf("%q under %q: %v, want %v", c.event, c.pattern, got, c.want)
}
}
// Design 47 G §347, §59, §462: a sent object is read by the reference that carried it, from a sender's sent store only; a
// state's objects, and a reference that is none are refused before the bus is asked.
t.Run("SentGetRefusesWhatIsNoSentObjectsReference", func(t *testing.T) {
digest := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
for _, c := range []struct {
name string
req buscalls.ObjectsSentGetRequest
want string
}{
{"no digest", buscalls.ObjectsSentGetRequest{Store: "m_sent-objects", Object: "o", Size: 3}, "named by the reference"},
{"a state's objects", buscalls.ObjectsSentGetRequest{Store: "m_asks-objects", Object: "o", Size: 3, Digest: digest}, "no sent store"},
} {
_, err := (&Conn{}).SentGet(c.req)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: got %v, want an error containing %q", c.name, err, c.want)
}
}
})
}
// A membership naming no stream publishes every event on EVENTS, exactly as before (issue 725).
func TestAnEventNoStreamCarriesIsPublishedOnEvents(t *testing.T) {
mail := &Membership{Module: "mail", Streams: []StreamIssued{{Stream: "MAIL", Events: []string{"mail.*.message.added"},
Subject: "mesh.stream.mail.event.{event}"}}}
for _, c := range []struct {
m *Membership
event string
want string
}{
{nil, "mail.home.message.added", "mesh.mod.mail.event.mail.home.message.added"},
{&Membership{Module: "mail"}, "mail.home.message.added", "mesh.mod.mail.event.mail.home.message.added"},
{mail, "mail.home.message.added", "mesh.stream.mail.event.mail.home.message.added"},
{mail, "status", "mesh.mod.mail.event.status"},
} {
if got := eventSubject(c.m, "mail", c.event); got != c.want {
t.Errorf("%s: %s, want %s", c.event, got, c.want)
}
}
if got := keyOf("mesh.stream.mail.event.mail.home.message.added"); got != "mail.mail.home.message.added" {
t.Errorf("a stream's event reaches its consumer as %q, not as on EVENTS", got)
}
}
+342
View File
@@ -0,0 +1,342 @@
package bus_test
import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// streamMesh is a mesh with the mail module's own stream MAIL, and a connection on anchor following the
// memberships of the modules named.
func streamMesh(t *testing.T, memberships ...bus.Membership) (*mt.Mesh, *bus.Conn) {
t.Helper()
mesh := mt.New(t)
mesh.ModuleStream(t, "mail", "MAIL")
for _, m := range memberships {
mesh.Issue(t, m)
}
conn, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
for _, m := range memberships {
conn.Follow(m.Module)
}
mt.Until(t, func() error {
for _, m := range memberships {
if conn.Membership(m.Module) == nil {
return errors.New(m.Module + " not issued yet")
}
}
return nil
})
return mesh, conn
}
// novox/hq ADR 0363 §2, §8 (issue 725): an event one of the module's streams carries is published on that
// stream's subject and kept by that stream alone; any other event of it goes to EVENTS as before.
func TestAnEventADeclaredStreamCarriesIsPublishedOnItsSubjectAlone(t *testing.T) {
mail := mt.MembershipOf("mail", "anchor", true, nil)
mail.Streams = []bus.StreamIssued{{Stream: "MAIL", Events: []string{"mail.*.message.added"},
Subject: "mesh.stream.mail.event.{event}"}}
mesh, conn := streamMesh(t, mail)
landed := mesh.NextEvent(t, "mesh.stream.mail.event.>")
if err := conn.PublishAs("mail", bus.Envelope{Key: "mail.home.message.added", Body: json.RawMessage(`{"id":1}`)}); err != nil {
t.Fatal(err)
}
select {
case got := <-landed:
if got != "mesh.stream.mail.event.mail.home.message.added" {
t.Errorf("published on %s", got)
}
case <-time.After(5 * time.Second):
t.Fatal("the event did not land on its stream's subject")
}
if err := conn.PublishAs("mail", bus.Envelope{Key: "status", Body: json.RawMessage(`{}`)}); err != nil {
t.Fatal(err)
}
if n := mesh.Messages(t, "MAIL"); n != 1 {
t.Errorf("MAIL holds %d messages, want the one event it carries", n)
}
if n := mesh.Messages(t, "EVENTS"); n != 1 {
t.Errorf("EVENTS holds %d messages, want only the event no stream carries", n)
}
}
// novox/hq ADR 0363 §8 (issue 725): a module hears, beside EVENTS, every module stream its membership names,
// one event at a time, with the key it reads on EVENTS; a membership naming none hears EVENTS alone.
func TestAModuleHearsEveryStreamItsMembershipNames(t *testing.T) {
reader := mt.MembershipOf("reader", "anchor", false, nil)
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.mail.home.>"}, 5*time.Second)
var mu sync.Mutex
heard := map[string]int{}
handling := 0
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
mu.Lock()
handling++
if handling > 1 {
t.Error("two events were handed to the module at once")
}
mu.Unlock()
time.Sleep(50 * time.Millisecond)
mu.Lock()
handling--
heard[env.Key]++
mu.Unlock()
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
count := func(key string) int {
mu.Lock()
defer mu.Unlock()
return heard[key]
}
// Its membership names no stream: EVENTS alone, as before.
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": 1})
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 1})
mt.Until(t, func() error {
if count("alpha.happened") != 1 {
return errors.New("the event on EVENTS was not heard")
}
return nil
})
if _, notYet := mesh.PendingOn(t, "MAIL", "anchor", "reader"); notYet != 1 || count("mail.mail.home.message.added") != 0 {
t.Fatalf("a module whose membership names no stream heard MAIL (%d not yet delivered)", notYet)
}
// Its membership now names its consumer on MAIL: bound, and what waited there is heard.
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh.Issue(t, reader)
mt.Until(t, func() error {
if count("mail.mail.home.message.added") != 1 {
return errors.New("the event on MAIL was not heard")
}
return nil
})
for n := 2; n <= 4; n++ {
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{"n": n})
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": n})
}
mt.Until(t, func() error {
if a, m := count("alpha.happened"), count("mail.mail.home.message.added"); a != 4 || m != 4 {
return fmt.Errorf("heard %d on EVENTS and %d on MAIL, want 4 and 4", a, m)
}
for _, stream := range []string{"EVENTS", "MAIL"} {
if ack, notYet := mesh.PendingOn(t, stream, "anchor", "reader"); ack != 0 || notYet != 0 {
return fmt.Errorf("%s still holds %d unacknowledged, %d undelivered", stream, ack, notYet)
}
}
return nil
})
// A membership that stops naming it stops what it reads there.
reader.Hears = nil
mesh.Issue(t, reader)
mt.Until(t, func() error {
if mesh.Consumers(t, "MAIL") != 1 {
return errors.New("MAIL's consumer is gone")
}
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 0})
time.Sleep(300 * time.Millisecond)
if _, notYet := mesh.PendingOn(t, "MAIL", "anchor", "reader"); notYet == 0 {
return errors.New("MAIL is still read")
}
return nil
})
// The membership as the grants part keeps it (ADR 0571 clause 7, 10): read first and applied live, and nothing
// published on the controller's subject is heard.
granted := reader
granted.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-grants"}}
mesh.Issue(t, granted)
again, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
again.Logf = func(string, ...any) {}
t.Cleanup(again.Close)
again.Follow("reader")
consumerOf := func() string {
if m := again.Membership("reader"); m != nil && len(m.Hears) == 1 {
return m.Hears[0].Consumer
}
return ""
}
if got := consumerOf(); got != "from-grants" {
t.Fatalf("the membership read at the follow names %q, want the grants part's", got)
}
granted.Hears[0].Consumer = "from-grants-2"
mesh.Issue(t, granted)
mt.Until(t, func() error {
if got := consumerOf(); got != "from-grants-2" {
return fmt.Errorf("the membership names %q, want the grants part's change", got)
}
return nil
})
// 774/S32/AC1: a copy said on the controller's subject (mesh.assignment) changes nothing; there is no stream to read it from.
streamed := reader
streamed.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "from-subject"}}
mesh.Publish(t, "mesh.assignment.anchor.reader", streamed)
again.Flush()
time.Sleep(200 * time.Millisecond)
if got := consumerOf(); got != "from-grants-2" {
t.Fatalf("a membership said on the controller's subject was applied: the membership names %q", got)
}
// Review g1 round 1: memberships that cannot be read at the follow (a bus not ready after its boot) are read again until
// they can be; the follow, which runs once per module, does not give the module up for good.
was := bus.GrantsRetry
bus.GrantsRetry = 50 * time.Millisecond
t.Cleanup(func() { bus.GrantsRetry = was })
mesh.DropGrants(t)
late, err := bus.Connect(bus.Credential{URL: mt.URL(t), Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
late.Logf = func(string, ...any) {}
t.Cleanup(late.Close)
late.Follow("reader")
if m := late.Membership("reader"); m != nil {
t.Fatalf("a membership was served from a state not there: %+v", m)
}
mesh.Issue(t, granted)
mt.Until(t, func() error {
if m := late.Membership("reader"); m == nil || len(m.Hears) != 1 || m.Hears[0].Consumer != "from-grants-2" {
return fmt.Errorf("the membership is %+v, want the one written once the state is there", m)
}
return nil
})
}
// novox/hq ADR 0363 §6 (issue 725): a consumer the membership names before the controller made it is bound once
// it exists, without the module's events on EVENTS waiting for it.
func TestAConsumerNamedBeforeItExistsIsBoundOnceItDoes(t *testing.T) {
restore := bus.RebindEvery
bus.RebindEvery = 200 * time.Millisecond
t.Cleanup(func() { bus.RebindEvery = restore })
reader := mt.MembershipOf("reader", "anchor", false, nil)
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
var mu sync.Mutex
heard := map[string]int{}
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
mu.Lock()
heard[env.Key]++
mu.Unlock()
return nil
})
if err != nil {
t.Fatalf("a consumer on a module stream not made yet kept the module from hearing EVENTS: %v", err)
}
t.Cleanup(stop)
mesh.Emit(t, "mesh.mod.alpha.event.happened", map[string]any{})
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{})
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if heard["alpha.happened"] != 1 || heard["mail.mail.home.message.added"] != 1 {
return fmt.Errorf("heard %v", heard)
}
return nil
})
}
// novox/hq ADR 0363 §6 (issue 725): a consumer the controller deletes and makes again under the same name, its
// membership unchanged, is bound again, and what is published after is heard — the module does not go deaf.
func TestAConsumerDeletedAndMadeAgainIsHeardAgain(t *testing.T) {
restore := bus.RebindEvery
bus.RebindEvery = 200 * time.Millisecond
t.Cleanup(func() { bus.RebindEvery = restore })
reader := mt.MembershipOf("reader", "anchor", false, nil)
reader.Hears = []bus.HeardIssued{{Stream: "MAIL", Consumer: "anchor_reader"}}
mesh, conn := streamMesh(t, reader)
mesh.Consumer(t, "anchor", "reader", []string{"mesh.mod.alpha.event.>"}, 5*time.Second)
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
logs := &syncLogs{}
conn.Logf = logs.Logf
var mu sync.Mutex
heard := 0
stop, err := conn.ConsumeAs("reader", func(env bus.Envelope) error {
if env.Key == "mail.mail.home.message.added" {
mu.Lock()
heard++
mu.Unlock()
}
return nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
count := func() int {
mu.Lock()
defer mu.Unlock()
return heard
}
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 1})
mt.Until(t, func() error {
if count() != 1 {
return errors.New("the first event on MAIL was not heard")
}
return nil
})
mesh.RemoveConsumer(t, "MAIL", "anchor", "reader")
mt.Until(t, func() error {
if !logs.Has("stopped hearing MAIL") {
return errors.New("the end of the loop was not said")
}
return nil
})
mesh.ConsumerOn(t, "MAIL", "anchor", "reader", []string{"mesh.stream.mail.event.>"}, 5*time.Second)
mesh.Emit(t, "mesh.stream.mail.event.mail.home.message.added", map[string]any{"n": 2})
mt.Until(t, func() error {
if count() != 2 {
return fmt.Errorf("heard %d events on MAIL after its consumer was made again, want 2", count())
}
return nil
})
}
// syncLogs keeps what the runtime says, from any goroutine.
type syncLogs struct {
mu sync.Mutex
lines []string
}
func (l *syncLogs) Logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *syncLogs) Has(fragment string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
if strings.Contains(line, fragment) {
return true
}
}
return false
}
+87
View File
@@ -0,0 +1,87 @@
package bus
import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"github.com/nats-io/nats.go/jetstream"
)
// SubscribeNew hears one event, named `<emitter>.<event>`, from the next message on, on an ordered consumer of its own on each
// stream that carries it, and hands every message to deliver as the envelope a module sees. The subjects the name lands on are
// mesh-sdk's one rule (buscompose.EventSubjects, design 47 G §559, A-153), so a name with a wildcard is refused, naming the rule;
// a name no stream on the bus carries is an error, never a subscription that hears nothing. stop ends every consumer.
func (c *Conn) SubscribeNew(name string, deliver func(json.RawMessage) error) (func(), error) {
subjects, err := buscompose.EventSubjects(name, "")
if err != nil {
return nil, err
}
js, err := c.JetStream()
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), StateTimeout)
defer cancel()
// The streams that carry each subject, found by the bus: a namespace no stream carries hears nothing and is skipped.
byStream := map[string][]string{}
for _, subject := range subjects {
if strings.HasSuffix(subject, ".>") {
continue
}
stream, err := js.StreamNameBySubject(ctx, subject)
if err != nil || stream == "" {
continue
}
byStream[stream] = append(byStream[stream], subject, subject+".>")
}
if len(byStream) == 0 {
return nil, fmt.Errorf("no stream on the bus carries %s (%s): nothing would be heard", name, strings.Join(subjects, ", "))
}
var stops []func()
stopAll := func() {
for _, s := range stops {
s()
}
}
for _, stream := range slices.Sorted(mapKeys(byStream)) {
cons, err := js.OrderedConsumer(ctx, stream, jetstream.OrderedConsumerConfig{FilterSubjects: byStream[stream],
DeliverPolicy: jetstream.DeliverNewPolicy})
if err != nil {
stopAll()
return nil, fmt.Errorf("an ordered consumer of %s on %s: %w", name, stream, err)
}
cc, err := cons.Consume(func(m jetstream.Msg) {
env, ok := envelopeOf(m.Subject(), m.Headers(), m.Data())
if !ok {
return
}
raw, err := json.Marshal(env)
if err != nil {
return
}
if err := deliver(raw); err != nil && c.Logf != nil {
c.Logf("[mesh-tools] a subscription for new %s messages did not take one: %v", name, err)
}
})
if err != nil {
stopAll()
return nil, fmt.Errorf("consuming %s on %s: %w", name, stream, err)
}
stops = append(stops, cc.Stop)
}
return stopAll, nil
}
func mapKeys(m map[string][]string) func(func(string) bool) {
return func(yield func(string) bool) {
for k := range m {
if !yield(k) {
return
}
}
}
}
+145
View File
@@ -0,0 +1,145 @@
package bus
import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// Waiting on the bus for one event (novox/hq ADR 0367): what the mesh MCP server's `mesh_wait` does.
//
// **A subscription for the call, and nothing kept** (§2): no durable consumer, no queue, no stream. It
// subscribes the subjects when called and unsubscribes them before it answers, so an event published
// between two calls is not heard — a session reads the state from its owner after every wake (§4).
// Heard is one event a wait heard: the subject it arrived on, its body, when it arrived, and when it was published, the
// event's own header `x-time` as its emitter wrote it, empty for an event that carries none (design 34 every-event §12).
type Heard struct {
Subject string
Body json.RawMessage
At time.Time
Published string
}
// HearFirst subscribes the subjects for this call alone and answers the first message whose body match
// accepts (every one, when match is nil), or ok false once `within` has passed. Every subscription is
// closed before it returns. A subscription the bus refuses is noted and the others go on waiting; refused lists
// those noted by the time it answers (design 34 every-event §16). When the bus refused every subscription of the
// call, it still waits out `within` and then answers the refusal, so a caller looping on it asks once a timeout and
// never at once again. A caller gone — ctx done — ends the wait at once with ctx's error, so nothing waits on for
// nobody.
func (c *Conn) HearFirst(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (heard Heard, ok bool, refused []string, err error) {
if len(subjects) == 0 {
return Heard{}, false, nil, fmt.Errorf("a wait names at least one subject")
}
var unique []string
for _, subject := range subjects {
if !slices.Contains(unique, subject) {
unique = append(unique, subject)
}
}
got := make(chan Heard, 1)
refusals := make(chan string, len(unique))
var subs []*nats.Subscription
defer func() {
for _, s := range subs {
_ = s.Unsubscribe()
}
c.mu.Lock()
for _, subject := range unique {
delete(c.waits[subject], refusals)
if len(c.waits[subject]) == 0 {
delete(c.waits, subject)
}
}
c.mu.Unlock()
}()
c.mu.Lock()
if c.waits == nil {
c.waits = map[string]map[chan string]bool{}
}
for _, subject := range unique {
if c.waits[subject] == nil {
c.waits[subject] = map[chan string]bool{}
}
c.waits[subject][refusals] = true
}
c.mu.Unlock()
for _, subject := range unique {
sub, err := c.nc.Subscribe(subject, func(msg *nats.Msg) {
body := json.RawMessage(msg.Data)
if !json.Valid(body) {
quoted, _ := json.Marshal(string(msg.Data))
body = quoted
}
if match != nil && !match(body) {
return
}
select {
case got <- Heard{Subject: msg.Subject, Body: body, At: time.Now().UTC(), Published: msg.Header.Get("x-time")}:
default:
}
})
if err != nil {
return Heard{}, false, nil, fmt.Errorf("could not wait on %s: %w", subject, err)
}
subs = append(subs, sub)
}
// The bus has every subscription — or has said which it refused — once this returns.
if err := c.nc.FlushTimeout(5 * time.Second); err != nil {
return Heard{}, false, nil, fmt.Errorf("the bus did not take the wait: %w", err)
}
timer := time.NewTimer(within)
defer timer.Stop()
for {
select {
case h := <-got:
return h, true, refused, nil
case subject := <-refusals:
if !slices.Contains(refused, subject) {
refused = append(refused, subject)
}
case <-timer.C:
if len(refused) == len(unique) {
return Heard{}, false, refused, fmt.Errorf("the bus refused this account a subscription on %s: the "+
"grants part grants this machine's tool runner every event subject (novox/hq ADR 0670 §1), so its "+
"grant is not composed yet; read the state from its owner meanwhile", strings.Join(refused, ", "))
}
return Heard{}, false, refused, nil
case <-ctx.Done():
return Heard{}, false, refused, ctx.Err()
}
}
}
// waitRefused tells every wait on a subject the bus refused that it was refused; true when one was waiting.
func (c *Conn) waitRefused(subject string) bool {
c.mu.Lock()
defer c.mu.Unlock()
told := false
for ch := range c.waits[subject] {
select {
case ch <- subject:
default:
}
told = true
}
return told
}
// Subscriptions is how many subscriptions this connection holds now: what a wait leaves behind is none.
func (c *Conn) Subscriptions() int { return c.nc.NumSubscriptions() }
// subjectRefused is the subject a subscription refusal names, or "".
func subjectRefused(err error) string {
m := refusedSubject.FindStringSubmatch(err.Error())
if len(m) < 2 {
return ""
}
return strings.Trim(m[1], `"`)
}
+117
View File
@@ -0,0 +1,117 @@
package bus_test
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq ADR 0670 §1 (design 34 every-event §12, §16): a subscription the bus refuses is noted and the others go on
// waiting, a heard event carries its own time; a wait whose every subscription is refused answers only after its
// timeout, in words naming the grants part's grant of every event subject, never a wait grant (E-2).
func TestTheBusRefusesAWaitOutsideTheGrant(t *testing.T) {
granted := "mesh.seat.operator-channel.event.decided.claude-code"
other := "mesh.seat.operator-channel.event.decided.shop"
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
Users: []*server.User{{Username: "desk.node-tools", Password: "pw", Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: []string{"$JS.API.DIRECT.GET.>", "mesh.>"}},
Subscribe: &server.SubjectPermission{Allow: []string{granted, "_INBOX.desk.node-tools.>", "mesh.assignment.desk.*"}},
}}}})
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
c := dial(t, s.ClientURL(), "node-tools", "desk", "desk.node-tools")
// Every subscription refused: answered after the timeout, never at once, in §16's words.
start := time.Now()
_, ok, refused, err := c.HearFirst(context.Background(), []string{other}, 300*time.Millisecond, nil)
if took := time.Since(start); took < 300*time.Millisecond || took > 3*time.Second {
t.Errorf("a wait refused whole answered after %s, want its timeout of 300ms", took)
}
if ok || err == nil || !strings.Contains(err.Error(), "the bus refused this account a subscription on "+other) ||
!strings.Contains(err.Error(), "every event subject (novox/hq ADR 0670 §1)") ||
strings.Contains(err.Error(), "wait grant") || strings.Contains(err.Error(), "0367") ||
len(refused) != 1 || refused[0] != other {
t.Errorf("a wait on another asker's decided: heard %v, refused %v, %v", ok, refused, err)
}
// One refused beside one granted: the granted one hears, with the event's own time, and the refused one is listed.
pub, err := nats.Connect(s.ClientURL(), nats.UserInfo("desk.node-tools", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(pub.Close)
for _, published := range []string{"2026-11-12T03:00:00.5Z", ""} {
type result struct {
h bus.Heard
refused []string
err error
}
done := make(chan result, 1)
go func() {
h, ok, refused, err := c.HearFirst(context.Background(), []string{other, granted}, 5*time.Second, nil)
if !ok && err == nil {
err = errors.New("timed out")
}
done <- result{h, refused, err}
}()
// The event is said again until the wait, subscribed in its own time, hears it.
say := time.NewTicker(20 * time.Millisecond)
var r result
heard:
for {
select {
case r = <-done:
break heard
case <-say.C:
msg := nats.NewMsg(granted)
msg.Data = []byte(`{"ask":"a1"}`)
if published != "" {
msg.Header.Set("x-time", published)
}
if err := pub.PublishMsg(msg); err != nil {
t.Fatal(err)
}
}
}
say.Stop()
if r.err != nil || r.h.Subject != granted || string(r.h.Body) != `{"ask":"a1"}` || r.h.Published != published {
t.Errorf("heard %+v, %v; want published %q", r.h, r.err, published)
}
if len(r.refused) > 1 || len(r.refused) == 1 && r.refused[0] != other {
t.Errorf("refused %v, want at most %s", r.refused, other)
}
}
}
// A wait whose caller is gone ends at once, its subscription closed (the review of issue 745).
func TestAWaitEndsWhenItsCallerIsGone(t *testing.T) {
c := dial(t, mt.URL(t), "node-tools", "desk", "")
before := c.Subscriptions()
ctx, cancel := context.WithCancel(context.Background())
time.AfterFunc(200*time.Millisecond, cancel)
start := time.Now()
_, ok, _, err := c.HearFirst(ctx, []string{"mesh.mod.mesh-knowledge.event.reviewed"}, time.Minute, nil)
if ok || !errors.Is(err, context.Canceled) {
t.Errorf("a cancelled wait answered %v, %v", ok, err)
}
if took := time.Since(start); took > 2*time.Second {
t.Errorf("a cancelled wait ended after %s", took)
}
if after := c.Subscriptions(); after != before {
t.Errorf("%d subscriptions before, %d after", before, after)
}
}
+385 -106
View File
@@ -2,8 +2,8 @@ package console
// The mesh's tools found by address, not announced whole (novox/hq ADR 0195, to-be 34 §3a).
//
// The console announces six tools — five to find and call, and mesh_runtimes to say which runtimes
// discovery heard. Everything the mesh answers is reached through them by one address per layer:
// The console announces seven tools — five to find and call, mesh_runtimes to say which runtimes
// discovery heard, and mesh_wait to wait on the bus for an event (novox/hq ADR 0367). Everything the mesh answers is reached through them by one address per layer:
//
// <seat>.<verb> a seat held once for the mesh — its holder answers
// <node>/<seat>.<verb> a seat held once per machine — that machine's holder answers
@@ -17,7 +17,14 @@ package console
// session — so a tool that arrived a minute ago is found without the client reconnecting.
import (
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
@@ -26,31 +33,35 @@ import (
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/firstview"
)
// IndexKept is how long what the mesh answered is kept before it is asked again: long enough that
// one agent turn's search, describe and call ask once, short enough that nothing goes stale.
var IndexKept = 5 * time.Second
// The six tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
// The seven tools the console announces. Names of the API's kind — letters, digits, `_`, `-`.
const (
verbOverview = "mesh_overview"
verbMachine = "mesh_machine"
verbSearch = "mesh_search"
verbDescribe = "mesh_describe"
verbCall = "mesh_call"
verbRuntimes = "mesh_runtimes"
verbOverview = "mesh_overview"
verbMachine = "mesh_machine"
verbSearch = "mesh_search"
verbDescribe = "mesh_describe"
verbCall = "mesh_call"
verbRuntimes = "mesh_runtimes"
verbFirstViews = "mesh_first_views"
verbFirstAgain = "mesh_first_view_again"
verbWait = "mesh_wait"
)
// searchCap is how many matches a search answers before it says how many more there were.
const searchCap = 25
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-controller.nodes`); " +
const grammar = "Addresses: `<seat>.<verb>` for a seat held once for the mesh (e.g. `mesh-module-assigner.assignments`); " +
"`<node>/<seat>.<verb>` for a seat every machine holds (e.g. `ace/node-packet-filter.rules`); " +
"`<node>/<module>.<tool>` for a module on one machine (e.g. `novox/postgres.postgres_list_databases`); " +
"and `<module>.<tool>` also for a module whose instances are interchangeable."
// discovery is the six tools as tools/list announces them.
// discovery is the seven tools as tools/list announces them.
func discovery() []map[string]any {
str := func(desc string) map[string]any { return map[string]any{"type": "string", "description": desc} }
obj := func(props map[string]any, required ...string) map[string]any {
@@ -84,12 +95,22 @@ func discovery() []map[string]any {
"arrive, its size in bytes, how many modules and tools it announced, whether it was shortened to fit the " +
"bus, and when it was last heard — and every runtime or machine expected that was not heard. Read-only; " +
"for when an address is said to be missing or on another machine."},
{"name": verbFirstViews, "inputSchema": obj(map[string]any{}),
"description": "The tool runner's first views: the assigner's assignments and seat-holders, and the catalog half's registrations, the gate's source and the view of each watched branch (which modules exist, each with its manifest): for each, the owner, the tool " +
"asked, the revision held and when it was made, how many keys, when it was asked and answered, how often " +
"it was asked and how often by hand, whether its condition is raised, and why it was last asked. Read-only."},
{"name": verbFirstAgain, "inputSchema": obj(map[string]any{
"why": map[string]any{"type": "string", "description": "why it is asked again, kept in the log"},
}), "description": "Ask the tool runner's four first views again, now, and say what each answers. A hand act, recorded in " +
"the runner's log with its reason; for when a picture is doubted, or after an owner's state was made again."},
waitTool(),
approvalsTool(),
}
}
func isDiscovery(name string) bool {
switch name {
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes:
case verbOverview, verbMachine, verbSearch, verbDescribe, verbCall, verbRuntimes, verbFirstViews, verbFirstAgain, verbWait, verbApprovals:
return true
}
return false
@@ -125,15 +146,55 @@ type index struct {
NotAnswering []string
Listing *Listing // the flat catalogue the call path resolves subjects with
Discovery announce.Discovery
// Recorded is where the controller's records place each module that declares tools.
// Recorded is where the assigner places each module that declares tools.
Recorded map[string][]string
// Unheard is every runtime whose answer this index lacks: it said it was there and did not say
// what it serves in time, or it answered before and not now. An address it might answer is never
// called missing while it is here (2026-10-05).
Unheard []unheard
// Replaces is what each verb and tool replaces, from the controller's records (novox/hq ADR 0245):
// Placed is the state seat-holders as the seat mesh-module-assigner's holder wrote it (novox/hq ADR 0571 clause 4), by its
// keys `<seat>` and `<seat>.<node>`, read with assigner.HoldersOf; PlacedWhy says why it could not be read.
Placed map[string][]byte
PlacedWhy string
// Replaces is what each verb and tool replaces (novox/hq ADR 0245):
// `seat:<seat>.<verb>` and `<module>.<tool>`.
Replaces map[string][]string
// Earlier is every runtime not heard this round whose last answer, within RuntimeForgotten, this
// index holds instead (novox/hq issue 554), by runtimeKey: when it was heard.
Earlier map[string]time.Time
// Stale is every module on a machine (`mod:<module>@<node>`) and every seat's holder
// (`seat:<seat>@<node>`) this index knows only from an earlier answer: from whom, and when.
Stale map[string]staleFrom
}
// staleFrom is where an address known only from an earlier round of discovery was heard.
type staleFrom struct {
Runtime string
Machine string
At time.Time
}
// say is the sentence a call or a description carries for it.
func (f staleFrom) say() string {
who := f.Runtime
if f.Machine != "" {
who += " on " + f.Machine
}
return fmt.Sprintf("found from what %s announced %s ago: it did not answer the latest round of discovery",
who, time.Since(f.At).Round(time.Second))
}
// earlier is a runtime's last answer, used for a round that did not hear it.
type earlier struct {
Heard announce.Heard
At time.Time
}
func endpointKey(e announce.Endpoint) string {
if e.Kind == announce.KindSeat {
return "seat:" + e.Seat + "@" + e.Node
}
return "mod:" + e.Module + "@" + e.Node
}
// unheard is one runtime discovery did not hear in full, and why.
@@ -185,42 +246,8 @@ func findTool(tools []Tool, name string) *Tool {
return nil
}
// controllerOutput is a controller seat verb's answer: the command's printed output.
func controllerOutput(conn *bus.Conn, verb string) (string, error) {
got, err := conn.Ask("seat:mesh-controller."+verb, map[string]any{}, "")
if err != nil {
return "", err
}
var r struct {
Output string `json:"output"`
OK *bool `json:"ok"`
}
if json.Unmarshal(got.Result, &r) != nil {
return "", fmt.Errorf("mesh-controller.%s answered something that is not its output", verb)
}
if r.OK != nil && !*r.OK {
return "", fmt.Errorf("mesh-controller.%s: %s", verb, strings.TrimSpace(r.Output))
}
return r.Output, nil
}
// jsonIn is the JSON document a command printed, after any lines it said first: a seat verb runs
// the controller's command, and a command may warn before it answers.
func jsonIn(output string) string {
t := strings.TrimSpace(output)
if strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") {
return t
}
for _, open := range []string{"\n{", "\n["} {
if i := strings.Index(output, open); i >= 0 {
return strings.TrimSpace(output[i+1:])
}
}
return ""
}
// recordedModule is a module as the controller's records hold it (`module list --json`): where the
// mesh assigned it, and whether it declares tools — what should announce itself, and where.
// recordedModule is a module as the bus holds it: where the assigner seat places it, and whether its manifest declares
// tools — what should announce itself, and where (novox/hq ADR 0571).
type recordedModule struct {
Module string `json:"module"`
On []string `json:"on"`
@@ -229,42 +256,68 @@ type recordedModule struct {
Replaces map[string][]string `json:"replaces,omitempty"`
}
// recordedMachine is a machine as the controller's records hold it (`node list --json`).
type recordedMachine struct {
Name string `json:"name"`
}
// indexOn asks the mesh what it holds (novox/hq ADR 0197): what answers, from every runtime's own
// announcement on the bus — one `$SRV.INFO` request — and what should, from the controller's records
// announcement on the bus — one `$SRV.INFO` request — and what should, from the indexer's records
// read as JSON. Nothing is inferred from a roster and nothing is parsed from print.
func indexOn(conn *bus.Conn) (*index, error) {
//
// `held` is what the console holds already, so an instance whose announcement is unchanged is not
// asked for it again; `before` is the last answer of every runtime this round did not hear, which the
// index holds instead, marked with its age (novox/hq issue 554).
func indexOn(conn *bus.Conn, src Source, held announce.Known, before func(announce.Discovery) []earlier) (*index, error) {
var wg sync.WaitGroup
var nodesOut, modulesOut string
var toolsAnswer json.RawMessage
wg.Add(3)
go func() { defer wg.Done(); nodesOut, _ = controllerOutput(conn, "nodes") }()
go func() { defer wg.Done(); modulesOut, _ = controllerOutput(conn, "modules") }()
// The seats' verbs as the records define them, for what each replaces (novox/hq ADR 0245): answered
// in-process from the records, never a command.
go func() {
defer wg.Done()
if got, err := conn.Ask("seat:mesh-controller.tools", map[string]any{}, ""); err == nil {
toolsAnswer = got.Result
}
}()
d, err := announce.Gather(conn)
var placed map[string][]byte
var placedErr error
var recorded []recordedModule
var known []string
var recordsWhy []string
wg.Add(2)
// Which modules there are and where they are placed, and which machines the mesh has, from the bus (ADR 0571
// clauses 4 and 5): the indexer's modules and manifests and the assigner seat's assignments, never any other record.
go func() { defer wg.Done(); recorded, known, recordsWhy = recordsOn(src) }()
// Who holds which seat on which node, from the assigner (ADR 0571 clause 4): no other call.
go func() { defer wg.Done(); placed, placedErr = seatHoldersOn(src) }()
d, err := announce.Gather(conn, held)
wg.Wait()
if err != nil {
return nil, err
}
var endpoints []announce.Endpoint
fresh := map[string]bool{}
for _, h := range d.Heard {
endpoints = append(endpoints, announce.Endpoints(h.Info)...)
for _, e := range announce.Endpoints(h.Info) {
fresh[endpointKey(e)] = true
endpoints = append(endpoints, e)
}
}
// **A runtime not heard this round is not gone** (novox/hq issue 554): a round in which no answer
// crossed the laptop's link in time replaced a full index with an empty one, and every seat and
// module call was refused. What it announced last, within RuntimeForgotten, stands — said with its age.
stale := map[string]staleFrom{}
earlierAt := map[string]time.Time{}
if before != nil {
for _, b := range before(d) {
earlierAt[runtimeKey(b.Heard.Info.Name, b.Heard.Info.ID)] = b.At
for _, e := range announce.Endpoints(b.Heard.Info) {
k := endpointKey(e)
if fresh[k] {
continue
}
if _, have := stale[k]; !have {
stale[k] = staleFrom{Runtime: b.Heard.Info.Name, Machine: b.Heard.Machine(), At: b.At}
}
endpoints = append(endpoints, e)
}
}
}
x, announced, machines := indexOfEndpoints(endpoints)
l := x.Listing
x.Discovery = d
x.Stale, x.Earlier = stale, earlierAt
x.Placed = placed
if placedErr != nil {
x.PlacedWhy = placedErr.Error()
}
for _, s := range d.Silent {
x.Unheard = append(x.Unheard, unheard{Runtime: s.Name, Machine: s.Machine,
Why: fmt.Sprintf("it answered PING; what it serves did not arrive within %s", announce.Patience)})
@@ -272,25 +325,24 @@ func indexOn(conn *bus.Conn) (*index, error) {
// What should have answered: every assignment of a module that declares tools. Silence is named;
// a module with no tools is never a name here.
var recorded []recordedModule
recordedOK := json.Unmarshal([]byte(jsonIn(modulesOut)), &recorded) == nil
x.Replaces = replacesOf(toolsAnswer, recorded)
if recordedOK {
for _, m := range recorded {
if !m.Tools {
continue
}
x.Recorded[m.Module] = append([]string{}, m.On...)
for _, n := range m.On {
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
// not a machine's — so what it announced without one answers for wherever it is assigned.
if !announced[m.Module][n] && !announced[m.Module][""] {
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
}
x.Replaces = replacesOf(catalogue.Seats(), recorded)
for _, m := range recorded {
if !m.Tools {
continue
}
x.Recorded[m.Module] = append([]string{}, m.On...)
for _, n := range m.On {
// A holder of a seat held once for the mesh announces no machine — the seat is the mesh's,
// not a machine's — so what it announced without one answers for wherever it is assigned.
if !announced[m.Module][n] && !announced[m.Module][""] {
l.NotAnswering = append(l.NotAnswering, m.Module+" on "+n)
}
}
} else {
l.NotAnswering = append(l.NotAnswering, "mesh-controller (its records of the modules did not answer, so what is missing cannot be said)")
}
l.NotAnswering = append(l.NotAnswering, recordsWhy...)
if x.PlacedWhy != "" {
l.NotAnswering = append(l.NotAnswering, firstview.SeatHoldersRead+" (who holds which seat by assignment could not be read: "+
x.PlacedWhy+")")
}
for _, u := range x.Unheard {
l.NotAnswering = append(l.NotAnswering, "the runtime "+sayUnheard([]unheard{u}))
@@ -298,13 +350,8 @@ func indexOn(conn *bus.Conn) (*index, error) {
sort.Strings(l.NotAnswering)
x.NotAnswering = l.NotAnswering
var known []recordedMachine
if json.Unmarshal([]byte(jsonIn(nodesOut)), &known) == nil {
for _, n := range known {
if n.Name != "" {
machines[n.Name] = true
}
}
for _, n := range known {
machines[n] = true
}
for n := range machines {
x.Machines = append(x.Machines, n)
@@ -313,6 +360,107 @@ func indexOn(conn *bus.Conn) (*index, error) {
return x, nil
}
// Source is where the console reads what it indexes the mesh by: the assigner's assignments and seat-holders, by
// `<owner>.<state>` (firstview.AssignmentsRead and SeatHoldersRead), and the modules of the catalog half with their entries.
// The tool runner's is its first views (firstview.Views), never the owners' state (novox/hq ADR 0615 §1; design 47 G §196, §594).
type Source interface {
// State is every current value of a state by key; an error says why it cannot be told yet.
State(name string) (map[string][]byte, error)
// Modules is every module the catalog half holds, by name, with its entry; an error says why it cannot be told yet.
Modules() (map[string]meshcatalog.CatalogEntry, error)
}
// seatHoldersOn reads the state seat-holders whole (novox/hq ADR 0571 clause 4), as firstview.SeatHoldersRead.
func seatHoldersOn(src Source) (map[string][]byte, error) {
return src.State(firstview.SeatHoldersRead)
}
// manifestFacts is what the console takes of a manifest: the tools it declares and what each replaces.
type manifestFacts struct {
Tools []string `json:"tools"`
Replaces map[string][]string `json:"replaces"`
}
// recordsOn is every module the indexer holds, with where the assigner seat places it and whether it declares tools,
// and every machine the assignments name (moduleNames, assigner.MachineNames). What cannot be read is said, one line
// per state, for the listing's not answering.
func recordsOn(src Source) (recorded []recordedModule, machines []string, why []string) {
held, err := src.State(firstview.AssignmentsRead)
if err != nil {
why = append(why, firstview.AssignmentsRead+" (where modules are placed could not be read, so what is missing cannot "+
"be said: "+err.Error()+")")
}
machines = assigner.MachineNames(held)
on := map[string][]string{}
// The assignments' keys in either form, a node key and its module keys or the form before (research 072 row 37).
assembled, err := assigner.AssignmentsOf(held)
if err != nil {
why = append(why, firstview.AssignmentsRead+" (not every key is an assignment: "+err.Error()+")")
}
for _, node := range machines {
for _, m := range assembled[node].Modules {
on[m.Module] = append(on[m.Module], node)
}
}
entries, err := src.Modules()
if err != nil {
return nil, machines, append(why, "the catalog half's modules (which modules there are could not be read: "+err.Error()+")")
}
names := make([]string, 0, len(entries))
for name := range entries {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
r := recordedModule{Module: name, On: on[name]}
if m := entries[name].Manifest; len(m) > 0 {
facts, err := manifestOf(m)
if err != nil {
why = append(why, name+" (its manifest could not be read, so whether it answers tools is not known: "+err.Error()+")")
} else {
r.Tools, r.Replaces = len(facts.Tools) > 0, facts.Replaces
}
}
recorded = append(recorded, r)
}
return recorded, machines, why
}
// manifestOf is what the console takes of the manifest an entry of the catalog half carries whole (design 47 G §443).
func manifestOf(manifest map[string]any) (manifestFacts, error) {
raw, err := json.Marshal(manifest)
if err != nil {
return manifestFacts{}, err
}
var facts manifestFacts
if err := json.Unmarshal(raw, &facts); err != nil {
return manifestFacts{}, err
}
return facts, nil
}
// placedHolders is who holds seat (on node, for a seat held per machine) by assignment, or none.
func (x *index) placedHolders(seat, node string) []meshmoduleassigner.SeatHoldersHoldersItem {
if len(x.Placed) == 0 {
return nil
}
hs, err := assigner.HoldersOf(x.Placed, seat, node)
if err != nil || len(hs) > 0 || node != "" {
return hs
}
// A seat held per machine, asked without one: every node's key of it.
var all []meshmoduleassigner.SeatHoldersHoldersItem
for key := range x.Placed {
if s, n, ok := strings.Cut(key, "."); ok && s == seat {
if more, err := assigner.HoldersOf(x.Placed, seat, n); err == nil {
all = append(all, more...)
}
}
}
sort.Slice(all, func(i, j int) bool { return all[i].Node+"/"+all[i].Module < all[j].Node+"/"+all[j].Module })
return all
}
func containsHolder(hs []holder, h holder) bool {
for _, x := range hs {
if x == h {
@@ -325,6 +473,10 @@ func containsHolder(hs []holder, h holder) bool {
func (s *Surface) index() (*index, error) { return s.indexAsked(false) }
// indexAsked is what the mesh answered, asked again when `fresh` or when the kept answer is old.
//
// **A round that heard nothing is not kept** (novox/hq issue 554): the index it gives holds what every
// runtime announced last, marked with its age, and the next question asks again rather than living
// with it for IndexKept.
func (s *Surface) indexAsked(fresh bool) (*index, error) {
s.mu.Lock()
if !fresh && s.idx != nil && time.Since(s.idxAt) <= IndexKept {
@@ -333,13 +485,17 @@ func (s *Surface) indexAsked(fresh bool) (*index, error) {
return x, nil
}
s.mu.Unlock()
x, err := indexOn(s.conn)
x, err := indexOn(s.conn, s.src, s.held, s.before)
if err != nil {
return nil, err
}
s.mu.Lock()
s.remember(x)
s.idx, s.idxAt = x, time.Now()
s.sayRound(x)
s.idx = x
if !x.Discovery.Empty() {
s.idxAt = time.Now()
}
s.mu.Unlock()
return x, nil
}
@@ -450,6 +606,26 @@ type target struct {
Node string
Tool Tool
Seat bool
// Stale is set when the target is known only from an earlier round of discovery (novox/hq issue 554).
Stale *staleFrom
}
// staleOf is where a target was heard, when every place it answers is known only from an earlier round.
func (x *index) staleOf(keys []string) *staleFrom {
if len(keys) == 0 || len(x.Stale) == 0 {
return nil
}
var first *staleFrom
for _, k := range keys {
f, ok := x.Stale[k]
if !ok {
return nil
}
if first == nil {
first = &f
}
}
return first
}
// resolve turns an address into exactly one target, or says why it cannot.
@@ -487,12 +663,17 @@ func resolve(x *index, address string) (target, error) {
return target{}, fmt.Errorf("%s is held once per machine: write <node>/%s — it is held on %s",
prefix, rest, orNobody(nodesOf(s.Holders)))
}
return target{Address: node + "/" + rest, Key: "seat:" + rest + "@" + node, Name: rest, Node: node, Tool: *verb, Seat: true}, nil
return target{Address: node + "/" + rest, Key: "seat:" + rest + "@" + node, Name: rest, Node: node, Tool: *verb, Seat: true,
Stale: x.staleOf([]string{"seat:" + prefix + "@" + node})}, nil
}
if node != "" {
return target{}, fmt.Errorf("%s is held once for the whole mesh: write %s, without a machine", prefix, rest)
}
return target{Address: rest, Key: "seat:" + rest, Name: rest, Tool: *verb, Seat: true}, nil
var holders []string
for _, h := range s.Holders {
holders = append(holders, "seat:"+prefix+"@"+h.Node)
}
return target{Address: rest, Key: "seat:" + rest, Name: rest, Tool: *verb, Seat: true, Stale: x.staleOf(holders)}, nil
}
m := x.Modules[prefix]
@@ -505,10 +686,23 @@ func resolve(x *index, address string) (target, error) {
return target{}, fmt.Errorf("%s is assigned to %s in the mesh's records, but nothing that answered discovery serves it: %s. "+
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(on, ", "), why)
}
if hs := x.placedHolders(prefix, node); len(hs) > 0 {
var held []string
for _, h := range hs {
held = append(held, h.Module+" on "+h.Node)
}
return target{}, fmt.Errorf("the seat %s is held by %s by assignment, but no holder of it announced itself on the bus. "+
"Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, strings.Join(held, ", "))
}
if len(x.Unheard) > 0 {
return target{}, fmt.Errorf("nothing that answered discovery is called %s, but not every runtime answered: %s — "+
"%s may be theirs. Ask again in a moment; mesh_runtimes shows which runtimes answered", prefix, sayUnheard(x.Unheard), prefix)
}
if x.Discovery.Empty() {
return target{}, fmt.Errorf("nothing answered discovery at all within %s, not even PING, and this console holds no "+
"earlier answer: whether anything is called %s is not known. Ask again in a moment; mesh_runtimes shows which "+
"runtimes answered", x.Discovery.Waited.Round(time.Millisecond), prefix)
}
return target{}, fmt.Errorf("nothing in the mesh is called %s: no seat, and no module that answers tools. "+
"mesh_search finds a tool by words", prefix)
}
@@ -521,7 +715,11 @@ func resolve(x *index, address string) (target, error) {
return target{}, fmt.Errorf("%s keeps state on each machine it runs on, so a call names the machine: "+
"write <node>/%s — it runs on %s", prefix, rest, orNobody(m.On))
}
return target{Address: rest, Key: rest, Name: rest, Tool: *tool}, nil
var on []string
for _, n := range m.On {
on = append(on, "mod:"+prefix+"@"+n)
}
return target{Address: rest, Key: rest, Name: rest, Tool: *tool, Stale: x.staleOf(on)}, nil
}
if len(m.On) > 0 && !contains(m.On, node) {
if us := x.unheardOn(node); len(us) > 0 || contains(x.Recorded[prefix], node) {
@@ -534,7 +732,8 @@ func resolve(x *index, address string) (target, error) {
}
return target{}, fmt.Errorf("%s does not run on %s; it runs on %s", prefix, node, orNobody(m.On))
}
return target{Address: node + "/" + rest, Key: rest + "@" + node, Name: rest, Node: node, Tool: *tool}, nil
return target{Address: node + "/" + rest, Key: rest + "@" + node, Name: rest, Node: node, Tool: *tool,
Stale: x.staleOf([]string{"mod:" + prefix + "@" + node})}, nil
}
// unheardOnAny is what is unheard on any of these machines.
@@ -600,7 +799,7 @@ func firstLine(s string) string {
// the machine, `node` is taken out: the machine is in the address. Where it does not — a seat held
// once for the mesh, or an interchangeable module — `node` is the tool's own argument, and is kept.
//
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the controller's
// **Taking it out everywhere made the mesh's own verbs lie** (novox/hq issue 244): the old controller verbs
// `push`, `plan`, `assign`, `pin` and `settings` take the machine they act on as `node`, and were
// described without it — `push` as taking nothing at all, so a push naming one machine arrived as a
// push of every machine behind.
@@ -633,6 +832,11 @@ func argumentsAsSent(t target, given map[string]any) (map[string]any, error) {
if !t.Seat {
return args, nil
}
t, held := seatDefinitionHeld(t)
if !held {
// Nothing here says what the verb takes: its holder judges the arguments (issue 1966).
return args, nil
}
declared := map[string]bool{}
var names []string
if p, ok := schemaAsPassed(t)["properties"].(map[string]any); ok {
@@ -660,6 +864,36 @@ func argumentsAsSent(t target, given map[string]any) (map[string]any, error) {
return args, nil
}
// seatDefinitionHeld is a seat verb's target with the input the console judges a call by, and whether anything here
// says what the verb takes (issue 1966): the schema its holder announced when it declares an argument, else the verb's
// input in the seat's definition this runner holds. Neither holds while the runner waits for the seat list holding
// only the bootstrapped definitions and the holder announced an empty schema: then a call's arguments are passed on to
// the holder, which judges its own input, and mesh_describe says the definition is not held.
func seatDefinitionHeld(t target) (target, bool) {
if p, ok := asSchema(t.Tool.Input)["properties"].(map[string]any); ok && len(p) > 0 {
return t, true
}
seat, ok := catalogue.SeatNamed(t.Tool.Module)
if !ok {
return t, false
}
for _, v := range seat.Serves {
if v.Name == t.Tool.Name && v.Input != nil {
if raw, err := json.Marshal(v.Input); err == nil {
t.Tool.Input = raw
}
}
}
return t, true
}
// notHeld is what mesh_describe says of a seat verb whose definition this runner does not hold.
func notHeld(seat string) string {
return "the seat list this tool runner holds has no definition of " + seat + " yet (it holds the bootstrapped ones " +
"until it reads the list): what the verb takes is not known here, so the arguments of a call are passed to " +
"its holder, which judges them"
}
// schemaWithoutNode is a tool's schema with `node` taken out, for an address that names the machine.
func schemaWithoutNode(raw json.RawMessage) map[string]any {
schema := asSchema(raw)
@@ -693,6 +927,21 @@ func schemaWithoutNode(raw json.RawMessage) map[string]any {
}
// answerText is a discovery verb's answer as MCP content: JSON, indented.
// callFailed is why a call failed, as mesh_call says it: a marked tool or verb that reaches no responder names `withheld`
// and the grants part's read (design 46 7-approval-mark §6), and a target known only from an earlier round says so.
func (s *Surface) callFailed(t target, err error) string {
why := whyItFailed(t.Address, err)
if ok, _ := s.markedTool(t); ok && errors.Is(err, bus.ErrNoResponders) {
// A marked subject withheld from a machine whose tool runner does not honour the mark (§6).
why += "; it needs the operator's approval, so it may be withheld where the machine's tool runner does not state " +
"approval-mark/1: mesh-grants.grants_user_list names it under withheld"
}
if t.Stale != nil {
why += " (" + t.Stale.say() + ")"
}
return why
}
func answerText(v any) map[string]any {
var b strings.Builder
enc := json.NewEncoder(&b)
@@ -707,10 +956,19 @@ func failure(text string) map[string]any {
}
// discover answers one of the discovery verbs.
func (s *Surface) discover(name string, args map[string]any) map[string]any {
func (s *Surface) discover(ctx context.Context, name string, args map[string]any) map[string]any {
if name == verbRuntimes {
return s.runtimesAnswer()
}
if name == verbFirstViews || name == verbFirstAgain {
return s.firstViewsAnswer(ctx, name, args)
}
if name == verbWait {
return s.waitAnswer(ctx, args)
}
if name == verbApprovals {
return s.approvalsAnswer(args)
}
x, err := s.index()
if err != nil {
return failure("the mesh's discovery failed: " + err.Error())
@@ -853,7 +1111,14 @@ func (s *Surface) discover(name string, args map[string]any) map[string]any {
if description == "" {
description = t.Name
}
held := true
if t.Seat {
t, held = seatDefinitionHeld(t)
}
described := map[string]any{"address": t.Address, "description": description, "arguments": schemaAsPassed(t)}
if !held {
described["arguments"] = notHeld(t.Tool.Module)
}
key := t.Tool.Module + "." + t.Tool.Name
if t.Seat {
key = "seat:" + key
@@ -861,6 +1126,13 @@ func (s *Surface) discover(name string, args map[string]any) map[string]any {
if r := x.Replaces[key]; len(r) > 0 {
described["instead_of"] = r
}
if t.Stale != nil {
described["found"] = t.Stale.say()
}
if ok, source := s.markedTool(t); ok {
described["needs-approval"] = map[string]any{"source": source,
"says": "the first call asks the operator and runs nothing; the same call again within 30 minutes of the Approve runs once"}
}
return answerText(described)
case verbCall:
@@ -880,12 +1152,19 @@ func (s *Surface) discover(name string, args map[string]any) map[string]any {
got, err = callTool(s.conn, t.Key, callArgs, nil, x.Listing)
}
if err != nil {
return failure(whyItFailed(t.Address, err))
return failure(s.callFailed(t, err))
}
result, unread := resolveSent(s.conn, got.Result)
content := []map[string]any{{"type": "text", "text": pretty(result)}}
if unread != "" {
content = append(content, map[string]any{"type": "text", "text": unread})
}
content := []map[string]any{{"type": "text", "text": pretty(got.Result)}}
if got.Node != "" {
content = append(content, map[string]any{"type": "text", "text": "answered by " + got.Node})
}
if t.Stale != nil {
content = append(content, map[string]any{"type": "text", "text": t.Stale.say()})
}
return map[string]any{"content": content}
}
return failure("no discovery verb " + name)
+38 -47
View File
@@ -36,9 +36,10 @@ func call(t *testing.T, endpoint, tool string, args map[string]any) (string, boo
return text(t, post(t, endpoint, string(body)))
}
// novox/hq ADR 0195: the console announces six tools, and everything the mesh answers is reached
// novox/hq ADR 0195 and ADR 0367: the console announces seven tools, and everything the mesh answers is reached
// through them by one address per layer.
func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
mt.NeedBundles(t)
was := IndexKept
IndexKept = 0 // every discovery asks the mesh, so a module arriving mid-test is found
t.Cleanup(func() { IndexKept = was })
@@ -69,35 +70,34 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Cleanup(stop)
}
// The controller: its records as JSON, as its seat verbs answer them, and its own seat's verbs
// announced on the bus like every runtime's.
controller := connect(t, "mesh-controller", "bench")
out := func(s string) map[string]any { return map[string]any{"output": s, "ok": true} }
serve := func(verb string, answer func() any) {
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
return answer(), nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
// The controller: its own seat's verbs announced on the bus like every runtime's; its records are asked by no one.
controller := connect(t, "mesh-module-assigner", "bench")
// Its verb nodes, called through the console as any seat's verb is.
stopNodes, err := controller.HandleSubject("mesh.seat.mesh-module-assigner.tool.nodes", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"name":"bench","mode":"converged"},{"name":"desk","mode":"converged"}]`}, nil
})
if err != nil {
t.Fatal(err)
}
serve("nodes", func() any {
return out("the bus's user list leaves out 2 user(s)\n" +
`[{"name":"bench","heard":"3m ago","mode":"converged","id":"1f2e"},{"name":"desk","heard":"here","mode":"converged","id":"9a8b"}]`)
t.Cleanup(stopNodes)
// What the mesh holds, as the indexer and the assigner write it (ADR 0571 clauses 4, 5): never the controller's records.
tools := map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "gamma": {"given"}, "delta": nil,
"epsilon": {"seven"}, "mesh-module-assigner": {"nodes"}}
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta"}, "bench": {"epsilon", "mesh-module-assigner"}})
console := mt.MembershipOf("node-tools", "desk", false, nil)
console.State = mt.ConsoleReads()
mesh.Issue(t, console)
mt.Until(t, func() error {
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
return fmt.Errorf("the console's reads are not issued yet")
}
return nil
})
var gammaOn atomic.Value
gammaOn.Store("[]")
serve("modules", func() any {
return out(fmt.Sprintf(`[{"module":"alpha","on":["desk"],"tools":true},{"module":"beta","on":["desk"],"tools":true},`+
`{"module":"gamma","on":%s,"tools":true},{"module":"delta","on":["desk"],"tools":false},`+
`{"module":"epsilon","on":["bench"],"tools":true},{"module":"mesh-controller","on":["bench"],"tools":true}]`, gammaOn.Load().(string)))
})
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-controller", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller", Scope: "mesh",
stopAnn, err := announce.Serve(controller, announce.Service{Name: "mesh-module-assigner", ID: "bench"}, func() []announce.Endpoint {
return []announce.Endpoint{{Kind: announce.KindSeat, Module: "mesh-module-assigner", Seat: "mesh-module-assigner", Scope: "mesh",
// As the live controller announces: a mesh seat's holder names no machine.
Tool: "nodes", Node: "", Description: "Every machine the mesh knows.", Schema: json.RawMessage(`{}`),
Subject: "mesh.seat.mesh-controller.tool.nodes"}}
Subject: "mesh.seat.mesh-module-assigner.tool.nodes"}}
})
if err != nil {
t.Fatal(err)
@@ -106,14 +106,14 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
controller.Flush()
watcher.Flush()
up, err := Serve(NewSurface(nodeTools, "desk.node-tools"), "127.0.0.1:0")
up, err := Serve(NewSurface(nodeTools, busSource{nodeTools}, "desk.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
// Six tools, nothing else.
// Seven tools, nothing else (ADR 0367).
listed := post(t, endpoint, `{"jsonrpc":"2.0","id":2,"method":"tools/list"}`)["result"].(map[string]any)
var names []string
for _, x := range listed["tools"].([]any) {
@@ -125,19 +125,19 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
}
}
}
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes" {
if got := strings.Join(names, ","); got != "mesh_overview,mesh_machine,mesh_search,mesh_describe,mesh_call,mesh_runtimes,mesh_first_views,mesh_first_view_again,mesh_wait" {
t.Errorf("announced %s", got)
}
// The overview names the mesh's seats, the machines' seats and the machines.
overview, isErr := call(t, endpoint, "mesh_overview", nil)
if isErr || !strings.Contains(overview, "mesh-controller.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
if isErr || !strings.Contains(overview, "mesh-module-assigner.nodes") || !strings.Contains(overview, "<node>/node-shelf.list") ||
!strings.Contains(overview, `"bench"`) || !strings.Contains(overview, `"desk"`) {
t.Errorf("overview: %s", overview)
}
// Silence is named only where tools should have answered: epsilon declares tools on bench and
// nothing there announced it; delta declares none and is never a name.
if strings.Contains(overview, "mesh-controller on bench") {
if strings.Contains(overview, "mesh-module-assigner on bench") {
t.Errorf("a mesh seat's holder that announced no machine is called silent on its machine:\n%s", overview)
}
if !strings.Contains(overview, "epsilon on bench") || strings.Contains(overview, "delta") {
@@ -150,7 +150,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
}
// A mesh seat, a node seat, an assignment and an interchangeable module, each by address.
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-controller.nodes"}); isErr || !strings.Contains(got, "converged") {
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "mesh-module-assigner.nodes"}); isErr || !strings.Contains(got, "converged") {
t.Errorf("mesh seat: %s", got)
}
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "desk/node-shelf.list"}); isErr || !strings.Contains(got, `"a"`) {
@@ -166,11 +166,11 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
// Refused, by name, where the address does not say enough or says the wrong thing.
for address, want := range map[string]string{
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-controller.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
"beta.three": "keeps state on each machine it runs on, so a call names the machine: write <node>/beta.three — it runs on desk",
"node-shelf.list": "held once per machine: write <node>/node-shelf.list — it is held on desk",
"desk/mesh-module-assigner.nodes": "held once for the whole mesh",
"bench/beta.three": "beta does not run on bench; it runs on desk",
"nonsense": "is not an address",
} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if !isErr || !strings.Contains(got, want) {
@@ -201,7 +201,7 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Fatalf("gamma was found before it served: %s", got)
}
mesh.Issue(t, mt.MembershipOf("gamma", "desk", false, nil))
gammaOn.Store(`["desk"]`)
mesh.Placed(t, tools, map[string][]string{"desk": {"alpha", "beta", "delta", "gamma"}, "bench": {"epsilon", "mesh-module-assigner"}})
late := connect(t, "node-tools", "desk")
stopLate, err := runtime.Run(late, []runtime.Served{{Module: "gamma", Entrypoints: []string{mt.Fixture("env-gamma.serve.mjs")}}},
nil, (&mt.Logs{}).Logf)
@@ -232,12 +232,3 @@ func TestTheMeshsToolsAreFoundByAddress(t *testing.T) {
t.Errorf("an old name: %s", got)
}
}
func TestTheControllersRecordsAreReadAsJSON(t *testing.T) {
if got := jsonIn("a warning printed first\n[{\"name\":\"ace\"}]"); got != `[{"name":"ace"}]` {
t.Errorf("an array after a warning: %q", got)
}
if got := jsonIn(`{"seats":[]}`); got != `{"seats":[]}` {
t.Errorf("an object: %q", got)
}
}
+99
View File
@@ -0,0 +1,99 @@
package console
import (
"log"
"strings"
"git.novox.be/novox/mesh-sdk/go/approval"
)
// verbApprovals is the mesh MCP server's tool that reads and steers what the gate asked on this machine (design 46
// 7-approval-mark §18).
const verbApprovals = "mesh_approvals"
// Approvals is the gate of this machine's tool runner, as mesh_approvals and mesh_describe read it.
type Approvals interface {
List() ([]approval.Listed, error)
Read(id string) (approval.Whole, error)
Applied() []approval.Marks
Describe(module, seat, name string) (bool, string)
Again(id string) error
Cancel(id string) error
// Fill is the state approvals' bytes against its cap, every machine's keys together.
Fill() (bytes uint64, max int64, err error)
}
func approvalsTool() map[string]any {
str := func(desc string) map[string]any { return map[string]any{"type": "string", "description": desc} }
return map[string]any{"name": verbApprovals, "inputSchema": map[string]any{"type": "object", "properties": map[string]any{
"ask": str("an ask's id: its whole arguments and the act's digest"),
"record": str("an ask's id: ask mesh-hand-acts.warranted again for its run, recorded in the hand-act log first"),
"cancel": str("an ask's id: take that open ask back, recorded in the hand-act log first"),
"why": str("why, for record and cancel, kept in the hand-act log"),
}}, "description": "The calls that needed the operator's approval on this machine: with no argument, those asked in " +
"the last day, newest first, with their state and fingerprint, the marks the tool runner applies (each module's " +
"marked tools and verbs with their source and the catalog revision read) and its state's fill; `ask` reads one " +
"whole, `record` asks its hand-act again, `cancel` takes an open ask back."}
}
// approvalsAnswer is mesh_approvals.
func (s *Surface) approvalsAnswer(args map[string]any) map[string]any {
if s.Approvals == nil {
return failure("this tool runner has no approval gate")
}
str := func(k string) string { v, _ := args[k].(string); return strings.TrimSpace(v) }
why := str("why")
if why == "" {
why = "asked by hand with no reason given"
}
act := func(id, what, cause string, do func(string) error) map[string]any {
// A hand act, recorded through the seat mesh-hand-acts before it is done (§18); one that cannot be recorded is refused.
log.Printf("[mesh-tools] hand act: %s %s ask %s: %s", s.who, what, id, why)
if err := s.recordAct(map[string]any{"what": what + " ask " + id, "why": why, "cause": cause, "verb": verbApprovals}); err != nil {
return failure("the hand act was not recorded through mesh-hand-acts, so nothing was done: " + err.Error())
}
if err := do(id); err != nil {
return failure(err.Error())
}
return answerText(map[string]any{"done": what + " ask " + id, "recorded": "mesh-hand-acts.hand-act"})
}
switch {
case str("record") != "":
return act(str("record"), "asked the hand-act of", "approval-record", s.Approvals.Again)
case str("cancel") != "":
return act(str("cancel"), "cancelled", "approval-cancel", s.Approvals.Cancel)
case str("ask") != "":
w, err := s.Approvals.Read(str("ask"))
if err != nil {
return failure(err.Error())
}
return answerText(w)
}
asks, err := s.Approvals.List()
out := map[string]any{"asks": asks, "marks": s.Approvals.Applied()}
if err != nil {
out["asks"], out["asks_why"] = []approval.Listed{}, "the state approvals cannot be read: "+err.Error()
}
if b, max, err := s.Approvals.Fill(); err != nil {
out["fill_why"] = err.Error()
} else {
out["fill"] = map[string]any{"bytes": b, "max_bytes": max, "of": "every machine's keys together"}
}
return answerText(out)
}
// marked is what mesh_describe and a call that reached no one say of a marked tool or verb (§6, §18).
func (s *Surface) marked(module, seat, name string) (bool, string) {
if s.Approvals == nil {
return false, ""
}
return s.Approvals.Describe(module, seat, name)
}
// markedTool is marked for a resolved address.
func (s *Surface) markedTool(t target) (bool, string) {
if t.Seat {
return s.marked("", t.Tool.Module, t.Tool.Name)
}
return s.marked(t.Tool.Module, "", t.Tool.Name)
}
+6 -13
View File
@@ -7,6 +7,8 @@ import (
"regexp"
"strings"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
@@ -61,8 +63,8 @@ func toolKey(name string, seats Seats) string {
// toolsOn is the flat catalogue (MESH_CONSOLE_FLAT=1): what announced itself on the bus, every
// tool and seat verb, and the assignments with tools that did not (novox/hq ADR 0197).
func toolsOn(conn *bus.Conn) (*Listing, error) {
x, err := indexOn(conn)
func toolsOn(conn *bus.Conn, src Source) (*Listing, error) {
x, err := indexOn(conn, src, nil, nil)
if err != nil {
return nil, err
}
@@ -109,7 +111,6 @@ func subjectListed(name, node string, l *Listing) string {
}
var (
noResponders = regexp.MustCompile(`(?i)no responders|503`)
// The bus's own two refusals, by their whole phrase. "authorization" alone also matched a tool's own
// answer that merely mentions the word — the runtime refusing a state value with an Authorization
// header (novox/hq ADR 0201) read as "this account may not call", which sent the reader the wrong way.
@@ -117,9 +118,6 @@ var (
timedOut = regexp.MustCompile(`(?i)timeout`)
)
// controllerSeat is the seat whose holder keeps what came of every call (novox/hq issue 265).
const controllerSeat = "mesh-controller"
// whyItFailed says why a call failed, so the remedy is in the words.
func whyItFailed(key string, err error) string {
if err == nil {
@@ -127,7 +125,8 @@ func whyItFailed(key string, err error) string {
}
msg := err.Error()
switch {
case noResponders.MatchString(msg):
// By type, not text: a tool's own error that mentions 503 is the tool's answer (issue 1529).
case errors.Is(err, bus.ErrNoResponders) || errors.Is(err, nats.ErrNoResponders):
extra := ""
if strings.HasPrefix(key, "seat:") {
extra = ", or nothing holds that seat"
@@ -143,12 +142,6 @@ func whyItFailed(key string, err error) string {
said := key + " did not answer within " + bus.RequestTimeout.String() + ". That is not a failure: " +
"something is serving it, so it may still be running and may already have done what was " +
"asked — check before repeating a call that changes the mesh."
if strings.HasPrefix(strings.TrimPrefix(key, "seat:"), controllerSeat+".") {
// The controller answers every call within seconds, with an id while it runs, so silence
// from it is an answer lost on the way — which it keeps.
said += " The controller answers every call within seconds, so its answer was lost on the " +
"way; " + controllerSeat + ".calls lists its recent calls and what came of each."
}
return said
}
return key + " failed: " + msg
+16 -9
View File
@@ -3,6 +3,7 @@ package console
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
@@ -42,6 +43,7 @@ func post(t *testing.T, endpoint string, body string) map[string]any {
// As node-tools, the runtime serves the bundles and is the console on loopback: the listing is what
// the modules and the mesh's records answered, and a call reaches the module on the machine named.
func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
mt.NeedBundles(t)
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("alpha", "desk", true, nil))
mesh.Issue(t, mt.MembershipOf("beta", "desk", false, map[string][]string{"node-shelf": {"list", "clear"}}))
@@ -55,16 +57,21 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
}
defer stop()
// The controller's records (ADR 0197): ghost declares tools on desk and nothing announces it.
controller := connect(t, "mesh-controller", "")
stopSeat, _ := controller.HandleSubject("mesh.seat.mesh-controller.tool.modules", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true, "output": `[{"module":"alpha","on":["desk"],"tools":true},` +
`{"module":"beta","on":["desk"],"tools":true},{"module":"ghost","on":["desk"],"tools":true}]`}, nil
// What the mesh holds, as the indexer and the assigner write it (ADR 0571): ghost declares tools on desk and nothing
// announces it.
mesh.Placed(t, map[string][]string{"alpha": {"one", "two"}, "beta": {"three"}, "ghost": {"boo"}},
map[string][]string{"desk": {"alpha", "beta", "ghost"}})
console := mt.MembershipOf("node-tools", "desk", false, nil)
console.State = mt.ConsoleReads()
mesh.Issue(t, console)
mt.Until(t, func() error {
if m := nodeTools.Membership("node-tools"); m == nil || len(m.State) == 0 {
return fmt.Errorf("the console's reads are not issued yet")
}
return nil
})
defer stopSeat()
controller.Flush()
flat := NewSurface(nodeTools, "desk.node-tools")
flat := NewSurface(nodeTools, busSource{nodeTools}, "desk.node-tools")
flat.Flat = true // the whole catalogue, as before ADR 0195: still reachable, no longer announced
up, err := Serve(flat, "127.0.0.1:0")
if err != nil {
@@ -125,7 +132,7 @@ func TestTheConsoleListsAndCallsOverHTTP(t *testing.T) {
}
func TestTheConsoleListensOnLoopbackAndNowhereElse(t *testing.T) {
if _, err := Serve(NewSurface(nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
if _, err := Serve(NewSurface(nil, nil, "x"), "0.0.0.0:0"); err == nil || !strings.Contains(err.Error(), "loopback and nowhere else") {
t.Errorf("a non-loopback console was not refused: %v", err)
}
}
@@ -0,0 +1,151 @@
package console
import (
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// novox/hq issue 554, reproduced: the laptop's console heard every runtime, then a round in which
// nothing answered discovery in time — and every seat and module call was refused ("it answered
// discovery 2s ago, not now") for the five seconds that round was kept, again and again. Now a call
// is resolved against what the runtime announced last, says how old that is, and reaches the tool; the
// empty round is logged once, is not kept, and the next question asks again.
func TestACallAfterARoundThatHeardNothingUsesWhatWasHeardBefore(t *testing.T) {
keptWas, patienceWas := IndexKept, announce.Patience
IndexKept, announce.Patience = 0, 1200*time.Millisecond
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
mt.New(t)
// The tools answer on their own connection; discovery is answered on another, which goes quiet.
tools := connect(t, "node-tools", "laptop")
stopTool, err := tools.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
return map[string]any{"ok": true}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopTool)
stopSeat, err := tools.HandleSubject("mesh.seat.issue-tracker.tool.list", func(json.RawMessage) (any, error) {
return map[string]any{"issues": []int{554}}, nil
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopSeat)
tools.Flush()
endpoints := []announce.Endpoint{
{Kind: announce.KindTool, Module: "slack", Tool: "slack_check", Node: "laptop", Subject: "mesh.mod.slack.tool.slack_check.laptop"},
{Kind: announce.KindSeat, Module: "issue-tracker", Seat: "issue-tracker", Tool: "list", Scope: "mesh", Node: "laptop",
Schema: json.RawMessage(`{}`), Subject: "mesh.seat.issue-tracker.tool.list"},
}
announcing := connect(t, "node-tools", "laptop")
stopAnnounce, err := announce.Serve(announcing, announce.Service{Name: "node-tools", ID: "laptop",
Metadata: map[string]string{"node": "laptop"}}, func() []announce.Endpoint { return endpoints })
if err != nil {
t.Fatal(err)
}
announcing.Flush()
console := connect(t, "node-tools", "laptop-console")
var mu sync.Mutex
var logged []string
console.Logf = func(f string, a ...any) {
mu.Lock()
defer mu.Unlock()
logged = append(logged, fmt.Sprintf(f, a...))
}
said := func(words string) int {
mu.Lock()
defer mu.Unlock()
n := 0
for _, l := range logged {
if strings.Contains(l, words) {
n++
}
}
return n
}
surface := NewSurface(console, busSource{console}, "laptop.node-tools")
up, err := Serve(surface, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer up.Close()
endpoint := "http://" + up.Address + "/mcp"
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/slack.slack_check"}); isErr || strings.Contains(got, "announced") {
t.Fatalf("while discovery hears the runtime: %s", got)
}
// Discovery goes quiet: no PING arrives at all.
stopAnnounce()
announcing.Flush()
for _, address := range []string{"laptop/slack.slack_check", "issue-tracker.list"} {
got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": address})
if isErr || !strings.Contains(got, "found from what node-tools on laptop announced") ||
!strings.Contains(got, "ago: it did not answer the latest round of discovery") {
t.Errorf("%s after a round that heard nothing: %s", address, got)
}
}
if got, isErr := call(t, endpoint, "mesh_describe", map[string]any{"address": "issue-tracker.list"}); isErr || !strings.Contains(got, `"found"`) {
t.Errorf("describe after a round that heard nothing: %s", got)
}
if n := said("discovery heard nothing at all"); n != 1 {
t.Errorf("the empty round was logged %d time(s), want once: %v", n, logged)
}
// A runtime really gone: the call is sent on what it announced, and fails with the bus's own refusal,
// saying where the address was found.
stopTool()
tools.Flush()
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "laptop/slack.slack_check"}); !isErr ||
!strings.Contains(got, "nothing serves") || !strings.Contains(got, "found from what node-tools on laptop announced") {
t.Errorf("a gone runtime's tool: %s", got)
}
surface.mu.Lock()
kept := surface.idx != nil && !surface.idx.Discovery.Empty() && time.Since(surface.idxAt) < time.Second
surface.mu.Unlock()
if kept {
t.Error("a round that heard nothing was kept as the answer")
}
// Discovery answers again: the calls are plain again, and that is logged once.
stopAgain, err := announce.Serve(announcing, announce.Service{Name: "node-tools", ID: "laptop",
Metadata: map[string]string{"node": "laptop"}}, func() []announce.Endpoint { return endpoints })
if err != nil {
t.Fatal(err)
}
t.Cleanup(stopAgain)
announcing.Flush()
if got, isErr := call(t, endpoint, "mesh_call", map[string]any{"address": "issue-tracker.list"}); isErr || strings.Contains(got, "announced") {
t.Errorf("once discovery hears the runtime again: %s", got)
}
if n := said("discovery hears every runtime again"); n != 1 {
t.Errorf("the recovery was logged %d time(s): %v", n, logged)
}
}
// An address known only from an earlier answer says how old it is; one heard fresh on its machine is
// plain, even where the same module is known only from an earlier answer on another.
func TestAnAddressKnownOnlyFromAnEarlierAnswerSaysHowOld(t *testing.T) {
at := time.Now().Add(-20 * time.Second)
x := &index{Modules: map[string]*moduleInfo{"slack": {Module: "slack", On: []string{"laptop"},
Tools: []Tool{{Module: "slack", Name: "slack_check"}}}}, Recorded: map[string][]string{},
Stale: map[string]staleFrom{"mod:slack@laptop": {Runtime: "node-tools", Machine: "laptop", At: at}},
Discovery: announce.Discovery{Pinged: 0}}
got, err := resolve(x, "laptop/slack.slack_check")
if err != nil || got.Stale == nil || !strings.Contains(got.Stale.say(), "announced 20s ago") {
t.Fatalf("%+v %v", got, err)
}
// A module heard fresh on one machine and only earlier on another: the fresh one is plain.
x.Modules["slack"].On = []string{"desk", "laptop"}
if got, err := resolve(x, "desk/slack.slack_check"); err != nil || got.Stale != nil {
t.Errorf("a fresh machine's module marked stale: %+v %v", got, err)
}
}
+81
View File
@@ -0,0 +1,81 @@
package console
import (
"context"
"errors"
"log"
"strings"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
)
// FirstViewer is the tool runner's four first views as the console shows them (novox/hq ADR 0615 §6; issue 1966 TLS-6): a
// Source that also is one is asked by mesh_first_views and mesh_first_view_again.
type FirstViewer interface {
Status() []views.Status
Again(ctx context.Context, why string) []error
}
// entryLister is a FirstViewer that also keeps entries by fingerprint (design 47 G A-103): mesh_first_views lists them with their
// state and since when.
type entryLister interface{ Entries() any }
// ownBuilder is a FirstViewer that also says the build the tool runner itself was started with (design 47 G §205, §424):
// mesh_first_views names it, or says `own build unknown` and why.
type ownBuilder interface{ OwnBuild() process.Build }
// firstViewsAnswer is mesh_first_views and mesh_first_view_again.
func (s *Surface) firstViewsAnswer(ctx context.Context, name string, args map[string]any) map[string]any {
fv, ok := s.src.(FirstViewer)
if !ok {
return failure("this tool runner holds no first views: it reads the indexer's and the assigner's states through " +
"another source")
}
if name == verbFirstViews {
out := map[string]any{"first views": fv.Status()}
if ob, ok := fv.(ownBuilder); ok {
out["own build"] = ob.OwnBuild().String()
}
if el, ok := fv.(entryLister); ok {
out["entries"] = el.Entries()
}
return answerText(out)
}
why, _ := args["why"].(string)
if why = strings.TrimSpace(why); why == "" {
why = "asked by hand with no reason given"
}
// A hand act, recorded through the seat mesh-hand-acts (novox/hq ADR 0615 §6, ADR 0565 §10) before it is done, and in the
// runner's log. One that cannot be recorded is said, and the ask is not made.
log.Printf("[mesh-tools] hand act: %s asked the first views again: %s", s.who, why)
if err := s.recordAct(map[string]any{"what": "asked the tool runner's first views again", "why": why,
"cause": "first-view-again", "verb": verbFirstAgain}); err != nil {
return failure("the hand act was not recorded through mesh-hand-acts, so the first views were not asked again: " + err.Error())
}
out := map[string]any{"asked again": why, "recorded": "mesh-hand-acts.hand-act"}
if errs := fv.Again(ctx, why); len(errs) > 0 {
var msgs []string
for _, e := range errs {
msgs = append(msgs, e.Error())
}
out["not answered"] = msgs
}
out["first views"] = fv.Status()
return answerText(out)
}
// recordAct records one act done by hand through the seat mesh-hand-acts (its verb hand-act); Surface.Record replaces the call
// in a test.
func (s *Surface) recordAct(args map[string]any) error {
if s.Record != nil {
return s.Record(args)
}
if s.conn == nil {
return errNoBus
}
_, err := s.conn.Ask("mesh-hand-acts.hand-act", args, "")
return err
}
var errNoBus = errors.New("no bus connection")
@@ -0,0 +1,74 @@
package console
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
)
// viewer is a Source that also holds first views.
type viewer struct {
busSource
asked []string
build process.Build
}
func (v *viewer) Status() []views.Status {
return []views.Status{{Owner: "mesh-delivery-assigner", Tool: "assignments", State: "assignments", Revision: 4, Asks: 1}}
}
// OwnBuild is the build the viewer was started with.
func (v *viewer) OwnBuild() process.Build { return v.build }
// Entries are the entries by fingerprint the viewer keeps (design 47 G A-103).
func (v *viewer) Entries() any { return []string{"fp:a asked"} }
func (v *viewer) Again(_ context.Context, why string) []error {
v.asked = append(v.asked, why)
return nil
}
// issue 1966 TLS-6: the console's mesh_first_views shows each first view, mesh_first_view_again asks them again with its
// reason, and a source holding none refuses both in words.
func TestTheConsoleShowsAndAsksAgainTheRunnersFirstViews(t *testing.T) {
text := func(r map[string]any) string {
c, _ := json.Marshal(r["content"])
return string(c)
}
v := &viewer{}
var recorded []map[string]any
s := &Surface{src: v, who: "desk.node-tools", Record: func(a map[string]any) error { recorded = append(recorded, a); return nil }}
if got := text(s.discover(context.Background(), "mesh_first_views", nil)); !strings.Contains(got, "mesh-delivery-assigner") || !strings.Contains(got, "assignments") || !strings.Contains(got, "fp:a asked") {
t.Errorf("mesh_first_views shows %s", got)
}
// The tool runner's own build is named, or said unknown with the reason (design 47 G §205, §424).
for build, want := range map[string]string{"abcd1234": "abcd1234", "": "own build unknown"} {
v.build = process.ReadOwnBuild(func(string) string { return build })
if got := text(s.discover(context.Background(), "mesh_first_views", nil)); !strings.Contains(got, `own build`) || !strings.Contains(got, want) {
t.Errorf("mesh_first_views with the build %q shows %s", build, got)
}
}
got := s.discover(context.Background(), "mesh_first_view_again", map[string]any{"why": "doubted"})
if len(v.asked) != 1 || v.asked[0] != "doubted" || !strings.Contains(text(got), "asked again") {
t.Errorf("mesh_first_view_again asked %v and answered %s", v.asked, text(got))
}
if len(recorded) != 1 || recorded[0]["why"] != "doubted" || recorded[0]["cause"] != "first-view-again" {
t.Errorf("the hand act recorded is %v", recorded)
}
// An act that cannot be recorded is not done.
refusing := &Surface{src: v, who: "desk.node-tools", Record: func(map[string]any) error { return errors.New("no answer") }}
if r := refusing.discover(context.Background(), "mesh_first_view_again", map[string]any{"why": "x"}); r["isError"] != true || len(v.asked) != 1 {
t.Errorf("an unrecorded hand act was done: %s (asked %v)", text(r), v.asked)
}
for _, name := range []string{"mesh_first_views", "mesh_first_view_again"} {
none := &Surface{src: busSource{}, who: "desk.node-tools"}
if r := none.discover(context.Background(), name, nil); r["isError"] != true || !strings.Contains(text(r), "holds no first views") {
t.Errorf("%s without first views answers %s", name, text(r))
}
}
}
+23 -4
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"strconv"
"strings"
"time"
"github.com/novox/mesh-tools/node-tools/internal/wire"
)
@@ -45,12 +46,30 @@ func Serve(s *Surface, listen string) (*Listening, error) {
if err != nil {
return nil, err
}
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) })}
// Every request's context ends when the console is closed, so a mesh_wait in flight ends at once rather
// than holding a stop or a restart for up to its ten minutes (the review of issue 745).
base, cancel := context.WithCancel(context.Background())
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { route(w, r, s) }),
BaseContext: func(net.Listener) context.Context { return base }}
go func() { _ = server.Serve(ln) }()
bound := ln.Addr().(*net.TCPAddr).Port
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error { return server.Shutdown(context.Background()) }}, nil
return &Listening{Address: host + ":" + strconv.Itoa(bound), Close: func() error {
cancel()
// A backstop: what is still answering after the cancel is cut off, never waited on for long.
ctx, done := context.WithTimeout(context.Background(), ShutdownWithin)
defer done()
err := server.Shutdown(ctx)
if errors.Is(err, context.DeadlineExceeded) {
return server.Close()
}
return err
}}, nil
}
// ShutdownWithin is how long closing the console waits for the requests still being answered after their
// context was cancelled.
var ShutdownWithin = 5 * time.Second
func route(w http.ResponseWriter, r *http.Request, s *Surface) {
switch r.URL.Path {
case "/":
@@ -89,7 +108,7 @@ func route(w http.ResponseWriter, r *http.Request, s *Surface) {
}
replies := []*Reply{}
for _, req := range batch {
if reply := s.Handle(req); reply != nil {
if reply := s.HandleContext(r.Context(), req); reply != nil {
replies = append(replies, reply)
}
}
@@ -105,7 +124,7 @@ func route(w http.ResponseWriter, r *http.Request, s *Surface) {
writeJSON(w, 400, map[string]any{"jsonrpc": "2.0", "id": nil, "error": map[string]any{"code": -32700, "message": "the body is not JSON"}})
return
}
reply := s.Handle(req)
reply := s.HandleContext(r.Context(), req)
if reply == nil {
w.WriteHeader(202)
return
@@ -0,0 +1,97 @@
package console
import (
"encoding/json"
"os"
"reflect"
"strings"
"testing"
)
// The tool runner's manifest names no events to wait on (novox/hq ADR 0670 §2): a session waits on any event by its
// name. It holds each change of design 34 every-event §11 (E-1): no `waits`, `operator-channel` in `uses` for the asks of
// marked waits, the per-machine state `approvals` in the old `state` section with its 48 hours and none in `requires`
// (design 46 7-approval-mark §13), and the finding stream (design 47 G §387).
func TestTheManifestNamesNoWaitsAndKeepsItsApprovals(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
t.Fatal(err)
}
if _, ok := keys["waits"]; ok {
t.Error("the manifest still carries waits (novox/hq ADR 0670 §2)")
}
var m struct {
Module string `json:"module"`
Uses []string `json:"uses"`
State []json.RawMessage `json:"state"`
Requires []json.RawMessage `json:"requires"`
Consumes []string `json:"consumes"`
Streams []struct {
Name string `json:"name"`
Events []string `json:"events"`
} `json:"streams"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if m.Module != "node-tools" || !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) {
t.Errorf("%s uses %v, want operator-channel (design 46 7-approval-mark §7)", m.Module, m.Uses)
}
type state struct {
Name string `json:"name"`
PerMachine bool `json:"per-machine"`
TTLSeconds int `json:"ttl-seconds"`
}
var approvals []state
for _, r := range m.State {
var s state
if json.Unmarshal(r, &s) == nil && s.Name == "approvals" {
approvals = append(approvals, s)
}
}
if want := []state{{Name: "approvals", PerMachine: true, TTLSeconds: 172800}}; !reflect.DeepEqual(approvals, want) {
t.Errorf("the old state section's approvals is %+v, want %+v (design 46 7-approval-mark §13)", approvals, want)
}
for _, r := range m.Requires {
if strings.Contains(string(r), `"approvals"`) {
t.Errorf("approvals is asked in requires too: %s", r)
}
}
if len(m.Streams) != 1 || m.Streams[0].Name != "node-tools-findings" ||
!reflect.DeepEqual(m.Streams[0].Events, []string{"finding-written.*", "finding-cleared.*", "findings-stated.*"}) {
t.Errorf("the finding stream is %+v, want node-tools-findings with the three finding events (design 47 G §387)", m.Streams)
}
// A wait is no consumer (ADR 0367 §2): what it consumes is the assigner's two views families (ADR 0615 §5, issue 1966
// TLS-5), the events of the catalog half's branch follower (design 47 G §196, §614: `catalog-revision`,
// `registration-changed`, and the gate's `source-moved`; never the indexer's `views.modules` or `views.manifests`), the two
// events of the seat that serves the seat list (ADR 0636 §4) and the keeper's `findings-asked`, and nothing else.
wantConsumes := []string{"mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-catalog.seat-definition-changed",
"mesh-catalog.seat-definition-removed", "mesh-conditions.findings-asked", "seat:mesh-module-assigner.views.assignments",
"seat:mesh-module-assigner.views.seat-holders", "mesh-delivery.source-moved"}
if !reflect.DeepEqual(m.Consumes, wantConsumes) {
t.Errorf("the tool runner consumes %v, want %v", m.Consumes, wantConsumes)
}
}
// issue 1966 TLS-1: the tool runner declares its own secret `ledgers`, each module's ledger credential, in the directory it
// keeps its broker credential in, which is where the runner reads it (runtime.LedgersFileOf).
func TestTheManifestDeclaresTheRunnersOwnSecretLedgers(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
OwnSecrets map[string]string `json:"own-secrets"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
want := map[string]string{"broker": "${dir:mesh-state}/broker", "ledgers": "${dir:mesh-state}/ledgers"}
if !reflect.DeepEqual(m.OwnSecrets, want) {
t.Errorf("own secrets %v, want %v", m.OwnSecrets, want)
}
}
+55 -7
View File
@@ -4,7 +4,10 @@
package console
import (
"context"
"encoding/json"
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"os"
"strings"
"sync"
@@ -44,6 +47,7 @@ type RPCError struct {
// Surface answers MCP requests over one bus connection, as one account.
type Surface struct {
conn *bus.Conn
src Source
who string
mu sync.Mutex
known *Listing
@@ -52,14 +56,48 @@ type Surface struct {
idxAt time.Time
// runtimes is every runtime discovery has heard, as it last heard it (mesh_runtimes).
runtimes map[string]*runtimeSeen
// roundSaid is what the last round logged was missing, so a round is logged once (novox/hq issue 554).
roundSaid string
// Flat announces the whole catalogue, as the console did before ADR 0195: for a person reading
// it or a client that wants it. Off by default; MESH_CONSOLE_FLAT=1 turns it on.
Flat bool
// Record, when set, records a hand act in place of the seat mesh-hand-acts' verb (a test's).
Record func(args map[string]any) error
// Approvals is this machine's approval gate, read by mesh_approvals and mesh_describe; nil when the runner has none.
Approvals Approvals
}
// MethodSeats is the console's JSON-RPC method beside MCP's that answers every seat's verbs and holders, in the shape
// the old `tools` answered: `{seats: [{seat, scope, tools: [{name, description, input, replaces}], holders:
// [{module, node}]}], holders_why}`.
const MethodSeats = "mesh/seats"
// seatsAnswer is MethodSeats' answer: mesh-sdk's seat definitions, with each seat's holders by assignment.
func (s *Surface) seatsAnswer() map[string]any {
placed, err := seatHoldersOn(s.src)
x := &index{Placed: placed}
seats := []map[string]any{}
for _, st := range catalogue.Seats() {
holders := x.placedHolders(st.Name, "")
if holders == nil {
holders = []meshmoduleassigner.SeatHoldersHoldersItem{}
}
tools := st.Serves
if tools == nil {
tools = []catalogue.Verb{}
}
seats = append(seats, map[string]any{"seat": st.Name, "scope": st.Scope, "tools": tools, "holders": holders})
}
out := map[string]any{"seats": seats}
if err != nil {
out["holders_why"] = "who holds which seat by assignment could not be read: " + err.Error()
}
return out
}
// NewSurface is the surface over a connection, as `who`.
func NewSurface(conn *bus.Conn, who string) *Surface {
return &Surface{conn: conn, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
func NewSurface(conn *bus.Conn, src Source, who string) *Surface {
return &Surface{conn: conn, src: src, who: who, Flat: os.Getenv("MESH_CONSOLE_FLAT") == "1"}
}
func (s *Surface) listing() (*Listing, error) {
@@ -70,7 +108,7 @@ func (s *Surface) listing() (*Listing, error) {
return l, nil
}
s.mu.Unlock()
l, err := toolsOn(s.conn)
l, err := toolsOn(s.conn, s.src)
if err != nil {
return nil, err
}
@@ -101,7 +139,11 @@ func idOrNull(id json.RawMessage) json.RawMessage {
}
// Handle answers one request; nil for a notification, which expects none.
func (s *Surface) Handle(r Request) *Reply {
func (s *Surface) Handle(r Request) *Reply { return s.HandleContext(context.Background(), r) }
// HandleContext answers one request for as long as ctx lasts: a wait whose client hung up or cancelled
// ends at once, its subscription closed (the review of issue 745).
func (s *Surface) HandleContext(ctx context.Context, r Request) *Reply {
notification := isNotification(r.ID)
switch r.Method {
case "initialize":
@@ -118,6 +160,11 @@ func (s *Surface) Handle(r Request) *Reply {
return nil
}
return answer(r.ID, map[string]any{})
case MethodSeats:
// The seats' contract and who holds each, for a client on this machine that has no seat definitions of its own
// (the TypeScript `mesh`): read in-process from mesh-sdk and from the assigner, never asked of anything else
// (novox/hq ADR 0571 clauses 4 and 5).
return answer(r.ID, s.seatsAnswer())
case "tools/list":
if !s.Flat {
return answer(r.ID, map[string]any{"tools": discovery()})
@@ -162,7 +209,7 @@ func (s *Surface) Handle(r Request) *Reply {
if p.Arguments == nil {
p.Arguments = map[string]any{}
}
return answer(r.ID, s.discover(p.Name, p.Arguments))
return answer(r.ID, s.discover(ctx, p.Name, p.Arguments))
}
args := map[string]any{}
for k, v := range p.Arguments {
@@ -296,14 +343,15 @@ func (s *Surface) instructions() string {
"that serves it; what may be called was fixed when this account was issued, so a " +
"refusal means the account, not the tool. The list is what the running modules " +
"answered, plus every role's tools from the mesh's records — the mesh's own verbs " +
"(mesh-controller.status, .push, .assign …) among them; a module that did not answer " +
"(mesh-delivery.deliveries, mesh-module-assigner.propose …) among them; a module that did not answer " +
"is named in the list's _meta and can still be called by <module>.<tool>."
}
return "The tools of a Novox mesh, reached as " + s.who + ", found by address rather than listed " +
"whole (novox/hq ADR 0195). mesh_overview shows the mesh's seats and machines; mesh_machine one " +
"machine's seats and modules; mesh_search finds a tool by words; mesh_describe gives one tool's " +
"arguments; mesh_call calls it; mesh_runtimes says which machines' runtimes answered discovery, how fast " +
"and how much — ask it when an address is said to be missing or on another machine. " + grammar + " What may be called was fixed when this account " +
"and how much — ask it when an address is said to be missing or on another machine; mesh_wait waits on the " +
"bus for an event instead of polling (novox/hq ADR 0367). " + grammar + " What may be called was fixed when this account " +
"was issued, so a refusal means the account, not the tool."
}
@@ -11,9 +11,9 @@ import (
// 2026-10-05 the controller's push was described as taking nothing, and a push naming one machine
// ran as a push of every machine behind.
func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
push := Tool{Module: "mesh-controller", Name: "push", Seat: true, Scope: "mesh",
push := Tool{Module: "mesh-module-assigner", Name: "push", Seat: true, Scope: "mesh",
Input: json.RawMessage(`{"type":"object","properties":{"node":{"type":"string"},"behind":{"type":"string"}}}`)}
mesh := target{Address: "mesh-controller.push", Tool: push, Seat: true}
mesh := target{Address: "mesh-module-assigner.push", Tool: push, Seat: true}
props := schemaAsPassed(mesh)["properties"].(map[string]any)
if _, has := props["node"]; !has {
@@ -44,6 +44,27 @@ func TestAMeshSeatsNodeIsItsOwnArgument(t *testing.T) {
t.Fatalf("another machine in the arguments than in the address was dropped: %v", err)
}
// Issue 1966: a seat verb its holder announced with an empty schema is judged by the definition this runner holds;
// where it holds none (it waits for the seat list, holding the bootstrapped ones), the arguments go to the holder,
// which judges them, and mesh_describe says the definition is not held.
empty := json.RawMessage(`{"type":"object","properties":{}}`)
propose := target{Address: "mesh-module-assigner.propose", Seat: true,
Tool: Tool{Module: "mesh-module-assigner", Name: "propose", Seat: true, Scope: "mesh", Input: empty}}
if sent, err := argumentsAsSent(propose, map[string]any{"changes": []any{}, "why": "w"}); err != nil || sent["why"] != "w" {
t.Fatalf("a held definition's arguments were refused: %v %v", sent, err)
}
if _, err := argumentsAsSent(propose, map[string]any{"chnages": 1}); err == nil || !strings.Contains(err.Error(), "chnages") {
t.Fatalf("an argument the held definition does not declare was sent: %v", err)
}
unheld := target{Address: "mesh-not-yet-read.act", Seat: true,
Tool: Tool{Module: "mesh-not-yet-read", Name: "act", Seat: true, Scope: "mesh", Input: empty}}
if sent, err := argumentsAsSent(unheld, map[string]any{"changes": 1}); err != nil || sent["changes"] != 1 {
t.Fatalf("a seat whose definition is not held refused its arguments: %v %v", sent, err)
}
if _, held := seatDefinitionHeld(unheld); held || !strings.Contains(notHeld("mesh-not-yet-read"), "no definition of mesh-not-yet-read") {
t.Fatal("a definition not held is said to be held")
}
// A module's tool is its own: what it takes beyond node is the module's to judge.
beta := target{Address: "desk/beta.three", Node: "desk", Tool: Tool{Module: "beta", Name: "three"}}
if sent, err := argumentsAsSent(beta, map[string]any{"node": "desk", "verbose": true}); err != nil || sent["verbose"] != true || sent["node"] != nil {
@@ -0,0 +1,43 @@
package console
import (
"errors"
"fmt"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// A marked tool or verb that reaches no responder names withheld and the grants part's read (design 46 7-approval-mark §6);
// an unmarked one, or a marked one failing otherwise, does not. A tool's own error that mentions 503 is the tool's answer, not a routing fault (issue 1529); a real no-responders still says
// nothing serves.
func TestARealNoRespondersStillSaysNothingServes(t *testing.T) {
got := whyItFailed("scanner.scan", errors.New("the scanner refused the scan job: 503 Service Unavailable"))
if strings.Contains(got, "nothing serves") || !strings.HasPrefix(got, "scanner.scan failed: ") || !strings.Contains(got, "503") {
t.Fatalf("got %q", got)
}
for _, err := range []error{bus.ErrNoResponders, fmt.Errorf("ask: %w", bus.ErrNoResponders), nats.ErrNoResponders} {
if got := whyItFailed("a.b", err); !strings.HasPrefix(got, "nothing serves a.b") {
t.Fatalf("%v: got %q", err, got)
}
}
s := &Surface{Approvals: &fakeApprovals{marked: "shell.wipe"}}
for _, c := range []struct {
target target
err error
hinted bool
}{
{target{Address: "shell.wipe", Tool: Tool{Module: "shell", Name: "wipe"}}, bus.ErrNoResponders, true},
{target{Address: "node-login-shell.execute", Seat: true, Tool: Tool{Module: "node-login-shell", Name: "execute"}}, bus.ErrNoResponders, false},
{target{Address: "shell.tidy", Tool: Tool{Module: "shell", Name: "tidy"}}, bus.ErrNoResponders, false},
{target{Address: "shell.wipe", Tool: Tool{Module: "shell", Name: "wipe"}}, errors.New("its own failure"), false},
} {
got := s.callFailed(c.target, c.err)
if hinted := strings.Contains(got, "approval-mark/1: mesh-grants.grants_user_list names it under withheld"); hinted != c.hinted {
t.Errorf("%s (%v): %q", c.target.Address, c.err, got)
}
}
}
+68 -7
View File
@@ -1,9 +1,18 @@
package console
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// A refusal is said only for the bus's own refusals; a tool's answer that mentions authorization is the
@@ -19,19 +28,71 @@ func TestARefusalIsSaidOnlyForTheBussOwn(t *testing.T) {
t.Errorf("%q read as a refusal: %v, want %v", msg, got, refusal)
}
}
// ADR 0666 §16: an answer a module sent as a reference is read from its sent store for the person asking; one the bus does
// not let the runner read is shown as its reference with its size and why it was not read.
t.Run("AnAnswerSentAsAReferenceIsReadForThePersonAsking", func(t *testing.T) {
nc, err := nats.Connect(meshtest.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, _ := jetstream.New(nc)
obs, err := js.CreateObjectStore(context.Background(), jetstream.ObjectStoreConfig{Bucket: "m_sent-objects"})
if err != nil {
t.Fatal(err)
}
conn, err := bus.Connect(bus.Credential{URL: meshtest.URL(t), Module: "console", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
defer conn.Close()
body := []byte(`{"lines":"` + strings.Repeat("x", 9<<20) + `"}`)
sum := sha256.Sum256(body)
ref := stateobjects.Ref{Store: "m_sent-objects", Object: "tool.lines." + hex.EncodeToString(sum[:]), Size: int64(len(body)),
Digest: hex.EncodeToString(sum[:]), Form: stateobjects.FormVersion}
if _, err := obs.PutBytes(context.Background(), ref.Object, body); err != nil {
t.Fatal(err)
}
if got, unread := resolveSent(conn, stateobjects.Wrap(ref)); unread != "" || string(got) != string(body) {
t.Fatalf("read %d bytes of %d, %q", len(got), len(body), unread)
}
missing := ref
missing.Store = "gone_sent-objects"
wrapped := stateobjects.Wrap(missing)
got, unread := resolveSent(conn, wrapped)
if string(got) != string(wrapped) || !strings.Contains(unread, "9437") || !strings.Contains(unread, "gone_sent-objects") {
t.Fatalf("an unread answer showed %q and %q", got, unread)
}
if got, unread := resolveSent(conn, []byte(`{"a":1}`)); unread != "" || string(got) != `{"a":1}` {
t.Fatalf("a plain answer showed %q, %q", got, unread)
}
})
}
// **A timeout says it is not a failure** (novox/hq issue 265): a push that outlasted the wait had
// pushed, and the caller read "did not answer in time". And from the controller, where its answer is.
// pushed, and the caller read "did not answer in time". No hint names a seat that is gone (novox/hq issue 1126):
// the grammar's example, the handshake's instructions and a timeout's remedy named the controller's verbs.
func TestATimeoutIsNotAFailure(t *testing.T) {
got := whyItFailed("seat:mesh-controller.push", errors.New("timeout"))
for _, want := range []string{"not a failure", "may already have done", "mesh-controller.calls"} {
got := whyItFailed("seat:mesh-delivery.deliveries", errors.New("timeout"))
for _, want := range []string{"not a failure", "may already have done"} {
if !strings.Contains(got, want) {
t.Errorf("a controller timeout does not say %q: %s", want, got)
t.Errorf("a seat's timeout does not say %q: %s", want, got)
}
}
if other := whyItFailed("novox/postgres.postgres_list_databases", errors.New("timeout")); !strings.Contains(other, "not a failure") ||
strings.Contains(other, "calls") {
t.Errorf("a module's timeout: %s", other)
for name, text := range map[string]string{
"grammar": grammar,
"flat instructions": (&Surface{Flat: true, who: "x"}).instructions(),
"deep instructions": (&Surface{who: "x"}).instructions(),
"timeout": got,
} {
if strings.Contains(text, "controller") {
t.Errorf("%s names the controller: %s", name, text)
}
// The seats were renamed (ADR 0634 §9): no hint names the seat that placed modules or enrolled a machine by its old name.
for _, old := range []string{"mesh-placement", "mesh-enrolment"} {
if strings.Contains(text, old) {
t.Errorf("%s names the seat %s, which is now mesh-module-assigner or mesh-node-enroller: %s", name, old, text)
}
}
}
}
+125 -3
View File
@@ -7,7 +7,9 @@ package console
import (
"fmt"
"log"
"sort"
"strings"
"time"
"github.com/novox/mesh-tools/node-tools/internal/announce"
@@ -31,6 +33,11 @@ type runtimeSeen struct {
Shortened string
Answered bool // in the latest round
Why string // why not, when it did not
// Reused is true when its PING said its announcement was unchanged, so it was not fetched again.
Reused bool
// Last is its last whole answer: what a round that does not hear it uses instead, and what an
// unchanged digest stands for (novox/hq issue 554).
Last announce.Heard
}
func runtimeKey(name, id string) string { return name + "/" + id }
@@ -50,7 +57,108 @@ func seenOf(h announce.Heard, at time.Time) *runtimeSeen {
}
}
return &runtimeSeen{Name: h.Info.Name, ID: h.Info.ID, Machine: h.Machine(), LastHeard: at, Took: h.Took,
Bytes: h.Bytes, Modules: len(modules), Tools: len(tools), SeatVerbs: len(verbs), Shortened: h.Shortened, Answered: true}
Bytes: h.Bytes, Modules: len(modules), Tools: len(tools), SeatVerbs: len(verbs), Shortened: h.Shortened, Answered: true,
Reused: h.Reused, Last: h}
}
// held is the answer this console holds for an instance at this digest (announce.Known): an instance
// whose announcement is unchanged is not asked for it again (novox/hq issue 554).
func (s *Surface) held(name, id, digest string) (announce.Heard, bool) {
s.mu.Lock()
defer s.mu.Unlock()
r := s.runtimes[runtimeKey(name, id)]
if r == nil || r.Last.Digest == "" || r.Last.Digest != digest || r.Last.Info.Name == "" {
return announce.Heard{}, false
}
h := r.Last
h.Took, h.Reused = 0, false
return h, true
}
// before is the last answer of every runtime this round did not hear, heard within RuntimeForgotten
// and not since restarted under another instance (novox/hq issue 554).
func (s *Surface) before(d announce.Discovery) []earlier {
heard := map[string]bool{}
running := map[string]bool{}
for _, h := range d.Heard {
heard[runtimeKey(h.Info.Name, h.Info.ID)] = true
if m := h.Machine(); m != "" {
running[h.Info.Name+"@"+m] = true
}
}
s.mu.Lock()
defer s.mu.Unlock()
var out []earlier
for k, r := range s.runtimes {
if heard[k] || r.Last.Info.Name == "" || r.LastHeard.IsZero() || time.Since(r.LastHeard) > RuntimeForgotten {
continue
}
if r.Machine != "" && running[r.Name+"@"+r.Machine] {
continue // restarted: the new instance answered
}
out = append(out, earlier{Heard: r.Last, At: r.LastHeard})
}
sort.Slice(out, func(i, j int) bool {
return runtimeKey(out[i].Heard.Info.Name, out[i].Heard.Info.ID) < runtimeKey(out[j].Heard.Info.Name, out[j].Heard.Info.ID)
})
return out
}
// sayRound logs a round of discovery that did not hear every runtime — once, when what it missed
// changes, and once when every runtime is heard again — so the next case leaves evidence in the
// journal (novox/hq issue 554). Called with s.mu held, after remember.
func (s *Surface) sayRound(x *index) {
d := x.Discovery
var missed []string
for _, u := range x.Unheard {
who := u.Runtime
if u.Machine != "" {
who += " on " + u.Machine
}
missed = append(missed, who)
}
sort.Strings(missed)
shape := strings.Join(missed, ", ")
if d.Empty() {
shape = "empty|" + shape
}
if shape == s.roundSaid {
return
}
s.roundSaid = shape
reused := 0
for _, h := range d.Heard {
if h.Reused {
reused++
}
}
logf := log.Printf
if s.conn != nil && s.conn.Logf != nil {
logf = s.conn.Logf
}
window := fmt.Sprintf("waited %s (window %s, patience %s)", d.Waited.Round(time.Millisecond), announce.Window, announce.Patience)
held := ""
if len(x.Earlier) > 0 {
held = fmt.Sprintf("; what %d of them announced earlier is used, with its age", len(x.Earlier))
}
switch {
case d.Empty():
logf("[mesh-tools] discovery heard nothing at all: no PING arrived; %s; not heard: %s%s; asking again at the next question",
window, orNone(missed), held)
case len(missed) > 0:
logf("[mesh-tools] discovery did not hear every runtime: %d PING answer(s), %d announcement(s) (%d unchanged, not fetched); %s; not heard: %s%s",
d.Pinged, len(d.Heard), reused, window, shape, held)
default:
logf("[mesh-tools] discovery hears every runtime again: %d PING answer(s), %d announcement(s) (%d unchanged, not fetched); %s",
d.Pinged, len(d.Heard), reused, window)
}
}
func orNone(xs []string) string {
if len(xs) == 0 {
return "none heard before either"
}
return strings.Join(xs, ", ")
}
// remember records a round of discovery and adds to the index every runtime heard lately that did not
@@ -101,14 +209,25 @@ func (s *Surface) remember(x *index) {
continue
}
r.Answered = false
used := ""
if at, ok := x.Earlier[k]; ok {
used = fmt.Sprintf("; what it announced %s ago is used", time.Since(at).Round(time.Second))
}
if why, ok := silent[k]; ok {
r.Why = why
if used != "" {
for i := range x.Unheard {
if x.Unheard[i].Runtime == r.Name && x.Unheard[i].Machine == r.Machine && x.Unheard[i].Why == why {
x.Unheard[i].Why += used
}
}
}
continue // already in the index's unheard
}
r.Why = "it did not answer this round of discovery"
if time.Since(r.LastHeard) <= RuntimeForgotten {
x.Unheard = append(x.Unheard, unheard{Runtime: r.Name, Machine: r.Machine,
Why: fmt.Sprintf("it answered discovery %s ago, not now", time.Since(r.LastHeard).Round(time.Second))})
Why: fmt.Sprintf("it answered discovery %s ago, not now%s", time.Since(r.LastHeard).Round(time.Second), used)})
}
}
for _, u := range x.Discovery.Silent {
@@ -157,6 +276,9 @@ func (s *Surface) runtimesAnswer() map[string]any {
if r.Shortened != "" {
entry["shortened, how"] = r.Shortened
}
if r.Reused && r.Answered {
entry["announcement"] = "unchanged (its PING's digest), so not fetched again; took is its PING's"
}
}
if r.Answered {
onMachine[r.Machine] = true
@@ -184,7 +306,7 @@ func (s *Surface) runtimesAnswer() map[string]any {
return answerText(map[string]any{
"asked": x.Discovery.At.Format(time.RFC3339),
"waited": x.Discovery.Waited.Round(time.Millisecond).String(),
"window": fmt.Sprintf("at least %s; up to %s for a runtime that answered PING", announce.Window, announce.Patience),
"window": fmt.Sprintf("at least %s; up to %s for a runtime that answered PING, or until one does", announce.Window, announce.Patience),
"answered": answered,
"not heard": notHeard,
})
+64 -15
View File
@@ -2,14 +2,19 @@ package console
import (
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
"time"
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
seatassigner "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
"github.com/nats-io/nats.go/micro"
"github.com/novox/mesh-tools/node-tools/internal/announce"
"github.com/novox/mesh-tools/node-tools/internal/bus"
"github.com/novox/mesh-tools/node-tools/internal/meshtest"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
@@ -23,18 +28,22 @@ func slowRuntime(t *testing.T, conn *bus.Conn, node string, lag time.Duration, e
t.Fatal(err)
}
t.Cleanup(stop)
stop, err = conn.Raw("$SRV.INFO", func(string, []byte) []byte {
// Asked of every service, and of this instance by name and id — as discovery asks it (issue 554).
info := func(string, []byte) []byte {
if lag < 0 {
return nil
}
time.Sleep(lag)
body, _ := json.Marshal(announce.Info(announce.Service{Name: id.Name, ID: id.ID, Metadata: id.Metadata}, endpoints))
return body
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
for _, subject := range []string{"$SRV.INFO", "$SRV.INFO." + id.Name + "." + id.ID} {
stop, err := conn.Raw(subject, info)
if err != nil {
t.Fatal(err)
}
t.Cleanup(stop)
}
conn.Flush()
}
@@ -48,7 +57,25 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
// Kept for the test: one round of discovery answers every question but mesh_runtimes, which asks anew.
IndexKept, announce.Patience = time.Minute, 2500*time.Millisecond
t.Cleanup(func() { IndexKept, announce.Patience = keptWas, patienceWas })
mt.New(t)
mesh := mt.New(t)
// What the mesh holds, as the indexer and the assigner write it (ADR 0571): never the controller's records.
mesh.Placed(t, map[string][]string{"slack": {"slack_check"}, "nap": {"doze"}, "nextcloud-client": {"nextcloud_check"}},
map[string][]string{"laptop": {"slack", "nextcloud-client"}, "sleeper": {"nap", "nextcloud-client"},
"desktop": {"nextcloud-client"}, "bench": nil})
// Who holds which seat, by assignment (ADR 0571 clause 4), as the seat mesh-module-assigner's holder composes it.
placedKV := mesh.Bucket(t, meshtest.SeatHoldersBucket)
holdersKey, err := assigner.SeatHoldersKey("mesh-wants", "")
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(seatassigner.SeatHolders{V: assigner.Version, Seat: "mesh-wants", Scope: "mesh",
Holders: []seatassigner.SeatHoldersHoldersItem{{Module: "mesh-delivery-assigner", Node: "bench"}}})
if _, err := placedKV.Put(holdersKey, raw); err != nil {
t.Fatal(err)
}
consoleMembership := mt.MembershipOf("node-tools", "laptop-console", false, nil)
consoleMembership.State = mt.ConsoleReads()
mesh.Issue(t, consoleMembership)
laptop := connect(t, "node-tools", "laptop")
stopTool, err := laptop.HandleSubject("mesh.mod.slack.tool.slack_check.laptop", func(json.RawMessage) (any, error) {
@@ -72,14 +99,17 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
Subject: "mesh.mod.nextcloud-client.tool.nextcloud_check.desktop"},
})
controller := connect(t, "mesh-controller", "bench")
controller := connect(t, "mesh-module-assigner", "bench")
var toolsAsked atomic.Int32
for verb, output := range map[string]string{
"nodes": `[{"name":"laptop"},{"name":"sleeper"},{"name":"desktop"},{"name":"bench"}]`,
"modules": `[{"module":"slack","on":["laptop"],"tools":true},{"module":"nap","on":["sleeper"],"tools":true},` +
`{"module":"nextcloud-client","on":["laptop","desktop","sleeper"],"tools":true}]`,
// The controller's records are served here only to see that none is asked (ADR 0571 clauses 4 and 5).
"nodes": `[{"name":"laptop"}]`,
"modules": `[]`,
"tools": `{"seats":[]}`,
} {
output := output
stop, err := controller.HandleSubject("mesh.seat.mesh-controller.tool."+verb, func(json.RawMessage) (any, error) {
output, verb := output, verb
stop, err := controller.HandleSubject("mesh.seat.mesh-module-assigner.tool."+verb, func(json.RawMessage) (any, error) {
toolsAsked.Add(1)
return map[string]any{"output": output, "ok": true}, nil
})
if err != nil {
@@ -90,7 +120,14 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
controller.Flush()
console := connect(t, "node-tools", "laptop-console")
up, err := Serve(NewSurface(console, "laptop.node-tools"), "127.0.0.1:0")
console.Follow("node-tools")
mt.Until(t, func() error {
if console.Membership("node-tools") == nil {
return fmt.Errorf("the console's membership is not issued yet")
}
return nil
})
up, err := Serve(NewSurface(console, busSource{console}, "laptop.node-tools"), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
@@ -109,8 +146,20 @@ func TestARuntimeThatAnswersLateIsFoundAndOneThatNeverDoesIsNamed(t *testing.T)
t.Errorf("the late runtime's machine lists nothing: %s", machine)
}
// The seats' contract and holders, for a client on this machine with no seat definitions (the TypeScript `mesh`).
seats := post(t, endpoint, `{"jsonrpc":"2.0","id":9,"method":"mesh/seats"}`)
if raw, _ := json.Marshal(seats["result"]); !strings.Contains(string(raw), `"holders":[{"module":"mesh-delivery-assigner","node":"bench"}],"scope":"mesh","seat":"mesh-wants"`) ||
!strings.Contains(string(raw), `"replaces":["journalctl"]`) {
t.Errorf("mesh/seats answered %s", raw)
}
if n := toolsAsked.Load(); n != 0 {
t.Errorf("the console asked the controller's records %d times", n)
}
// The silent one is named, never called missing.
for address, want := range map[string]string{
// A seat whose holder announced nothing is named with its holder by assignment, never called missing.
"mesh-wants.read": "the seat mesh-wants is held by mesh-delivery-assigner on bench by assignment",
"sleeper/nap.doze": "nap is assigned to sleeper in the mesh's records, but nothing that answered discovery serves it: discovery did not hear in full from node-tools on sleeper (it answered PING",
"sleeper/nextcloud-client.nextcloud_check": "nextcloud-client on sleeper did not answer discovery: discovery did not hear in full from node-tools on sleeper",
} {
@@ -175,7 +224,7 @@ func TestARuntimeHeardBeforeAndNotNowIsNamed(t *testing.T) {
s.remember(before)
now := &index{Modules: map[string]*moduleInfo{"nextcloud-client": {Module: "nextcloud-client", On: []string{"desktop"},
Tools: []Tool{{Module: "nextcloud-client", Name: "nextcloud_check"}}}}, Recorded: map[string][]string{},
Discovery: announce.Discovery{At: time.Now().UTC()}}
Discovery: announce.Discovery{At: time.Now().UTC(), Pinged: 1}} // desktop's runtime answered
s.remember(now)
if len(now.Unheard) != 1 || now.Unheard[0].Machine != "laptop" {
t.Fatalf("unheard: %+v", now.Unheard)
@@ -190,7 +239,7 @@ func TestARuntimeHeardBeforeAndNotNowIsNamed(t *testing.T) {
t.Errorf("%v", err)
}
// Nobody unheard: the plain answers stand.
quiet := &index{Modules: now.Modules, Recorded: map[string][]string{}}
quiet := &index{Modules: now.Modules, Recorded: map[string][]string{}, Discovery: now.Discovery}
if _, err := resolve(quiet, "laptop/nextcloud-client.nextcloud_check"); err == nil || !strings.Contains(err.Error(), "does not run on laptop; it runs on desktop") {
t.Errorf("%v", err)
}
+40 -22
View File
@@ -19,7 +19,8 @@ package console
// Ties go to a seat — the place tools belong — then to the shorter name, then by address.
import (
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"path"
"sort"
"strings"
@@ -44,6 +45,31 @@ var synonyms = map[string][]string{
"hosts": {"hostname"},
}
// filler are the function words a question carries and no tool is named for: a query's words must all
// match, so "did our weekly upgrade run on all our nodes" would otherwise find nothing for "did", "our",
// "on" and "all". Kept short and here only; a word that names something (`every`, `list`, `not`) is not
// in it. A query of nothing but these matches nothing.
var filler = map[string]bool{}
func init() {
for _, w := range strings.Fields("a an the did do does our my your we i is are was on in at of for to all any and or with it its there what which how has have been") {
filler[w] = true
}
}
// queryWords are a query's words to match: split at spaces, the punctuation of a sentence trimmed from
// each end (`nodes?` is `nodes`), filler left out.
func queryWords(query string) []string {
var out []string
for _, w := range strings.Fields(query) {
w = strings.Trim(w, "?!.,;:\"'()")
if w != "" && !filler[w] {
out = append(out, w)
}
}
return out
}
// hit is one match as mesh_search answers it.
type hit struct {
Address string `json:"address"`
@@ -169,8 +195,11 @@ func scoreOf(query string, qwords []string, name, description string, replaces [
// search is mesh_search's answer: every match, best first.
func (x *index) search(query string) []hit {
query = strings.ToLower(strings.TrimSpace(query))
qwords := strings.Fields(query)
qwords := queryWords(query)
var hits []hit
if len(qwords) == 0 {
return nil
}
for _, st := range x.Seats {
for _, v := range st.Verbs {
replaces := x.Replaces["seat:"+st.Seat+"."+v.Name]
@@ -237,27 +266,16 @@ func (x *index) search(query string) []hit {
return hits
}
// replacesOf is what every verb and tool replaces, from the controller's records (novox/hq ADR 0245):
// a seat's verbs from its `tools` answer, keyed `seat:<seat>.<verb>`, and a module's own tools from
// its `modules` answer, keyed `<module>.<tool>`. A controller older than the field answers neither,
// and search matches names and descriptions as before.
func replacesOf(toolsAnswer json.RawMessage, modules []recordedModule) map[string][]string {
// replacesOf is what every verb and tool replaces (novox/hq ADR 0245): a seat's verbs from the seat's contract in
// mesh-sdk, keyed `seat:<seat>.<verb>`, read in-process and asked of no part (ADR 0571 clause 5: no part keeps a
// central record of tools); and a module's own tools from its records, keyed `<module>.<tool>`. A verb or a tool with
// no `replaces` is matched by its name and description as before.
func replacesOf(seats []catalogue.Seat, modules []recordedModule) map[string][]string {
out := map[string][]string{}
var seats struct {
Seats []struct {
Seat string `json:"seat"`
Tools []struct {
Name string `json:"name"`
Replaces []string `json:"replaces"`
} `json:"tools"`
} `json:"seats"`
}
if json.Unmarshal(toolsAnswer, &seats) == nil {
for _, s := range seats.Seats {
for _, t := range s.Tools {
if len(t.Replaces) > 0 {
out["seat:"+s.Seat+"."+t.Name] = t.Replaces
}
for _, s := range seats {
for _, v := range s.Serves {
if len(v.Replaces) > 0 {
out["seat:"+s.Name+"."+v.Name] = append([]string(nil), v.Replaces...)
}
}
}
+52 -29
View File
@@ -1,7 +1,8 @@
package console
import (
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"strings"
"testing"
@@ -19,6 +20,7 @@ func meshToSearch(t *testing.T) *index {
"status": "One unit as the service manager sees it now: its states, whether it starts at boot.",
"restart": "Restart one unit.",
"journal": "The last lines of one unit's journal (at most 2000).",
"failed": "The units that failed.",
}
for _, node := range []string{"anchor", "laptop"} {
for verb, does := range verbs {
@@ -31,8 +33,8 @@ func meshToSearch(t *testing.T) *index {
"node": "What one machine reported it can do, what it is assigned, and why.",
"nodes": "Every machine the mesh knows, with whether it is converged or adopted.",
} {
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-controller", Seat: "mesh-controller",
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-controller.tool." + verb})
endpoints = append(endpoints, announce.Endpoint{Kind: announce.KindSeat, Module: "mesh-module-assigner", Seat: "mesh-module-assigner",
Tool: verb, Scope: "mesh", Description: does, Subject: "mesh.seat.mesh-module-assigner.tool." + verb})
}
for tool, does := range map[string]string{
"docker_list": "Every container on this machine: name, image, state, ports.",
@@ -44,13 +46,7 @@ func meshToSearch(t *testing.T) *index {
Description: does, Subject: "mesh.mod.docker.tool." + tool + ".anchor"})
}
x, _, _ := indexOfEndpoints(endpoints)
tools := json.RawMessage(`{"seats":[{"seat":"node-service-manager","scope":"node","tools":[
{"name":"units","replaces":["systemctl list-units","systemctl --failed"]},
{"name":"status","replaces":["systemctl status","systemctl is-active"]},
{"name":"restart","replaces":["systemctl restart"]},
{"name":"journal","replaces":["journalctl"]}]},
{"seat":"mesh-controller","scope":"mesh","tools":[{"name":"node","replaces":["hostnamectl","uptime"]},{"name":"nodes"}]}]}`)
x.Replaces = replacesOf(tools, []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
x.Replaces = replacesOf(catalogue.Seats(), []recordedModule{{Module: "docker", On: []string{"anchor"}, Tools: true,
Replaces: map[string][]string{"docker_list": {"docker ps"}, "docker_logs": {"docker logs"}, "docker_restart": {"docker restart"}}}})
return x
}
@@ -60,22 +56,21 @@ func meshToSearch(t *testing.T) *index {
func TestSearchFindsTheToolThatReplacesACommandFirst(t *testing.T) {
x := meshToSearch(t)
for query, want := range map[string]string{
"journalctl": "anchor/node-service-manager.journal",
"journalctl -u mesh-controller -n 50": "anchor/node-service-manager.journal",
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
"logs": "anchor/node-service-manager.journal",
"systemctl status": "anchor/node-service-manager.status",
"systemctl status nats.service": "anchor/node-service-manager.status",
"systemctl restart mesh-controller": "anchor/node-service-manager.restart",
"systemctl --failed": "anchor/node-service-manager.units",
"docker ps": "anchor/docker.docker_list",
"docker ps -a": "anchor/docker.docker_list",
"docker logs": "anchor/docker.docker_logs",
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
"docker restart nats": "anchor/docker.docker_restart",
"container logs": "anchor/docker.docker_logs",
"uptime": "mesh-controller.node",
"journalctl": "anchor/node-service-manager.journal",
"journalctl -u mesh-module-assigner -n 50": "anchor/node-service-manager.journal",
"sudo journalctl -fu sshd": "anchor/node-service-manager.journal",
"/usr/bin/journalctl": "anchor/node-service-manager.journal",
"logs": "anchor/node-service-manager.journal",
"systemctl status": "anchor/node-service-manager.status",
"systemctl status nats.service": "anchor/node-service-manager.status",
"systemctl restart mesh-module-assigner": "anchor/node-service-manager.restart",
"systemctl --failed": "anchor/node-service-manager.failed",
"docker ps": "anchor/docker.docker_list",
"docker ps -a": "anchor/docker.docker_list",
"docker logs": "anchor/docker.docker_logs",
"docker logs --tail 100 nats": "anchor/docker.docker_logs",
"docker restart nats": "anchor/docker.docker_restart",
"container logs": "anchor/docker.docker_logs",
} {
hits := x.search(query)
if len(hits) == 0 {
@@ -116,7 +111,7 @@ func TestASearchByCommandSaysWhatItReplaces(t *testing.T) {
// A synonym is a word, never part of one: `logs` matches through `log`, and finds no `login`.
func TestSearchStillMatchesEveryWord(t *testing.T) {
x := meshToSearch(t)
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-controller.nodes" {
if hits := x.search("machine converged"); len(hits) != 1 || hits[0].Address != "mesh-module-assigner.nodes" {
t.Errorf("words in a description: %+v", hits)
}
if hits := x.search("journalctl kubernetes"); len(hits) != 1 || hits[0].Address != "anchor/node-service-manager.journal" {
@@ -134,10 +129,10 @@ func TestSearchStillMatchesEveryWord(t *testing.T) {
}
}
// A controller older than `replaces` answers without it: search matches names and descriptions as before.
// A seat's contract and a module's records without `replaces`: search matches names and descriptions as before.
func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
x := meshToSearch(t)
x.Replaces = replacesOf(json.RawMessage(`{"seats":[{"seat":"node-service-manager","tools":[{"name":"journal"}]}]}`),
x.Replaces = replacesOf([]catalogue.Seat{{Name: "node-service-manager", Serves: []catalogue.Verb{{Name: "journal"}}}},
[]recordedModule{{Module: "docker"}})
if len(x.Replaces) != 0 {
t.Fatalf("replaces from records that have none: %v", x.Replaces)
@@ -149,3 +144,31 @@ func TestSearchWithoutReplacesMatchesAsBefore(t *testing.T) {
t.Errorf("a tool's own name: %+v", hits)
}
}
// A question's filler words do not have to match (every other word still does): the operator's own words
// find the tool whose description says it in other words; a query of filler alone finds nothing.
func TestSearchIgnoresFillerWords(t *testing.T) {
x := meshToSearch(t)
x.Modules["pacman"] = &moduleInfo{Module: "pacman", On: []string{"anchor"}, Tools: []Tool{{Module: "pacman", Name: "pacman_round",
Description: "Whether this week's system upgrade ran on every node (the weekly upgrade round, last run of all nodes)."}}}
for _, c := range []struct {
query string
want string // the first hit's address, "" for none
}{
{"did our weekly system upgrade run on all our nodes?", "anchor/pacman.pacman_round"},
{"the weekly upgrade", "anchor/pacman.pacman_round"},
{"did our", ""},
{"what is the", ""},
{"", ""},
{"did our weekly kubernetes upgrade run?", ""},
} {
hits := x.search(c.query)
got := ""
if len(hits) > 0 {
got = hits[0].Address
}
if got != c.want || (c.want == "" && len(hits) != 0) {
t.Errorf("%q: first hit %q, want %q (%+v)", c.query, got, c.want, hits)
}
}
}
@@ -0,0 +1,10 @@
package console
import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/seatlist/seatlisttest"
)
// The seats the tests describe are given to catalogue as the tool runner gives the ones it read from the bus
// (novox/hq ADR 0635 §1); catalogue compiles none.
func init() { catalogue.UseSeats(catalogue.SeatsFromDefinitions(seatlisttest.Definitions())) }
+31
View File
@@ -0,0 +1,31 @@
package console
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// resolveSent is the answer a person reads (ADR 0666 §16): an answer a module sent as a reference to an object of its
// sent store is read from it when the bus grants this runner's account the read (the standing grant the manifests give
// the callers of the verb), else shown as its reference with its size and why it was not read.
func resolveSent(conn *bus.Conn, result json.RawMessage) (json.RawMessage, string) {
ref, ok := stateobjects.ReadRef(result)
if !ok {
return result, ""
}
whole, err := stateobjects.Receive(result, func(r stateobjects.Ref) ([]byte, error) {
got, err := conn.SentGet(buscalls.ObjectsSentGetRequest{Store: r.Store, Object: r.Object, Size: r.Size, Digest: r.Digest})
if err != nil {
return nil, err
}
return got.Data, nil
})
if err != nil {
return result, fmt.Sprintf("the answer is %d bytes, held as the object %s of %s, and was not read: %v", ref.Size, ref.Object, ref.Store, err)
}
return whole, ""
}
@@ -0,0 +1,60 @@
package console
import (
"encoding/json"
"sync"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// busSource is a Source over a bus connection's own states, which a test's fixture mesh fills: the console's tests stand a
// membership that reads the four states beside it, where the tool runner itself holds first views (firstview.Views).
type busSource struct{ conn *bus.Conn }
func (b busSource) State(name string) (map[string][]byte, error) {
var mu sync.Mutex
keys := map[string][]byte{}
stop, err := b.conn.StateWatch(b.conn.Credential().Module, name, "", func(c bus.StateChange) error {
if c.Current && c.Op == "put" {
mu.Lock()
keys[c.Key] = c.Value
mu.Unlock()
}
return nil
})
if err != nil {
return nil, err
}
stop()
mu.Lock()
defer mu.Unlock()
return keys, nil
}
func (b busSource) Entry(name, key string) ([]byte, error) {
entry, err := b.conn.StateGet(b.conn.Credential().Module, name, key)
if err != nil || entry == nil {
return nil, err
}
return entry.Value, nil
}
// Modules reads the test bed's entries of the catalog half (meshtest.ModulesRead), as the tool runner's first views hold them.
func (b busSource) Modules() (map[string]meshcatalog.CatalogEntry, error) {
raw, err := b.State(mt.ModulesRead)
if err != nil {
return nil, err
}
out := map[string]meshcatalog.CatalogEntry{}
for module, value := range raw {
var e meshcatalog.CatalogEntry
if err := json.Unmarshal(value, &e); err != nil {
return nil, err
}
out[module] = e
}
return out, nil
}
+336
View File
@@ -0,0 +1,336 @@
package console
// mesh_wait: a session waits on the bus for one event instead of polling (novox/hq ADR 0367).
//
// **Any event, by its name** (novox/hq ADR 0670 §1, ADR 0671 §1; design 34 every-event §13, §26): the grants part
// grants this machine's tool runner every event subject, a seat's, a module's and a module stream's, and a name
// becomes its subjects through mesh-sdk's buscompose.EventSubjects; no membership and no manifest names a list.
// **An event only wakes the session** (§4): what it says is read again from the verb that owns it.
import (
"bytes"
"context"
"encoding/json"
"fmt"
"math"
"slices"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/approval"
)
// WaitDefault is how long a wait lasts when no timeout is given, and WaitLongest the most it may last
// (ADR 0367 §1): four minutes and ten.
var (
WaitDefault = 4 * time.Minute
WaitLongest = 10 * time.Minute
)
// waitMostNames is the most event names one call gives (design 34 every-event §13, §26): each name is six
// subscriptions for the call, so 16 names are 96, far above what a watcher names.
const waitMostNames = 16
func waitTool() map[string]any {
return map[string]any{"name": verbWait, "inputSchema": map[string]any{"type": "object", "properties": map[string]any{
"subjects": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": fmt.Sprintf("one or more event names, `<seat or module>.<event>`, such as "+
"`mesh-delivery-view.delivered`, at most %d; an event whose subject goes on after the name (an asker, a "+
"kind, a module) is heard by the name too", waitMostNames)},
"from": map[string]any{"type": "string", "enum": []string{"seat", "module", "stream"},
"description": "narrows every name to a seat's events, a module's or a module stream's, for an emitter " +
"whose name is more than one of them (optional)"},
"filter": map[string]any{"type": "object",
"description": "fields of the event's body that must equal the given values, e.g. {\"kind\": \"decision\", \"number\": 367}: the filter narrows by the body's fields (optional)"},
"timeout": map[string]any{"type": "number",
"description": fmt.Sprintf("seconds to wait: %d when absent, %d at most", int(WaitDefault.Seconds()), int(WaitLongest.Seconds()))},
}, "required": []string{"subjects"}},
"description": "Wait for the first event of the named events that matches the filter, and answer it with its " +
"name, where it came from, body and times — or say it timed out. Call it in a loop from a background command, " +
"and after every answer read the state from the verb that owns it: an event only wakes you, and one " +
"published between two waits is not heard (novox/hq ADR 0367). An operator's answer is read only with " +
"claude_code_answer, never from the event."}
}
// waitAnswer answers one mesh_wait call.
func (s *Surface) waitAnswer(ctx context.Context, args map[string]any) map[string]any {
names, err := waitNames(args["subjects"])
if err != nil {
return failure(err.Error())
}
timeout, err := waitTimeout(args["timeout"])
if err != nil {
return failure(err.Error())
}
filter, _ := args["filter"].(map[string]any)
if args["filter"] != nil && filter == nil {
return failure("mesh_wait's filter is an object of body fields to the values they must equal")
}
from, _ := args["from"].(string)
if args["from"] != nil && from != "seat" && from != "module" && from != "stream" {
return failure(fmt.Sprintf("mesh_wait's from is seat, module or stream, not %v", args["from"]))
}
if len(names) > waitMostNames {
return failure(fmt.Sprintf("mesh_wait names at most %d events in one call, not %d: each name is six "+
"subscriptions for the call (design 34 every-event §13)", waitMostNames, len(names)))
}
known, unchecked := s.emitters()
var subjects, undeclared []string
for _, name := range names {
subs, err := buscompose.EventSubjects(name, from)
if err != nil {
return failure("mesh_wait may not wait on it: " + err.Error())
}
// A misspelt emitter is refused, never waited on for ever; a known emitter's undeclared event is waited on and
// its timed-out answer says so (design 34 every-event §14).
if unchecked == "" {
emitter, event, _ := strings.Cut(name, ".")
seatEvents, isSeat := known.seats[emitter]
moduleEvents, isModule := known.modules[emitter]
isSeat = isSeat && from != "module" && from != "stream"
isModule = isModule && from != "seat"
if !isSeat && !isModule {
return failure(fmt.Sprintf("mesh_wait may not wait on %q: %s is neither a seat nor a module of %s",
name, emitter, known.source))
}
if !(isSeat && declares(seatEvents, event)) && !(isModule && declares(moduleEvents, event)) {
undeclared = append(undeclared, name)
}
}
for _, subject := range subs {
if !slices.Contains(subjects, subject) {
subjects = append(subjects, subject)
}
}
}
listen := func(within time.Duration) (any, error) {
return s.hear(ctx, names, subjects, undeclared, known.source, unchecked, filter, within)
}
// A wait whose subjects overlap a marked event's runs only on the operator's warrant for that one wait (design 34
// every-event §34 to §36), judged by the runner's approval gate.
var answer any
if g, ok := s.Approvals.(waitGate); ok {
given := map[string]any{}
for _, k := range []string{"subjects", "from", "filter", "timeout"} {
if v, has := args[k]; has {
given[k] = v
}
}
raw, _ := json.Marshal(given)
answer, err = g.Waited(ctx, approval.Call{Address: verbWait, Name: verbWait, Caller: s.who, Args: raw},
approval.Wait{Names: names, From: from, Subjects: subjects, Filter: filter, Timeout: timeout}, listen)
} else {
answer, err = listen(timeout)
}
if err != nil {
return failure(err.Error())
}
return answerText(answer)
}
// waitGate is the runner's approval gate as mesh_wait goes through it (design 34 every-event §34 to §36).
type waitGate interface {
Waited(ctx context.Context, c approval.Call, w approval.Wait, hear func(within time.Duration) (any, error)) (any, error)
}
// hear subscribes the wait's subjects for within and answers the first matching event, or that it timed out.
func (s *Surface) hear(ctx context.Context, names, subjects, undeclared []string, source, unchecked string, filter map[string]any,
timeout time.Duration) (map[string]any, error) {
heard, ok, refused, err := s.conn.HearFirst(ctx, subjects, timeout, func(body json.RawMessage) bool { return bodyMatches(body, filter) })
if err != nil {
return nil, fmt.Errorf("mesh_wait could not wait: %w", err)
}
var answer map[string]any
if !ok {
answer = map[string]any{"timed-out": true, "waited": timeout.String(), "subjects": names,
"read": "nothing matching was heard; read the state once from the verb that owns it, then wait again"}
if len(undeclared) > 0 {
answer["undeclared"] = fmt.Sprintf("no declaration names %s: no seat definition and no module's entry in %s "+
"declares that event", strings.Join(undeclared, ", "), source)
}
} else {
name, from, _ := buscompose.EventName(heard.Subject)
h := map[string]any{"subject": name, "from": from, "body": heard.Body, "at": heard.At.Format(time.RFC3339Nano)}
if heard.Published != "" {
h["published"] = heard.Published
}
read := "an event only wakes you: read the state from the verb that owns it before acting (novox/hq ADR 0367 §4)"
if strings.HasPrefix(name, "operator-channel.decided") {
read = "an event only wakes you: the operator's answer is read only with claude_code_answer, never from " +
"the event's body (novox/hq ADR 0670 §5)"
}
answer = map[string]any{"heard": h, "read": read}
}
if unchecked != "" {
answer["unchecked"] = "the names were not checked: " + unchecked
}
// A subscription the bus refused while others stood is said (design 34 every-event §16).
if len(refused) > 0 {
answer["refused"] = refused
}
return answer, nil
}
// halfReader is a Source that also holds the catalog half of each followed branch (firstview.Views.Halves): mesh_wait
// checks an event name's emitter against it (design 34 every-event §14).
type halfReader interface {
Halves() (map[string]meshcatalogclient.Half, error)
}
// knownEmitters is what the catalog half says emits what: each seat's declared events and each module's, and the
// source a refusal names, with its revisions.
type knownEmitters struct {
seats, modules map[string][]string
source string
}
// emitters is the catalog half's emitters, or why they are not checked: the half not yet read (its first view,
// mesh_first_views), or no half held at all.
func (s *Surface) emitters() (known knownEmitters, unchecked string) {
hr, ok := s.src.(halfReader)
if !ok {
return known, "this tool runner holds no catalog half"
}
halves, err := hr.Halves()
if err != nil {
return known, "the catalog half is not yet read (" + err.Error() + "); its first view is in mesh_first_views"
}
seatEvents := map[string][]string{} // a seat's events, as its definition in the seat list declares them
for _, st := range catalogue.Seats() {
seatEvents[st.Name] = st.Emits
}
known.seats, known.modules = map[string][]string{}, map[string][]string{}
branches := make([]string, 0, len(halves))
for b := range halves {
branches = append(branches, b)
}
slices.Sort(branches)
var revisions []string
for _, b := range branches {
half := halves[b]
revisions = append(revisions, fmt.Sprintf("revision %d of %s", half.Revision.Revision, b))
for _, d := range half.Revision.Definitions {
known.seats[d.Seat] = seatEvents[d.Seat]
}
for m, entry := range half.Entries {
if _, taken := known.modules[m]; !taken {
known.modules[m] = eventsDeclared(entry.Manifest)
}
}
}
known.source = "the catalog half (mesh-catalog.revision, " + strings.Join(revisions, "; ") + ")"
return known, ""
}
// eventsDeclared is the events a module's manifest declares: its `emits` and its streams' `events`.
func eventsDeclared(manifest map[string]any) []string {
var out []string
add := func(list any) {
items, _ := list.([]any)
for _, x := range items {
if e, ok := x.(string); ok {
out = append(out, e)
}
}
}
add(manifest["emits"])
streams, _ := manifest["streams"].([]any)
for _, st := range streams {
if m, ok := st.(map[string]any); ok {
add(m["events"])
}
}
return out
}
// declares says whether one of the declared events is the event a name gives, or one its name is followed by.
func declares(declared []string, event string) bool {
for _, d := range declared {
if buscompose.Overlaps(d, event) || buscompose.Overlaps(d, event+".>") {
return true
}
}
return false
}
// waitNames is the event names a call gives: a list of names, or one name.
func waitNames(given any) ([]string, error) {
var names []string
switch v := given.(type) {
case string:
names = []string{v}
case []any:
for _, x := range v {
name, ok := x.(string)
if !ok {
return nil, fmt.Errorf("mesh_wait's subjects are event names, such as mesh-delivery-view.delivered; %v is not one", x)
}
names = append(names, name)
}
}
for i := range names {
names[i] = strings.TrimSpace(names[i])
}
if len(names) == 0 {
return nil, fmt.Errorf("mesh_wait names at least one event in `subjects`, such as mesh-delivery-view.delivered")
}
return names, nil
}
// waitTimeout is the call's timeout: WaitDefault when absent, refused beyond WaitLongest or below a second.
func waitTimeout(given any) (time.Duration, error) {
if given == nil {
return WaitDefault, nil
}
longest := int(WaitLongest.Seconds())
seconds, ok := given.(float64)
if !ok {
if n, isInt := given.(int); isInt {
seconds, ok = float64(n), true
}
}
if !ok || math.IsNaN(seconds) || seconds < 1 || seconds > float64(longest) {
return 0, fmt.Errorf("mesh_wait's timeout is a number of seconds from 1 to %d (novox/hq ADR 0367 §1), not %v", longest, given)
}
return time.Duration(seconds * float64(time.Second)), nil
}
// bodyMatches says whether every field the filter names equals, in the event's body, the value given.
func bodyMatches(body json.RawMessage, filter map[string]any) bool {
if len(filter) == 0 {
return true
}
var fields map[string]json.RawMessage
if json.Unmarshal(body, &fields) != nil {
return false
}
for k, want := range filter {
got, has := fields[k]
if !has {
return false
}
wanted, _ := json.Marshal(want)
if !sameJSON(got, wanted) {
return false
}
}
return true
}
// sameJSON compares two JSON values as values: 367 and 367.0 are one number, key order does not matter.
func sameJSON(a, b []byte) bool {
var x, y any
da, db := json.NewDecoder(bytes.NewReader(a)), json.NewDecoder(bytes.NewReader(b))
if da.Decode(&x) != nil || db.Decode(&y) != nil {
return false
}
ja, _ := json.Marshal(x)
jb, _ := json.Marshal(y)
return bytes.Equal(ja, jb)
}
+689
View File
@@ -0,0 +1,689 @@
package console
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"reflect"
"sort"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
opseat "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// waitingRuntime is the tool runner on a machine, issued no waits (novox/hq ADR 0670 §2): the surface over its
// connection, and the mesh to publish on.
func waitingRuntime(t *testing.T) (*Surface, *mt.Mesh) {
t.Helper()
mesh := mt.New(t)
mesh.Issue(t, mt.MembershipOf("node-tools", "desk", false, nil))
conn := connect(t, "node-tools", "desk")
mt.Until(t, func() error {
if conn.Membership("node-tools") == nil {
return os.ErrNotExist
}
return nil
})
return NewSurface(conn, busSource{conn}, "desk.node-tools"), mesh
}
// halfSource is a Source that also holds a catalog half, as the tool runner's first views do.
type halfSource struct {
Source
halves map[string]meshcatalogclient.Half
}
func (h halfSource) Halves() (map[string]meshcatalogclient.Half, error) { return h.halves, nil }
// listening waits until the surface's connection holds n more subscriptions than before: a wait has subscribed.
func listening(t *testing.T, s *Surface, before, n int) {
t.Helper()
mt.Until(t, func() error {
if s.conn.Subscriptions() < before+n {
return os.ErrNotExist
}
return nil
})
}
// callWait is one tools/call of mesh_wait, answered as an agent host reads it.
func callWait(s *Surface, args map[string]any) (text string, isError bool) {
raw, _ := json.Marshal(map[string]any{"name": verbWait, "arguments": args})
r := s.Handle(Request{JSONRPC: "2.0", ID: json.RawMessage("1"), Method: "tools/call", Params: raw})
result, _ := r.Result.(map[string]any)
content, _ := result["content"].([]map[string]any)
var parts []string
for _, c := range content {
parts = append(parts, c["text"].(string))
}
isErr, _ := result["isError"].(bool)
return strings.Join(parts, "\n"), isErr
}
// novox/hq ADR 0367 §1, ADR 0670 §1 (design 34 every-event §13, §15, §19): mesh_wait answers the first event that
// matches — by its name and its filter — with no membership naming it: its whole name, the namespace it arrived on, its
// body, when it arrived and when it was published, within a second of its publication, and ignores the others.
func TestMeshWaitAnswersTheFirstMatchingEvent(t *testing.T) {
s, mesh := waitingRuntime(t)
type answer struct {
text string
isErr bool
at time.Time
}
type heardAnswer struct {
Heard struct {
Subject string `json:"subject"`
From string `json:"from"`
Body map[string]any `json:"body"`
At time.Time `json:"at"`
Published *string `json:"published"`
} `json:"heard"`
Read string `json:"read"`
}
nc, err := nats.Connect(mt.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
done := make(chan answer, 1)
before := s.conn.Subscriptions()
go func() {
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-delivery-view.delivered"},
"filter": map[string]any{"module": "netcheck"}, "timeout": 20})
done <- answer{text, isErr, time.Now()}
}()
// Once it listens: another module's delivery, another event, then the one it waits for, with its own time.
listening(t, s, before, 6)
mesh.Emit(t, "mesh.mod.mesh-delivery-view.event.delivered", map[string]any{"module": "grafana"})
mesh.Emit(t, "mesh.mod.mesh-delivery-view.event.planned", map[string]any{"module": "netcheck"})
msg := nats.NewMsg("mesh.mod.mesh-delivery-view.event.delivered")
msg.Header.Set("x-time", "2026-11-12T03:00:00.5Z")
msg.Data = []byte(`{"module":"netcheck"}`)
published := time.Now()
if err := nc.PublishMsg(msg); err != nil {
t.Fatal(err)
}
var got answer
select {
case got = <-done:
case <-time.After(10 * time.Second):
t.Fatal("mesh_wait did not answer the matching event")
}
if got.isErr {
t.Fatalf("mesh_wait failed: %s", got.text)
}
if got.at.Sub(published) > time.Second {
t.Errorf("answered %s after the event was published, want within a second", got.at.Sub(published))
}
var heard heardAnswer
if err := json.Unmarshal([]byte(got.text), &heard); err != nil {
t.Fatalf("not an answer: %s", got.text)
}
if h := heard.Heard; h.Subject != "mesh-delivery-view.delivered" || h.From != "module" || h.Body["module"] != "netcheck" ||
h.At.IsZero() || h.Published == nil || *h.Published != "2026-11-12T03:00:00.5Z" || !strings.Contains(heard.Read, "only wakes you") {
t.Errorf("answered %s, want netcheck's delivery from a module with its times", got.text)
}
// And a seat's `decided` of one asker, heard by `operator-channel.decided`: its whole name, from the seat, with no
// time of its own, and the answer read only with claude_code_answer.
before = s.conn.Subscriptions()
go func() {
text, isErr := callWait(s, map[string]any{"subjects": "operator-channel.decided", "timeout": 20})
done <- answer{text, isErr, time.Now()}
}()
listening(t, s, before, 6)
// A seat's event, said by the router on core NATS here: no stream of the test's bus carries the seat.
if err := connect(t, "messenger", "").Say("mesh.seat.operator-channel.event.decided.claude-code", []byte(`{"ask":"a1"}`)); err != nil {
t.Fatal(err)
}
select {
case got = <-done:
heard = heardAnswer{}
_ = json.Unmarshal([]byte(got.text), &heard)
if h := heard.Heard; got.isErr || h.Subject != "operator-channel.decided.claude-code" || h.From != "seat" ||
h.Published != nil || !strings.Contains(heard.Read, "claude_code_answer") {
t.Errorf("the asker's decided: %s", got.text)
}
case <-time.After(10 * time.Second):
t.Fatal("mesh_wait did not answer the asker's decided")
}
}
// It says it timed out at its timeout, and leaves no subscription behind (ADR 0367 §1, §2).
func TestMeshWaitSaysItTimedOutAndKeepsNothing(t *testing.T) {
s, _ := waitingRuntime(t)
before := s.conn.Subscriptions()
start := time.Now()
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": 1})
if took := time.Since(start); took < time.Second || took > 3*time.Second {
t.Errorf("a one-second wait took %s", took)
}
// Before the catalog half is read, names are not checked, and the answer says so (design 34 every-event §14).
if isErr || !strings.Contains(text, `"timed-out": true`) || !strings.Contains(text, `"unchecked"`) {
t.Errorf("a wait that heard nothing: %s", text)
}
if after := s.conn.Subscriptions(); after != before {
t.Errorf("%d subscriptions before the wait, %d after it", before, after)
}
}
// Its timeout is four minutes when none is given and ten at most: a longer one, or none at all, is refused
// with the bound, never shortened in silence.
func TestMeshWaitsTimeoutIsBounded(t *testing.T) {
s, _ := waitingRuntime(t)
if WaitDefault != 4*time.Minute || WaitLongest != 10*time.Minute {
t.Errorf("the bounds are %s and %s, want 4m and 10m (ADR 0367 §1)", WaitDefault, WaitLongest)
}
for _, timeout := range []any{601, 0, -5, "soon"} {
text, isErr := callWait(s, map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "timeout": timeout})
if !isErr || !strings.Contains(text, "600") {
t.Errorf("a timeout of %v: %s", timeout, text)
}
}
}
// novox/hq ADR 0670 §1, §2 (design 34 every-event §13, §16, §19): mesh_wait refuses loudly and never loops: a name with
// a wildcard, of one token, more than 16 names, none, or a from of another word is refused at once naming the rule; a
// wait whose every subscription the bus refuses answers after its timeout, in words naming no wait grant.
func TestMeshWaitRefusesLoudlyAndNeverLoops(t *testing.T) {
s, _ := waitingRuntime(t)
var seventeen []any
for i := 0; i < 17; i++ {
seventeen = append(seventeen, fmt.Sprintf("mesh-knowledge.reviewed%d", i))
}
for _, c := range []struct {
args map[string]any
says string
}{
{map[string]any{"subjects": []any{"mesh-knowledge.*"}}, "[A-Za-z0-9_-]+"},
{map[string]any{"subjects": []any{"operator-channel.decided.>"}}, "[A-Za-z0-9_-]+"},
{map[string]any{"subjects": []any{"mesh-knowledge"}}, "fewer than two tokens"},
{map[string]any{"subjects": seventeen}, "at most 16"},
{map[string]any{"subjects": []any{}}, "at least one event"},
{map[string]any{"subjects": []any{"mesh-knowledge.reviewed"}, "from": "bench"}, "seat, module or stream"},
} {
c.args["timeout"] = 1
start := time.Now()
text, isErr := callWait(s, c.args)
if !isErr || !strings.Contains(text, c.says) || strings.Contains(text, "manifest") {
t.Errorf("%v: %s, want refused saying %q", c.args, text, c.says)
}
if took := time.Since(start); took > 500*time.Millisecond {
t.Errorf("%v was refused after %s, want at once", c.args, took)
}
}
t.Run("the catalog half", func(t *testing.T) {
t.Parallel()
// E §14: against the catalog half, a misspelt emitter is refused naming the half and its revision, a known emitter's
// undeclared event is waited on and its timeout says so, and a declared one says nothing of it.
half := NewSurface(s.conn, halfSource{busSource{s.conn}, map[string]meshcatalogclient.Half{"main": {
Revision: catalogseat.CatalogRevision{Revision: 7, Definitions: []catalogseat.CatalogRevisionDefinitionsItem{{Seat: "mesh-catalog"}}},
Entries: map[string]catalogseat.CatalogEntry{"mesh-delivery-view": {Module: "mesh-delivery-view",
Manifest: map[string]any{"emits": []any{"delivered"}}}},
}}}, "desk.node-tools")
for _, c := range []struct {
name, says string
refused bool
}{
{"mesh-delivry-view.delivered", "mesh-delivry-view is neither a seat nor a module of the catalog half (mesh-catalog.revision, revision 7 of main)", true},
{"mesh-delivery-view.planned", "no declaration names mesh-delivery-view.planned", false},
{"mesh-catalog.catalog-revision", "", false},
} {
text, isErr := callWait(half, map[string]any{"subjects": []any{c.name}, "timeout": 1})
if isErr != c.refused || c.says != "" && !strings.Contains(text, c.says) ||
c.says == "" && (strings.Contains(text, "undeclared") || strings.Contains(text, "unchecked")) {
t.Errorf("%s against the catalog half: %s", c.name, text)
}
}
})
t.Run("a bus that refuses every subscription", func(t *testing.T) {
t.Parallel()
// The bus refusing every subscription of the call: answered after the timeout, in §16's words.
b, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, NoLog: true, NoSigs: true,
Users: []*server.User{{Username: "desk.node-tools", Password: "pw", Permissions: &server.Permissions{
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX.desk.node-tools.>", "mesh.assignment.desk.*"}},
}}}})
if err != nil {
t.Fatal(err)
}
go b.Start()
if !b.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(b.Shutdown)
conn, err := bus.Connect(bus.Credential{URL: b.ClientURL(), Module: "node-tools", Node: "desk", User: "desk.node-tools", Password: "pw"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
start := time.Now()
text, isErr := callWait(NewSurface(conn, nil, "desk.node-tools"), map[string]any{"subjects": []any{"mesh-delivery-view.delivered"}, "timeout": 1})
if took := time.Since(start); took < time.Second || took > 3*time.Second {
t.Errorf("a wait the bus refused whole answered after %s, want its timeout of 1s", took)
}
if !isErr || !strings.Contains(text, "the bus refused this account a subscription on mesh.seat.mesh-delivery-view.event.delivered") ||
!strings.Contains(text, "novox/hq ADR 0670 §1") || strings.Contains(text, "wait grant") || strings.Contains(text, "0367 §2") {
t.Errorf("a wait the bus refused whole: %s", text)
}
})
// Design 34 every-event §37: a wait whose subjects overlap a marked event's, or a stream's with a `grant`, runs only on the
// operator's warrant for that one wait; with none it subscribes nothing, asks once and answers at its timeout or the choice.
// Each group of rows has a runtime of its own, so the waits of the groups run side by side (A-311: the one-second floor of
// a wait's timeout stands, and no clock is injected).
for name, group := range map[string]func(*testing.T, *Surface, *mt.Mesh){
"no warrant": markedWaitsWithNoWarrant, "a warrant": markedWaitsOnAWarrant, "the catalog half unread": markedWaitsBeforeTheHalfIsRead,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
s, mesh := waitingRuntime(t)
group(t, s, mesh)
})
}
}
// markedWaitsWithNoWarrant are §37's rows of a wait with no warrant, against a gate on the test bus, the router's records written there as the router writes them.
func markedWaitsWithNoWarrant(t *testing.T, s *Surface, mesh *mt.Mesh) {
gw := newGatedWaits(t, s, mesh, "src2:runner")
g := gw.surface
took := func(args map[string]any) (string, bool, time.Duration) {
start := time.Now()
text, isErr := callWait(g, args)
return text, isErr, time.Since(start)
}
// A fixture module's marked event, no warrant: nothing of it subscribed (only its ask's decided), asked once, §12's
// words at its timeout; the same wait again asks nothing new.
before := g.conn.Subscriptions()
go func() {
listening(t, g, before, 1)
mesh.Emit(t, "mesh.mod.mail.event.message", map[string]any{"n": 1})
}()
text, isErr, d := took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1})
if !isErr || !strings.Contains(text, "this wait needs the operator's approval. Asked on telegram") || gw.asks() != 1 ||
d < time.Second || strings.Contains(text, "heard") {
t.Errorf("a marked event's wait, no warrant: %s after %s, %d asks", text, d, gw.asks())
}
if text, isErr, _ = took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
!strings.Contains(text, "still waiting") || gw.asks() != 1 {
t.Errorf("the same marked wait again: %s, %d asks", text, gw.asks())
}
// A stream with a `grant` is marked whole; a name with the node after a marked event's name is marked.
for i, name := range []string{"mail.inbox", "mail.message.desk"} {
if text, isErr, _ := took(map[string]any{"subjects": []any{name}, "timeout": 1}); !isErr || !strings.Contains(text, "Not run") || gw.asks() != 2+i {
t.Errorf("%s: %s, %d asks", name, text, gw.asks())
}
}
// The same names from a seat ask nothing.
if text, isErr, _ := took(map[string]any{"subjects": []any{"mail.message"}, "from": "seat", "timeout": 1}); isErr ||
!strings.Contains(text, `"timed-out": true`) || gw.asks() != 3 {
t.Errorf("from a seat: %s, %d asks", text, gw.asks())
}
}
// markedWaitsOnAWarrant are §37's rows of a wait chosen while it waits and of one on a warrant.
func markedWaitsOnAWarrant(t *testing.T, s *Surface, mesh *mt.Mesh) {
gw := newGatedWaits(t, s, mesh, "src2:runner")
g := gw.surface
took := func(args map[string]any) (string, bool, time.Duration) {
start := time.Now()
text, isErr := callWait(g, args)
return text, isErr, time.Since(start)
}
// The most timeout a wait takes, its ask chosen while it waits: answered at the choice; the same call made then
// subscribes and hears the event published after it, answered with its approval; a warrant for other names, another
// filter or another timeout is refused, asked anew.
long := map[string]any{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 2}, "timeout": 600}
done := make(chan string, 1)
go func() { text, _, d := took(long); done <- fmt.Sprintf("%s after %s", text, d) }()
mt.Until(t, func() error {
if gw.asks() < 1 {
return os.ErrNotExist
}
return nil
})
gw.approve(t)
select {
case got := <-done:
if !strings.Contains(got, "make this wait again within 30 minutes") {
t.Errorf("a long wait chosen while it waits: %s", got)
}
case <-time.After(10 * time.Second):
t.Fatal("a long marked wait was not answered at the choice")
}
before := g.conn.Subscriptions()
go func() {
listening(t, g, before, 6)
mesh.Emit(t, "mesh.mod.mail.event.message", map[string]any{"n": 2})
}()
if text, isErr, _ := took(long); isErr || !strings.Contains(text, `"approval": "run on the operator's approval`) ||
!strings.Contains(text, `"subject": "mail.message"`) {
t.Errorf("the same wait on its warrant: %s", text)
}
for _, other := range []map[string]any{
{"subjects": []any{"mail.message", "mail.inbox"}, "filter": map[string]any{"n": 2}, "timeout": 1},
{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 3}, "timeout": 1},
{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 2}, "timeout": 2},
} {
asked := gw.asks()
if text, isErr, _ := took(other); !isErr || !strings.Contains(text, "Not run") || gw.asks() != asked+1 {
t.Errorf("another wait than the warrant's %v: %s", other, text)
}
}
}
// markedWaitsBeforeTheHalfIsRead are §37's rows of a wait before the catalog half is read, and of one whose runner build is unknown.
func markedWaitsBeforeTheHalfIsRead(t *testing.T, s *Surface, mesh *mt.Mesh) {
gw := newGatedWaits(t, s, mesh, "src2:runner")
g := gw.surface
took := func(args map[string]any) (string, bool, time.Duration) {
start := time.Now()
text, isErr := callWait(g, args)
return text, isErr, time.Since(start)
}
// Before the catalog half is read nothing is asked: judged when its first view comes within the timeout, refused at
// the timeout when it does not.
gw.cat.mu.Lock()
gw.cat.ready = errors.New("no first view yet")
gw.cat.mu.Unlock()
asked := gw.asks()
if text, isErr, d := took(map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
!strings.Contains(text, "the marks are not yet read") || gw.asks() != asked || d < time.Second {
t.Errorf("before the catalog half is read: %s after %s", text, d)
}
gw.cat.came = make(chan chan struct{})
go func() { close(<-gw.cat.came) }()
if text, isErr, _ := took(map[string]any{"subjects": []any{"mail.message"}, "filter": map[string]any{"n": 9}, "timeout": 1}); !isErr ||
!strings.Contains(text, "this wait needs the operator's approval") || gw.asks() != asked+1 {
t.Errorf("the catalog half read within the timeout: %s", text)
}
// While the runner's own fingerprint is unknown a marked wait is refused, saying so.
unknown := newGatedWaits(t, s, mesh, "")
if text, isErr := callWait(unknown.surface, map[string]any{"subjects": []any{"mail.message"}, "timeout": 1}); !isErr ||
!strings.Contains(text, "own fingerprint is unknown") || unknown.asks() != 0 {
t.Errorf("the runner's fingerprint unknown: %s", text)
}
}
// gatedWaits is the surface with the runner's approval gate: the fixture module mail marks its event message and declares
// the stream inbox with a grant; the router takes each ask as it is published, its records on the test bus.
type gatedWaits struct {
surface *Surface
cat *waitCatalog
records nats.KeyValue
mu sync.Mutex
asked []opseat.Ask
say *bus.Conn
}
func newGatedWaits(t *testing.T, s *Surface, mesh *mt.Mesh, build string) *gatedWaits {
gw := &gatedWaits{cat: &waitCatalog{entries: []catalogseat.CatalogEntry{{Module: "mail", Fp: "src2:mail", Manifest: map[string]any{
"emits": []any{"message"}, "needs-approval": []any{"message"},
"streams": []any{map[string]any{"name": "inbox", "events": []any{"inbox.*"}, "grant": map[string]any{"subscribe": []any{"x"}}}}}}}},
records: mesh.Bucket(t, "operator-channel-asks"), say: connect(t, "messenger", "")}
state := mt.NewState()
gate := approval.New(approval.Deps{Node: "desk", Runner: "node-tools", Build: build, Catalog: gw.cat, Wait: gw.cat.wait,
State: state, Holder: func(string) (bool, error) { return true, nil },
Hear: func(ctx context.Context, subjects []string, within time.Duration, match func(json.RawMessage) bool) (bool, error) {
_, ok, _, err := s.conn.HearFirst(ctx, subjects, within, match)
return ok, err
},
Channel: &operatorchannelclient.GateChannel{State: state, KeyPrefix: "desk.", Publish: func(module, subject string, body any, id string) error {
a, ok := body.(opseat.Ask)
if subject != opseat.AcceptSubjectAsk(module) || !ok {
return nil
}
gw.mu.Lock()
gw.asked = append(gw.asked, a)
gw.mu.Unlock()
gw.write(t, opseat.AsksByAskerIDValue{Asker: module, Key: module + "." + a.ID, State: "open", Headline: a.Headline,
Expires: a.Expires, Opened: operatorchannelclient.FormatTime(time.Now()), OfferedOn: []string{"telegram"}})
return nil
},
ReadRecord: func(module, id string) (*stdio.StateEntry, error) {
e, err := gw.records.Get(module + "." + id)
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, nil
} else if err != nil {
return nil, err
}
return &stdio.StateEntry{Key: e.Key(), Value: e.Value(), Revision: e.Revision()}, nil
}}})
gw.surface = NewSurface(s.conn, busSource{s.conn}, "desk.node-tools")
gw.surface.Approvals = gateApprovals{gate}
return gw
}
func (gw *gatedWaits) write(t *testing.T, rec opseat.AsksByAskerIDValue) {
raw, _ := json.Marshal(rec)
if _, err := gw.records.Put(rec.Key, raw); err != nil {
t.Error(err)
}
}
func (gw *gatedWaits) asks() int {
gw.mu.Lock()
defer gw.mu.Unlock()
return len(gw.asked)
}
// approve writes the router's record of the newest ask as a person's Approve, and says its decided.
func (gw *gatedWaits) approve(t *testing.T) {
gw.mu.Lock()
a := gw.asked[len(gw.asked)-1]
gw.mu.Unlock()
gw.write(t, opseat.AsksByAskerIDValue{Asker: "node-tools", Key: "node-tools." + a.ID, State: "chosen", Expires: a.Expires,
OfferedOn: []string{"telegram"}, Warrant: &opseat.Warrant{Ask: a.ID, Asker: "node-tools", Outcome: operatorchannelclient.OutcomeChosen,
Option: approval.OptionApprove, Level: operatorchannelclient.Approve, At: operatorchannelclient.FormatTime(time.Now()),
AskDigest: operatorchannelclient.Digest(a), Channel: "telegram",
By: &opseat.Person{Who: operatorchannelclient.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}})
if err := gw.say.Say(opseat.SubjectDecided("node-tools"), []byte(`{"ask":"`+a.ID+`"}`)); err != nil {
t.Fatal(err)
}
}
// gateApprovals is the gate as the surface holds it: mesh_approvals' reads unused here, mesh_wait's gate.
type gateApprovals struct{ *approval.Gate }
func (gateApprovals) Fill() (uint64, int64, error) { return 0, 0, nil }
// waitCatalog is a catalog half as a fixture: its entries, read or not; with came set, its first view comes once a wait
// waits for it (waiting is told on it, then it closes).
type waitCatalog struct {
entries []catalogseat.CatalogEntry
mu sync.Mutex
ready error
came chan chan struct{}
}
func (c *waitCatalog) Ready() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.ready
}
func (c *waitCatalog) wait(within time.Duration) error {
if c.Ready() == nil {
return nil
}
if c.came != nil {
view := make(chan struct{})
c.came <- view
<-view
c.mu.Lock()
c.ready = nil
c.mu.Unlock()
return nil
}
<-time.After(within) // the bound of the runner's own wait for the first view
return c.Ready()
}
func (c *waitCatalog) Revision() string { return "7" }
func (c *waitCatalog) Served(string) (approval.Entry, bool, string) {
return approval.Entry{}, false, "not-found"
}
func (c *waitCatalog) Newest(string) (approval.Entry, bool) {
return approval.Entry{}, false
}
func (c *waitCatalog) Definitions() map[string]seatdef.Definition { return nil }
func (c *waitCatalog) Entries() []approval.Entry {
var out []approval.Entry
for _, e := range c.entries {
out = append(out, runnerapproval.Entry(e))
}
return out
}
// The mesh MCP server announces seven tools (ADR 0367 v13): ADR 0195's five, mesh_runtimes and mesh_wait, its timeout bounded.
func TestMeshWaitIsAnnouncedAmongSeven(t *testing.T) {
var names []string
var wait map[string]any
for _, tool := range discovery() {
names = append(names, tool["name"].(string))
if tool["name"] == verbWait {
wait = tool
}
}
sort.Strings(names)
want := []string{"mesh_approvals", "mesh_call", "mesh_describe", "mesh_first_view_again", "mesh_first_views", "mesh_machine", "mesh_overview", "mesh_runtimes", "mesh_search", "mesh_wait"}
if !reflect.DeepEqual(names, want) {
t.Errorf("announced %v, want %v", names, want)
}
if wait == nil || !isDiscovery(verbWait) {
t.Fatal("mesh_wait is not announced")
}
schema, _ := json.Marshal(wait["inputSchema"])
for _, word := range []string{`"subjects"`, `"filter"`, `"timeout"`} {
if !strings.Contains(string(schema), word) {
t.Errorf("mesh_wait's schema has no %s: %s", word, schema)
}
}
// Design 46 7-approval-mark §18: mesh_approvals lists, and records a hand act before it cancels; refused without a record.
gate := &fakeApprovals{}
var recorded []string
s := &Surface{who: "n1.node-tools", Approvals: gate, Record: func(a map[string]any) error {
if a["why"] == "unrecordable" {
return errors.New("mesh-hand-acts does not answer")
}
recorded = append(recorded, a["what"].(string))
return nil
}}
if out := s.discover(context.Background(), verbApprovals, map[string]any{}); out["isError"] == true || !strings.Contains(fmtAnswer(out), "every machine") {
t.Errorf("the list %v", out)
}
if out := s.discover(context.Background(), verbApprovals, map[string]any{"cancel": "a1", "why": "no longer needed"}); out["isError"] == true ||
gate.cancelled != "a1" || len(recorded) != 1 || recorded[0] != "cancelled ask a1" {
t.Errorf("a cancel %v, recorded %v", out, recorded)
}
if out := s.discover(context.Background(), verbApprovals, map[string]any{"cancel": "a2", "why": "unrecordable"}); out["isError"] != true || gate.cancelled != "a1" {
t.Errorf("a cancel without a record %v", out)
}
}
func fmtAnswer(out map[string]any) string { raw, _ := json.Marshal(out); return string(raw) }
// fakeApprovals is a gate as mesh_approvals reads it.
type fakeApprovals struct {
cancelled string
// marked is the one `<module>.<tool>` its Describe says is marked.
marked string
}
func (f *fakeApprovals) List() ([]approval.Listed, error) {
return []approval.Listed{{Ask: "a1", State: "asked"}}, nil
}
func (f *fakeApprovals) Read(id string) (approval.Whole, error) { return approval.Whole{}, nil }
func (f *fakeApprovals) Applied() []approval.Marks { return nil }
func (f *fakeApprovals) Describe(module, _, name string) (bool, string) {
return f.marked != "" && f.marked == module+"."+name, "entry " + module
}
func (f *fakeApprovals) Again(string) error { return nil }
func (f *fakeApprovals) Cancel(id string) error { f.cancelled = id; return nil }
func (f *fakeApprovals) Fill() (uint64, int64, error) { return 10, 64 << 20, nil }
// A client that hangs up or cancels ends its wait at once: the subscription is closed and nothing is left
// waiting for the timeout (the review of issue 745). Over the console's own HTTP, as an agent host calls it.
func TestMeshWaitEndsWhenItsClientHangsUp(t *testing.T) {
s, _ := waitingRuntime(t)
up, err := Serve(s, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = up.Close() })
before := s.conn.Subscriptions()
ctx, cancel := context.WithCancel(context.Background())
body := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"mesh_wait",` +
`"arguments":{"subjects":["mesh-knowledge.reviewed"],"timeout":600}}}`
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, "http://"+up.Address+"/mcp", strings.NewReader(body))
req.Header.Set("content-type", "application/json")
go func() { _, _ = http.DefaultClient.Do(req) }()
mt.Until(t, func() error {
if s.conn.Subscriptions() <= before {
return os.ErrNotExist
}
return nil
})
cancel()
start := time.Now()
for s.conn.Subscriptions() > before {
if time.Since(start) > 3*time.Second {
t.Fatalf("%d subscriptions 3s after the client hung up, %d before the wait", s.conn.Subscriptions(), before)
}
time.Sleep(20 * time.Millisecond)
}
}
// The tool runner stopping ends every wait in flight at once (the second review of issue 745): Close cancels
// the requests' context rather than waiting up to ten minutes for a wait to time out.
func TestMeshWaitEndsWhenTheConsoleCloses(t *testing.T) {
s, _ := waitingRuntime(t)
up, err := Serve(s, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
before := s.conn.Subscriptions()
body := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"mesh_wait",` +
`"arguments":{"subjects":["mesh-knowledge.reviewed"],"timeout":600}}}`
go func() { _, _ = http.Post("http://"+up.Address+"/mcp", "application/json", strings.NewReader(body)) }()
mt.Until(t, func() error {
if s.conn.Subscriptions() <= before {
return os.ErrNotExist
}
return nil
})
closed := make(chan error, 1)
start := time.Now()
go func() { closed <- up.Close() }()
select {
case <-closed:
case <-time.After(5 * time.Second):
t.Fatal("closing the console waited on a mesh_wait in flight")
}
if took := time.Since(start); took > 3*time.Second {
t.Errorf("closing took %s", took)
}
mt.Until(t, func() error {
if s.conn.Subscriptions() > before {
return os.ErrExist
}
return nil
})
}
+196
View File
@@ -0,0 +1,196 @@
package firstview
import (
"context"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/seatdef"
runnerapproval "github.com/novox/mesh-tools/node-tools/internal/approval"
)
// Findings is the runner's held finding writer, the one the gate writes its findings through too (design 46 7-approval-mark
// §4, §16, §19); nil when it was not made.
func (v *Views) Findings() runnerapproval.HeldWriter {
if v.deps.Findings == nil {
return nil
}
return v.deps.Findings
}
// ApprovalCatalog is the catalog half as the gate reads its marks (design 46 7-approval-mark §5): the entries the branch
// follower holds, an entry by fingerprint asked of `mesh-catalog.entry` when no branch lists it, and the seat definitions
// the seat list holds (defs).
func (v *Views) ApprovalCatalog(defs func() map[string]seatdef.Definition) approval.Catalog {
return approvalCatalog{v: v, defs: defs}
}
type approvalCatalog struct {
v *Views
defs func() map[string]seatdef.Definition
}
func (c approvalCatalog) Ready() error {
if c.v.branches == nil {
return errNotFollowed
}
return c.v.branches.Ready()
}
func (c approvalCatalog) Revision() string {
if c.v.branches == nil {
return ""
}
for _, branch := range c.v.branches.Followed() {
if cat, ok := c.v.branches.Catalog(branch); ok {
if half, ok := cat.View(); ok {
return strconv.FormatInt(half.Revision.Revision, 10)
}
}
}
return ""
}
func (c approvalCatalog) Served(fingerprint string) (approval.Entry, bool, string) {
if c.v.branches == nil {
return approval.Entry{}, false, errNotFollowed.Error()
}
e, state := c.v.branches.Resolve(context.Background(), fingerprint)
return runnerapproval.Entry(e), state == meshcatalogclient.EntryAnswered, string(state)
}
func (c approvalCatalog) Newest(module string) (approval.Entry, bool) {
if c.v.branches == nil {
return approval.Entry{}, false
}
e, ok, err := c.v.branches.Entry(module)
return runnerapproval.Entry(e), ok && err == nil
}
func (c approvalCatalog) Definitions() map[string]seatdef.Definition {
if c.defs == nil {
return nil
}
return c.defs()
}
func (c approvalCatalog) Entries() []approval.Entry {
halves, err := c.v.Halves()
if err != nil {
return nil
}
var out []approval.Entry
for _, half := range halves {
for _, e := range half.Entries {
out = append(out, runnerapproval.Entry(e))
}
}
return out
}
type notFollowed struct{}
func (notFollowed) Error() string { return "the catalog half is not followed by this runner" }
var errNotFollowed error = notFollowed{}
// heardOne wakes every wait on the picture.
func (v *Views) heardOne() {
v.mu.Lock()
defer v.mu.Unlock()
if v.heard != nil {
close(v.heard)
}
v.heard = make(chan struct{})
}
// heardNext is closed at the next event heard.
func (v *Views) heardNext() <-chan struct{} {
v.mu.Lock()
defer v.mu.Unlock()
if v.heard == nil {
v.heard = make(chan struct{})
}
return v.heard
}
// WaitCatalog waits within the bound for the catalog half's first view of every watched branch (design 46 7-approval-mark
// §19), waking at each event the runner hears and at each first view answered; nil once it is read, else why not.
func (v *Views) WaitCatalog(within time.Duration) error {
c := approvalCatalog{v: v}
timer := time.NewTimer(within)
defer timer.Stop()
for {
next := v.heardNext()
err := c.Ready()
if err == nil {
return nil
}
var ready []<-chan struct{}
if v.branches != nil {
for _, branch := range v.branches.Followed() {
if cat, ok := v.branches.Catalog(branch); ok {
select {
case <-cat.Ready(): // read already: waits on nothing
default:
ready = append(ready, cat.Ready())
}
}
}
}
if !waitAny(next, timer.C, ready) {
return c.Ready()
}
}
}
// waitAny waits for next or one of ready, true, or for the bound, false.
func waitAny(next <-chan struct{}, bound <-chan time.Time, ready []<-chan struct{}) bool {
woke := make(chan struct{}, 1)
stop := make(chan struct{})
defer close(stop)
for _, r := range ready {
go func(r <-chan struct{}) {
select {
case <-r:
woke <- struct{}{}
case <-stop:
}
}(r)
}
select {
case <-next:
case <-woke:
case <-bound:
return false
}
return true
}
// OnSeatHolder calls f whenever the assigner's seat-holders view changes for the seat (lead answer A-157: the gate asks
// mesh-hand-acts.warranted again when the hand-act seat's holder is heard again).
func (v *Views) OnSeatHolder(seat string, f func()) {
v.mu.Lock()
defer v.mu.Unlock()
if v.onKey == nil {
v.onKey = map[string][]func(){}
}
k := "seat-holders " + seat
v.onKey[k] = append(v.onKey[k], f)
}
// heardKey calls the hearers of a key of an owner's state just changed: a seat's holders, under the seat's own key and
// its `<seat>.<node>` keys.
func (v *Views) heardKey(state, key string) {
seat, _, _ := strings.Cut(key, ".")
v.mu.Lock()
fs := append([]func(){}, v.onKey[state+" "+seat]...)
v.mu.Unlock()
for _, f := range fs {
go f()
}
}
+605
View File
@@ -0,0 +1,605 @@
// Package firstview is the tool runner's picture of what it reads of the assigner and the catalog half: the assigner's
// assignments and seat-holders (novox/hq ADR 0615 §1, §3; ADR 0621 §2; issue 1966 TLS-5), and which modules exist with the
// manifest each entry carries, followed per watched branch through meshcatalogclient's branch follower on this process's own
// consumer (design 47 G §196, §594, §614; lead answers A-5, A-27, A-103), never the indexer's state `modules` or `manifests`.
//
// **It never reads the owners' state.** At start it asks each owner's first-view tool until answered (the owner answers in a
// runtime of its own, never through this one: ADR 0621 §1), then follows the owner's views family, one event per changed key;
// a gap in the revisions, or an owner whose state was made again, asks the first view once more, said in the log. Past two
// minutes without an answer it writes a finding through its held finding writer (design 47 G A-27, A-28), which keeps it in its
// own state `findings` and says it; the answer clears it.
package firstview
import (
"context"
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/approval"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// AssignerSeat is the seat whose holder's views the runner follows: the holder is read from the catalog half's claims of the
// seat, never named in code (design 47 G §616, §428), and the names the console reads the two states by are the seat's
// (`<seat>.<state>`; the manifest's `consumes` names their views families by the seat, as `seat:<seat>.views.<state>`).
const AssignerSeat = meshmoduleassigner.Seat
const (
AssignmentsRead = AssignerSeat + ".assignments"
SeatHoldersRead = AssignerSeat + ".seat-holders"
)
// Spec is one first view the runner holds of the seat's holder: its first-view tool and the state.
type Spec struct{ Tool, State string }
// Specs are the two states asked of the holder's tool; the catalog half's views are adopted by the branch follower.
var Specs = []Spec{
{"assignments", "assignments"},
{"seat-holders", "seat-holders"},
}
// Name is the name a state is read by, `<seat>.<state>`: AssignmentsRead and SeatHoldersRead.
func (s Spec) Name() string { return AssignerSeat + "." + s.State }
// Views is the first views: the assigner's two, and the catalog half's per watched branch.
type Views struct {
logf func(string, ...any)
mu sync.Mutex
followers map[string]*views.Follower // by `<owner>.<state>`
byState map[string]*views.Follower // by the state an event names, the owners' views families
adopted map[string]*adoption // by `<owner>.<state>`, the views the branch follower adopted
unheardKeys map[string]bool // event keys said as heard by no view
hearers map[string][]*adoption // event key `<seat>.<event>` -> the adopted views that hear it
branches *meshcatalogclient.Branches
holder *holding // the module that claims the assigner seat, once the catalog half names it
ctx context.Context
cancel context.CancelFunc
wg sync.WaitGroup
deps Deps
// heard is closed and made again at every event heard, so a wait for the picture wakes on events, never on a timer.
heard chan struct{}
// onKey are the hearers of one key of an owner's state, by `<state> <key>` (OnSeatHolder).
onKey map[string][]func()
}
// FindingsState is the runner's own state the findings are written in.
const FindingsState = "findings"
// Deps is what the views reach the bus through: Start makes them from the runner's connection, a test from fakes.
type Deps struct {
// Module is the runner's own name, the observer of its findings.
Module string
// Ask calls an owner's tool, `<owner>.<tool>`.
Ask func(address string, body any) (json.RawMessage, error)
// Consume hands each event of the owners' views families to deliver, until the stop is called.
Consume func(deliver func(bus.Envelope) error) (stop func(), err error)
// Findings is the runner's held finding writer (views.HeldWriter): a follower's unanswered first view is written and cleared
// through it with the contract's Finding (design 47 G A-28). Nil says it in the log only.
Findings FindingWriter
// StartFindings starts that writer, and its stop ends it; nil starts none.
StartFindings func(ctx context.Context) (stop func(), err error)
// Build is the build this process was started with (process.OwnBuild, read once at the start): its fingerprint, or
// why it is unknown (design 47 G §205, §424).
Build process.Build
// Node is the machine the runner is on, named in its findings' subject.
Node string
// Bound and Wait are the follower's (zero: two minutes, five seconds).
Bound, Wait time.Duration
}
// FindingWriter is the writer the runner's findings go through: views.HeldWriter over the contract's Finding.
type FindingWriter interface {
Write(ctx context.Context, finding condseat.Finding) error
Clear(ctx context.Context, scope, id, kind, why string) error
}
// Start asks the first views, each until answered, and follows the owners' views families on the runner's own
// consumers. It returns at once: the picture fills as the owners answer.
func Start(parent context.Context, conn *bus.Conn, logf func(string, ...any)) (*Views, func()) {
return StartWith(parent, DepsOf(conn, logf), logf)
}
// DepsOf is the views' reach through the runner's own connection: its tool calls, its consumer and its held finding writer, whose
// state adapter, JetStream and node are this connection's, in this one place (design 47 G A-16, A-27).
func DepsOf(conn *bus.Conn, logf func(string, ...any)) Deps {
me := conn.Module()
d := Deps{Module: me, Node: conn.Node(), Build: process.OwnBuild(),
Ask: func(address string, body any) (json.RawMessage, error) {
got, err := conn.Ask(address, body, "")
return got.Result, err
},
Consume: func(deliver func(bus.Envelope) error) (func(), error) { return conn.ConsumeAs(me, deliver) },
}
js, err := conn.JetStream()
if err != nil {
logf("[mesh-tools] the runner's findings are not written (%v): the first views' findings are in the log only", err)
return d
}
w, err := views.NewHeldWriter(views.WriterConfig[condseat.Finding, condseat.FindingCleared, condseat.FindingsStated, condseat.FindingsAskedEvent]{
Asked: condseat.SubjectFindingsAsked, Module: me, Node: conn.Node(), State: findingsState{conn: conn, module: me}, JS: js, Logf: logf,
Kinds: append([]string{views.UnansweredKind}, approval.Kinds...)})
if err != nil {
logf("[mesh-tools] the runner's findings are not written (%v): the first views' findings are in the log only", err)
return d
}
d.Findings = w
d.StartFindings = func(ctx context.Context) (func(), error) { return w.Close, w.Start(ctx) }
return d
}
// findingsState is the runner's own state `findings` (views.FindingState) over its connection.
type findingsState struct {
conn *bus.Conn
module string
}
func (s findingsState) Get(key string) (*stdio.StateEntry, error) {
e, err := s.conn.StateGet(s.module, FindingsState, key)
if err != nil || e == nil {
return nil, err
}
return &stdio.StateEntry{Key: e.Key, Value: e.Value, Revision: e.Revision, At: e.At}, nil
}
func (s findingsState) Put(key string, value any) (uint64, error) {
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
return s.conn.StatePut(s.module, FindingsState, key, raw)
}
func (s findingsState) Delete(key string) error {
return s.conn.StateDelete(s.module, FindingsState, key)
}
func (s findingsState) Keys() ([]string, error) { return s.conn.StateKeys(s.module, FindingsState) }
// StartWith is Start over deps.
func StartWith(parent context.Context, d Deps, logf func(string, ...any)) (*Views, func()) {
ctx, cancel := context.WithCancel(parent)
v := &Views{logf: logf, ctx: ctx, cancel: cancel, followers: map[string]*views.Follower{}, byState: map[string]*views.Follower{},
adopted: map[string]*adoption{}, hearers: map[string][]*adoption{}, deps: d}
stopFindings := func() {}
if d.StartFindings != nil {
if stop, err := d.StartFindings(ctx); err != nil {
logf("[mesh-tools] the runner's finding writer does not start (%v): the first views' findings are in the log only", err)
} else {
stopFindings = stop
}
}
// The catalog half is followed by the shared branch follower on this process's own wiring (design 47 G §614, A-103): the
// registrations, the gate's source and a view per watched branch, each heard on this consumer.
branches, err := meshcatalogclient.NewBranches(wiring{v}, func(key string, body any) (json.RawMessage, error) { return d.Ask(key, body) })
if err != nil {
logf("[mesh-tools] the catalog half is not followed (%v): the console cannot say which modules exist", err)
} else {
branches.Logf = func(format string, args ...any) { logf("[mesh-tools] "+format, args...) }
branches.Changed = func(_, module string, entry json.RawMessage, deleted bool, _ time.Time) {
v.claimed(module, entry, deleted)
}
v.branches = branches
}
stop, err := d.Consume(v.hear)
if err != nil {
logf("[mesh-tools] the owners' views are not followed (%v): the pictures are the first views alone until the runner starts again", err)
stop = func() {}
}
return v, func() { stop(); cancel(); v.wg.Wait(); stopFindings() }
}
// hear applies one views event; a gap asks the first view once more. Every event heard wakes what waits on the picture
// (heardOne).
func (v *Views) hear(env bus.Envelope) error {
defer v.heardOne()
v.mu.Lock()
hs := v.hearersOf(env.Key)
v.mu.Unlock()
if len(hs) > 0 { // an event of the catalog half or the gate's source, heard by the views the branch follower adopted
for _, a := range hs {
if err := a.hear(v.ctx, env.Body); err != nil {
v.logf("[mesh-tools] %v", err)
}
}
return nil
}
ev, err := views.DecodeEvent(env.Body)
if err != nil {
v.unheard(env.Key)
return nil // not a views event: nothing to hold, nothing to retry
}
v.mu.Lock()
f := v.byState[ev.State]
v.mu.Unlock()
if f == nil || f.Owner != ev.Owner {
return nil
}
defer v.heardKey(ev.State, ev.Key)
if gap, why := f.Apply(ev); gap {
v.logf("[mesh-tools] a gap in %s's %s (%s): its first view %s is asked once more", f.Owner, f.State, why, f.Tool)
go func() { _ = f.Again(v.ctx, "a gap: "+why) }()
}
return nil
}
// hearersOf are the adopted views that hear the event key `<seat>.<event>` or, as its subject carries further tokens
// (`<event>.<branch>`, the mesh-catalog seat's catalog-revision), the key's shorter prefixes: the way mesh-sdk's views reader
// finds its adopted hears. The caller holds v.mu.
func (v *Views) hearersOf(key string) []*adoption {
for k := key; k != ""; {
if hs := v.hearers[k]; len(hs) > 0 {
return append([]*adoption(nil), hs...)
}
i := strings.LastIndex(k, ".")
if i <= 0 {
break
}
k = k[:i]
}
return nil
}
// unheard says once per key an event no adopted view and no owner's state hears, never dropping it silently.
func (v *Views) unheard(key string) {
v.mu.Lock()
if v.unheardKeys == nil {
v.unheardKeys = map[string]bool{}
}
seen := v.unheardKeys[key]
v.unheardKeys[key] = true
v.mu.Unlock()
if !seen {
v.logf("[mesh-tools] an event %s is heard by no view of this tool runner: it is dropped (said once per key)", key)
}
}
// finding writes or clears the runner's finding of a first view unanswered past its bound, through its writer (design 47 G A-28).
func (v *Views) finding(f condseat.Finding, tool string, raised bool) {
if v.deps.Findings == nil {
v.logf("[mesh-tools] %s", f.Summary)
return
}
var err error
if raised {
err = v.deps.Findings.Write(context.Background(), f)
} else {
err = v.deps.Findings.Clear(context.Background(), f.Scope, f.ID, f.Kind, "the owner answered")
}
if err != nil {
v.logf("[mesh-tools] the finding of the first view %s is not kept: %v", tool, err)
}
}
// State is the picture of the state spec names (`<owner>.<state>`), every current value by key; an error says the owner has
// not answered yet, so nothing is claimed of it.
func (v *Views) State(name string) (map[string][]byte, error) {
v.mu.Lock()
if state, ok := strings.CutPrefix(name, AssignerSeat+"."); ok { // the seat's name stands for its holder's
if v.holder == nil {
v.mu.Unlock()
return nil, fmt.Errorf("the catalog half names no holder of the seat %s yet, so %s has no first view", AssignerSeat, name)
}
name = v.holder.module + "." + state
}
f := v.followers[name]
v.mu.Unlock()
if f == nil {
return nil, fmt.Errorf("the tool runner holds no first view of %s", name)
}
select {
case <-f.Ready():
default:
return nil, fmt.Errorf("%s's first view %s has not been answered yet", f.Owner, f.Tool)
}
out := map[string][]byte{}
for k, raw := range f.Values() {
out[k] = raw
}
return out, nil
}
// Entry is one value of the picture, nil when the owner holds none.
func (v *Views) Entry(name, key string) ([]byte, error) {
all, err := v.State(name)
if err != nil {
return nil, err
}
return all[key], nil
}
// Status is each first view as it stands, by `<owner>.<state>`, sorted.
func (v *Views) Status() []views.Status {
var out []views.Status
for _, f := range v.all() {
out = append(out, f.Status())
}
sort.Slice(out, func(i, j int) bool { return out[i].Owner+out[i].State < out[j].Owner+out[j].State })
return out
}
// OwnBuild is the build the tool runner was started with: the fingerprint of the want its node-engine applied, or why it is
// unknown (`own build unknown: started without MESH_FINGERPRINT`).
func (v *Views) OwnBuild() process.Build { return v.deps.Build }
// Entries are the entries by fingerprint the catalog half's follower keeps, with their state and since when (design 47 G A-103).
func (v *Views) Entries() any {
if v.branches == nil {
return []meshcatalogclient.EntryStatus{}
}
return v.branches.EntryStatuses()
}
// all is every first view held now: the owners', then the adopted ones.
func (v *Views) all() []*views.Follower {
v.mu.Lock()
defer v.mu.Unlock()
out := make([]*views.Follower, 0, len(v.followers))
for _, f := range v.followers {
out = append(out, f)
}
return out
}
// Again asks every first view once more, by hand, for why.
func (v *Views) Again(ctx context.Context, why string) []error {
var errs []error
for _, f := range v.all() {
if err := f.Again(ctx, why); err != nil {
errs = append(errs, err)
}
}
return errs
}
// unansweredFinding is the finding of a first view unanswered past its bound, in the contract's form (design 47 G A-28): about the
// runner's module, a warning, its id the view's name escaped as a key's name is.
func unansweredFinding(module, node string, st views.Status, at time.Time) condseat.Finding {
return condseat.Finding{Kind: views.UnansweredKind, Scope: scopeModule, ID: keys.EscapeName(module + "." + st.Owner + "." + st.State),
Severity: condseat.FindingSeverityWarning, Source: "views", Observer: module,
Subject: condseat.Subject{Scope: scopeModule, ID: module, Machine: node},
Summary: fmt.Sprintf("%s has no first view of %s's %s: %s has not answered since %s (%s)", module, st.Owner, st.State, st.Tool,
st.Asked.Format(time.RFC3339), strings.TrimSpace(st.Error)),
Headline: fmt.Sprintf("%s waits for %s", module, st.Owner), At: at.UTC()}
}
// scopeModule is the scope of a finding about a module.
const scopeModule = "module"
// viewWaitsFinding is the finding of a view waited for past views.WaitsBound, in the contract's form: about the runner's module,
// a warning, its id the view's name escaped as a key's name is.
func viewWaitsFinding(module, node string, st views.Status, at time.Time) condseat.Finding {
return condseat.Finding{Kind: views.ViewWaitsKind, Scope: scopeModule, ID: keys.EscapeName(module + "." + st.Owner + "." + st.State),
Severity: condseat.FindingSeverityWarning, Source: "views", Observer: module,
Subject: condseat.Subject{Scope: scopeModule, ID: module, Machine: node},
Summary: fmt.Sprintf("%s waits for the view %s.%s at revision %s since %s and acts on nothing new until it holds it; it holds %s",
module, st.Owner, st.State, st.WaitsFor, st.WaitingSince.UTC().Format(time.RFC3339), orNone(st.View)),
Headline: fmt.Sprintf("%s waits for a view of %s", module, st.Owner), At: at.UTC()}
}
func orNone(s string) string {
if s == "" {
return "none"
}
return s
}
// adoption is a view the branch follower put on this process's consumer: its follower, the event it hears and its handler.
type adoption struct {
name, key string
f *views.Follower
hear func(context.Context, []byte) error
cancel context.CancelFunc
}
// wiring is the branch follower's wiring over this process's own consumer (meshcatalogclient.Wiring; design 47 G §614, A-103):
// an adopted view is held with the owners', takes its first view at once, hears its seat's event on the runner's consumer,
// writes its findings through the runner's held writer, and a dropped one clears them.
type wiring struct{ v *Views }
func (w wiring) Adopt(f *views.Follower, seat, event string, hear func(context.Context, []byte) error) error {
v := w.v
if f == nil || f.Owner == "" || f.State == "" || seat == "" || event == "" || hear == nil {
return fmt.Errorf("the tool runner adopts a view that names its owner, state, seat's event and handler")
}
name := f.Owner + "." + f.State
key := seat + "." + event
v.mu.Lock()
if v.followers[name] != nil {
v.mu.Unlock()
return fmt.Errorf("%s is followed already", name)
}
if f.Logf == nil {
f.Logf = v.logf
}
if f.Bound == 0 {
f.Bound = v.deps.Bound
}
if f.Wait == 0 {
f.Wait = v.deps.Wait
}
held := func() bool { v.mu.Lock(); defer v.mu.Unlock(); return v.followers[name] == f }
f.Raise = func(st views.Status) {
if held() {
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, true)
}
}
f.Clear = func(st views.Status) {
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
}
f.RaiseWaits = func(st views.Status) {
if held() {
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, true)
}
}
f.ClearWaits = func(st views.Status) {
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
}
ctx, cancel := context.WithCancel(v.ctx)
a := &adoption{name: name, key: key, f: f, hear: hear, cancel: cancel}
v.followers[name] = f
v.adopted[name] = a
v.hearers[key] = append(v.hearers[key], a)
v.mu.Unlock()
v.wg.Add(1)
go func() { defer v.wg.Done(); _ = f.Start(ctx) }()
return nil
}
func (w wiring) Drop(owner, state string) error {
v := w.v
name := owner + "." + state
v.mu.Lock()
a := v.adopted[name]
if a == nil {
v.mu.Unlock()
return fmt.Errorf("%s is not a view this tool runner adopted", name)
}
delete(v.followers, name)
delete(v.adopted, name)
v.hearers[a.key] = slices.DeleteFunc(v.hearers[a.key], func(o *adoption) bool { return o == a })
v.mu.Unlock()
a.cancel()
st := a.f.Status()
if st.Raised {
v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
}
if st.WaitsRaised {
v.finding(viewWaitsFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
}
return nil
}
// Halves is the catalog half of each followed branch, by branch: its revision, with the seat definitions it was judged
// with, and its entries; mesh_wait checks an event name's emitter against them (design 34 every-event §14). An error says
// the half has no first view yet, never that no module exists.
func (v *Views) Halves() (map[string]meshcatalogclient.Half, error) {
if v.branches == nil {
return nil, fmt.Errorf("the tool runner does not follow the catalog half")
}
if err := v.branches.Ready(); err != nil {
return nil, err
}
out := map[string]meshcatalogclient.Half{}
for _, branch := range v.branches.Followed() {
if c, ok := v.branches.Catalog(branch); ok {
out[branch], _ = c.View() // a branch at none-yet is ready and empty (§614)
}
}
return out, nil
}
// Modules is every module the catalog half holds, by name, with its entry: the entries of the watched branches' newest trunk
// revisions, the branch followed first naming a module more than one hold (design 47 G §196, §594). An error says the half has
// no first view yet, never that no module exists.
func (v *Views) Modules() (map[string]catalogseat.CatalogEntry, error) {
if v.branches == nil {
return nil, fmt.Errorf("the tool runner does not follow the catalog half")
}
if err := v.branches.Ready(); err != nil {
return nil, err
}
out := map[string]catalogseat.CatalogEntry{}
for _, branch := range v.branches.Followed() {
c, ok := v.branches.Catalog(branch)
if !ok {
continue
}
half, _ := c.View() // a branch at none-yet is ready and empty (§614)
for m, e := range half.Entries {
if _, taken := out[m]; !taken {
out[m] = e
}
}
}
return out, nil
}
// holding is the views of the module that claims the assigner seat, and what stops them.
type holding struct {
module string
cancel context.CancelFunc
views []*views.Follower
}
// claimed is the branch follower's word of one entry changed: an entry that serves a verb of the assigner seat makes its module
// the holder whose views the runner follows (a holder before it is dropped), the deletion of the holder's entry drops them
// (design 47 G §616, §428). It acts on the event alone; nothing waits on a timer for the claim.
func (v *Views) claimed(module string, entry json.RawMessage, deleted bool) {
claims := false
if !deleted {
var e catalogseat.CatalogEntry
if json.Unmarshal(entry, &e) == nil {
claims = slices.ContainsFunc(e.Serves, func(s string) bool { return strings.HasPrefix(s, AssignerSeat+".") })
}
}
v.mu.Lock()
cur := v.holder
switch {
case claims && cur != nil && cur.module == module, !claims && (cur == nil || cur.module != module), v.ctx.Err() != nil:
v.mu.Unlock()
return
}
if cur != nil {
v.dropHolderLocked()
}
if claims {
v.startHolderLocked(module)
}
v.mu.Unlock()
}
// startHolderLocked follows the first views of module, the holder of the seat, on the runner's own consumer.
func (v *Views) startHolderLocked(module string) {
ctx, cancel := context.WithCancel(v.ctx)
h := &holding{module: module, cancel: cancel}
d := v.deps
for _, s := range Specs {
s := s
call := func(tool string, body any) (json.RawMessage, error) { return d.Ask(module+"."+tool, body) }
f := &views.Follower{Owner: module, Tool: s.Tool, State: s.State, Ask: views.AskVia(call, s.Tool), Logf: v.logf,
Bound: d.Bound, Wait: d.Wait,
Raise: func(st views.Status) {
v.finding(unansweredFinding(d.Module, d.Node, st, time.Now()), st.Tool, true)
},
Clear: func(st views.Status) {
v.finding(unansweredFinding(d.Module, d.Node, st, time.Now()), st.Tool, false)
}}
v.followers[module+"."+s.State] = f
v.byState[s.State] = f
h.views = append(h.views, f)
v.wg.Add(1)
go func() { defer v.wg.Done(); _ = f.Start(ctx) }()
}
v.holder = h
v.logf("[mesh-tools] the seat %s is held by %s: its views are followed", AssignerSeat, module)
}
// dropHolderLocked stops the views of the holder and clears the findings they raised.
func (v *Views) dropHolderLocked() {
h := v.holder
v.holder = nil
h.cancel()
for _, f := range h.views {
delete(v.followers, f.Owner+"."+f.State)
delete(v.byState, f.State)
if st := f.Status(); st.Raised {
go v.finding(unansweredFinding(v.deps.Module, v.deps.Node, st, time.Now()), st.Tool, false)
}
}
v.logf("[mesh-tools] %s no longer holds the seat %s: its views are no longer followed", h.module, AssignerSeat)
}
@@ -0,0 +1,204 @@
//go:build integration
package firstview
import (
"context"
"encoding/json"
"os"
"strings"
"sync"
"testing"
"time"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
delivery "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"github.com/novox/mesh-tools/node-tools/internal/bus"
mt "github.com/novox/mesh-tools/node-tools/internal/meshtest"
)
// issue 1966 TLS-5 [i] (ADR 0615 §2, §4, §6): against the real bus, with the owners down and then up, the runner's own
// consumers on the owners' own views streams carry the manifest's four `consumes`: the first views are asked until answered,
// the unanswered ones are findings that the answer clears, and each owner's next event reaches its picture.
func TestTheRunnersConsumerCarriesTheOwnersViewsFamiliesWhenTheOwnersAreDownThenUp(t *testing.T) {
mesh := mt.New(t)
url := mt.URL(t)
// The owners publish their views on their own streams (views.Emit: `mesh.stream.<owner>.event.views.<state>`, on
// assigner-views and indexer-views), never on EVENTS. The consumers on them are the mesh's, one filter per family the
// manifest's `consumes` names, and the runner's membership names them as the mesh issues it (ADR 0363 §8).
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var manifest struct {
Consumes []string `json:"consumes"`
}
if err := json.Unmarshal(raw, &manifest); err != nil {
t.Fatal(err)
}
// the manifest names the assigner's views by the seat; the module the catalog half says claims it is the fixture's
const holder = "assigner-fixture"
streamOf := map[string]string{"seat:" + AssignerSeat: catalogue.StreamBusName("assigner-views")}
filters := map[string][]string{}
families := 0
for _, c := range manifest.Consumes {
emitter, event, isView := strings.Cut(c, ".views.")
if !isView { // the seats' events (the seat list's, the catalog half's), on EVENTS (cmd/node-tools/seatlist.go)
continue
}
stream := streamOf[emitter]
if stream == "" {
t.Fatalf("the manifest consumes %s, whose views stream the test does not know", c)
}
filters[stream] = append(filters[stream], process.ModuleStreamSubject(holder, views.EventPrefix+event))
families++
}
if families != len(Specs) {
t.Fatalf("the manifest consumes %d families, the runner holds %d first views", families, len(Specs))
}
runner := mt.MembershipOf("node-tools", "anchor", false, nil)
for _, seat := range []string{"seat:" + AssignerSeat} {
stream := streamOf[seat]
mesh.ModuleStream(t, holder, stream)
mesh.ConsumerOn(t, stream, "anchor", "node-tools", filters[stream], 2*time.Second)
runner.Hears = append(runner.Hears, bus.HeardIssued{Stream: stream, Consumer: "anchor_node-tools"})
}
mesh.Consumer(t, "anchor", "node-tools", []string{"mesh.mod.mesh-knowledge.event.reviewed"}, 2*time.Second) // its `waits`, on EVENTS
mesh.Issue(t, runner)
conn, err := bus.Connect(bus.Credential{URL: url, Module: "node-tools", Node: "anchor"})
if err != nil {
t.Fatal(err)
}
conn.Logf = func(string, ...any) {}
t.Cleanup(conn.Close)
conn.Follow("node-tools")
mt.Until(t, func() error {
if conn.Membership("node-tools") == nil {
return errFindings(-1)
}
return nil
})
var mu sync.Mutex
findings := map[string]bool{}
deps := DepsOf(conn, t.Logf)
deps.Bound, deps.Wait = 100*time.Millisecond, 20*time.Millisecond
deps.Findings = writes{&mu, findings}
deps.StartFindings = nil
v, stop := StartWith(context.Background(), deps, func(string, ...any) {})
t.Cleanup(stop)
// The owners are down: no tool answers, nothing is claimed, and past the bound each view is a finding.
if _, err := v.State(AssignmentsRead); err == nil {
t.Fatal("a state its owner has not answered is claimed")
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if len(findings) != catalogViews {
return errFindings(len(findings))
}
return nil
})
// The owners come up and answer their first views.
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
respond := func(subject string, answer any) {
if _, err := nc.Subscribe(subject, func(m *nats.Msg) {
raw, _ := json.Marshal(answer)
body, _ := json.Marshal(map[string]json.RawMessage{"result": raw})
_ = m.Respond(body)
}); err != nil {
t.Fatal(err)
}
}
// the catalog half answers: the gate's source names main, whose entries say the fixture claims the assigner seat
respond("mesh.seat.mesh-delivery.tool.source", delivery.SourceAnswer{Branches: []string{"main"}})
respond("mesh.seat.mesh-catalog.tool.registrations", catalogseat.RegistrationsAnswer{Registrations: []catalogseat.Registration{}})
respond("mesh.seat.mesh-catalog.tool.revision", catalogseat.RevisionAnswer{
Revision: &catalogseat.CatalogRevision{Revision: 3, Kind: meshcatalogclient.KindTrunk, Branch: "main"},
Entries: []catalogseat.CatalogEntry{{Module: holder, Serves: []string{AssignerSeat + ".assignments"}}}})
for _, s := range Specs {
s := s
if _, err := nc.Subscribe("mesh.mod."+holder+".tool."+s.Tool, func(m *nats.Msg) {
answer, _ := json.Marshal(views.Answer{V: 1, Owner: holder, State: s.State, Revision: 4, Created: time.Unix(1000, 0).UTC(),
Values: map[string]json.RawMessage{"first": json.RawMessage(`{"n":1}`)}})
body, _ := json.Marshal(map[string]json.RawMessage{"result": answer})
_ = m.Respond(body)
}); err != nil {
t.Fatal(err)
}
}
_ = nc.Flush()
for _, s := range Specs {
name := s.Name()
mt.Until(t, func() error { _, err := v.Entry(name, "first"); return err })
}
mt.Until(t, func() error {
mu.Lock()
defer mu.Unlock()
if len(findings) != 0 { // every view answered clears its finding
return errFindings(len(findings))
}
return nil
})
// Each owner's next event arrives on the runner's consumer: all four families are bound.
for _, s := range Specs {
ev, _ := json.Marshal(views.Event{V: 1, Owner: holder, State: s.State, Key: "next", Revision: 5, Created: time.Unix(1000, 0).UTC(),
Value: json.RawMessage(`{"n":2}`)})
mesh.Emit(t, process.ModuleStreamSubject(holder, views.EventPrefix+s.State), json.RawMessage(ev))
}
for _, s := range Specs {
name := s.Name()
mt.Until(t, func() error {
got, err := v.Entry(name, "next")
if err != nil || string(got) != `{"n":2}` {
return errFindings(-1)
}
return nil
})
}
}
// catalogViews is the first views the branch follower adopts before any branch is known: the registrations and the gate's source
// (design 47 G §614), each a finding while the seats' verbs go unanswered.
const catalogViews = 2
type errFindings int
func (e errFindings) Error() string { return "not yet" }
// writes is the runner's finding writer in this test: the findings written and not cleared.
type writes struct {
mu *sync.Mutex
by map[string]bool
}
func (w writes) Write(_ context.Context, f condseat.Finding) error {
w.mu.Lock()
defer w.mu.Unlock()
w.by[views.FindingKey(f.Scope, f.ID, f.Kind)] = true
return nil
}
func (w writes) Clear(_ context.Context, scope, id, kind, _ string) error {
w.mu.Lock()
defer w.mu.Unlock()
delete(w.by, views.FindingKey(scope, id, kind))
return nil
}
@@ -0,0 +1,302 @@
package firstview
import (
"context"
"encoding/json"
"errors"
"os"
"slices"
"strings"
"sync"
"testing"
"time"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
catalogseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
delivery "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-sdk/go/views"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
// owners is the four owners' first-view tools as a test answers them: a state's values at a revision, or nothing while down.
type owners struct {
mu sync.Mutex
down bool
state map[string]map[string]string // tool → key → value
rev map[string]uint64
asked map[string]int
deliver func(bus.Envelope) error
finds map[string]bool
// catalog is the revision of the watched branch main the seat mesh-catalog answers, and modules its entries
catalog int64
modules []string
// holder is the module among them that claims the assigner seat
holder string
}
func newOwners() *owners {
return &owners{state: map[string]map[string]string{"assignments": {"anchor": `{"p":1}`}, "seat-holders": {"seat": `{"h":1}`}},
rev: map[string]uint64{"assignments": 4, "seat-holders": 2}, asked: map[string]int{}, finds: map[string]bool{},
catalog: 3, modules: []string{"web", fixtureHolder}, holder: fixtureHolder}
}
// Write and Clear are the runner's finding writer in a test: the findings written and not cleared, by their key.
func (o *owners) Write(_ context.Context, f condseat.Finding) error {
o.mu.Lock()
defer o.mu.Unlock()
o.finds[views.FindingKey(f.Scope, f.ID, f.Kind)] = true
return nil
}
func (o *owners) Clear(_ context.Context, scope, id, kind, _ string) error {
o.mu.Lock()
defer o.mu.Unlock()
delete(o.finds, views.FindingKey(scope, id, kind))
return nil
}
func (o *owners) ask(address string, _ any) (json.RawMessage, error) {
o.mu.Lock()
defer o.mu.Unlock()
tool := address[strings.LastIndex(address, ".")+1:]
o.asked[tool]++
if o.down {
return nil, errors.New("503 no responders")
}
switch address {
case "seat:mesh-delivery.source":
return json.Marshal(delivery.SourceAnswer{Branches: []string{"main"}})
case "seat:mesh-catalog.registrations":
return json.Marshal(catalogseat.RegistrationsAnswer{Registrations: []catalogseat.Registration{}})
case "seat:mesh-catalog.revision":
a := catalogseat.RevisionAnswer{Revision: &catalogseat.CatalogRevision{Revision: o.catalog, Kind: meshcatalogclient.KindTrunk, Branch: "main"}}
for _, m := range o.modules {
e := catalogseat.CatalogEntry{Module: m, Manifest: map[string]any{"module": m, "tools": []string{m + "_check"}}}
if m == o.holder {
e.Serves = []string{AssignerSeat + ".assignments", AssignerSeat + ".seat-holders"}
}
a.Entries = append(a.Entries, e)
}
return json.Marshal(a)
}
vals := map[string]json.RawMessage{}
for k, v := range o.state[tool] {
vals[k] = json.RawMessage(v)
}
return json.Marshal(views.Answer{V: 1, Owner: "o", State: tool, Revision: o.rev[tool], Created: time.Unix(1000, 0).UTC(), Values: vals})
}
func (o *owners) say(t *testing.T, state, key, value string, revision uint64, gap bool) {
t.Helper()
ev := views.Event{V: 1, Owner: owner(state), State: state, Key: key, Revision: revision, Created: time.Unix(1000, 0).UTC(), Gap: gap}
if value != "" {
ev.Value = json.RawMessage(value)
}
raw, _ := json.Marshal(ev)
if err := o.deliver(bus.Envelope{Key: ev.Owner + ".views." + state, Body: raw}); err != nil {
t.Fatal(err)
}
}
// fixtureHolder is the module the fixture catalog half says claims the assigner seat: no name is in the code.
const fixtureHolder = "assigner-fixture"
func owner(state string) string { return fixtureHolder }
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
for end := time.Now().Add(3 * time.Second); !ok(); time.Sleep(5 * time.Millisecond) {
if time.Now().After(end) {
t.Fatal(what)
}
}
}
// issue 1966 TLS-5 (ADR 0615 §2, §4, §5, §6): the runner holds the four states from the owners' first views and their
// events alone; nothing is claimed of a state not yet answered; a gap asks the first view once more; an owner that stays down
// past the bound is a finding, cleared by the answer; the status and the hand ask show them.
func TestTheRunnerHoldsTheFourStatesFromTheOwnersFirstViewsAndTheirEvents(t *testing.T) {
o := newOwners()
o.down = true
var logs []string
var lmu sync.Mutex
deps := Deps{Module: "node-tools", Ask: o.ask, Bound: 30 * time.Millisecond, Wait: 10 * time.Millisecond,
Consume: func(deliver func(bus.Envelope) error) (func(), error) { o.deliver = deliver; return func() {}, nil },
Findings: o}
v, stop := StartWith(context.Background(), deps, func(f string, a ...any) { lmu.Lock(); logs = append(logs, f); lmu.Unlock() })
defer stop()
// Down: nothing is claimed, and past the bound each unanswered view is a finding.
if _, err := v.State(AssignmentsRead); err == nil || !strings.Contains(err.Error(), "names no holder") {
t.Fatalf("a state whose holder is not known is claimed: %v", err)
}
eventually(t, "no finding past the bound", func() bool { o.mu.Lock(); defer o.mu.Unlock(); return len(o.finds) == 2 })
// Up: answered, the findings are cleared, and the pictures hold the owners' values.
o.mu.Lock()
o.down = false
o.mu.Unlock()
eventually(t, "not answered", func() bool { _, err := v.State(SeatHoldersRead); return err == nil })
eventually(t, "findings not cleared", func() bool { o.mu.Lock(); defer o.mu.Unlock(); return len(o.finds) == 0 })
if got, _ := v.Entry(AssignerSeat+".seat-holders", "none"); got != nil {
t.Errorf("an absent key is %q", got)
}
// The catalog half is followed by the shared branch follower on the runner's own consumer: the gate's source names main,
// main's view is adopted, and its entries are the modules, each with its manifest (design 47 G §196, §614).
eventually(t, "the catalog half holds no modules", func() bool { m, err := v.Modules(); return err == nil && len(m) == 2 })
m, _ := v.Modules()
if m["web"].Manifest["module"] != "web" {
t.Errorf("the module web is %+v", m["web"])
}
// A catalog-revision of the branch is heard on the same consumer and asks the view once more.
o.mu.Lock()
o.catalog, o.modules = 4, []string{"web", "db", fixtureHolder}
o.mu.Unlock()
ev, _ := json.Marshal(catalogseat.CatalogRevisionEvent{V: 1, Branch: "main", Kind: meshcatalogclient.KindTrunk, Revision: 4})
if err := o.deliver(bus.Envelope{Key: catalogseat.Seat + "." + catalogseat.EventCatalogRevision + ".main", Body: ev}); err != nil {
t.Fatal(err)
}
eventually(t, "the catalog-revision was not taken", func() bool { m, err := v.Modules(); return err == nil && len(m) == 3 })
// An event no view hears is said once per key, not dropped silently.
for i := 0; i < 2; i++ {
_ = o.deliver(bus.Envelope{Key: "some-seat.unknown-event.x", Body: json.RawMessage(`"x"`)})
}
// And the halves by branch, with their revision, which mesh_wait names checking an emitter (design 34 every-event §14).
if h, err := v.Halves(); err != nil || h["main"].Revision.Revision != 4 || len(h["main"].Entries) != 3 {
t.Errorf("the halves: %+v, %v", h, err)
}
// Events: the next revision is applied; an old one passed over; a deletion removes; none of it asks again.
asked := func(tool string) int { o.mu.Lock(); defer o.mu.Unlock(); return o.asked[tool] }
before := asked("assignments")
o.say(t, "assignments", "laptop", `{"p":2}`, 5, false)
o.say(t, "assignments", "laptop", `{"p":0}`, 5, false)
o.say(t, "assignments", "anchor", "", 6, false)
// (a deleted mark carries no value: Deleted is the encoding's, set below)
if got, _ := v.Entry(AssignerSeat+".assignments", "laptop"); string(got) != `{"p":2}` {
t.Errorf("the next revision is not applied: %q", got)
}
if n := asked("assignments"); n != before {
t.Errorf("an event in order asked the first view again (%d asks, was %d)", n, before)
}
// A gap (revision 9 where 7 was next) asks the first view once more, said in the log.
o.say(t, "assignments", "x", `{"p":9}`, 9, false)
eventually(t, "a gap did not ask the first view again", func() bool { return asked("assignments") > before })
// The status says owner, tool, revision and the asks; the hand ask asks every view again.
var assignments views.Status
for _, s := range v.Status() {
if s.State == "assignments" {
assignments = s
}
}
if assignments.Owner != fixtureHolder || assignments.Tool != "assignments" || assignments.Asks < 2 || assignments.Answered.IsZero() {
t.Errorf("the status of assignments is %+v", assignments)
}
manifests := asked("seat-holders")
if errs := v.Again(context.Background(), "doubted"); len(errs) != 0 {
t.Fatalf("the hand ask failed: %v", errs)
}
if asked("seat-holders") != manifests+1 {
t.Errorf("the hand ask did not ask the seat-holders again")
}
for _, st := range v.Status() {
if st.State == "seat-holders" && st.ByHand != 1 {
t.Errorf("the hand ask is not counted: %+v", st)
}
}
lmu.Lock()
defer lmu.Unlock()
if n := strings.Count(strings.Join(logs, "\n"), "heard by no view"); n != 1 {
t.Errorf("an unheard event was said %d times, want once: %v", n, logs)
}
if !strings.Contains(strings.Join(logs, "\n"), "a gap in") {
t.Errorf("the gap was not said in the log: %v", logs)
}
}
// The four first views are the manifest's `consumes` families, and the console reads each by the name here.
func TestTheFirstViewsAreTheManifestsViews(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Consumes []string `json:"consumes"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
want := map[string]bool{AssignmentsRead: true, SeatHoldersRead: true}
for _, s := range Specs {
if !want[s.Name()] {
t.Errorf("the first view %s is read by no name here", s.Name())
}
delete(want, s.Name())
// the family is named by the seat, never by its holder (design 47 G §616)
if view := "seat:" + AssignerSeat + ".views." + s.Tool; !slices.Contains(m.Consumes, view) {
t.Errorf("the first view %s has no family in the manifest's consumes %v", view, m.Consumes)
}
}
if len(want) > 0 {
t.Errorf("no first view reads %v", want)
}
for _, c := range m.Consumes {
if strings.Contains(c, "assigner") && !strings.HasPrefix(c, "seat:") {
t.Errorf("the manifest names the assigner by a module: %s", c)
}
}
// the catalog half's follower hears these events and asks these verbs, and the manifest says so (design 47 G §614)
for _, ev := range []string{"mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-delivery.source-moved"} {
if !slices.Contains(m.Consumes, ev) {
t.Errorf("the catalog half's event %s is not in the manifest's consumes %v", ev, m.Consumes)
}
}
for _, ev := range []string{"mesh-delivery-indexer.views.modules", "mesh-delivery-indexer.views.manifests"} {
if slices.Contains(m.Consumes, ev) {
t.Errorf("the manifest still consumes the indexer's %s", ev)
}
}
// design 47 G §616: the holder of the assigner seat is whichever module the catalog half says serves it; when another one
// claims it the views are of the new holder, and when none does the seat's names answer that the catalog half names no holder.
t.Run("TheAssignersViewsFollowTheModuleThatClaimsTheSeat", func(t *testing.T) {
o := newOwners()
deps := Deps{Module: "node-tools", Ask: o.ask, Bound: time.Second, Wait: 10 * time.Millisecond,
Consume: func(deliver func(bus.Envelope) error) (func(), error) { o.deliver = deliver; return func() {}, nil }, Findings: o}
v, stop := StartWith(context.Background(), deps, func(string, ...any) {})
defer stop()
eventually(t, "not answered", func() bool { _, err := v.State(AssignmentsRead); return err == nil })
ownerOf := func() string {
for _, s := range v.Status() {
if s.State == "assignments" {
return s.Owner
}
}
return ""
}
if ownerOf() != fixtureHolder {
t.Fatalf("the views are of %q", ownerOf())
}
revise := func(rev int64, holder string, modules ...string) {
o.mu.Lock()
o.catalog, o.modules, o.holder = rev, modules, holder
o.mu.Unlock()
ev, _ := json.Marshal(catalogseat.CatalogRevisionEvent{V: 1, Branch: "main", Kind: meshcatalogclient.KindTrunk, Revision: rev})
if err := o.deliver(bus.Envelope{Key: catalogseat.Seat + "." + catalogseat.EventCatalogRevision, Body: ev}); err != nil {
t.Fatal(err)
}
}
revise(4, "other-holder", "web", "other-holder")
eventually(t, "the views did not move to the new holder", func() bool { return ownerOf() == "other-holder" })
eventually(t, "not answered", func() bool { _, err := v.State(SeatHoldersRead); return err == nil })
revise(5, "", "web")
eventually(t, "the views stayed after the claim went", func() bool {
_, err := v.State(AssignmentsRead)
return err != nil && strings.Contains(err.Error(), "names no holder")
})
})
}
@@ -3,7 +3,12 @@ package launch
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
type lostWrite struct{}
@@ -28,3 +33,84 @@ func TestALostWriteIsAnsweredWithItsOwnCodeAndNothingElseIs(t *testing.T) {
t.Errorf("a plain refusal is answered %d", got)
}
}
// A bundle that answers the handshake and an empty tool list, and notes each start and its pid in $OUT.
const notingBundle = `#!/bin/sh
printf 'started %s\n' "$$" >> "$OUT"
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id" ;;
esac
done
`
// The service manager stops the runtime by signalling its whole group, every bundle with it (issue 1126, ace's
// reboot of 2026-10-28: every bundle "exited (SIGTERM); started again in 500ms" for the 15 s the runtime took
// to stop, the power bundle among them dialling the system bus again, which restarted the bus during the
// shutdown and held it 29 minutes). A bundle that exits while it serves is started again; once the runtime
// itself is stopping, none is, by a restart or by a call.
func TestABundleIsStartedAgainUnlessTheRuntimeIsStopping(t *testing.T) {
saved := RestartFirst
RestartFirst = 10 * time.Millisecond
t.Cleanup(func() { RestartFirst = saved; shuttingDown.Store(false) })
for _, row := range []struct {
name string
stopping bool
starts int
}{
{"a bundle that dies while the runtime serves", false, 2},
{"a bundle signalled with the stopping runtime", true, 1},
} {
t.Run(row.name, func(t *testing.T) {
shuttingDown.Store(false)
dir := t.TempDir()
entry, out := filepath.Join(dir, "bundle"), filepath.Join(dir, "out")
if err := os.WriteFile(entry, []byte(notingBundle), 0o755); err != nil {
t.Fatal(err)
}
l, err := Start("noting", entry, append(os.Environ(), "OUT="+out), &subscribing{}, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
said, _ := os.ReadFile(out)
var pid int
if _, err := fmt.Sscanf(string(said), "started %d", &pid); err != nil {
t.Fatalf("the bundle noted %q", said)
}
if row.stopping {
ShutDown()
}
if err := syscall.Kill(pid, syscall.SIGTERM); err != nil {
t.Fatal(err)
}
// The restart row ends at its second start; the stopping row watches a window many restart pauses long.
deadline := time.Now().Add(250 * time.Millisecond)
for {
said, _ = os.ReadFile(out)
n := strings.Count(string(said), "started")
if n > row.starts {
t.Fatalf("started %d times, want %d", n, row.starts)
}
if n == row.starts && !row.stopping {
break
}
if time.Now().After(deadline) {
if n != row.starts {
t.Fatalf("started %d times, want %d", n, row.starts)
}
break
}
time.Sleep(5 * time.Millisecond)
}
if row.stopping {
// And a bundle first loaded now (a reload) is not started either.
if _, err := Start("late", entry, append(os.Environ(), "OUT="+out), &subscribing{}, t.Logf); err == nil {
t.Error("a bundle was started after the runtime began stopping")
}
}
})
}
}
@@ -19,6 +19,9 @@ while IFS= read -r line; do
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/subscribe","params":{"pattern":"#"}}\n' ;;
*'"method":"mesh/heartbeat"'*)
params=$(printf '%s' "$line" | sed -n 's/.*"params":\(.*\)}$/\1/p')
printf '{"jsonrpc":"2.0","id":"h1","method":"mesh/publish","params":%s}\n' "$params" ;;
*'"method":"mesh/event"'*)
case "$ANSWER" in
error) printf '{"jsonrpc":"2.0","id":%s,"error":{"code":-32000,"message":"Unexpected end of JSON input"}}\n' "$id" ;;
@@ -30,23 +33,41 @@ done
`
type subscribing struct {
mu sync.Mutex
deliver func(json.RawMessage) error
mu sync.Mutex
deliver func(json.RawMessage) error
beat func(json.RawMessage)
published []json.RawMessage
}
func (b *subscribing) Publish(json.RawMessage) error { return nil }
func (b *subscribing) Publish(p json.RawMessage) error {
b.mu.Lock()
b.published = append(b.published, p)
b.mu.Unlock()
return nil
}
func (b *subscribing) Ask(json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) State(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Log(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) Watch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Seat(string, json.RawMessage, func(string, any) (json.RawMessage, error)) (json.RawMessage, error) {
return nil, nil
}
func (b *subscribing) Subscribe(d func(json.RawMessage) error) error {
func (b *subscribing) Provision(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) ProvisionWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) BusState(string, json.RawMessage) (json.RawMessage, error) { return nil, nil }
func (b *subscribing) BusStateWatch(string, json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
func (b *subscribing) Begin(string) (string, func()) { return "", func() {} }
func (b *subscribing) Subscribe(d func(json.RawMessage) error, beat func(json.RawMessage)) error {
b.mu.Lock()
defer b.mu.Unlock()
b.deliver = d
b.beat = beat
return nil
}
@@ -90,3 +111,49 @@ func TestAnEventIsTakenByAnyAnswerThatIsNotAnError(t *testing.T) {
t.Fatalf("an error answer was not the bundle's refusal of mesh/event: %#v", err)
}
}
func (b *subscribing) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
// The runtime's word on the module's consumer between events reaches the child that subscribed as `mesh/heartbeat`,
// unchanged (stdio.OnHeartbeat reads it in the bundle).
func TestTheWordOnTheConsumerBetweenEventsReachesTheBundleThatSubscribed(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(answeringBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &subscribing{}
l, err := Start("plex", entry, append(os.Environ(), "ANSWER=result"), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
var beat func(json.RawMessage)
for i := 0; beat == nil; i++ {
b.mu.Lock()
beat = b.beat
b.mu.Unlock()
if i > 100 {
t.Fatal("the bundle never subscribed")
}
time.Sleep(20 * time.Millisecond)
}
word := json.RawMessage(`{"at":"2026-11-04T13:00:00Z","missed":true,"why":"the bus ended it"}`)
beat(word)
for i := 0; ; i++ {
b.mu.Lock()
got := b.published
b.mu.Unlock()
if len(got) == 1 {
if string(got[0]) != string(word) {
t.Fatalf("the bundle was told %s, not %s", got[0], word)
}
return
}
if i > 100 {
t.Fatal("the bundle was told nothing")
}
time.Sleep(20 * time.Millisecond)
}
}
+341 -31
View File
@@ -15,9 +15,12 @@ import (
"regexp"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"golang.org/x/sys/unix"
"github.com/novox/mesh-tools/node-tools/internal/wire"
@@ -37,6 +40,9 @@ type Tool struct {
Name string
Description string
Input json.RawMessage
// Effect is what the tool does, reads, acts or dangerous (novox/hq ADR 0387 §1), where the runtime knows it: its own
// tools say it; a bundle's tool is marked by its manifest.
Effect string
// Run calls the tool for caller: the bus user the call's subject named, or "" when it named none. A
// caller is handed to the bundle in the call's `_meta` (MetaCaller), never among its arguments.
Run func(caller string, args json.RawMessage) (json.RawMessage, error)
@@ -47,12 +53,21 @@ type Tool struct {
// stdio.CallerOf, and says Unproven when it is absent.
const MetaCaller = "mesh/caller"
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and — only when the
// subject named one — the caller, in `_meta`, which the runtime writes and no caller can.
func callParams(name, caller string, args json.RawMessage) map[string]any {
// callParams is a `tools/call`'s params: the tool, its arguments as the caller sent them, and in `_meta`, which the
// runtime writes and no caller can, the caller — only when the subject named one — and the id of the call in flight
// (mesh-sdk buscalls.MetaCall), which a ledger's apply names so the runtime stamps the caller it proved (novox/hq ADR
// 0597 §7).
func callParams(name, caller, call string, args json.RawMessage) map[string]any {
params := map[string]any{"name": name, "arguments": args}
meta := map[string]any{}
if caller != "" {
params["_meta"] = map[string]any{MetaCaller: caller}
meta[MetaCaller] = caller
}
if call != "" {
meta[buscalls.MetaCall] = call
}
if len(meta) > 0 {
params["_meta"] = meta
}
return params
}
@@ -66,7 +81,9 @@ type Registration struct {
// Bus is what a launched bundle reaches the mesh through (novox/hq ADR 0193, ADR 0198): the runtime
// publishes, asks and subscribes on the module's behalf. Subscribe is asked each time the module's
// code subscribes; the runtime binds the module's consumer once and calls deliver for every event,
// acknowledging it on the bus when deliver returns nil.
// acknowledging it on the bus when deliver returns nil. beat is the runtime's word on that consumer between
// events (a bus.Beat as JSON), which the launch hands the child that subscribed as `mesh/heartbeat`, so a bundle
// can tell a silent source from a quiet one (stdio.OnHeartbeat).
//
// State and Watch reach the module's state (novox/hq ADR 0201): State answers `get`, `put`, `delete`
// and `keys`; Watch hands deliver the current values and then every change, and returns once the
@@ -74,15 +91,45 @@ type Registration struct {
type Bus interface {
Publish(params json.RawMessage) error
Ask(params json.RawMessage) (json.RawMessage, error)
Subscribe(deliver func(envelope json.RawMessage) error) error
Subscribe(deliver func(envelope json.RawMessage) error, beat func(heartbeat json.RawMessage)) error
State(verb string, params json.RawMessage) (json.RawMessage, error)
Watch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Log answers a bundle's `append`, `read`, `last` and `info` on its module's log (novox/hq ADR 0297).
Log(verb string, params json.RawMessage) (json.RawMessage, error)
// LogWatch hands deliver each entry of a log from where the bundle asked, one at a time (novox/hq ADR 0571).
LogWatch(params json.RawMessage, deliver func(entry json.RawMessage) error) (stop func(), err error)
// Seat answers a bundle's seat traffic (novox/hq ADR 0259 §3): `publish`, `prove` and `record` at once;
// `take` and `answer` bind once and hand each message or proof on through handOn, to whichever child
// of the module asked last — a restarted child asks again as its code runs again.
Seat(verb string, params json.RawMessage, handOn func(method string, params any) (json.RawMessage, error)) (json.RawMessage, error)
// Provision answers a bundle's `get` of one of its module's values at runtime, and ProvisionWatch hands deliver
// that value now and each change after, as the module whose bundle asks and never a module the bundle names
// (novox/hq ADR 0349 §14, ADR 0370 §4): its config, and its bindings' values that are not secret.
Provision(verb string, params json.RawMessage) (json.RawMessage, error)
ProvisionWatch(params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// BusState answers a bundle's calls on its module's bus state in the new form (mesh-sdk buscalls; novox/hq ADR 0595
// §9, design 25 § 12.5): its objects, a stream's read-back and fill, a work queue's size and dead letters, and its
// ledgers, by the method the bundle asked, from its own module's bindings alone.
BusState(method string, params json.RawMessage) (json.RawMessage, error)
// BusStateWatch hands deliver each change of an objects or a ledger watch, the current ones first.
BusStateWatch(method string, params json.RawMessage, deliver func(change json.RawMessage) error) (stop func(), err error)
// Begin says a tool call of the module is in flight for the caller the runtime proved, and answers its id, which the
// bundle is handed in the call's `_meta`, and the end to call once the call is answered (novox/hq ADR 0597 §7).
Begin(caller string) (id string, end func())
}
// busStateCalls are the methods of mesh-sdk buscalls a bundle asks and BusState answers.
var busStateCalls = map[string]bool{
buscalls.ObjectsPut: true, buscalls.ObjectsGet: true, buscalls.ObjectsList: true, buscalls.ObjectsDelete: true,
buscalls.ObjectsInfo: true, buscalls.StreamRead: true, buscalls.StreamFill: true, buscalls.StreamPublish: true, buscalls.WorkQueueSize: true,
buscalls.WorkQueueDead: true, buscalls.LedgerApply: true, buscalls.LedgerGet: true, buscalls.LedgerHistory: true,
buscalls.LedgerQuery: true, buscalls.LedgerRebuild: true, buscalls.LedgerReindex: true, buscalls.LedgerSnapshot: true,
buscalls.LedgerLag: true, buscalls.ObjectsSentGet: true, buscalls.ObjectsPutChunk: true, buscalls.ObjectsGetChunk: true,
}
// busStateWatches are the watches of mesh-sdk buscalls, each with the method its changes are handed to the bundle by.
var busStateWatches = map[string]string{buscalls.ObjectsWatch: buscalls.ObjectsWatched, buscalls.LedgerWatch: buscalls.LedgerWatched}
// EventTimeout bounds how long a bundle has to handle one event before it is offered again.
var EventTimeout = 2 * time.Minute
@@ -105,6 +152,21 @@ var (
RestartSettle = time.Minute
)
// shuttingDown is set once the runtime itself is stopping (ShutDown): from then on no bundle is started again.
var shuttingDown atomic.Bool
// ShutDown says the runtime itself is stopping, before it stops its bundles one by one. The service manager stops
// the runtime by signalling its whole group, so every bundle receives SIGTERM with it, and a bundle's exit must not
// be taken for a fault: started again, it would run on through the machine's shutdown with nothing left to stop it
// (issue 1126: on ace's reboot of 2026-10-28 the restarted bundles dialled the system bus again and started it
// during the shutdown, which then waited 29 minutes). After ShutDown no bundle is started again, by a restart or a
// call.
func ShutDown() { shuttingDown.Store(true) }
// StopWait bounds how long Launched.Stop waits for its bundle's exit to be handled; a bundle that
// ignores SIGTERM longer than this is left to exit on its own (novox/hq issue 476).
var StopWait = 5 * time.Second
// Launched is a running bundle: what it registered, and how to stop it.
type Launched struct {
Registrations []Registration
@@ -141,12 +203,37 @@ type child struct {
pending map[int64]chan message
next int64
dead chan struct{}
// handled is closed once the child's exit has been handled whole — its state cleared and its
// restart decided — which Stop waits for (novox/hq issue 476).
handled chan struct{}
why error
// endedFor is why the runner itself ended the bundle (a line over the cap); its calls fail with it.
endedFor string
// watches are the state watches this child asked for, stopped when it exits (ADR 0201): a child
// that starts again watches again, as its code runs again.
watches []func()
}
// maxLine is the longest line the runner reads from a bundle; a variable so a test need not write megabytes.
var maxLine = 16 << 20
// errNoAnswer is a call the bundle did not answer within its bound.
var errNoAnswer = errors.New("the bundle did not answer in time")
// readCapped reads one line of at most max bytes. tooLong says it was longer; the rest is not read.
func readCapped(r *bufio.Reader, max int) (line []byte, tooLong bool, err error) {
for {
part, isPrefix, e := r.ReadLine()
line = append(line, part...)
if len(line) > max {
return nil, true, nil
}
if e != nil || !isPrefix {
return line, false, e
}
}
}
func (c *child) write(m any) error {
b, err := wire.Marshal(m)
if err != nil {
@@ -160,9 +247,17 @@ func (c *child) write(m any) error {
var stackLine = regexp.MustCompile(`^\s+at\s`)
// colour is a terminal's colour or style sequence, which node writes into a stack trace when its
// environment asks for colour (FORCE_COLOR): taken out before a line is judged or kept as a bundle's
// reason, so the reason reads the same in every environment.
var colour = regexp.MustCompile("\x1b\\[[0-9;]*m")
// Start launches one bundle and learns its tools. It fails when the child cannot be started or does
// not complete the handshake. A child that exits later is started again on its next call.
func Start(module, entry string, env []string, mesh Bus, logf func(string, ...any)) (*Launched, error) {
if shuttingDown.Load() {
return nil, fmt.Errorf("%s's bundle is not started: the runtime is stopping", module)
}
var mu sync.Mutex
var current *child
// The child that last subscribed is the one events are handed to: a restarted child subscribes
@@ -191,9 +286,24 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
return err
}
// The heartbeat of the module's consumer goes to the child that last subscribed, as a notification: the
// bundle answers nothing, and a child not running misses it, as it misses the events it did not take.
beat := func(heartbeat json.RawMessage) {
mu.Lock()
c := subscriber
mu.Unlock()
if c == nil {
return
}
_ = c.write(map[string]any{"jsonrpc": "2.0", "method": "mesh/heartbeat", "params": heartbeat})
}
var restart func(after time.Duration)
var startedAt time.Time
pause := RestartFirst
// The restart pauses are read once, here: the exit handling runs on its own goroutine and must not
// read the package's settings while a caller (a test) changes them (novox/hq issue 476).
pauseFirst, pauseMost, settle := RestartFirst, RestartMost, RestartSettle
lineCap := maxLine
pause := pauseFirst
start := func() (*child, error) {
cmd := exec.Command(entry)
@@ -213,7 +323,8 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
if err := cmd.Start(); err != nil {
return nil, err
}
c := &child{cmd: cmd, stdin: stdin, pending: map[int64]chan message{}, next: 1, dead: make(chan struct{})}
c := &child{cmd: cmd, stdin: stdin, pending: map[int64]chan message{}, next: 1, dead: make(chan struct{}),
handled: make(chan struct{})}
var lastSaid string
var saidMu sync.Mutex
stderrDone := make(chan struct{})
@@ -227,18 +338,35 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
continue
}
logf("[%s] %s", module, line)
if !stackLine.MatchString(line) && !strings.HasPrefix(line, "Node.js v") {
plain := colour.ReplaceAllString(line, "")
if !stackLine.MatchString(plain) && !strings.HasPrefix(plain, "Node.js v") {
saidMu.Lock()
lastSaid = strings.TrimSpace(line)
lastSaid = strings.TrimSpace(plain)
saidMu.Unlock()
}
}
}()
go func() {
scan := bufio.NewScanner(stdout)
scan.Buffer(make([]byte, 64*1024), 16<<20)
for scan.Scan() {
line := strings.TrimSpace(scan.Text())
defer close(c.handled)
rd := bufio.NewReaderSize(stdout, 64*1024)
for {
raw, tooLong, rerr := readCapped(rd, lineCap)
if tooLong {
// Never left frozen (issue 1966): a reader that stops while the bundle lives leaves the bundle's
// next write blocked for good. Said, and the bundle ended; the exit path below fails what waits
// on it and starts it again.
reason := fmt.Sprintf("%s's bundle wrote a line over %d bytes (a tool's answer the bus cannot carry)", module, lineCap)
logf("[mesh-tools] %s; ending the bundle", reason)
c.mu.Lock()
c.endedFor = reason
c.mu.Unlock()
_ = cmd.Process.Kill()
break
}
if rerr != nil && len(raw) == 0 {
break
}
line := strings.TrimSpace(string(raw))
if line == "" {
continue
}
@@ -256,20 +384,85 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
go func(m message) {
var result any = map[string]any{}
var err error
asked := time.Now()
switch m.Method {
case "mesh/publish":
err = mesh.Publish(m.Params)
var said struct {
Ack bool `json:"ack"`
}
_ = json.Unmarshal(m.Params, &said)
if !said.Ack {
err = mesh.Publish(m.Params)
break
}
// With ack the runtime answers only once the bus acknowledged the event: its sequence, or {error}
// (design 47 G §559). A refusal is an answer, so a writer says it as a finding's resend.
acker, ok := mesh.(interface {
PublishAcked(params json.RawMessage) (uint64, error)
})
switch {
case !ok:
result = map[string]any{"error": "this tool runner's bus publishes no event with an acknowledgement"}
default:
if seq, perr := acker.PublishAcked(m.Params); perr != nil {
result = map[string]any{"error": perr.Error()}
} else {
result = map[string]any{"sequence": seq}
}
}
case "mesh/ask":
var asked json.RawMessage
if asked, err = mesh.Ask(m.Params); err == nil {
result = asked
}
case stdio.CapabilitiesCall:
// The capability words this runner states (design 47 G §576); a Bus that states none answers none.
words := []string{}
if w, ok := mesh.(interface{ CapabilityWords() []string }); ok {
words = w.CapabilityWords()
}
var raw []byte
if raw, err = json.Marshal(map[string]any{"words": words}); err == nil {
result = json.RawMessage(raw)
}
case "mesh/subscribe":
var said struct {
Pattern string `json:"pattern"`
Deliver string `json:"deliver"`
}
_ = json.Unmarshal(m.Params, &said)
if said.Deliver == "new" {
// An ordered consumer of its own from the next message on (design 47 G §559, A-153): each message
// is handed to this child as an event is, and the consumer ends with the child.
newer, ok := mesh.(interface {
SubscribeNew(pattern string, deliver func(json.RawMessage) error) (func(), error)
})
if !ok {
err = errors.New("this tool runner's bus makes no subscription for new messages")
break
}
var stop func()
stop, err = newer.SubscribeNew(said.Pattern, func(envelope json.RawMessage) error {
_, err := c.ask(module, "mesh/event", map[string]any{"envelope": envelope}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
break
}
mu.Lock()
subscriber = c
mu.Unlock()
err = mesh.Subscribe(deliver)
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds":
err = mesh.Subscribe(deliver, beat)
case "mesh/seat.publish", "mesh/seat.prove", "mesh/seat.record", "mesh/seat.kinds", "mesh/seat.pull", "mesh/seat.settle":
var answered json.RawMessage
if answered, err = mesh.Seat(strings.TrimPrefix(m.Method, "mesh/seat."), m.Params, nil); err == nil {
result = answered
@@ -283,11 +476,59 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
result = answered
}
case "mesh/state.get", "mesh/state.put", "mesh/state.delete", "mesh/state.keys",
"mesh/state.create", "mesh/state.update":
"mesh/state.create", "mesh/state.update", "mesh/state.history", "mesh/state.info", "mesh/state.page":
var answered json.RawMessage
if answered, err = mesh.State(strings.TrimPrefix(m.Method, "mesh/state."), m.Params); err == nil {
result = answered
}
case "mesh/log.append", "mesh/log.read", "mesh/log.newest", "mesh/log.last", "mesh/log.info":
var answered json.RawMessage
if answered, err = mesh.Log(strings.TrimPrefix(m.Method, "mesh/log."), m.Params); err == nil {
result = answered
}
case "mesh/provision.get":
var answered json.RawMessage
if answered, err = mesh.Provision("get", m.Params); err == nil {
result = answered
}
case "mesh/provision.watch":
// As a state's watch: each change asked of this child in order, naming the watch by the
// id the child asked it with, and answered once the current value is handed over.
var stop func()
watch := m.ID
stop, err = mesh.ProvisionWatch(m.Params, func(change json.RawMessage) error {
_, err := c.ask(module, "mesh/provision", map[string]any{"watch": watch, "change": change}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
case "mesh/log.watch":
// As a state's watch: each entry asked of this child in order, `mesh/log {watch, entry}`,
// naming the watch by the id the child asked it with (novox/hq ADR 0571).
var stop func()
watch := m.ID
stop, err = mesh.LogWatch(m.Params, func(entry json.RawMessage) error {
_, err := c.ask(module, "mesh/log", map[string]any{"watch": watch, "entry": entry}, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
case "mesh/state.watch":
// Each change is asked of this child, in order, naming the watch it is for by
// the id the child asked it with — two watches of one state are two handlers;
@@ -309,13 +550,47 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
c.mu.Unlock()
}
}
case buscalls.ObjectsWatch, buscalls.LedgerWatch:
// As a state's watch: each change asked of this child in order, by the method of its
// family; the change names the watch by the id the bundle gave in its request.
var stop func()
told := busStateWatches[m.Method]
stop, err = mesh.BusStateWatch(m.Method, m.Params, func(change json.RawMessage) error {
_, err := c.ask(module, told, change, EventTimeout)
return err
})
if err == nil {
c.mu.Lock()
if c.why != nil {
c.mu.Unlock()
stop()
} else {
c.watches = append(c.watches, stop)
c.mu.Unlock()
}
}
default:
if busStateCalls[m.Method] {
var answered json.RawMessage
if answered, err = mesh.BusState(m.Method, m.Params); err == nil {
result = answered
}
break
}
if len(m.ID) > 0 {
_ = c.write(map[string]any{"jsonrpc": "2.0", "id": m.ID,
"error": map[string]any{"code": -32601, "message": "the runtime answers no " + m.Method + " from a bundle"}})
}
return
}
// A state or log operation the bus refused or let time out reaches a caller as a call that
// does not answer: say it here, with how long it took, so the next hang is readable.
// A lost compare-and-set is the normal answer of two writers to one key, retried by its caller: not said.
if err != nil && errorCode(err) != CodeChanged && (strings.HasPrefix(m.Method, "mesh/state.") ||
strings.HasPrefix(m.Method, "mesh/log.") || busStateCalls[m.Method]) {
logf("[mesh-tools] %s's bundle asked %s and it failed after %s: %v", module, m.Method,
time.Since(asked).Round(time.Millisecond), err)
}
if len(m.ID) == 0 {
return
}
@@ -357,6 +632,9 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
}
saidMu.Unlock()
c.mu.Lock()
if c.endedFor != "" {
why = c.endedFor + "; the bundle was ended (" + code + ")"
}
c.why = errors.New(why)
watches := c.watches
c.watches = nil
@@ -364,6 +642,10 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
for _, stop := range watches {
stop()
}
// A session of the bundle's put or get ends with its channel (design 47 G §518).
if d, ok := mesh.(interface{ DropSessions() }); ok {
d.DropSessions()
}
close(c.dead)
mu.Lock()
// Only a child that was serving is brought back; one that died in its own handshake was
@@ -380,17 +662,20 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
}
wasStopped := stopped
// Started again at once if it ran a while; after a growing pause while it keeps exiting.
wait := RestartFirst
if time.Since(startedAt) < RestartSettle {
wait := pauseFirst
if time.Since(startedAt) < settle {
wait = pause
if pause *= 2; pause > RestartMost {
pause = RestartMost
if pause *= 2; pause > pauseMost {
pause = pauseMost
}
} else {
pause = RestartFirst
pause = pauseFirst
}
mu.Unlock()
if !wasStopped && wasServing {
if wasServing && !wasStopped && shuttingDown.Load() {
logf("[mesh-tools] %s; not started again: the runtime is stopping", why)
}
if !wasStopped && wasServing && !shuttingDown.Load() {
// Every bundle stays up (ADR 0198): code that runs long — a handler, a provisioner —
// is not waiting for a call to bring it back, and a tool bundle back early costs nothing.
logf("[mesh-tools] %s; started again in %s", why, wait.Round(100*time.Millisecond))
@@ -408,14 +693,27 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
asking := func(method string, params any, timeout time.Duration) (json.RawMessage, error) {
mu.Lock()
c := current
c, gone := current, stopped
mu.Unlock()
if gone {
// A stopped bundle is never started again by a call: a process started now would be one
// nothing stops (issue 1945).
return nil, fmt.Errorf("%s's bundle is stopped; it answers no call", module)
}
if c == nil {
if shuttingDown.Load() {
return nil, fmt.Errorf("%s's bundle is not started again: the runtime is stopping", module)
}
fresh, err := start()
if err != nil {
return nil, err
}
mu.Lock()
if stopped {
mu.Unlock()
_ = fresh.cmd.Process.Signal(syscall.SIGTERM)
return nil, fmt.Errorf("%s's bundle is stopped; it answers no call", module)
}
if current == nil {
current = fresh
startedAt = time.Now()
@@ -433,7 +731,7 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
go func() {
time.Sleep(after)
mu.Lock()
if stopped || current != nil {
if stopped || current != nil || shuttingDown.Load() {
mu.Unlock()
return
}
@@ -442,10 +740,10 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
mu.Lock()
defer mu.Unlock()
if err != nil {
if !stopped {
if !stopped && !shuttingDown.Load() {
wait := pause
if pause *= 2; pause > RestartMost {
pause = RestartMost
if pause *= 2; pause > pauseMost {
pause = pauseMost
}
logf("[mesh-tools] %s's bundle did not start again: %v; trying in %s", module, err, wait)
go restart(wait)
@@ -505,8 +803,14 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
if len(args) == 0 || string(args) == "null" {
args = json.RawMessage("{}")
}
res, err := asking("tools/call", callParams(full, caller, args), CallTimeout)
call, end := mesh.Begin(caller)
defer end()
res, err := asking("tools/call", callParams(full, caller, call, args), CallTimeout)
if err != nil {
if errors.Is(err, errNoAnswer) {
// Its caller is gone by now and nobody else would hear of it (issue 1966).
logf("[mesh-tools] %s.%s called by %q: %v", module, full, caller, err)
}
return nil, err
}
var called struct {
@@ -550,6 +854,12 @@ func Start(module, entry string, env []string, mesh Bus, logf func(string, ...an
mu.Unlock()
if c != nil && c.cmd.Process != nil {
_ = c.cmd.Process.Signal(syscall.SIGTERM)
// Returns once the bundle's exit is handled, so nothing of it runs on after Stop
// (novox/hq issue 476); bounded, for a bundle that does not exit on SIGTERM.
select {
case <-c.handled:
case <-time.After(StopWait):
}
}
}}
for _, under := range order {
@@ -598,7 +908,7 @@ func (c *child) ask(module, method string, params any, timeout time.Duration) (j
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds", module, method, int(timeout/time.Second))
return nil, fmt.Errorf("%s's bundle did not answer %s in %ds: %w", module, method, int(timeout/time.Second), errNoAnswer)
}
}
@@ -0,0 +1,111 @@
package launch
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
)
// A bundle that asks for one of its values and watches it, and writes each change it is handed to $OUT.
const provisionBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"g1","method":"mesh/provision.get","params":{"kind":"config","key":"role"}}\n'
printf '{"jsonrpc":"2.0","id":"w1","method":"mesh/provision.watch","params":{"kind":"config","key":"role"}}\n' ;;
*'"method":"mesh/provision"'*)
printf '%s\n' "$line" >> "$OUT"
printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"id":"g1"'*) printf '%s\n' "$line" >> "$OUT" ;;
esac
done
`
type provisioning struct {
subscribing
mu sync.Mutex
asked []string
}
func (b *provisioning) Provision(verb string, params json.RawMessage) (json.RawMessage, error) {
b.mu.Lock()
defer b.mu.Unlock()
b.asked = append(b.asked, verb+" "+string(params))
return json.RawMessage(`"the build machine"`), nil
}
func (b *provisioning) ProvisionWatch(params json.RawMessage, deliver func(json.RawMessage) error) (func(), error) {
b.mu.Lock()
b.asked = append(b.asked, "watch "+string(params))
b.mu.Unlock()
return func() {}, deliver(json.RawMessage(`{"kind":"config","key":"role","value":"the build machine"}`))
}
// novox/hq ADR 0349 §14, ADR 0370 §4: a bundle's ask for its values reaches the runtime's bus from its own channel,
// answered there, and a watch's change is handed to it naming the watch it asked.
func TestABundleAsksForItsValuesAndIsHandedItsWatchsChanges(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(provisionBundle), 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(t.TempDir(), "out")
b := &provisioning{}
l, err := Start("claude-code", entry, append(os.Environ(), "OUT="+out), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
deadline := time.Now().Add(10 * time.Second)
for {
said, _ := os.ReadFile(out)
if strings.Count(string(said), "\n") >= 2 {
// Each line as the bundle read it, by what it is: the get's answer, and the watch's change.
got := map[string]string{}
for _, line := range strings.Split(strings.TrimSpace(string(said)), "\n") {
var m struct {
ID any `json:"id"`
Method string `json:"method"`
Result json.RawMessage `json:"result"`
Params json.RawMessage `json:"params"`
}
if err := json.Unmarshal([]byte(line), &m); err != nil {
t.Fatalf("the bundle read %s", line)
}
if m.Method != "" {
got[m.Method] = string(m.Params)
} else {
got["answer"] = string(m.Result)
}
}
if got["answer"] != `"the build machine"` {
t.Errorf("the get was answered %s", got["answer"])
}
if got["mesh/provision"] != `{"change":{"kind":"config","key":"role","value":"the build machine"},"watch":"w1"}` {
t.Errorf("the watch was handed %s", got["mesh/provision"])
}
break
}
if time.Now().After(deadline) {
t.Fatalf("the bundle was handed %q", said)
}
time.Sleep(20 * time.Millisecond)
}
b.mu.Lock()
defer b.mu.Unlock()
slices.Sort(b.asked)
if len(b.asked) != 2 || b.asked[0] != `get {"kind":"config","key":"role"}` || b.asked[1] != `watch {"kind":"config","key":"role"}` {
t.Errorf("the runtime's bus was asked %q", b.asked)
}
}
func (b *provisioning) LogWatch(json.RawMessage, func(json.RawMessage) error) (func(), error) {
return func() {}, nil
}
+102
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
@@ -68,4 +69,105 @@ func TestABundleTakesASeatsWorkThroughItsOwnChannel(t *testing.T) {
if err != nil || string(got) != `{"taken":true}` {
t.Fatalf("the work was not handed to the bundle: %s %v", got, err)
}
// Design 47 G §576, §518: a bundle reads the runner's capability words with mesh/capabilities, then runs its put session
// through the runner's put-chunk, and the sessions of its channel are dropped when the channel closes.
t.Run("ABundleReadsTheCapabilityWordsRunsAPutSessionAndItsChannelsClosingDropsThem", func(t *testing.T) {
dir := t.TempDir()
entry := filepath.Join(dir, "bundle")
if err := os.WriteFile(entry, []byte(puttingBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &sessionBus{dropped: make(chan struct{})}
l, err := Start("putter", entry, os.Environ(), b, func(string, ...any) {})
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
select {
case <-b.dropped:
case <-time.After(5 * time.Second):
t.Fatal("the bundle's sessions were not dropped when its channel closed")
}
b.mu.Lock()
defer b.mu.Unlock()
if len(b.chunks) != 1 || !strings.HasPrefix(b.chunks[0], "mesh/objects.put-chunk ") || !strings.Contains(b.chunks[0], `"seq":1`) {
t.Fatalf("the runner took %v", b.chunks)
}
})
// Design 47 G §559: `mesh/publish` with `ack` is answered with its sequence once the bus acknowledged it, or `{error}`;
// `mesh/subscribe` with `deliver: new` makes a subscription for new messages that ends with the bundle's channel.
t.Run("APublishWithAckIsAnsweredWithItsSequenceAndASubscriptionForNewMessagesEndsWithTheChannel", func(t *testing.T) {
dir := t.TempDir()
entry, out := filepath.Join(dir, "bundle"), filepath.Join(dir, "out")
if err := os.WriteFile(entry, []byte(findingBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &findingBus{stopped: make(chan struct{})}
l, err := Start("writer", entry, append(os.Environ(), "FINDING_OUT="+out), b, func(string, ...any) {})
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
select {
case <-b.stopped:
case <-time.After(5 * time.Second):
t.Fatal("the subscription for new messages was not ended with the bundle's channel")
}
got, _ := os.ReadFile(out)
for _, want := range []string{`"sequence":42`, `"error":"the bus refused it"`} {
if !strings.Contains(string(got), want) {
t.Errorf("the bundle was answered %s, want it to hold %s", got, want)
}
}
b.mu.Lock()
defer b.mu.Unlock()
if b.pattern != "mesh-conditions.findings-asked" {
t.Errorf("the subscription was made for %q", b.pattern)
}
if len(b.published) != 0 {
t.Errorf("a publish with ack went the unacknowledged way: %v", b.published)
}
})
}
// A bundle that pulls one ask from its worker when it is listed.
const pullingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*)
printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"p1","method":"mesh/seat.pull","params":{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker","wait_ms":0}}\n' ;;
esac
done
`
// 1887/A11: a bundle's pull reaches the runtime's bus from its own channel, as its other seat traffic does.
func TestABundlePullsASeatsWorkThroughItsOwnChannel(t *testing.T) {
entry := filepath.Join(t.TempDir(), "bundle")
if err := os.WriteFile(entry, []byte(pullingBundle), 0o755); err != nil {
t.Fatal(err)
}
b := &seating{}
l, err := Start("mesh-build-go", entry, os.Environ(), b, t.Logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
for i := 0; ; i++ {
b.mu.Lock()
verb, params := b.verb, b.params
b.mu.Unlock()
if verb == "pull" {
if params != `{"worker":"SEAT_NODE_BUILD_GO_DELIVERY_worker","wait_ms":0}` {
t.Fatalf("pulled with %s", params)
}
return
}
if i > 100 {
t.Fatal("the bundle's pull never reached the runtime")
}
time.Sleep(20 * time.Millisecond)
}
}
@@ -0,0 +1,78 @@
package launch
import (
"encoding/json"
"errors"
"strings"
"sync"
)
// puttingBundle asks the capability words, and once they are answered with put-sessions sends the one chunk of a put
// session, as mesh-sdk's bundle does; it then ends, which closes its channel.
const puttingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"w1","method":"mesh/capabilities","params":{}}\n' ;;
*'"id":"w1"'**put-sessions*) printf '{"jsonrpc":"2.0","id":"p1","method":"mesh/objects.put-chunk","params":{"objects":"o","name":"n","call":"c","seq":1,"last":true,"data":"aGk="}}\n' ;;
*'"id":"p1"'*) exit 0 ;;
esac
done
`
type sessionBus struct {
subscribing
mu sync.Mutex
chunks []string
dropped chan struct{}
}
func (b *sessionBus) CapabilityWords() []string { return []string{"put-sessions"} }
func (b *sessionBus) BusState(method string, params json.RawMessage) (json.RawMessage, error) {
b.mu.Lock()
b.chunks = append(b.chunks, method+" "+string(params))
b.mu.Unlock()
return json.RawMessage(`{"name":"n"}`), nil
}
func (b *sessionBus) DropSessions() { close(b.dropped) }
// findingBundle asks, as mesh-sdk's finding writer does, a publish with ack that the bus takes, one it refuses, and a
// subscription for new messages; each answer it gets is written to $FINDING_OUT, and it ends after the subscription's.
const findingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"a1","method":"mesh/publish","params":{"key":"taken","ack":true,"body":{}}}\n' ;;
*'"id":"a1"'*) printf '%s\n' "$line" >>"$FINDING_OUT"
printf '{"jsonrpc":"2.0","id":"a2","method":"mesh/publish","params":{"key":"refused","ack":true,"body":{}}}\n' ;;
*'"id":"a2"'*) printf '%s\n' "$line" >>"$FINDING_OUT"
printf '{"jsonrpc":"2.0","id":"s1","method":"mesh/subscribe","params":{"pattern":"mesh-conditions.findings-asked","deliver":"new"}}\n' ;;
*'"id":"s1"'*) printf '%s\n' "$line" >>"$FINDING_OUT"; exit 0 ;;
esac
done
`
type findingBus struct {
subscribing
mu sync.Mutex
pattern string
stopped chan struct{}
}
func (b *findingBus) PublishAcked(params json.RawMessage) (uint64, error) {
if strings.Contains(string(params), `"refused"`) {
return 0, errors.New("the bus refused it")
}
return 42, nil
}
func (b *findingBus) SubscribeNew(pattern string, _ func(json.RawMessage) error) (func(), error) {
b.mu.Lock()
b.pattern = pattern
b.mu.Unlock()
return func() { close(b.stopped) }, nil
}
@@ -0,0 +1,152 @@
package launch
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// changedError is a lost compare-and-set as the bus answers it: told by its code (novox/hq ADR 0259).
type changedError struct{}
func (changedError) Error() string { return "the key changed since revision 3" }
func (changedError) ErrorCode() int { return CodeChanged }
// refusingState answers the state "lost" with a lost compare-and-set and every other state with a time-out.
type refusingState struct{ subscribing }
func (*refusingState) State(_ string, params json.RawMessage) (json.RawMessage, error) {
if strings.Contains(string(params), `"state":"lost"`) {
return nil, changedError{}
}
return nil, errors.New("the bus did not answer in 10s")
}
// A bundle that asks the runtime for the state "lost" once it is started, and for the state "copied" once that is
// answered, so the first answer's handling has ended before the second is asked.
const askingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[]}}\n' "$id"
printf '{"jsonrpc":"2.0","id":"q1","method":"mesh/state.update","params":{"state":"lost","key":"k"}}\n' ;;
*'"id":"q1"'*) printf '{"jsonrpc":"2.0","id":"q2","method":"mesh/state.get","params":{"state":"copied","key":"k"}}\n' ;;
esac
done
`
// A state operation the bus refuses or lets time out is said in the runtime's log with its method, its bundle and its
// time (issue 1966: a hand-act seat whose calls hung said nothing); a lost compare-and-set, asked first, is not.
func TestAFailedStateOperationIsSaidInTheRuntimesLog(t *testing.T) {
dir := t.TempDir()
entry := filepath.Join(dir, "bundle")
if err := os.WriteFile(entry, []byte(askingBundle), 0o755); err != nil {
t.Fatal(err)
}
said := make(chan string, 16)
logf := func(f string, a ...any) { said <- fmt.Sprintf(f, a...) }
l, err := Start("asker", entry, os.Environ(), &refusingState{}, logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
var heard []string
timeout := time.After(5 * time.Second)
for {
select {
case line := <-said:
heard = append(heard, line)
if strings.Contains(line, "mesh/state.update") {
t.Fatalf("a lost compare-and-set was said as a failure: %q", line)
}
if strings.Contains(line, "asker's bundle asked mesh/state.get and it failed after") && strings.Contains(line, "did not answer in 10s") {
return
}
case <-timeout:
t.Fatalf("the failed state operation was not said; the log: %v", heard)
}
}
}
// A bundle that lists one tool, "t", and answers a call to it as its MODE says: "long" with a line over the runner's cap,
// "silent" with nothing at all.
const callingBundle = `#!/bin/sh
while IFS= read -r line; do
id=$(printf '%s' "$line" | sed -n 's/.*"id":\([0-9]*\)[,}].*/\1/p')
case "$line" in
*'"method":"initialize"'*) echo started >&2; printf '{"jsonrpc":"2.0","id":%s,"result":{}}\n' "$id" ;;
*'"method":"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[{"name":"t","description":"d"}]}}\n' "$id" ;;
*'"method":"tools/call"'*) [ "$MODE" = long ] && printf '%4096s\n' x ;;
esac
done
`
// Issue 1966: a bundle's line over the runner's cap, and a call the bundle never answers, are said in the runtime's log;
// the over-long line ends the bundle and fails the call at once, never leaving the runner reading nothing while the
// bundle's next write blocks.
func TestAnOverlongLineAndAnUnansweredCallAreSaid(t *testing.T) {
oldMax, oldCall, oldFirst := maxLine, CallTimeout, RestartFirst
maxLine, RestartFirst = 1024, 10*time.Millisecond
t.Cleanup(func() { maxLine, CallTimeout, RestartFirst = oldMax, oldCall, oldFirst })
for _, row := range []struct {
mode string
timeout time.Duration
}{
// The call's own bound is long: only the runner ending the bundle fails it within a second.
{"long", 10 * time.Second},
{"silent", 300 * time.Millisecond},
} {
t.Run(row.mode, func(t *testing.T) {
CallTimeout = row.timeout
dir := t.TempDir()
entry := filepath.Join(dir, "bundle")
if err := os.WriteFile(entry, []byte(callingBundle), 0o755); err != nil {
t.Fatal(err)
}
said := make(chan string, 32)
logf := func(f string, a ...any) { said <- fmt.Sprintf(f, a...) }
l, err := Start("caller", entry, append(os.Environ(), "MODE="+row.mode), &refusingState{}, logf)
if err != nil {
t.Fatal(err)
}
t.Cleanup(l.Stop)
begun := time.Now()
_, err = l.Registrations[0].Tools[0].Run("caller-1", nil)
if err == nil {
t.Fatal("the call was answered")
}
var heard []string
timeout := time.After(5 * time.Second)
want := "caller.t called by \"caller-1\""
if row.mode == "long" {
want = "ending the bundle"
if took := time.Since(begun); took > time.Second {
t.Fatalf("the call hung for %s", took)
}
if !strings.Contains(err.Error(), "over 1024 bytes") || strings.Contains(err.Error(), "did not answer") {
t.Fatalf("the call failed with %q, not the cap's reason", err)
}
}
// For "long" the bundle is also started again: it says "started" on each start, so twice in all.
started, found := 0, false
for !found || (row.mode == "long" && started < 2) {
select {
case line := <-said:
heard = append(heard, line)
found = found || strings.Contains(line, want)
if line == "[caller] started" {
started++
}
case <-timeout:
t.Fatalf("not said, or the bundle not started again; the log: %v", heard)
}
}
})
}
}
@@ -0,0 +1,14 @@
package launch
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// A-42: a launched bundle may ask mesh/stream.publish.
func TestABundleMayAskStreamPublish(t *testing.T) {
if !busStateCalls[buscalls.StreamPublish] {
t.Fatal("mesh/stream.publish is not among the methods a bundle may ask")
}
}
+576
View File
@@ -0,0 +1,576 @@
// Package ledger is the tool runner's side of a ledger (novox/hq ADR 0597, design 25 § 12.4; issue 1966): a storage type
// whose state is derived from its history. `apply` checks an operation against its type, appends it to the instance's
// history with the last sequence expected, derives the state by the operation's effect and writes it as an update of the
// revision read, and writes the instance's snapshot. What the history, the state and the snapshots are kept in is given
// (History, State, Snapshots): the bus in the runtime, fakes in a test.
package ledger
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// The refusals the stores answer, told by errors.Is.
var (
// ErrRaced is an append or a snapshot written with an expected sequence that is no longer the last: another writer
// wrote first.
ErrRaced = errors.New("another writer wrote the instance's history first")
// ErrChanged is a state written at a revision that is no longer the key's.
ErrChanged = errors.New("the state was written by another writer first")
// ErrFull is a state bucket that refuses a write because it is full.
ErrFull = errors.New("the state bucket is full")
)
// CodeChanged is the error code a bundle is answered a lost race with, as a lost state update (mesh-sdk stdio.CodeChanged).
const CodeChanged = -32010
// raced is an apply refused because another writer appended to the instance first (ADR 0597 §5).
type raced struct{ instance string }
func (e raced) Error() string {
return "the operation on " + e.instance + " is not taken: another writer appended to its history first; read it and apply again (novox/hq ADR 0597 §5)"
}
func (e raced) Is(target error) bool { return target == ErrRaced }
func (e raced) ErrorCode() int { return CodeChanged }
// Kept is one message a history keeps: its sequence and its bytes.
type Kept struct {
Sequence uint64
Data []byte
}
// History is a ledger's history: a stream with a subject per instance, each instance named by its key.
type History interface {
// Last is the sequence of the instance's last operation, 0 when it has none.
Last(key string) (uint64, error)
// Append writes one operation, expecting the instance's last sequence to be expect (0: it has none); ErrRaced when
// it is not.
Append(key string, op []byte, expect uint64) (uint64, error)
// Read is the operations of the instances keys names (a key, with `*` for one name and `>` for the rest) from a
// sequence on, oldest first, at most limit of them (every one when limit is 0).
Read(keys string, from uint64, limit int) ([]Kept, error)
// Purge removes every operation of the instance key names, and nothing of another (an instance's removal).
Purge(key string) error
// Fill is the history's fill against its cap, the ledger's cap (buscalls.LedgerFill; design 47 G §587).
Fill() (buscalls.Fill, error)
}
// State is a ledger's state bucket: each instance's record under its key.
type State interface {
// Get is the key's value and revision; nil and 0 when it has none.
Get(key string) ([]byte, uint64, error)
// Write writes the key's value at the revision read (0: only when it has none); ErrChanged when the key is no longer
// at it, ErrFull when the bucket is full.
Write(key string, value []byte, revision uint64) (uint64, error)
// Keys is every key the bucket holds.
Keys() ([]string, error)
// Delete removes the key; a key it does not hold is no fault.
Delete(key string) error
}
// Snapshots are a ledger's snapshots: the latest record of each instance, on a subject of its own, never expiring.
type Snapshots interface {
// Get is the key's latest snapshot and the sequence it was written at; nil and 0 when it has none.
Get(key string) ([]byte, uint64, error)
// Put writes the key's snapshot, expecting its last to be at expect (0: none); ErrRaced when it is not.
Put(key string, value []byte, expect uint64) error
// Purge removes the key's snapshot; one it does not hold is no fault.
Purge(key string) error
}
// Ledger is one ledger of one module: its types and where it is kept.
type Ledger struct {
Module string
Name string
// Account is the owning module's authenticated account, which every operation it applies is stamped with.
Account string
Types []catalogue.LedgerType
History History
State State
Snapshots Snapshots
// Indexes are its declared indexes' buckets, by index name.
Indexes map[string]Index
// Now is the clock an operation is stamped by; time.Now when nil.
Now func() time.Time
// Logf says what an apply could not do after its operation was taken; nothing is said when nil.
Logf func(format string, args ...any)
}
func (l *Ledger) say(format string, args ...any) {
if l.Logf != nil {
l.Logf(format, args...)
}
}
// Record is what the state and a snapshot keep of an instance: its state and the sequence of the last operation in it.
type Record struct {
Sequence uint64 `json:"sequence"`
State json.RawMessage `json:"state"`
}
// segment is one part of an instance's path.
var segment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
// KeyOf is the key an instance is kept under, in the state and as its history's and snapshot's subject token: its path's
// parts joined by dots (`decision/0290` is `decision.0290`).
func KeyOf(instance string) (string, error) {
parts := strings.Split(instance, "/")
if len(parts) < 2 {
return "", fmt.Errorf("%q is not an instance: an instance is named by a path whose first part is its type's name, "+
"such as decision/0290 (novox/hq ADR 0597 §1)", instance)
}
for _, p := range parts {
if !segment.MatchString(p) {
return "", fmt.Errorf("%q is not an instance: each part of its path holds letters, digits, - and _ alone", instance)
}
}
return strings.Join(parts, "."), nil
}
// InstanceOf is the instance a key names.
func InstanceOf(key string) string { return strings.ReplaceAll(key, ".", "/") }
func (l *Ledger) now() time.Time {
if l.Now != nil {
return l.Now().UTC()
}
return time.Now().UTC()
}
// typeOf is the instance's type, by its path's first part.
func (l *Ledger) typeOf(instance string) (catalogue.LedgerType, error) {
name, _, _ := strings.Cut(instance, "/")
for _, t := range l.Types {
if t.Name == name {
return t, nil
}
}
return catalogue.LedgerType{}, fmt.Errorf("%s's ledger %s has no type %q, which the instance %s names", l.Module, l.Name, name, instance)
}
// operationOf is the operation a type allows by its name.
func operationOf(t catalogue.LedgerType, name string) (catalogue.LedgerOperation, error) {
for _, op := range t.Operations {
if op.Name == name {
return op, nil
}
}
return catalogue.LedgerOperation{}, fmt.Errorf("the type %s allows no operation %q", t.Name, name)
}
// Digest is the digest a type pairs a text field with (ADR 0597 §2): sha256 of the text, as the knowledge base writes it.
func Digest(text string) string {
sum := sha256.Sum256([]byte(text))
return "sha256:" + hex.EncodeToString(sum[:])
}
// kinds is the JSON kind each schema type name is held to; a name not here is the judge's to know.
var kinds = map[string]func(any) bool{
"string": func(v any) bool { _, ok := v.(string); return ok },
"number": func(v any) bool { _, ok := v.(float64); return ok },
"integer": func(v any) bool { f, ok := v.(float64); return ok && f == float64(int64(f)) },
"boolean": func(v any) bool { _, ok := v.(bool); return ok },
"object": func(v any) bool { _, ok := v.(map[string]any); return ok },
"array": func(v any) bool { _, ok := v.([]any); return ok },
}
// checked is an operation as apply checked it: its instance's key, its operation and its body.
type checked struct {
key string
op catalogue.LedgerOperation
body map[string]any
}
// check holds an operation to its type (ADR 0597 §2): a type the instance names, an operation it allows, a body that is an
// object whose fields fit the schema, the key its effect names, and every digest matching its text.
func (l *Ledger) check(r buscalls.ApplyRequest) (checked, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return checked{}, err
}
t, err := l.typeOf(r.Instance)
if err != nil {
return checked{}, err
}
op, err := operationOf(t, r.Operation)
if err != nil {
return checked{}, err
}
body := map[string]any{}
if len(r.Body) > 0 && string(r.Body) != "null" {
if err := json.Unmarshal(r.Body, &body); err != nil {
return checked{}, fmt.Errorf("the operation %s's body is a JSON object: %w", r.Operation, err)
}
}
for name, field := range t.Schema {
v, has := body[name]
if !has {
continue
}
if fits, known := kinds[field.Type]; known && !fits(v) {
return checked{}, fmt.Errorf("the operation %s's field %s is a %s in the type %s", r.Operation, name, field.Type, t.Name)
}
}
for name, field := range t.Schema {
if field.DigestOf == "" {
continue
}
digest, hasDigest := body[name]
text, hasText := body[field.DigestOf]
if !hasDigest && !hasText {
continue
}
s, isText := text.(string)
if !hasDigest || !hasText || !isText || digest != Digest(s) {
return checked{}, fmt.Errorf("the operation %s's %s is not the digest of its %s, so it is refused (novox/hq ADR 0597 §2)",
r.Operation, name, field.DigestOf)
}
}
switch op.Effect {
case catalogue.EffectSetFields, catalogue.EffectAppend:
case catalogue.EffectSetEntry, catalogue.EffectRemove:
if op.Effect == catalogue.EffectRemove && op.Into == "" {
break
}
if _, ok := body[op.Key].(string); !ok {
return checked{}, fmt.Errorf("the operation %s names its entry by its field %s, a text, which its body lacks", r.Operation, op.Key)
}
default:
return checked{}, fmt.Errorf("the operation %s's effect %q is not one the mesh offers (novox/hq ADR 0597 §2)", r.Operation, op.Effect)
}
return checked{key: key, op: op, body: body}, nil
}
// effect is an operation's effect on a state (ADR 0597 §2): it copies only the fields the operation names, and the
// operation's sequence and proven author where it says so; null is a removed instance.
func effect(state json.RawMessage, op catalogue.LedgerOperation, body map[string]any, seq uint64, author buscalls.Author) (json.RawMessage, error) {
s := map[string]any{}
if len(state) > 0 && string(state) != "null" {
if err := json.Unmarshal(state, &s); err != nil {
return nil, fmt.Errorf("the state held is not an object: %w", err)
}
}
copied := map[string]any{}
for _, f := range op.Fields {
if v, ok := body[f]; ok {
copied[f] = v
}
}
if op.Sequence {
copied["sequence"] = seq
}
// Written last, so a body's own `author` never stands for the proven one (ADR 0597 §7).
if op.Author {
copied["author"] = author
}
switch op.Effect {
case catalogue.EffectSetFields:
for k, v := range copied {
s[k] = v
}
case catalogue.EffectSetEntry:
m, _ := s[op.Into].(map[string]any)
if m == nil {
m = map[string]any{}
}
m[body[op.Key].(string)] = copied
s[op.Into] = m
case catalogue.EffectAppend:
list, _ := s[op.Into].([]any)
s[op.Into] = append(list, copied)
case catalogue.EffectRemove:
if op.Into == "" {
return json.RawMessage("null"), nil
}
if m, ok := s[op.Into].(map[string]any); ok {
delete(m, body[op.Key].(string))
}
}
return json.Marshal(s)
}
// fold applies one kept operation to a record, as apply derived it.
func (l *Ledger) fold(rec Record, k Kept) (Record, error) {
var o buscalls.Operation
if err := json.Unmarshal(k.Data, &o); err != nil {
return rec, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
}
t, err := l.typeOf(o.Instance)
if err != nil {
return rec, err
}
op, err := operationOf(t, o.Operation)
if err != nil {
return rec, err
}
body := map[string]any{}
if len(o.Body) > 0 && string(o.Body) != "null" {
if err := json.Unmarshal(o.Body, &body); err != nil {
return rec, err
}
}
if (op.Effect == catalogue.EffectSetEntry || (op.Effect == catalogue.EffectRemove && op.Into != "")) && !isText(body[op.Key]) {
return rec, fmt.Errorf("the history's operation %d names no entry", k.Sequence)
}
state, err := effect(rec.State, op, body, k.Sequence, o.Author)
if err != nil {
return rec, err
}
return Record{Sequence: k.Sequence, State: state}, nil
}
func isText(v any) bool { _, ok := v.(string); return ok }
// replay folds the instance's operations after rec's sequence, up to upto (every one when 0), onto rec.
func (l *Ledger) replay(key string, rec Record, upto uint64) (Record, error) {
kept, err := l.History.Read(key, rec.Sequence+1, 0)
if err != nil {
return rec, err
}
for _, k := range kept {
if upto != 0 && k.Sequence > upto {
break
}
if k.Sequence <= rec.Sequence {
continue
}
if rec, err = l.fold(rec, k); err != nil {
return rec, err
}
}
return rec, nil
}
// readState is the instance's record in the state and its revision; an empty record when it has none.
func (l *Ledger) readState(key string) (Record, uint64, error) {
raw, rev, err := l.State.Get(key)
if err != nil || raw == nil {
return Record{}, rev, err
}
var rec Record
if err := json.Unmarshal(raw, &rec); err != nil {
return Record{}, rev, fmt.Errorf("the state of %s is not a record: %w", key, err)
}
return rec, rev, nil
}
// readSnapshot is the instance's snapshot and the sequence it was written at; an empty record when it has none.
func (l *Ledger) readSnapshot(key string) (Record, uint64, error) {
raw, at, err := l.Snapshots.Get(key)
if err != nil || raw == nil {
return Record{}, at, err
}
var rec Record
if err := json.Unmarshal(raw, &rec); err != nil {
return Record{}, at, fmt.Errorf("the snapshot of %s is not a record: %w", key, err)
}
return rec, at, nil
}
// derive is the instance's record caught up with its history up to last: from the state, or from the snapshot when it
// is further on (an instance whose state was lost, or whose operations expired, ADR 0597 §8).
func (l *Ledger) derive(key string, base Record, last uint64) (Record, error) {
if base.Sequence >= last {
return base, nil
}
snap, _, err := l.readSnapshot(key)
if err != nil {
return base, err
}
if snap.Sequence > base.Sequence {
base = snap
}
return l.replay(key, base, last)
}
// settle writes want into the state at the revision read (ADR 0597 §5): a lost update re-reads, and catches the state
// read up to want's sequence from the history, never appending again; a state already further on is left as it is. It
// answers what the state holds after it, and whether it is behind because the bucket is full (ADR 0597 §6).
func (l *Ledger) settle(key string, want Record, rev uint64) (Record, bool, error) {
for tries := 0; tries < 16; tries++ {
raw, err := json.Marshal(want)
if err != nil {
return want, false, err
}
_, err = l.State.Write(key, raw, rev)
switch {
case err == nil:
return want, false, nil
case errors.Is(err, ErrFull):
return want, true, nil
case !errors.Is(err, ErrChanged):
return want, false, err
}
cur, curRev, err := l.readState(key)
if err != nil {
return want, false, err
}
if cur.Sequence >= want.Sequence {
return cur, false, nil
}
if want, err = l.replay(key, cur, want.Sequence); err != nil {
return want, false, err
}
rev = curRev
}
return want, false, fmt.Errorf("the state of %s kept being written by other writers; the operation is in its history and the next apply catches it up", key)
}
// writeSnapshot writes rec as the instance's snapshot unless the snapshot is already as far on (ADR 0597 §8).
func (l *Ledger) writeSnapshot(key string, rec Record) error {
for tries := 0; tries < 16; tries++ {
snap, at, err := l.readSnapshot(key)
if err != nil {
return err
}
if at != 0 && snap.Sequence >= rec.Sequence {
return nil
}
raw, err := json.Marshal(rec)
if err != nil {
return err
}
if err = l.Snapshots.Put(key, raw, at); err == nil || !errors.Is(err, ErrRaced) {
return err
}
}
return fmt.Errorf("the snapshot of %s kept being written by other writers", key)
}
// Apply applies one operation to one instance, stamped with author (ADR 0597 §3, §5, §6, §8; design 25 § 12.4 steps 1-5):
// checked against its type, appended with the instance's last sequence expected, the state derived by its effect and
// written at the revision read, and the snapshot written. A racing writer's append is refused; a full state bucket
// answers taken with the state behind.
func (l *Ledger) Apply(r buscalls.ApplyRequest, author buscalls.Author) (buscalls.Applied, error) {
c, err := l.check(r)
if err != nil {
return buscalls.Applied{}, err
}
last, err := l.History.Last(c.key)
if err != nil {
return buscalls.Applied{}, err
}
held, rev, err := l.readState(c.key)
if err != nil {
return buscalls.Applied{}, err
}
base, err := l.derive(c.key, held, last)
if err != nil {
return buscalls.Applied{}, err
}
body := r.Body
if len(body) == 0 {
body = json.RawMessage("{}")
}
op, err := json.Marshal(buscalls.Operation{Instance: r.Instance, Operation: r.Operation, Body: body, Author: author, At: l.now()})
if err != nil {
return buscalls.Applied{}, err
}
seq, err := l.History.Append(c.key, op, last)
if errors.Is(err, ErrRaced) {
return buscalls.Applied{}, raced{r.Instance}
}
if err != nil {
return buscalls.Applied{}, err
}
state, err := effect(base.State, c.op, c.body, seq, author)
if err != nil {
return buscalls.Applied{}, err
}
// From here the operation is taken: it is in the history, and a fault below is the state's to catch up, said here
// and answered as the state behind, never as a refusal of what later takes effect (ADR 0597 §6).
now, behind, err := l.settle(c.key, Record{Sequence: seq, State: state}, rev)
if err != nil {
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its state could not follow: %v; the next apply "+
"or a rebuild catches it up", l.Module, l.Name, seq, r.Instance, err)
behind, now = true, Record{Sequence: seq, State: state}
}
if behind {
now = Record{Sequence: seq, State: state}
}
if err := l.writeSnapshot(c.key, now); err != nil {
l.say("[mesh-tools] %s's ledger %s took operation %d on %s and its snapshot could not be written: %v; the next "+
"apply or a snapshot repair writes it", l.Module, l.Name, seq, r.Instance, err)
}
return buscalls.Applied{Taken: true, Sequence: seq, StateBehind: behind}, nil
}
// Fill is the fill of the ledger's history against the ledger's cap (buscalls.LedgerFill; design 47 G §587): its messages,
// its bytes and the cap in bytes. The history holds every operation, and the cap is its alone: neither the fullest of the
// stores nor their sum is answered. A history that cannot say its fill is an error naming the ledger, never a fill of
// nothing.
func (l *Ledger) Fill(buscalls.LedgerRequest) (buscalls.Fill, error) {
f, err := l.History.Fill()
if err != nil {
return buscalls.Fill{}, fmt.Errorf("%s's ledger %s cannot say the fill of its history: %w", l.Module, l.Name, err)
}
return f, nil
}
// Remove removes one instance with everything it kept (buscalls.LedgerRemove; design 47 G §587, §155): its operations, its
// snapshot, its key in the state bucket and its keys in every declared index, and answers how many operations and how many
// bytes of the history that was. An instance the ledger does not hold answers 0 and 0, and a removal that left something
// (said, and answered as an error) is made again by calling it again, which removes the rest.
func (l *Ledger) Remove(r buscalls.LedgerRemoveRequest) (buscalls.LedgerRemoved, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return buscalls.LedgerRemoved{}, err
}
out := buscalls.LedgerRemoved{Instance: r.Instance}
before, err := l.History.Fill()
if err != nil {
return out, fmt.Errorf("%s's ledger %s cannot say the fill of its history, so a removal is not made: %w", l.Module, l.Name, err)
}
ops, err := l.History.Read(key, 1, 0)
if err != nil {
return out, err
}
for _, step := range []struct {
what string
remove func(string) error
}{{"operations", l.History.Purge}, {"snapshot", l.Snapshots.Purge}, {"state", l.State.Delete}, {"index entries", l.dropFromIndexes}} {
if err := step.remove(key); err != nil {
l.say("[mesh-tools] %s's ledger %s removes %s and its %s could not be removed: %v; removing it again removes what "+
"is left", l.Module, l.Name, r.Instance, step.what, err)
return out, fmt.Errorf("%s's ledger %s removed part of %s and its %s are left: %w", l.Module, l.Name, r.Instance, step.what, err)
}
}
after, err := l.History.Fill()
if err != nil {
return out, fmt.Errorf("%s's ledger %s removed %s and cannot say the fill of its history after: %w", l.Module, l.Name, r.Instance, err)
}
out.Operations = uint64(len(ops))
if before.Bytes > after.Bytes {
out.Bytes = before.Bytes - after.Bytes
}
return out, nil
}
// dropFromIndexes removes the entries of the instance key names from every index bucket the ledger is bound to.
func (l *Ledger) dropFromIndexes(key string) error {
instance := InstanceOf(key)
for name, ix := range l.Indexes {
held, err := ix.Entries("", 0)
if err != nil {
return fmt.Errorf("the index %s: %w", name, err)
}
for _, e := range held {
var v buscalls.IndexEntry
if json.Unmarshal(e.Value, &v) == nil && v.Instance == instance {
if err := ix.Delete(e.Key); err != nil {
return fmt.Errorf("the index %s: %w", name, err)
}
}
}
}
return nil
}
+148
View File
@@ -0,0 +1,148 @@
package ledger
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
var owner = buscalls.Author{Account: "ledger.anchor.m"}
// 1966/AC13, 1966/AC14 and the checks of an operation against its type (ADR 0597 §2, §5, §6, §8; design 25 § 12.4).
func TestApplyAppendsDerivesAndSnapshots(t *testing.T) {
cases := []struct {
name string
run func(t *testing.T, l *Ledger, f fakes)
}{
{"1966/AC13 an instance at sequence 4 is taken with 5, its state and snapshot the effect's result", func(t *testing.T, l *Ledger, f fakes) {
for i := 1; i <= 4; i++ {
applied(t, l, "decision/0290", "note", fmt.Sprintf(`{"says":"n%d"}`, i), owner)
}
if last, _ := f.history.Last("decision.0290"); last != 4 {
t.Fatalf("the instance is at %d, want 4", last)
}
got := applied(t, l, "decision/0290", "title", `{"title":"A ledger","ignored":"x"}`, owner)
if !got.Taken || got.Sequence != 5 || got.StateBehind {
t.Fatalf("answered %+v, want taken with 5", got)
}
raw, _, _ := f.state.Get("decision.0290")
state := record(t, raw)
want := `{"author":{"account":"ledger.anchor.m"},"notes":[{"says":"n1"},{"says":"n2"},{"says":"n3"},{"says":"n4"}],"sequence":5,"title":"A ledger"}`
if state.Sequence != 5 || string(state.State) != want {
t.Fatalf("the state is %d %s, want 5 %s", state.Sequence, state.State, want)
}
snap, _, _ := f.snapshots.Get("decision.0290")
if string(snap) != string(raw) {
t.Fatalf("the snapshot is %s, the state %s", snap, raw)
}
}},
{"1966/AC14 a full state bucket answers taken and state behind, and the next apply after room catches up", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/0290", "note", `{"says":"first"}`, owner)
f.state.Full = true
got := applied(t, l, "decision/0290", "note", `{"says":"while full"}`, owner)
if !got.Taken || got.Sequence != 2 || !got.StateBehind {
t.Fatalf("answered %+v, want taken with 2 and state behind", got)
}
if raw, _, _ := f.state.Get("decision.0290"); record(t, raw).Sequence != 1 {
t.Fatalf("the full state moved: %s", raw)
}
f.state.Full = false
got = applied(t, l, "decision/0290", "note", `{"says":"after room"}`, owner)
raw, _, _ := f.state.Get("decision.0290")
state := record(t, raw)
if got.StateBehind || state.Sequence != 3 || string(state.State) != `{"notes":[{"says":"first"},{"says":"while full"},{"says":"after room"}]}` {
t.Fatalf("after room: %+v, the state %d %s", got, state.Sequence, state.State)
}
}},
{"each effect: set an entry, remove it, remove the instance", func(t *testing.T, l *Ledger, f fakes) {
text := "the section's text"
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"a","text":%q,"digest":%q}`, text, Digest(text)), owner)
applied(t, l, "decision/1", "section", fmt.Sprintf(`{"id":"b","text":"b","digest":%q}`, Digest("b")), owner)
applied(t, l, "decision/1", "drop-section", `{"id":"a"}`, owner)
raw, _, _ := f.state.Get("decision.1")
if got := string(record(t, raw).State); got != fmt.Sprintf(`{"sections":{"b":{"digest":%q}}}`, Digest("b")) {
t.Fatalf("the state is %s", got)
}
applied(t, l, "decision/1", "withdraw", `{}`, owner)
raw, _, _ = f.state.Get("decision.1")
if got := record(t, raw); got.Sequence != 4 || string(got.State) != "null" {
t.Fatalf("the removed instance is %d %s", got.Sequence, got.State)
}
}},
{"an operation outside its type, a body off its schema and a digest off its text are refused", func(t *testing.T, l *Ledger, f fakes) {
for _, c := range []struct{ instance, operation, body, refusal string }{
{"listing/a17", "title", `{}`, `no type "listing"`},
{"decision/0290", "accept", `{}`, `no operation "accept"`},
{"decision/0290", "title", `{"title":7}`, "field title is a string"},
{"decision/0290", "section", `{"id":"a","text":"x","digest":"sha256:00"}`, "is not the digest of its text"},
{"decision/0290", "section", `{"text":"x"}`, "is not the digest"},
{"decision/0290", "drop-section", `{}`, "names its entry by its field id"},
{"decision", "title", `{}`, "is not an instance"},
{"decision/a.b", "title", `{}`, "is not an instance"},
} {
_, err := l.Apply(buscalls.ApplyRequest{Instance: c.instance, Operation: c.operation, Body: json.RawMessage(c.body)}, owner)
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s %s %s: %v, want %q", c.instance, c.operation, c.body, err, c.refusal)
}
}
if last, _ := f.history.Last("decision.0290"); last != 0 {
t.Fatalf("a refused operation was appended: %d", last)
}
}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
l, f := newLedger()
c.run(t, l, f)
})
}
// design 47 G §587: a ledger's fill is its history's against the ledger's cap, a store that cannot say it is an error naming
// the ledger; a removal takes an instance's operations, snapshot, state and index entries and answers what that was, an
// unknown instance answers none, and nothing of another instance goes.
t.Run("TheLedgerSaysItsFillAndRemovesAnInstanceWhole", func(t *testing.T) {
l, f := newLedger()
f.history.MaxBytes = 1 << 20
for _, n := range []string{"a", "b"} {
applied(t, l, "decision/0290", "note", `{"says":"`+n+`"}`, owner)
}
applied(t, l, "decision/0291", "note", `{"says":"other"}`, owner)
entry, _ := json.Marshal(buscalls.IndexEntry{Instance: "decision/0290", Sequence: 2})
other, _ := json.Marshal(buscalls.IndexEntry{Instance: "decision/0291", Sequence: 1})
_ = f.index.Put("t.decision.0290", entry)
_ = f.index.Put("t.decision.0291", other)
fill, err := l.Fill(buscalls.LedgerRequest{Ledger: "records"})
if err != nil || fill.Count != 3 || fill.Bytes == 0 || fill.MaxBytes != 1<<20 {
t.Fatalf("the fill is %+v (%v), want 3 messages against 1 MiB", fill, err)
}
got, err := l.Remove(buscalls.LedgerRemoveRequest{Ledger: "records", Instance: "decision/0290"})
if err != nil || got.Instance != "decision/0290" || got.Operations != 2 || got.Bytes == 0 {
t.Fatalf("the removal answered %+v (%v), want 2 operations and their bytes", got, err)
}
if kept, _ := f.history.Read("decision.0290", 1, 0); len(kept) != 0 {
t.Errorf("operations are left: %d", len(kept))
}
if raw, _, _ := f.state.Get("decision.0290"); raw != nil {
t.Errorf("the state is left: %s", raw)
}
if raw, _, _ := f.snapshots.Get("decision.0290"); raw != nil {
t.Errorf("the snapshot is left: %s", raw)
}
if held, _ := f.index.Entries("", 0); len(held) != 1 || held[0].Key != "t.decision.0291" {
t.Errorf("the index holds %v, want the other instance's entry alone", held)
}
if kept, _ := f.history.Read("decision.0291", 1, 0); len(kept) != 1 {
t.Errorf("another instance lost operations: %d left", len(kept))
}
if again, err := l.Remove(buscalls.LedgerRemoveRequest{Ledger: "records", Instance: "decision/0290"}); err != nil || again.Operations != 0 || again.Bytes != 0 {
t.Errorf("a removal said twice answered %+v (%v), want none", again, err)
}
f.history.FillFails = true
if _, err := l.Fill(buscalls.LedgerRequest{}); err == nil || !strings.Contains(err.Error(), "ledger records cannot say the fill") {
t.Errorf("a history that cannot say its fill answered %v", err)
}
})
}
+251
View File
@@ -0,0 +1,251 @@
package ledger
import (
"errors"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// The memory stores keep what the bus keeps, as the bus answers it, for a test of a ledger without a bus: a stream whose
// sequence counts across every instance, a bucket with revisions that may be full, and snapshots written with an expected
// sequence.
// MemoryHistory is a history in memory.
type MemoryHistory struct {
// MaxBytes is the cap its fill is said against.
MaxBytes uint64
// FillFails makes Fill an error, as a store that cannot say its fill.
FillFails bool
mu sync.Mutex
seq uint64
kept []struct {
key string
Kept
}
}
// Matches is whether a key pattern (`*` one name, `>` the rest) names a key.
func Matches(pattern, key string) bool {
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
for i, part := range p {
if part == ">" {
return len(k) > i
}
if i >= len(k) || (part != "*" && part != k[i]) {
return false
}
}
return len(p) == len(k)
}
func (h *MemoryHistory) Last(key string) (uint64, error) {
h.mu.Lock()
defer h.mu.Unlock()
last := uint64(0)
for _, k := range h.kept {
if k.key == key {
last = k.Sequence
}
}
return last, nil
}
func (h *MemoryHistory) Append(key string, op []byte, expect uint64) (uint64, error) {
last, _ := h.Last(key)
h.mu.Lock()
defer h.mu.Unlock()
if last != expect {
return 0, ErrRaced
}
h.seq++
h.kept = append(h.kept, struct {
key string
Kept
}{key, Kept{Sequence: h.seq, Data: op}})
return h.seq, nil
}
func (h *MemoryHistory) Read(keys string, from uint64, limit int) ([]Kept, error) {
h.mu.Lock()
defer h.mu.Unlock()
var out []Kept
for _, k := range h.kept {
if k.Sequence >= from && Matches(keys, k.key) && (limit == 0 || len(out) < limit) {
out = append(out, k.Kept)
}
}
return out, nil
}
func (h *MemoryHistory) Purge(key string) error {
h.mu.Lock()
defer h.mu.Unlock()
kept := h.kept[:0]
for _, k := range h.kept {
if k.key != key {
kept = append(kept, k)
}
}
h.kept = kept
return nil
}
func (h *MemoryHistory) Fill() (buscalls.Fill, error) {
h.mu.Lock()
defer h.mu.Unlock()
if h.FillFails {
return buscalls.Fill{}, errors.New("the history does not say its fill")
}
f := buscalls.Fill{Count: uint64(len(h.kept)), MaxBytes: h.MaxBytes}
for _, k := range h.kept {
f.Bytes += uint64(len(k.Data))
}
return f, nil
}
// Expire drops every operation of the history, as a ledger's keeping days end them.
func (h *MemoryHistory) Expire() {
h.mu.Lock()
defer h.mu.Unlock()
h.kept = nil
}
type memoryValue struct {
value []byte
rev uint64
}
// MemoryState is a state bucket in memory.
type MemoryState struct {
mu sync.Mutex
rev uint64
// Full makes every write refused as a full bucket's.
Full bool
kv map[string]memoryValue
}
func (s *MemoryState) Get(key string) ([]byte, uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
v := s.kv[key]
return v.value, v.rev, nil
}
func (s *MemoryState) Write(key string, value []byte, revision uint64) (uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.Full {
return 0, ErrFull
}
if s.kv[key].rev != revision {
return 0, ErrChanged
}
if s.kv == nil {
s.kv = map[string]memoryValue{}
}
s.rev++
s.kv[key] = memoryValue{value, s.rev}
return s.rev, nil
}
func (s *MemoryState) Keys() ([]string, error) {
s.mu.Lock()
defer s.mu.Unlock()
var out []string
for k := range s.kv {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (s *MemoryState) Delete(key string) error {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.kv, key)
return nil
}
// MemorySnapshots are snapshots in memory.
type MemorySnapshots struct {
mu sync.Mutex
at uint64
kv map[string]memoryValue
}
func (s *MemorySnapshots) Get(key string) ([]byte, uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
v := s.kv[key]
return v.value, v.rev, nil
}
func (s *MemorySnapshots) Put(key string, value []byte, expect uint64) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.kv[key].rev != expect {
return ErrRaced
}
if s.kv == nil {
s.kv = map[string]memoryValue{}
}
s.at++
s.kv[key] = memoryValue{value, s.at}
return nil
}
func (s *MemorySnapshots) Purge(key string) error {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.kv, key)
return nil
}
// MemoryIndex is an index's bucket in memory.
type MemoryIndex struct {
mu sync.Mutex
kv map[string][]byte
}
func (x *MemoryIndex) Entries(pattern string, limit int) ([]Indexed, error) {
x.mu.Lock()
defer x.mu.Unlock()
if pattern == "" {
pattern = ">"
}
var keys []string
for k := range x.kv {
if Matches(pattern, k) {
keys = append(keys, k)
}
}
sort.Strings(keys)
var out []Indexed
for _, k := range keys {
if limit > 0 && len(out) >= limit {
break
}
out = append(out, Indexed{Key: k, Value: x.kv[k]})
}
return out, nil
}
func (x *MemoryIndex) Put(key string, value []byte) error {
x.mu.Lock()
defer x.mu.Unlock()
if x.kv == nil {
x.kv = map[string][]byte{}
}
x.kv[key] = value
return nil
}
func (x *MemoryIndex) Delete(key string) error {
x.mu.Lock()
defer x.mu.Unlock()
delete(x.kv, key)
return nil
}
+166
View File
@@ -0,0 +1,166 @@
package ledger
import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// Reading a ledger (ADR 0597 §10, design 25 § 12.5; issue 1966 T3): an instance's state, its history, a watch of its
// states, and a declared index. Only the owning module reads its ledger: the runtime opens a ledger for the module whose
// bundle asks, never one the request names.
// Index is one declared index's bucket, kept by a watcher in the owning module's process and rebuilt by reindex (ADR
// 0597 §4).
type Index interface {
// Entries is the entries whose key a pattern names (`*` one name, `>` the rest; empty is every key), at most limit
// (every one when 0), each as the bucket holds it.
Entries(pattern string, limit int) ([]Indexed, error)
// Put writes one entry; Delete removes one.
Put(key string, value []byte) error
Delete(key string) error
}
// Indexed is one key of an index's bucket and its value, a buscalls.IndexEntry as JSON.
type Indexed struct {
Key string
Value []byte
}
// Watcher is a state bucket that hands each instance's record as it is and as it changes, under a key pattern.
type Watcher interface {
Watch(pattern string, deliver func(key string, value []byte, current bool) error) (func(), error)
}
// HistoryMost is the most operations one history read answers.
const HistoryMost = 1000
// Get is an instance's state as the state holds it, and the sequence of its last operation in it; null and 0 for an
// instance with none.
func (l *Ledger) Get(r buscalls.InstanceRequest) (buscalls.InstanceState, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return buscalls.InstanceState{}, err
}
rec, _, err := l.readState(key)
if err != nil {
return buscalls.InstanceState{}, err
}
state := rec.State
if len(state) == 0 {
state = json.RawMessage("null")
}
return buscalls.InstanceState{Instance: r.Instance, State: state, Sequence: rec.Sequence}, nil
}
// operationsOf reads kept operations as the history answers them, each with its sequence.
func operationsOf(kept []Kept) ([]buscalls.Operation, error) {
out := make([]buscalls.Operation, 0, len(kept))
for _, k := range kept {
var o buscalls.Operation
if err := json.Unmarshal(k.Data, &o); err != nil {
return nil, fmt.Errorf("the history's operation %d is not one: %w", k.Sequence, err)
}
o.Sequence = k.Sequence
out = append(out, o)
}
return out, nil
}
// ReadHistory is an instance's operations from a sequence on, oldest first, each with its sequence and its proven author.
func (l *Ledger) ReadHistory(r buscalls.HistoryRequest) (buscalls.HistoryPage, error) {
key, err := KeyOf(r.Instance)
if err != nil {
return buscalls.HistoryPage{}, err
}
limit := r.Limit
if limit <= 0 || limit > HistoryMost {
limit = HistoryMost
}
last, err := l.History.Last(key)
if err != nil {
return buscalls.HistoryPage{}, err
}
kept, err := l.History.Read(key, r.From, limit+1)
if err != nil {
return buscalls.HistoryPage{}, err
}
page := buscalls.HistoryPage{Last: last}
if len(kept) > limit {
page.Next = kept[limit].Sequence
kept = kept[:limit]
}
if page.Operations, err = operationsOf(kept); err != nil {
return buscalls.HistoryPage{}, err
}
return page, nil
}
// Watch hands deliver the state of every instance Instances names (`*` one name, `>` the rest, in an instance's path),
// the current ones first, then each change.
func (l *Ledger) Watch(r buscalls.LedgerWatchRequest, deliver func(buscalls.InstanceChange) error) (func(), error) {
w, ok := l.State.(Watcher)
if !ok {
return nil, errors.New("this ledger's state cannot be watched")
}
pattern := ">"
if r.Instances != "" {
pattern = keyPattern(r.Instances)
}
return w.Watch(pattern, func(key string, value []byte, current bool) error {
var rec Record
if value != nil {
if err := json.Unmarshal(value, &rec); err != nil {
return fmt.Errorf("the state of %s is not a record: %w", key, err)
}
}
state := rec.State
if len(state) == 0 {
state = json.RawMessage("null")
}
return deliver(buscalls.InstanceChange{Watch: r.Watch, Instance: InstanceOf(key), State: state, Sequence: rec.Sequence,
Current: current})
})
}
// keyPattern is an instance pattern as a key pattern: its path's parts joined by dots.
func keyPattern(instances string) string {
out := []byte(instances)
for i, c := range out {
if c == '/' {
out[i] = '.'
}
}
return string(out)
}
// index is a declared index by its name, or the reason there is none.
func (l *Ledger) index(name string) (Index, error) {
if ix, ok := l.Indexes[name]; ok {
return ix, nil
}
return nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, name)
}
// Query is a declared index's entries whose key Key names.
func (l *Ledger) Query(r buscalls.QueryRequest) (buscalls.QueryAnswer, error) {
ix, err := l.index(r.Index)
if err != nil {
return buscalls.QueryAnswer{}, err
}
held, err := ix.Entries(r.Key, r.Limit)
if err != nil {
return buscalls.QueryAnswer{}, err
}
entries := make([]buscalls.IndexEntry, 0, len(held))
for _, h := range held {
var e buscalls.IndexEntry
if err := json.Unmarshal(h.Value, &e); err != nil {
return buscalls.QueryAnswer{}, fmt.Errorf("the index %s's entry %s is not one: %w", r.Index, h.Key, err)
}
entries = append(entries, e)
}
return buscalls.QueryAnswer{Entries: entries}, nil
}
+120
View File
@@ -0,0 +1,120 @@
package ledger
import (
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// 1966/AC15, the other reads of a ledger (get, query) and its repairs (ADR 0597 §8, §10; design 25 § 12.5).
func TestReadsAndRepairsOfALedger(t *testing.T) {
cases := []struct {
name string
run func(t *testing.T, l *Ledger, f fakes)
}{
{"1966/AC15 history answers an instance's operations oldest first with sequence and author", func(t *testing.T, l *Ledger, f fakes) {
op := buscalls.Author{Account: "ledger.anchor.m", Caller: "op@shanks"}
applied(t, l, "decision/1", "note", `{"says":"one"}`, owner)
applied(t, l, "decision/2", "note", `{"says":"elsewhere"}`, owner)
applied(t, l, "decision/1", "title", `{"title":"Two"}`, op)
applied(t, l, "decision/1", "note", `{"says":"three"}`, owner)
page, err := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", Limit: 2})
if err != nil {
t.Fatal(err)
}
var got []string
for _, o := range page.Operations {
got = append(got, fmt.Sprintf("%s@%d %s %s", o.Operation, o.Sequence, o.Author.Account, o.Author.Caller))
}
if strings.Join(got, "; ") != "note@1 ledger.anchor.m ; title@3 ledger.anchor.m op@shanks" || page.Next != 4 || page.Last != 4 {
t.Fatalf("the history is %q, next %d, last %d", got, page.Next, page.Last)
}
rest, _ := l.ReadHistory(buscalls.HistoryRequest{Instance: "decision/1", From: page.Next})
if len(rest.Operations) != 1 || rest.Operations[0].Sequence != 4 || rest.Next != 0 {
t.Fatalf("read on: %+v", rest)
}
}},
{"get answers the state and its sequence, null for an instance with none", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
got, err := l.Get(buscalls.InstanceRequest{Instance: "decision/1"})
if err != nil || got.Sequence != 1 || !strings.Contains(string(got.State), `"title":"One"`) {
t.Fatalf("got %+v, %v", got, err)
}
if none, err := l.Get(buscalls.InstanceRequest{Instance: "decision/9"}); err != nil || string(none.State) != "null" {
t.Fatalf("none is %+v, %v", none, err)
}
}},
{"query reads a declared index by key, and an undeclared one is refused", func(t *testing.T, l *Ledger, f fakes) {
_ = f.index.Put("One.decision.1", []byte(`{"instance":"decision/1","sequence":1}`))
_ = f.index.Put("Two.decision.2", []byte(`{"instance":"decision/2","sequence":2}`))
got, err := l.Query(buscalls.QueryRequest{Index: "by-title", Key: "One.>"})
if err != nil || len(got.Entries) != 1 || got.Entries[0] != (buscalls.IndexEntry{Instance: "decision/1", Sequence: 1}) {
t.Fatalf("queried %+v, %v", got, err)
}
if _, err := l.Query(buscalls.QueryRequest{Index: "by-date"}); err == nil || !strings.Contains(err.Error(), `no index "by-date"`) {
t.Fatalf("an undeclared index: %v", err)
}
}},
{"a lost state rebuilds from the history, an expired one from its snapshot, and lag says what is behind", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
applied(t, l, "decision/1", "note", `{"says":"kept"}`, owner)
want, _, _ := f.state.Get("decision.1")
f.state.kv = nil
lag, _ := l.Lag(buscalls.LedgerRequest{})
if len(lag.Instances) != 1 || lag.Instances[0].History != 2 || lag.Instances[0].State != 0 || lag.Instances[0].Snapshot != 2 {
t.Fatalf("lag says %+v", lag)
}
if got, err := l.Rebuild(buscalls.RebuildRequest{}); err != nil || got.Written != 1 {
t.Fatalf("rebuilt %+v, %v", got, err)
}
if got, _, _ := f.state.Get("decision.1"); string(record(t, got).State) != string(record(t, want).State) {
t.Fatalf("rebuilt from the history %s, want %s", got, want)
}
f.state.kv = nil
f.history.Expire()
if got, err := l.Rebuild(buscalls.RebuildRequest{Instance: "decision/1", From: buscalls.RebuildFromSnapshot}); err != nil || got.Written != 1 {
t.Fatalf("rebuilt %+v, %v", got, err)
}
if got, _, _ := f.state.Get("decision.1"); string(got) != string(want) {
t.Fatalf("rebuilt from the snapshot %s, want %s", got, want)
}
}},
{"reindex builds a declared index from the states, and lag says an index behind until it does", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
applied(t, l, "decision/2", "note", `{"says":"untitled"}`, owner)
_ = f.index.Put("stale.decision.9", []byte(`{}`))
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 1 || lag.Indexes[0].Behind != 1 {
t.Fatalf("lag says %+v", lag)
}
if got, err := l.Reindex(buscalls.ReindexRequest{Index: "by-title"}); err != nil || got.Written != 1 {
t.Fatalf("reindexed %+v, %v", got, err)
}
entries, _ := f.index.Entries("", 0)
if len(entries) != 1 || entries[0].Key != buscalls.IndexKey([]string{"One"}, "decision/1") || string(entries[0].Value) != `{"instance":"decision/1","sequence":1}` {
t.Fatalf("the index holds %+v", entries)
}
if lag, _ := l.Lag(buscalls.LedgerRequest{}); len(lag.Indexes) != 0 || len(lag.Instances) != 0 {
t.Fatalf("lag after reindex says %+v", lag)
}
}},
{"snapshot rewrites a snapshot behind its state", func(t *testing.T, l *Ledger, f fakes) {
applied(t, l, "decision/1", "title", `{"title":"One"}`, owner)
f.snapshots.kv = nil
if got, err := l.Snapshot(buscalls.InstanceRequest{}); err != nil || got.Written != 1 {
t.Fatalf("snapshot %+v, %v", got, err)
}
state, _, _ := f.state.Get("decision.1")
if snap, _, _ := f.snapshots.Get("decision.1"); string(snap) != string(state) {
t.Fatalf("the snapshot %s, the state %s", snap, state)
}
}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
l, f := newLedger()
c.run(t, l, f)
})
}
}
+304
View File
@@ -0,0 +1,304 @@
package ledger
import (
"encoding/json"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// The ledger's repairs (ADR 0597 §8, §10; design 25 § 12.5; issue 1966 T4): rebuild an instance's state from its history
// or its snapshot, rewrite a snapshot behind its state, build a declared index again, and say what is behind its history.
// The tool runner serves them as tools of every module owning a ledger, its own ledgers only (the runtime's repairs.go).
// instances is every instance the ledger knows, by key: those its state holds and those its history names.
func (l *Ledger) instances() ([]string, error) {
seen := map[string]bool{}
keys, err := l.State.Keys()
if err != nil {
return nil, err
}
for _, k := range keys {
seen[k] = true
}
kept, err := l.History.Read(">", 1, 0)
if err != nil {
return nil, err
}
ops, err := operationsOf(kept)
if err != nil {
return nil, err
}
for _, o := range ops {
if k, err := KeyOf(o.Instance); err == nil {
seen[k] = true
}
}
out := make([]string, 0, len(seen))
for k := range seen {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
// keysOf is the one instance a request names, or every instance when it names none.
func (l *Ledger) keysOf(instance string) ([]string, error) {
if instance == "" {
return l.instances()
}
k, err := KeyOf(instance)
if err != nil {
return nil, err
}
return []string{k}, nil
}
// Rebuild derives the state of an instance (every instance when it names none) again, from its whole history or from its
// snapshot and the operations after it, and writes it, and its snapshot when that is behind. A derivation further back
// than the state holds is not written: the state never goes back (ADR 0597 §5).
func (l *Ledger) Rebuild(r buscalls.RebuildRequest) (buscalls.Repaired, error) {
from := r.From
if from == "" {
from = buscalls.RebuildFromHistory
}
if from != buscalls.RebuildFromHistory && from != buscalls.RebuildFromSnapshot {
return buscalls.Repaired{}, fmt.Errorf("a rebuild is from %q or %q, not %q", buscalls.RebuildFromHistory, buscalls.RebuildFromSnapshot, from)
}
keys, err := l.keysOf(r.Instance)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, key := range keys {
base := Record{}
if from == buscalls.RebuildFromSnapshot {
if base, _, err = l.readSnapshot(key); err != nil {
return out, err
}
}
derived, err := l.replay(key, base, 0)
if err != nil {
return out, err
}
held, rev, err := l.readState(key)
if err != nil {
return out, err
}
if derived.Sequence < held.Sequence {
continue
}
now, _, err := l.settle(key, derived, rev)
if err != nil {
return out, err
}
if err := l.writeSnapshot(key, now); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// Snapshot rewrites the snapshot of an instance (every instance when it names none) that is behind its state.
func (l *Ledger) Snapshot(r buscalls.InstanceRequest) (buscalls.Repaired, error) {
keys, err := l.keysOf(r.Instance)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, key := range keys {
held, _, err := l.readState(key)
if err != nil {
return out, err
}
snap, at, err := l.readSnapshot(key)
if err != nil {
return out, err
}
if held.Sequence == 0 || (at != 0 && snap.Sequence >= held.Sequence) {
continue
}
if err := l.writeSnapshot(key, held); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// indexEntryOf is an instance's entry in a declared index, as reindex writes it and the owner's watcher keeps it
// (mesh-sdk buscalls.IndexKey, buscalls.IndexEntry): its key the values of the fields the index is on, in their order,
// then the instance; its value the instance and the sequence of the state it was taken from. False when the state holds
// none of those fields. A value that is not text is written as its JSON.
func indexEntryOf(on []string, key string, rec Record) (string, []byte, bool) {
var state map[string]any
if json.Unmarshal(rec.State, &state) != nil || state == nil {
return "", nil, false
}
values := make([]string, 0, len(on))
found := false
for _, f := range on {
v, has := state[f]
found = found || has
s, ok := v.(string)
if !ok {
raw, _ := json.Marshal(v)
s = string(raw)
}
values = append(values, s)
}
if !found {
return "", nil, false
}
instance := InstanceOf(key)
value, _ := json.Marshal(buscalls.IndexEntry{Instance: instance, Sequence: rec.Sequence})
return buscalls.IndexKey(values, instance), value, true
}
// declared is the fields an index is on and the types declaring it.
func (l *Ledger) declared(index string) ([]string, map[string]bool, error) {
types := map[string]bool{}
var on []string
for _, t := range l.Types {
for _, ix := range t.Indexes {
if ix.Name == index {
types[t.Name] = true
on = ix.On
}
}
}
if len(types) == 0 {
return nil, nil, fmt.Errorf("%s's ledger %s declares no index %q", l.Module, l.Name, index)
}
return on, types, nil
}
// wanted is every entry an index should hold, from the states of the types declaring it.
func (l *Ledger) wanted(index string) (map[string][]byte, map[string]uint64, error) {
on, types, err := l.declared(index)
if err != nil {
return nil, nil, err
}
keys, err := l.instances()
if err != nil {
return nil, nil, err
}
want, seqs := map[string][]byte{}, map[string]uint64{}
for _, key := range keys {
name, _, _ := strings.Cut(key, ".")
if !types[name] {
continue
}
held, _, err := l.readState(key)
if err != nil {
return nil, nil, err
}
if k, v, ok := indexEntryOf(on, key, held); ok {
want[k], seqs[k] = v, held.Sequence
}
}
return want, seqs, nil
}
// Reindex builds a declared index again from the ledger's states, derived from its history: every entry it should hold
// written, every other removed.
func (l *Ledger) Reindex(r buscalls.ReindexRequest) (buscalls.Repaired, error) {
ix, err := l.index(r.Index)
if err != nil {
return buscalls.Repaired{}, err
}
want, _, err := l.wanted(r.Index)
if err != nil {
return buscalls.Repaired{}, err
}
held, err := ix.Entries("", 0)
if err != nil {
return buscalls.Repaired{}, err
}
var out buscalls.Repaired
for _, e := range held {
if _, keep := want[e.Key]; !keep {
if err := ix.Delete(e.Key); err != nil {
return out, err
}
}
}
keys := make([]string, 0, len(want))
for k := range want {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if err := ix.Put(k, want[k]); err != nil {
return out, err
}
out.Written++
}
return out, nil
}
// Lag is every instance whose state or snapshot is behind its history, and how many entries each declared index lacks
// or holds from an older state; nothing when nothing is behind.
func (l *Ledger) Lag(buscalls.LedgerRequest) (buscalls.Lagged, error) {
keys, err := l.instances()
if err != nil {
return buscalls.Lagged{}, err
}
var out buscalls.Lagged
for _, key := range keys {
last, err := l.History.Last(key)
if err != nil {
return out, err
}
held, _, err := l.readState(key)
if err != nil {
return out, err
}
snap, _, err := l.readSnapshot(key)
if err != nil {
return out, err
}
if held.Sequence < last || snap.Sequence < last {
out.Instances = append(out.Instances, buscalls.InstanceLag{Instance: InstanceOf(key), History: last,
State: held.Sequence, Snapshot: snap.Sequence})
}
}
names := make([]string, 0, len(l.Indexes))
for name := range l.Indexes {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if _, _, err := l.declared(name); err != nil {
// A bucket bound for an index no type declares has nothing it should hold.
continue
}
want, seqs, err := l.wanted(name)
if err != nil {
return out, err
}
held, err := l.Indexes[name].Entries("", 0)
if err != nil {
return out, err
}
at := map[string]uint64{}
for _, e := range held {
var v buscalls.IndexEntry
_ = json.Unmarshal(e.Value, &v)
at[e.Key] = v.Sequence
}
behind := uint64(0)
for k := range want {
if s, has := at[k]; !has || s < seqs[k] {
behind++
}
}
if behind > 0 {
out.Indexes = append(out.Indexes, buscalls.IndexLag{Index: name, Behind: behind})
}
}
return out, nil
}
+61
View File
@@ -0,0 +1,61 @@
package ledger
import (
"encoding/json"
"testing"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// records is a ledger type of each effect: a title set, a section set as an entry, a note appended, a section dropped,
// and the whole instance removed.
var records = catalogue.LedgerType{
Name: "decision",
Schema: map[string]catalogue.LedgerField{
"title": {Type: "string"}, "text": {Type: "string"}, "digest": {Type: "string", DigestOf: "text"},
},
Operations: []catalogue.LedgerOperation{
{Name: "title", Effect: catalogue.EffectSetFields, Fields: []string{"title"}, Sequence: true, Author: true},
{Name: "section", Effect: catalogue.EffectSetEntry, Into: "sections", Key: "id", Fields: []string{"digest"}},
{Name: "note", Effect: catalogue.EffectAppend, Into: "notes", Fields: []string{"says"}},
{Name: "drop-section", Effect: catalogue.EffectRemove, Into: "sections", Key: "id"},
{Name: "withdraw", Effect: catalogue.EffectRemove},
},
Indexes: []catalogue.LedgerIndex{{Name: "by-title", On: []string{"title"}}},
}
type fakes struct {
history *MemoryHistory
state *MemoryState
snapshots *MemorySnapshots
index *MemoryIndex
}
func newLedger() (*Ledger, fakes) {
f := fakes{&MemoryHistory{}, &MemoryState{}, &MemorySnapshots{}, &MemoryIndex{}}
return &Ledger{Module: "m", Name: "records", Types: []catalogue.LedgerType{records}, History: f.history, State: f.state,
Snapshots: f.snapshots, Indexes: map[string]Index{"by-title": f.index}}, f
}
// applied applies an operation and fails the test on a refusal.
func applied(t *testing.T, l *Ledger, instance, operation, body string, author buscalls.Author) buscalls.Applied {
t.Helper()
got, err := l.Apply(buscalls.ApplyRequest{Ledger: l.Name, Instance: instance, Operation: operation, Body: json.RawMessage(body)}, author)
if err != nil {
t.Fatalf("%s %s: %v", operation, body, err)
}
return got
}
// record is the instance's record a store holds, read as a Record.
func record(t *testing.T, raw []byte) Record {
t.Helper()
var r Record
if raw != nil {
if err := json.Unmarshal(raw, &r); err != nil {
t.Fatal(err)
}
}
return r
}
+38
View File
@@ -0,0 +1,38 @@
package meshtest
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"testing"
)
// bundlesWhy returns the reason bundles cannot be launched from a checkout rooted at root, or "" when they can: the
// TypeScript and Python bundles the runtime and the console launch need node and the package
// @novox/mesh-sdk, which comes from the package registry and is installed beside node-tools/package.json
// (novox/hq ADR 0529 clause 1: what merge-check.sh said "NOT TESTED HERE" about is a skip with its reason).
func bundlesWhy(root string, haveNode bool) string {
if !haveNode {
return "node is not installed"
}
if _, err := os.Stat(filepath.Join(root, "node_modules", "@novox", "mesh-sdk")); err != nil {
return "@novox/mesh-sdk is not installed in " + filepath.Join(root, "node_modules")
}
return ""
}
// NeedBundles skips the test, saying why, where the bundles it launches cannot run here.
func NeedBundles(t *testing.T) {
t.Helper()
_, err := exec.LookPath("node")
if why := bundlesWhy(moduleRoot(), err == nil); why != "" {
t.Skip("NOT TESTED HERE: this test launches bundles, and " + why)
}
}
// moduleRoot is node-tools/, where package.json and node_modules are.
func moduleRoot() string {
_, here, _, _ := runtime.Caller(0)
return filepath.Join(filepath.Dir(here), "..", "..")
}
+255 -16
View File
@@ -1,7 +1,7 @@
// Package meshtest raises what the controller would, for tests against a real bus: the ASSIGNMENTS
// Package meshtest raises what the mesh would, for tests against a real bus: the ASSIGNMENTS
// and EVENTS streams, memberships issued by hand, and a fixture's path.
//
// **Every test has a bus of its own** (novox/hq ADR 0237, the controller's internal/testbus): a server
// **Every test has a bus of its own** (novox/hq ADR 0237, the old testbus): a server
// linked in at the release go.mod pins — the release the mesh runs, held so by a test beside this one —
// started for the one test and gone after it. The packages shared one bus and had to run one at a time:
// each test raised the streams afresh, and the console discovers every runtime that announces itself on
@@ -11,6 +11,7 @@
package meshtest
import (
"context"
"encoding/json"
"os"
"path/filepath"
@@ -20,8 +21,14 @@ import (
"testing"
"time"
assigner "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
seatassigner "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/membership"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-tools/node-tools/internal/bus"
)
@@ -67,7 +74,7 @@ func URL(t *testing.T) string {
// buses are the running tests' buses, by test.
var buses sync.Map
// Mesh is the controller's job, done by hand.
// Mesh is the mesh's job, done by hand.
type Mesh struct {
nc *nats.Conn
js nats.JetStreamContext
@@ -81,11 +88,10 @@ func New(t *testing.T) *Mesh {
t.Fatal(err)
}
js, _ := nc.JetStream()
for _, s := range []string{"ASSIGNMENTS", "EVENTS"} {
_ = js.DeleteStream(s)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "ASSIGNMENTS", Subjects: []string{"mesh.assignment.>"},
MaxMsgsPerSubject: 1, AllowDirect: true}); err != nil {
_ = js.DeleteStream("EVENTS")
// The grants part's state memberships, where a runtime reads what is issued it (ADR 0571 clause 10).
_ = js.DeleteKeyValue(membership.GrantsBucket)
if _, err := js.CreateKeyValue(&nats.KeyValueConfig{Bucket: membership.GrantsBucket, History: 1}); err != nil {
t.Fatal(err)
}
if _, err := js.AddStream(&nats.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
@@ -95,15 +101,124 @@ func New(t *testing.T) *Mesh {
return &Mesh{nc: nc, js: js}
}
// Issue publishes a membership.
// Issue writes a membership where the grants part keeps it, its state memberships, key `<node>.<module>` (novox/hq ADR
// 0571 clause 7 and 10; mesh-sdk membership.GrantsBucket), the one place a runtime reads it. It writes with
// mesh-sdk's encoder, the grants part's own, never json of this package's type: a value built here by hand agreed with
// the reader's mistake and hid that the part wrote another shape (issue 774).
func (m *Mesh) Issue(t *testing.T, mem bus.Membership) {
t.Helper()
body, _ := json.Marshal(mem)
if _, err := m.js.Publish(bus.MembershipSubject(mem.Node, mem.Module), body); err != nil {
m.putGrants(t, mem, membership.EncodeGrantsMembership(sdkMembership(t, mem)))
}
// IssueInGrantsWrapped writes a membership as the grants part wrote it before mesh-sdk's codec (until 2026-10-30, what
// the live mesh holds until the part writes again): its pass record, the membership inside, published saying what
// became of it ("published", "unchanged", or why it was held back).
func (m *Mesh) IssueInGrantsWrapped(t *testing.T, mem bus.Membership, published string) {
t.Helper()
body, _ := json.Marshal(map[string]any{"node": mem.Node, "module": mem.Module, "digest": "sha256:0",
"membership": sdkMembership(t, mem), "published": published})
m.putGrants(t, mem, body)
}
func (m *Mesh) putGrants(t *testing.T, mem bus.Membership, body []byte) {
t.Helper()
kv, err := m.js.KeyValue(membership.GrantsBucket)
if err != nil {
if kv, err = m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: membership.GrantsBucket, History: 1}); err != nil {
t.Fatal(err)
}
}
if _, err := kv.Put(membership.GrantsKey(mem.Node, mem.Module), body); err != nil {
t.Fatal(err)
}
}
// DropGrants deletes the grants part's state memberships, as a bus that has not made it yet holds none; Issue makes it again.
func (m *Mesh) DropGrants(t *testing.T) {
t.Helper()
if err := m.js.DeleteKeyValue(membership.GrantsBucket); err != nil {
t.Fatal(err)
}
}
// sdkMembership is a membership of this package's type as mesh-sdk's, which the grants part composes and writes.
func sdkMembership(t *testing.T, mem bus.Membership) membership.Membership {
t.Helper()
raw, _ := json.Marshal(mem)
var out membership.Membership
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
// The states the console reads, and their buckets (the test bed's own names: the indexer's client names
// no holder).
const (
catalogHolder = "mesh-catalog"
assignerHolder = "mesh-module-assigner" // the seat's name stands for its holder in the console's state names (firstview.AssignmentsRead)
// AssignmentsBucket and SeatHoldersBucket are the assigner's states assignments and seat-holders.
AssignmentsBucket = assignerHolder + "_assignments"
SeatHoldersBucket = assignerHolder + "_seat-holders"
// ModulesBucket holds the catalog half's entries by module, the test bed's own stand-in for the runner's first views of it.
ModulesBucket = catalogHolder + "_modules"
)
// ModulesRead names the test bed's state of the catalog half's entries.
const ModulesRead = catalogHolder + ".modules"
// ConsoleReads are the states a console's runtime reads to say what the mesh holds (novox/hq ADR 0571 clauses 4, 5):
// assignments, the catalog half's entries, and the seat holders.
func ConsoleReads() []bus.StateIssued {
return []bus.StateIssued{
{Name: assignerHolder + ".assignments", Bucket: AssignmentsBucket},
{Name: ModulesRead, Bucket: ModulesBucket},
{Name: assignerHolder + ".seat-holders", Bucket: SeatHoldersBucket},
}
}
// Placed writes what the mesh holds as its parts write it, with mesh-sdk's encoders: each module the indexer holds with
// its manifest (tools, the tools it declares; none for a module without), and each node's assignment of modules.
func (m *Mesh) Placed(t *testing.T, tools map[string][]string, on map[string][]string) {
t.Helper()
kv := func(bucket string) nats.KeyValue {
_ = m.js.DeleteKeyValue(bucket)
b, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: bucket, History: 1, MaxValueSize: 256 * 1024})
if err != nil {
t.Fatal(err)
}
return b
}
modules, assigned := kv(ModulesBucket), kv(AssignmentsBucket)
kv(SeatHoldersBucket)
for module, declared := range tools {
manifest := map[string]any{"module": module, "tools": declared}
raw, err := json.Marshal(meshcatalog.CatalogEntry{Module: module, Manifest: manifest, Commit: strings.Repeat("a", 40)})
if err != nil {
t.Fatal(err)
}
if _, err := modules.Put(module, raw); err != nil {
t.Fatal(err)
}
}
for node, placed := range on {
p := seatassigner.Assignments{V: assigner.Version, Node: node}
for _, module := range placed {
p.Modules = append(p.Modules, seatassigner.PlacedModule{Module: module})
}
// As the seat writes it: its node key and a key per module (research 072 row 37).
values, err := assigner.AssignmentsValues(p)
if err != nil {
t.Fatal(err)
}
for key, raw := range values {
if _, err := assigned.Put(key, raw); err != nil {
t.Fatal(err)
}
}
}
}
// NextEvent is the subject the next event under a pattern lands on.
func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
t.Helper()
@@ -122,7 +237,7 @@ func (m *Mesh) NextEvent(t *testing.T, pattern string) <-chan string {
return ch
}
// MembershipOf is a membership as the controller issues one on a machine.
// MembershipOf is a membership as the mesh issues one on a machine.
func MembershipOf(module, node string, plain bool, seats map[string][]string) bus.Membership {
own := "mesh.mod." + module
serves := []bus.Served{{Subject: own + ".tool.{tool}." + node}}
@@ -183,12 +298,12 @@ func (l *Logs) Has(fragments ...string) bool {
// All is every line.
func (l *Logs) All() string { return strings.Join(l.lines, "\n") }
// Consumer makes a module's durable consumer on a machine as the controller does: pull, on the
// Consumer makes a module's durable consumer on a machine as the mesh does: pull, on the
// EVENTS stream, filtered to what the module consumes, with a short ack wait so a test sees a
// redelivery in seconds rather than the mesh's minutes.
func (m *Mesh) Consumer(t *testing.T, node, module string, filters []string, ackWait time.Duration) {
t.Helper()
cfg := &nats.ConsumerConfig{Durable: node + "_" + module, AckPolicy: nats.AckExplicitPolicy,
cfg := &nats.ConsumerConfig{Durable: node + "_" + module + "_tools", AckPolicy: nats.AckExplicitPolicy,
AckWait: ackWait, MaxDeliver: 10, DeliverPolicy: nats.DeliverNewPolicy}
if len(filters) == 1 {
cfg.FilterSubject = filters[0]
@@ -209,18 +324,41 @@ func (m *Mesh) Emit(t *testing.T, subject string, body any) {
}
}
// Publish says body on a plain subject, as a core publish with no stream behind it.
func (m *Mesh) Publish(t *testing.T, subject string, body any) {
t.Helper()
data, _ := json.Marshal(body)
if err := m.nc.Publish(subject, data); err != nil {
t.Fatal(err)
}
if err := m.nc.Flush(); err != nil {
t.Fatal(err)
}
}
// Pending is what a module's consumer still holds: delivered and not acknowledged, and not yet
// delivered.
func (m *Mesh) Pending(t *testing.T, node, module string) (ackPending, notDelivered uint64) {
t.Helper()
info, err := m.js.ConsumerInfo("EVENTS", node+"_"+module)
info, err := m.js.ConsumerInfo("EVENTS", node+"_"+module+"_tools")
if err != nil {
t.Fatal(err)
}
return uint64(info.NumAckPending), info.NumPending
}
// Bucket makes a module's state afresh as the controller does from the catalogue (novox/hq ADR 0201),
// BucketKeeping makes a module's state afresh as Bucket does, keeping up to history values a key.
func (m *Mesh) BucketKeeping(t *testing.T, name string, history uint8) nats.KeyValue {
t.Helper()
_ = m.js.DeleteKeyValue(name)
kv, err := m.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: name, History: history, MaxValueSize: 256 * 1024})
if err != nil {
t.Fatal(err)
}
return kv
}
// Bucket makes a module's state afresh as the mesh does from the catalogue (novox/hq ADR 0201),
// and answers it for writing what is there before a bundle starts.
func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
t.Helper()
@@ -231,3 +369,104 @@ func (m *Mesh) Bucket(t *testing.T, name string) nats.KeyValue {
}
return kv
}
// BucketKeyTTL makes a module's state afresh as the mesh does for one declared `key-ttl` (mesh-sdk buscompose:
// a time to live per key allowed, its end leaving a marker kept a minute; issue 1952).
func (m *Mesh) BucketKeyTTL(t *testing.T, name string) {
t.Helper()
_ = m.js.DeleteKeyValue(name)
js, err := jetstream.New(m.nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 1, MaxValueSize: 256 * 1024,
LimitMarkerTTL: time.Minute}); err != nil {
t.Fatal(err)
}
}
// Log makes a module's log afresh as the mesh does from the catalogue (novox/hq ADR 0297): a file
// stream named LOG_<module>_<name> on mesh.log.<module>.<name>.>, limits retention, no age, no cap per
// key, the declared cap, 260 KiB a message (a 256 KiB value and its headers), refusing new entries when full, direct get, no delete or
// purge. It answers the membership entry the mesh issues for it.
func (m *Mesh) Log(t *testing.T, module, name string, maxBytes int64) bus.LogIssued {
t.Helper()
stream := "LOG_" + module + "_" + name
subject := "mesh.log." + module + "." + name
_ = m.js.DeleteStream(stream)
if _, err := m.js.AddStream(&nats.StreamConfig{Name: stream, Subjects: []string{subject + ".>"},
Storage: nats.FileStorage, Retention: nats.LimitsPolicy, MaxAge: 0, MaxMsgsPerSubject: -1,
MaxBytes: maxBytes, MaxMsgSize: 260 * 1024, Discard: nats.DiscardNew, AllowDirect: true,
DenyDelete: true, DenyPurge: true, Replicas: 1,
Description: module + "'s log " + name + ", kept as its record (novox/hq ADR 0297)"}); err != nil {
t.Fatal(err)
}
return bus.LogIssued{Name: name, Stream: stream, Subject: subject, Writes: true}
}
// Consumers is how many consumers a stream has.
func (m *Mesh) Consumers(t *testing.T, stream string) int {
t.Helper()
info, err := m.js.StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
return info.State.Consumers
}
// ModuleStream makes a module's own stream afresh as the mesh does from the catalogue (novox/hq ADR
// 0363 §1): named in capitals, taking `mesh.stream.<module>.event.>`, a subject space EVENTS does not take.
func (m *Mesh) ModuleStream(t *testing.T, module, name string) {
t.Helper()
_ = m.js.DeleteStream(name)
if _, err := m.js.AddStream(&nats.StreamConfig{Name: name, Subjects: []string{"mesh.stream." + module + ".event.>"},
Retention: nats.LimitsPolicy, Discard: nats.DiscardOld}); err != nil {
t.Fatal(err)
}
}
// ConsumerOn makes a module's durable consumer on a machine on any stream, as Consumer does on EVENTS.
func (m *Mesh) ConsumerOn(t *testing.T, stream, node, module string, filters []string, ackWait time.Duration) {
t.Helper()
cfg := &nats.ConsumerConfig{Durable: node + "_" + module + "_tools", AckPolicy: nats.AckExplicitPolicy,
AckWait: ackWait, MaxDeliver: 10, DeliverPolicy: nats.DeliverNewPolicy}
if len(filters) == 1 {
cfg.FilterSubject = filters[0]
} else {
cfg.FilterSubjects = filters
}
if _, err := m.js.AddConsumer(stream, cfg); err != nil {
t.Fatal(err)
}
}
// PendingOn is what a module's consumer on a stream still holds, as Pending is on EVENTS.
func (m *Mesh) PendingOn(t *testing.T, stream, node, module string) (ackPending, notDelivered uint64) {
t.Helper()
info, err := m.js.ConsumerInfo(stream, node+"_"+module+"_tools")
if err != nil {
t.Fatal(err)
}
return uint64(info.NumAckPending), info.NumPending
}
// Messages is how many messages a stream holds.
func (m *Mesh) Messages(t *testing.T, stream string) uint64 {
t.Helper()
info, err := m.js.StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
return info.State.Msgs
}
// RemoveConsumer deletes a module's consumer on a machine on a stream, as the mesh does when a grant
// it cannot narrow is withdrawn (novox/hq ADR 0363 §6).
func (m *Mesh) RemoveConsumer(t *testing.T, stream, node, module string) {
t.Helper()
if err := m.js.DeleteConsumer(stream, node+"_"+module+"_tools"); err != nil {
t.Fatal(err)
}
}
@@ -73,3 +73,56 @@ func TestEachTestHasABusOfItsOwn(t *testing.T) {
}
})
}
// What merge-check.sh said was "NOT TESTED HERE" is a skip with its reason: no node, or no package, each says so.
func TestBundlesCannotBeLaunchedWithoutNodeOrTheSDKAndTheSkipSaysWhy(t *testing.T) {
root := t.TempDir()
if why := bundlesWhy(root, false); !strings.Contains(why, "node is not installed") {
t.Errorf("no node said %q", why)
}
if why := bundlesWhy(root, true); !strings.Contains(why, "@novox/mesh-sdk") {
t.Errorf("no package said %q", why)
}
if err := os.MkdirAll(filepath.Join(root, "node_modules", "@novox", "mesh-sdk"), 0o755); err != nil {
t.Fatal(err)
}
if why := bundlesWhy(root, true); why != "" {
t.Errorf("node and the package present said %q", why)
}
}
// The checker runs node-tools' Go level by Go's convention and its Node level by package.json's test script
// (novox/hq ADR 0529): no script of the repository's own runs beside them, and no module but node-tools is built from
// here (ADR 0524 §5, the toolchain image retired).
func TestTheRepositoryHoldsNoCheckScriptAndNodeToolsIsItsOnlyModule(t *testing.T) {
root := filepath.Join("..", "..", "..")
if _, err := os.Stat(filepath.Join(root, "merge-check.sh")); err == nil {
t.Errorf("merge-check.sh is back: the checker's levels run the tests")
}
var found []string
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && (d.Name() == "node_modules" || d.Name() == ".git") {
return filepath.SkipDir
}
if !d.IsDir() && d.Name() == "module.json" {
found = append(found, filepath.ToSlash(p))
}
// ADR 0604 section 4: the TypeScript that called the old catalog module's tools is gone, and nothing
// under node-tools names one (the needle is built so this file does not name it either).
if !d.IsDir() && strings.Contains(filepath.ToSlash(p), "/node-tools/") {
if body, rerr := os.ReadFile(p); rerr == nil && strings.Contains(string(body), "catalog"+"_modules") {
t.Errorf("%s names a catalog tool of the retired module", p)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || !strings.HasSuffix(found[0], "/node-tools/module.json") {
t.Errorf("the module manifests are %v, want node-tools' alone", found)
}
}
+113
View File
@@ -0,0 +1,113 @@
package meshtest
import (
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// State is a module's state as a fixture, the shape a gate's Store takes: Fail makes every read and write fail, as a bus
// refusing them; Vals are its values by key. A key the bus refuses (a name outside letters, digits and -/_=, in
// dot-separated names) is refused here too, so no test passes a key the live bus refuses (lead answer A-209).
type State struct {
mu sync.Mutex
Vals map[string]json.RawMessage
revs map[string]uint64
n uint64
Fail error
}
// checkKey refuses a key the bus would.
func checkKey(k string) error {
for _, part := range strings.Split(k, ".") {
if keys.CheckName("key", part) != nil {
return fmt.Errorf("%q is not a key the bus can hold: letters, digits and -/_=, in dot-separated names", k)
}
}
return nil
}
// NewState is an empty State.
func NewState() *State { return &State{Vals: map[string]json.RawMessage{}, revs: map[string]uint64{}} }
func (m *State) Get(k string) (*stdio.StateEntry, error) {
m.mu.Lock()
defer m.mu.Unlock()
if err := checkKey(k); err != nil {
return nil, err
}
if m.Fail != nil {
return nil, m.Fail
}
v, ok := m.Vals[k]
if !ok {
return nil, nil
}
return &stdio.StateEntry{Key: k, Value: v, Revision: m.revs[k]}, nil
}
func (m *State) Create(k string, v any) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if err := checkKey(k); err != nil {
return 0, err
}
if m.Fail != nil {
return 0, m.Fail
}
if _, ok := m.Vals[k]; ok {
return 0, stdio.ErrChanged
}
return m.put(k, v)
}
func (m *State) Update(k string, v any, rev uint64) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if err := checkKey(k); err != nil {
return 0, err
}
if m.Fail != nil {
return 0, m.Fail
}
if rev == 0 || m.revs[k] != rev {
return 0, stdio.ErrChanged
}
return m.put(k, v)
}
func (m *State) Put(k string, v any) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if err := checkKey(k); err != nil {
return 0, err
}
if m.Fail != nil {
return 0, m.Fail
}
return m.put(k, v)
}
func (m *State) Keys() ([]string, error) {
m.mu.Lock()
defer m.mu.Unlock()
if m.Fail != nil {
return nil, m.Fail
}
var out []string
for k := range m.Vals {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func (m *State) put(k string, v any) (uint64, error) {
raw, ok := v.(json.RawMessage)
if !ok {
raw, _ = json.Marshal(v)
}
m.n++
m.Vals[k], m.revs[k] = raw, m.n
return m.n, nil
}

Some files were not shown because too many files have changed in this diff Show More