Compare commits

...
79 Commits
Author SHA1 Message Date
mesh-tools f0c21ca033 Merge pull request 'feh: the second single-module proof (one comment line)' (#702) from proof/feh-two into main 2026-11-17 19:49:31 +00:00
jschoubben c6ec42bc2e feh: mark the wallpaper file with the second single-module proof
The single-module proof of design 47 G (the operator, 2026-11-17), after pack49:
one change to one leaf module, followed to delivered on g14 and shanks with no hand act.
2026-11-17 20:49:20 +01:00
mesh-tools f65d789052 Merge pull request 'feh: the wallpaper file is replaced at every delivery (single-module proof)' (#701) from proof/feh-one-line into main 2026-11-17 16:02:47 +00:00
jschoubben 62b8251b99 feh: say the wallpaper file is replaced at every delivery, the glossary's word
The single-module proof of design 47 G (the operator, 2026-11-17): one change to one module,
followed through its pipeline to delivered on g14 and shanks.
2026-11-17 17:01:16 +01:00
mesh-tools 90c88fc7b5 Merge pull request 'A-471: the build modules' tests expect the language version in a toolchain report' (#700) from fix/a471-toolchain-language-version into main 2026-11-17 14:56:47 +00:00
jschoubben 740c218139 A-471: mesh-build-go start test expects the language version
Builds: design 47 G A-471 (G-22-lead-answers-2.md)
2026-11-17 15:53:05 +01:00
jschoubben 1481465ec7 A-471: the build modules' tests expect the language version in a toolchain report
The toolchain a build module reports (guard key and toolchain-reported event) names the
language version alone (mesh-delivery-sdk steprun Guard); the build module's own
Toolchain func keeps the installed text. Delivered with or after the sdk change.

Builds: design 47 G A-471 (G-22-lead-answers-2.md)
2026-11-17 15:53:05 +01:00
mesh-tools d2b8f64c94 Merge pull request 'G-24 tests on main: mesh-build-go keeps no step-code bridge, uses a stand-in composer, and runs within its level bounds' (#699) from fix/g24-tests-on-main into main 2026-11-17 14:35:32 +00:00
jschoubben eed95cd63b mesh-build-go: the integration level runs its slowest independent tests side by side, inside its 60 s bound
The level took 67 s locally and 74 s in the pipeline, over IntegrationRun's 60 s. The eight slowest tests share no state and run in
parallel: 31 s locally with the fixture's store and bus. The real-store test makes its work directory under the test temp
directory, not /tmp.

Builds: ADR 0445 §3 (the integration level's 60 s bound)
2026-11-17 15:31:00 +01:00
jschoubben 84106a9338 mesh-build-go: the compose ask's test builds a stand-in composer from its own testdata, reading no other repository
A pipeline's test step places no repository beside the tree and a module's tests read nothing outside their module (lead answer
A-474), so the test no longer needs MESH_CHECK_BESIDE or the node-engine's tree: it fetches a small program of the composer's
contract (testdata/composer) and still proves the build once, the kept composer on the second ask, and the verdict by case.

Builds: design 47 G lead answer A-474 (principles 10 and 9)
2026-11-17 15:31:00 +01:00
jschoubben 9ed13cd9dc mesh-build-go: the empty buildstep bridge goes (G's indexer runs), and the unit level runs its digest tests side by side
The bridge kept a directory for the pre-G indexer's package publishing; its own commit says it goes once G's indexer runs, and
the test of the delivery SDK's step code refuses it. The unit level took 6.3 s over its bound of 5 s in the take-in's pipeline:
the digest tests ran their go commands one after another; the marker table is a subtest per case in a fixture of its own, the
list half a test of its own, and the six slowest independent tests run in parallel (4.6 s locally).

Builds: design 47 G §464, §680 (the build modules keep no copy of the delivery SDK step code); ADR 0445 §3 (the unit level's 5 s bound)
2026-11-17 15:31:00 +01:00
mesh-tools d76c3588e3 Merge pull request 'gate: the start comparison ends only when every head was compared (A-464e)' (#698) from fix/47-G-start-compare into main 2026-11-17 08:22:03 +00:00
jschoubben 377d42f5ea gate: the start comparison ends only when every head was compared (A-464e)
Builds: design 47 G-22 A-464e, A-464
2026-11-17 09:03:29 +01:00
mesh-tools 9ade421e12 Merge pull request 'gate: the source of a watched branch follows every merge, forward only (A-464..A-464d)' (#697) from fix/47-G-untaken-merge into main 2026-11-17 01:13:07 +00:00
jschoubben 46f3dc8cd7 gate: take-in.taken moves a head only forward too (A-464d)
Also corrects the manifest test's comment, which said the gate does not hear merged (A-464).

Builds: design 47 G-22 A-464d, A-464
2026-11-17 02:10:36 +01:00
jschoubben 7da6dd20a1 gate: a head moves only forward and only when it changes (A-464c)
Builds: design 47 G-22 A-464c, A-464
2026-11-17 02:02:07 +01:00
jschoubben c87cf1ace5 gate: the source of a watched branch follows every merge into it (A-464, A-464b)
The forge bridge's merged moves the repository's head and says source-moved; at its start each head is compared with the forge's branch head; gate_source_follow does it for one repository by hand.

Builds: design 47 G-22 A-464, A-464b
2026-11-17 01:59:09 +01:00
mesh-tools 3f3f6789dd Merge pull request 'nats: the tools bundle answers initialize again' (#696) from fix/nats-starts-again into main 2026-11-16 21:29:12 +00:00
jschoubben 8b5cc8be8d nats: the tools bundle answers initialize again (keep days read when asked)
bundleKeepDays read the config value snapshot-keep-days while allTools
listed the tools, before stdio.Serve; a config read waits for the runtime's
channel, which opens only once the bundle serves, so the bundle never
answered initialize and node-tools served none of its tools (novox
2026-11-16 22:21 and 22:26, after the first build from main since A-112).
The setting is now read when nats_snapshots is called.
2026-11-16 22:28:40 +01:00
mesh-tools 7c6aed9d69 Merge pull request 'nats: a tool purges a stream's messages below a sequence' (#695) from fix/nats-stream-purge into main 2026-11-16 21:07:49 +00:00
jschoubben 4f9f307eaf nats: a tool purges a stream's messages below a sequence
A build seat's work queue (SEAT_NODE_BUILD_GO) held about 370 asks of
pipelines nobody waits for, ahead of the one pipeline wanted, and no tool
could empty it short of removing the stream. nats_stream_purge removes the
messages below a given sequence, keeping the rest, the stream and its
consumers; it needs why, reads first, offers dry_run, refuses a purge past
the stream's end and the module-kept data nats_drop owns, and says
stream-purged as its audit event, like the three removal tools.
2026-11-16 22:07:18 +01:00
mesh-tools e87c6cd3b6 Merge pull request 'mesh-build-go: an asked version brings the requirements and sums of its go.mod into the modfile' (#694) from fix/build-go-asked-versions-every-dir into main 2026-11-16 20:10:03 +00:00
jschoubben 73c2286ec9 mesh-build-go: an asked version brings the requirements and sums of its go.mod into the modfile
Kind: a version written into a workspace's modfile without what that version's graph needs, judged against files the build does not use.

Builds: design 47 G (ADR 0573 §2, §4, as written in versions.go: asked versions hold whatever go.mod names)
2026-11-16 21:06:17 +01:00
mesh-tools 3be2f79921 Merge pull request 'mesh-hand-acts: a thirteenth change through its own pipeline' (#693) from probe/thirteenth-pipeline-hand-acts into main 2026-11-16 15:42:06 +00:00
jschoubben cf5ea429a2 mesh-hand-acts: a thirteenth change through its own pipeline 2026-11-16 16:37:58 +01:00
mesh-tools 1419dcef7d Merge pull request 'mesh-hand-acts: a twelfth change through its own pipeline' (#692) from probe/twelfth-pipeline-hand-acts into main 2026-11-16 14:44:00 +00:00
jschoubben 8c96a02a01 mesh-hand-acts: a twelfth change through its own pipeline 2026-11-16 15:43:38 +01:00
mesh-tools 03cc8eac90 Merge pull request 'Design 47 G start: catalog fixes placed by hand and the Go build per module (A-447)' (#691) from fix/47-G-start-merge into main 2026-11-16 14:15:15 +00:00
jschoubben f31d6c645d Merge commit 'cdf16870' into fix/47-G-start-merge 2026-11-16 15:11:01 +01:00
jschoubben bbc9eb7c93 Merge remote-tracking branch 'origin/main' into fix/47-G-start-merge 2026-11-16 15:11:01 +01:00
mesh-tools 1310c28102 Merge pull request 'mesh-hand-acts: an eleventh change through its own pipeline' (#690) from probe/eleventh-pipeline-hand-acts into main 2026-11-16 14:02:55 +00:00
jschoubben affffa9c23 mesh-hand-acts: an eleventh change through its own pipeline 2026-11-16 14:58:00 +01:00
mesh-tools 94f3cde557 Merge pull request 'mesh-hand-acts: a tenth change through its own pipeline' (#689) from probe/tenth-pipeline-hand-acts into main 2026-11-16 13:46:58 +00:00
jschoubben 6962da2f98 mesh-hand-acts: a tenth change through its own pipeline 2026-11-16 14:46:48 +01:00
mesh-tools 0aee792af6 Merge pull request 'mesh-hand-acts: a ninth change through its own pipeline' (#688) from probe/ninth-pipeline-hand-acts into main 2026-11-16 13:32:32 +00:00
jschoubben 86367f65b2 mesh-hand-acts: a ninth change through its own pipeline 2026-11-16 14:32:22 +01:00
mesh-tools 932e5c4e4b Merge pull request 'mesh-hand-acts: an eighth change through its own pipeline' (#687) from probe/eighth-pipeline-hand-acts into main 2026-11-16 13:21:00 +00:00
jschoubben 930c0fb783 mesh-hand-acts: an eighth change through its own pipeline 2026-11-16 14:20:51 +01:00
mesh-tools 79bd15884d Merge pull request 'mesh-hand-acts: a seventh change through its own pipeline' (#686) from probe/seventh-pipeline-hand-acts into main 2026-11-16 13:14:34 +00:00
jschoubben 13a0277f8c mesh-hand-acts: a seventh change through its own pipeline 2026-11-16 14:14:21 +01:00
mesh-tools fcdf21d615 Merge pull request 'mesh-hand-acts: a sixth change through its own pipeline' (#685) from probe/sixth-pipeline-hand-acts into main 2026-11-16 13:10:33 +00:00
jschoubben d6b75616f5 mesh-hand-acts: a sixth change through its own pipeline 2026-11-16 14:10:25 +01:00
mesh-tools 096668a14a Merge pull request 'mesh-hand-acts: a fifth change through its own pipeline' (#684) from probe/fifth-pipeline-hand-acts into main 2026-11-16 12:50:56 +00:00
jschoubben 2b1bc1d302 mesh-hand-acts: a fifth change through its own pipeline 2026-11-16 13:50:47 +01:00
mesh-tools a55241c30b Merge pull request 'mesh-hand-acts: a fourth change through its own pipeline' (#683) from probe/fourth-pipeline-hand-acts into main 2026-11-16 12:21:38 +00:00
jschoubben e1ee303a1c mesh-hand-acts: a fourth change through its own pipeline 2026-11-16 13:21:29 +01:00
mesh-tools c9f8e6e8cd Merge pull request 'mesh-hand-acts: a third change through its own pipeline, each step asked for this module's directory' (#682) from probe/third-pipeline-hand-acts into main 2026-11-16 11:24:27 +00:00
jschoubben 0afca54a0f mesh-hand-acts: a third change through its own pipeline, each step asked for this module's directory 2026-11-16 12:24:18 +01:00
mesh-tools 8bb73eb61f Merge pull request 'mesh-hand-acts: a second change through its own pipeline, each step on this module alone' (#681) from probe/second-pipeline-hand-acts into main 2026-11-16 11:13:39 +00:00
jschoubben 229c887999 mesh-hand-acts: a second change through its own pipeline, each step on this module alone 2026-11-16 12:13:31 +01:00
jschoubben cdf168706b Go steps test a module's own directories alone, not what its go.mod replaces from outside (A-447c)
Builds: design 47 G (G-22 lead answers 2) A-447c
2026-11-16 11:42:56 +01:00
jschoubben ae78fb039f Go fetch with no dir fetches the tree whole again, for the checker until G removes it (A-447e)
Builds: design 47 G (G-22 lead answers 2) A-447e
2026-11-16 11:40:18 +01:00
jschoubben 928de07c43 Go build module: a step for a module compares its own module.json's generated files alone (A-447c)
Builds: design 47 G (G-22 lead answers 2) A-447c
2026-11-16 11:37:23 +01:00
jschoubben 666be37916 Go fetch is refused with no dir, naming its asker; the checking program's fetch names its directory (A-447b)
Builds: design 47 G (G-22 lead answers 2) A-447b
2026-11-16 11:34:17 +01:00
jschoubben ccab80d726 Go fetch with dir: a directory with no go.mod chooses every go.mod under it; none chosen fails naming it (A-447a)
Builds: design 47 G (G-22 lead answers 2) A-447a
2026-11-16 11:16:16 +01:00
jschoubben 1168376930 Go build module: a fetch asked with dir chooses that module and its local replaces alone (A-447)
Builds: design 47 G (G-22 lead answers 2) A-447
2026-11-16 11:09:26 +01:00
mesh-tools 2679399d60 Merge pull request 'mesh-hand-acts: the first change delivered through its own pipeline' (#680) from probe/first-pipeline-hand-acts into main 2026-11-16 09:17:04 +00:00
jschoubben ce30601b0c mesh-hand-acts: say in its header that its changes are delivered through its own pipeline (the first to run one end to end) 2026-11-16 10:16:53 +01:00
jschoubben d2c1a24a24 nats: the bus keeper no longer reads the seat mesh-catalog's states before the first views (reverts ed536102)
A-379 replaced A-376 for the start, and the seat mesh-catalog offers neither the states modules and manifests as views
nor verbs of those names, so the validator refused the manifest's two consumes and two invokes.

Builds: design 47 G A-379, A-445 (G-22-lead-answers-2.md)
2026-11-16 09:51:48 +01:00
jschoubben 88ff34e855 distribution: a step given up at its bound is said given up, and its call's end is said later
Builds: design 47 G A-443, A-443a
2026-11-16 09:46:12 +01:00
jschoubben 76195e421f mesh-delivery-gate: the start pass runs once, asking again only while the registrations cannot be read
Builds: design 47 G A-442, A-401
2026-11-16 09:41:45 +01:00
jschoubben 59704456c3 messenger: offer every ask when it is raised; hold none back for room
The operator channel kept at most three asks of one asker offered and held its oldest back behind newer ones, so asks
waited unseen (the pipeline's proposal behind two mailu asks). It keeps no count any more and holds nothing back: every
ask is offered when it is raised, and lapses at its expiry or ends at its answer. An ask held back before the change is
offered again at the router's next tick; ask-again offers one so and holds nothing back in its place.

Builds: design 47 G A-424
2026-11-15 15:52:48 +01:00
jschoubben 4a99f7044e distribution: a publish with a version and no parts tags the digest the store holds
The builder's publish again of mesh-delivery-pipeline 0.1.6 sends the version and the digest with no parts, as A-73
has it; the build stored the bundle untagged, and the holder answered 'publish: no parts' unless the tag existed
already. The holder now checks the digest against the store under the name (the artifact store, A-50), asks the
builder's reservation, tags it with the version and says published; a digest it holds none of is refused as not
stored, naming the digest.

Builds: design 47 G A-417, A-73, A-50, §588
2026-11-15 02:39:28 +01:00
jschoubben 7d17441a9a gofmt 2026-11-15 01:38:29 +01:00
jschoubben c006538064 Distribution: the incoming-full finding is said off the publish's path
Builds: design 47 G (publish path of oci-artifact-registry; no design line covers it)
2026-11-15 01:38:27 +01:00
jschoubben bfc877209f Distribution: each bus step of a publish bounded at 10 s and logged by name
Builds: design 47 G (publish path of oci-artifact-registry; no design line covers the bounds or log lines)
2026-11-15 01:30:05 +01:00
jschoubben 99f3e23492 Distribution: each wait of a publish logged with its time; the window's learning no longer holds the answer
Builds: design 47 G (publish path of oci-artifact-registry; no design line covers the log lines)
2026-11-15 01:20:04 +01:00
jschoubben e8204a9c27 Gate: the start pass asks again until no repository is left out (A-401)
The forge's git verbs may serve after the gate does; one pass with every head read refused wrote nothing and was never retried.

Builds: design 47 G A-401
2026-11-14 22:34:54 +01:00
jschoubben 89397466aa Gate: source-moved for started heads names no group; tests for the registration-changed branch and the start pass (A-401, A-404)
Builds: design 47 G A-401, A-404
2026-11-14 22:16:30 +01:00
jschoubben 28f616e379 Gate: a held source key gets the registered repositories it lacks, heads unchanged (A-403)
Builds: design 47 G A-403, A-401, A-34
2026-11-14 22:09:06 +01:00
jschoubben 8ea8ef9768 Gate: a watched branch starts from the forge, as a clone does; tool gate_source_start (A-401)
At its start and on each registration-changed of the seat mesh-catalog, the gate writes source.<branch> for every watched
branch the registrations name that no source key holds, each registered repository at its head from the seat git's verb
branch, group empty, then says source-moved; a repository whose head cannot be read is left out and said; a held key is
never rewritten. gate_source_start (branch, why) does it by hand through the hand-act log. The manifest consumes
registration-changed and invokes seat:git.branch and seat:mesh-catalog.registrations.

Builds: design 47 G A-401, §430, §15, A-34
2026-11-14 22:08:22 +01:00
jschoubben fb24d384a2 Build seat: says each toolchain-reported it said, not only the ones it could not 2026-11-14 19:00:01 +01:00
jschoubben 5236c2aa72 gitea: the seat git's branch with no branch named answers the repository's default branch (A-399)
The seed and a registration with no branches read the default branch through the seat git (design 47 G-1 §104) by
calling branch with an empty branch; gitea refused it as no branch for a path, and the seed stopped on novox 2026-11-14.
2026-11-14 14:57:27 +01:00
jschoubben 5df0fb64f5 mesh-build-go: a finding writer that cannot start is said, and the module serves on (A-394)
Live on shanks the bundle exited 39 times on 'an ordered consumer of mesh-conditions.findings-asked: context deadline exceeded', so the seat node-build-go never answered steps and the seed waited.

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-14 11:52:23 +01:00
jschoubben ed536102ac A-376: the bus keeper makes the catalog half's own holder before the first views
Before the catalog half's first views are in (a read refused, or none-yet), the keeper reads the manifest on the trunk of the
holder of the seat mesh-catalog from the seat's states modules and manifests (the holder from the seat that places modules)
and makes its kept data as any module's, each entry said as ever; every other module waits as A-367 says.

Builds: design 47 G A-376 (G-22-lead-answers-2.md)

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 23:11:57 +01:00
jschoubben a98b6ba77e S6: the bus keeper treats a none-yet catalog half as not yet viewed
mesh/delivery ready: it is merged on the Approve
mesh/delivery-group group fix/47-G-start-2 checking: 2 of 2 member(s) ready
PlanCatalog plans nothing, deletes no plans key and orphans no entry while the
catalog half lists no module before any was seen; the first revision with
entries plans as before.

Builds: design 47 G A-367 (G-22-lead-answers-2.md), W22/Q4 of start-waits.md

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 20:48:04 +01:00
mesh-tools b1125c02cf Merge pull request 'Bridge for G's take-in: an empty modules/mesh-build-go/buildstep package' (#678) from fix/47-G-bridge into main 2026-11-13 13:40:27 +00:00
jschoubben a4bc3d95ed Bridge for G's take-in: an empty modules/mesh-build-go/buildstep package
mesh/delivery merged: source.main moved to novox/mesh-catalog@b1125c02cfe6
mesh/delivery-group group fix/47-G-bridge merged: every member is merged
mesh/check checking a4bc3d95edef on b1125c02cfe6 | novox/mesh-catalog#678 in group fix/47-G-bridge
The running pre-G indexer publishes mesh-catalog's packages only while every package directory it published before is
in the tree; G removed this one, so mesh-catalog's packages are never published (2026-11-13). An empty package keeps
the directory; it goes once G's indexer runs.

Claude-Session: https://claude.ai/code/session_01NcbgfGKaxw984XyAPfLm4h
2026-11-13 14:34:40 +01:00
mesh-tools 6ba73d4e2a Merge pull request 'Milestone 47 G: mesh-catalog's part of the one release' (#677) from feat/47-G-join12 into main 2026-11-13 13:08:20 +00:00
71 changed files with 2242 additions and 322 deletions
@@ -0,0 +1,123 @@
package main
import (
"fmt"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/parts"
"distribution/internal/holder"
)
// stepBound bounds each bus operation of a publish well under the tool runner's 30 s call bound, so a hang is an error
// naming its step and the publish answers with it.
const stepBound = 10 * time.Second
// step runs f, logs its start and its end with its time, and gives up on it after bound: the error names the step. A step
// given up on is not cancelled (the SDK's bus calls take no context): its call ends at the SDK's own bound of the ask, and
// the log says, by the step's name, that it was given up and later when the call behind it ended and how (A-443, A-443a).
func step[T any](bound time.Duration, what string, f func() (T, error)) (T, error) {
type res struct {
v T
err error
}
start := time.Now()
logf("step %s starts", what)
ch := make(chan res, 1)
go func() { v, err := f(); ch <- res{v, err} }()
select {
case r := <-ch:
logf("step %s ended after %s (%v)", what, time.Since(start).Round(time.Millisecond), r.err)
return r.v, r.err
case <-time.After(bound):
var zero T
err := fmt.Errorf("%s did not answer within %s", what, bound)
logf("step %s: %v; given up, its call is left to end at the SDK's own bound", what, err)
go func() {
r := <-ch
logf("step %s, given up after %s: its call ended after %s (%v)", what, bound, time.Since(start).Round(time.Millisecond), r.err)
}()
return zero, err
}
}
func stepErr(bound time.Duration, what string, f func() error) error {
_, err := step(bound, what, func() (struct{}, error) { return struct{}{}, f() })
return err
}
// boundedObjects is the incoming object store with each operation bounded and logged.
type boundedObjects struct{ parts.Objects }
func (b boundedObjects) Used() (uint64, error) {
return step(stepBound, "the incoming object store's fill (objects)", b.Objects.Used)
}
func (b boundedObjects) Put(name string, data []byte) error {
return stepErr(stepBound, "the put of a chunk in the incoming object store (objects)", func() error { return b.Objects.Put(name, data) })
}
func (b boundedObjects) Get(name string) ([]byte, error) {
return step(stepBound, "the get of a chunk from the incoming object store (objects)", func() ([]byte, error) { return b.Objects.Get(name) })
}
func (b boundedObjects) Delete(name string) error {
return stepErr(stepBound, "the delete of a chunk in the incoming object store (objects)", func() error { return b.Objects.Delete(name) })
}
// boundedTransfers is the transfers state with its reads and writes bounded and logged.
type boundedTransfers struct {
parts.Transfers[holder.Opening]
}
func (b boundedTransfers) Get(id string) (parts.Transfer[holder.Opening], bool, error) {
type r struct {
t parts.Transfer[holder.Opening]
ok bool
}
x, err := step(stepBound, "the read of the transfer in the transfers state", func() (r, error) {
t, ok, err := b.Transfers.Get(id)
return r{t, ok}, err
})
return x.t, x.ok, err
}
func (b boundedTransfers) Put(t parts.Transfer[holder.Opening]) error {
return stepErr(stepBound, "the write of the transfer in the transfers state", func() error { return b.Transfers.Put(t) })
}
func (b boundedTransfers) Delete(id string) error {
return stepErr(stepBound, "the delete of the transfer in the transfers state", func() error { return b.Transfers.Delete(id) })
}
// boundedCap is the incoming cap read from its binding, bounded and logged.
func boundedCap(c func() (uint64, error)) func() (uint64, error) {
return func() (uint64, error) {
return step(stepBound, "the read of incoming's cap from its binding (objects)", c)
}
}
// boundedArm is the idle key's write, bounded and logged.
func boundedArm(a func(string, time.Duration) error) func(string, time.Duration) error {
return func(id string, ttl time.Duration) error {
return stepErr(stepBound, "the write of the idle key in the idle state", func() error { return a(id, ttl) })
}
}
// boundedFinding is the findings writer's say moved off the publish's path: the receiver says the incoming fill on every
// open, and a publish never waits for it. One worker says them in order, each bounded and logged; one said while the worker
// is full is dropped and logged (the next open says the fill again).
func boundedFinding(f holder.Finding) holder.Finding {
type said struct {
kind, subject, words string
raised bool
}
queue := make(chan said, 64)
go func() {
for s := range queue {
_ = stepErr(stepBound, "the finding "+s.kind+" of "+s.subject, func() error { f(s.kind, s.subject, s.words, s.raised); return nil })
}
}()
return func(kind, subject, words string, raised bool) {
select {
case queue <- said{kind, subject, words, raised}:
default:
logf("the finding %s of %s is not said now: the writer is behind, the next open says it again", kind, subject)
}
}
}
@@ -0,0 +1,70 @@
package main
import (
"fmt"
"strings"
"sync"
"testing"
"time"
)
// A bus operation that never answers becomes an error naming its step, within the bound.
func TestAHungStepAnswersWithItsName(t *testing.T) {
block := make(chan struct{})
defer close(block)
_, err := step(10*time.Millisecond, "the write of the idle key", func() (int, error) { <-block; return 0, nil })
if err == nil || !strings.Contains(err.Error(), "the write of the idle key did not answer within") {
t.Fatalf("err = %v", err)
}
if v, err := step(time.Second, "x", func() (int, error) { return 7, nil }); v != 7 || err != nil {
t.Fatalf("%d %v", v, err)
}
}
// A step given up on is said given up by name, and the log says later when the call behind it ended and how (A-443a).
func TestAGivenUpStepSaysWhenItsCallEnded(t *testing.T) {
var mu sync.Mutex
var lines []string
ended := make(chan struct{})
w := func(f string, a ...any) {
l := fmt.Sprintf(f, a...)
mu.Lock()
lines = append(lines, l)
mu.Unlock()
if strings.Contains(l, "the put of a chunk") && strings.Contains(l, "its call ended after") {
close(ended)
}
}
old := logTo.Swap(&w)
defer logTo.Store(old)
release := make(chan struct{})
if _, err := step(10*time.Millisecond, "the put of a chunk", func() (int, error) { <-release; return 0, fmt.Errorf("bus closed") }); err == nil {
t.Fatal("not given up")
}
close(release)
<-ended
mu.Lock()
all := strings.Join(lines, "\n")
mu.Unlock()
if !strings.Contains(all, "given up") || !strings.Contains(all, "bus closed") {
t.Fatalf("log: %s", all)
}
}
// The finding is said off the caller's path: a writer that never answers does not hold the call that says it.
func TestAFindingIsSaidOffThePath(t *testing.T) {
block := make(chan struct{})
defer close(block)
say := boundedFinding(func(string, string, string, bool) { <-block })
done := make(chan struct{})
go func() {
say("incoming-full", "incoming", "w", false)
say("incoming-full", "incoming", "w", false)
close(done)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("the call was held by the finding")
}
}
@@ -16,6 +16,7 @@ import (
"os" "os"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient" "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
@@ -29,7 +30,15 @@ import (
"distribution/internal/holder" "distribution/internal/holder"
) )
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, "[distribution-tools] "+format+"\n", a...) } // logTo is where logf writes; a test replaces it (atomically: a step given up on logs from its own goroutine).
var logTo atomic.Pointer[func(string, ...any)]
func init() {
w := func(format string, a ...any) { fmt.Fprintf(os.Stderr, "[distribution-tools] "+format+"\n", a...) }
logTo.Store(&w)
}
func logf(format string, a ...any) { (*logTo.Load())(format, a...) }
// kept is one state of distribution's, through the runtime. // kept is one state of distribution's, through the runtime.
type kept[T any] struct { type kept[T any] struct {
@@ -249,10 +258,21 @@ func tools(v *holder.Verbs, windows kept[holder.Window]) []stdio.Tool {
if err != nil { if err != nil {
return nil, err return nil, err
} }
a, err := v.Publish(seat, stdio.CallerOf(args), publishOf(in)) caller := stdio.CallerOf(args)
start := time.Now()
logf("%s.publish received from %s: %s %s (%s, %d parts)", seat, caller, in.Name, in.Version, in.Kind, len(in.Parts))
a, err := v.Publish(seat, caller, publishOf(in))
if err == nil { if err == nil {
learnPublishing(windows, stdio.CallerOf(args)) // The window's learning asks the delivery view (up to 60 s, past the runner's 30 s call bound),
// so it never holds the answer.
go func() {
t := time.Now()
logf("learning the publishing build of %s starts (mesh-delivery-view.node_haves)", caller)
learnPublishing(windows, caller)
logf("learning the publishing build of %s ended after %s", caller, time.Since(t).Round(time.Millisecond))
}()
} }
logf("%s.publish from %s answered after %s (error: %v)", seat, caller, time.Since(start).Round(time.Millisecond), err)
return publishAnswer(a), err return publishAnswer(a), err
}}, }},
stdio.Tool{Name: seat + "." + holder.VerbPublishPart, Description: "Send one chunk (at most 1 MiB) of one part of an open transfer, from the caller that opened it.", stdio.Tool{Name: seat + "." + holder.VerbPublishPart, Description: "Send one chunk (at most 1 MiB) of one part of an open transfer, from the caller that opened it.",
@@ -468,8 +488,8 @@ func main() {
}() }()
base, _ := url.Parse("http://127.0.0.1:5001") base, _ := url.Parse("http://127.0.0.1:5001")
store := &holder.OCI{Base: base, Upstream: &holder.Upstream{}} store := &holder.OCI{Base: base, Upstream: &holder.Upstream{}}
h := &holder.Holder{Objects: parts.BusObjects(holder.IncomingObjects), Builder: holder.Reserved{Ask: stdio.Ask}, Store: store, Say: say, Finding: findings.Say, h := &holder.Holder{Logf: logf, Objects: boundedObjects{parts.BusObjects(holder.IncomingObjects)}, Builder: holder.Reserved{Ask: stdio.Ask}, Store: store, Say: say, Finding: boundedFinding(findings.Say),
Transfers: parts.BusTransfers[holder.Opening](holder.TransfersState), Cap: parts.BusCap(holder.IncomingObjects), Record: recordAct, Idle: parts.BusArm(holder.IdleState)} Transfers: boundedTransfers{parts.BusTransfers[holder.Opening](holder.TransfersState)}, Cap: boundedCap(parts.BusCap(holder.IncomingObjects)), Record: recordAct, Idle: boundedArm(parts.BusArm(holder.IdleState))}
files := &holder.Files{Container: or(os.Getenv("MESH_REGISTRY_CONTAINER"), "mesh-registry"), Run: holder.ExecRunner, UID: os.Getuid()} files := &holder.Files{Container: or(os.Getenv("MESH_REGISTRY_CONTAINER"), "mesh-registry"), Run: holder.ExecRunner, UID: os.Getuid()}
v := &holder.Verbs{Holder: h, Catalog: store, Holds: kept[holder.Hold]{state: "holds", key: holder.Hold.Key}, Mirrors: mirrors{}, v := &holder.Verbs{Holder: h, Catalog: store, Holds: kept[holder.Hold]{state: "holds", key: holder.Hold.Key}, Mirrors: mirrors{},
Files: files, SweepHolder: func() (string, error) { return seatlist.HolderOf(stdio.Ask, holder.SweepSeat) }} Files: files, SweepHolder: func() (string, error) { return seatlist.HolderOf(stdio.Ask, holder.SweepSeat) }}
@@ -71,6 +71,7 @@ func (s *fakeStore) Put(t PublishRequest, parts map[string][]byte) (string, erro
return t.Name + "@" + t.Digest, nil return t.Name + "@" + t.Digest, nil
} }
func (s *fakeStore) Held(string, string, string) (string, bool, error) { return "", false, nil } func (s *fakeStore) Held(string, string, string) (string, bool, error) { return "", false, nil }
func (s *fakeStore) Holds(r PublishRequest) (bool, error) { return s.has[r.Digest], nil }
type said struct { type said struct {
seat, event string seat, event string
@@ -231,6 +232,21 @@ func TestTheHolderOfTheTwoImageSeats(t *testing.T) {
return PublishRequest{Transfer: id, Name: "m/program", Version: "0.1.4", Kind: KindBundle, Digest: part.Digest, Parts: []Part{part}} return PublishRequest{Transfer: id, Name: "m/program", Version: "0.1.4", Kind: KindBundle, Digest: part.Digest, Parts: []Part{part}}
} }
t.Run("a publish says each of its waits with its time", func(t *testing.T) {
b := newBed(true)
var lines []string
b.h.Logf = func(f string, a ...any) { lines = append(lines, fmt.Sprintf(f, a...)) }
if _, err := b.h.Publish(ArtifactSeat, builder, req("tl")); err != nil {
t.Fatal(err)
}
all := strings.Join(lines, "\n")
for _, want := range []string{"the store's read of what it holds (the program's registry) starts", "(the program's registry) ended after", "the opening of the transfer on the bus"} {
if !strings.Contains(all, want) {
t.Errorf("no log line %q in:\n%s", want, all)
}
}
})
t.Run("a transfer in parts, cut and resumed, hashed, numbered by the builder, published", func(t *testing.T) { t.Run("a transfer in parts, cut and resumed, hashed, numbered by the builder, published", func(t *testing.T) {
b := newBed(true) b := newBed(true)
a, err := b.h.Publish(ArtifactSeat, builder, req("t1")) a, err := b.h.Publish(ArtifactSeat, builder, req("t1"))
@@ -650,30 +666,37 @@ func TestTheHolderOfTheTwoImageSeats(t *testing.T) {
} }
} }
}) })
t.Run("a late publish names a held bundle with no size: tagged at the store's size, nothing sent", func(t *testing.T) { t.Run("a publish with a version and no parts tags the held digest at the store's size; one not held is not stored", func(t *testing.T) {
// Design 47 G A-417, A-73, A-50: the build stored the bundle untagged; the builder's publish of its version with that
// digest and no parts tags it and says published, nothing sent; a digest the store holds none of is refused as not
// stored, naming the digest.
store := newFakeRegistry() store := newFakeRegistry()
sts := httptest.NewServer(store) sts := httptest.NewServer(store)
defer sts.Close() defer sts.Close()
base, _ := url.Parse(sts.URL) base, _ := url.Parse(sts.URL)
o := &OCI{Base: base} b := newBed(true)
b.h.Store = &OCI{Base: base}
blob := []byte(strings.Repeat("b", 4321)) blob := []byte(strings.Repeat("b", 4321))
store.blobs["m/tools@"+digestOf(blob)] = blob store.blobs["m/tools@"+digestOf(blob)] = blob
ref, err := o.Put(PublishRequest{Name: "m/tools", Version: "0.1.4", Kind: KindBundle, Digest: digestOf(blob), got, err := b.h.Publish(ArtifactSeat, "x.mesh-delivery-builder", PublishRequest{Name: "m/tools", Version: "0.1.4", Kind: KindBundle, Digest: digestOf(blob),
Parts: []Part{{Digest: digestOf(blob)}}}, map[string][]byte{}) Transfer: "republish-m-0-1-4-tools"})
if err != nil || ref != "m/tools@"+digestOf(blob) { if err != nil || !got.Ended || len(b.said) != 1 || b.said[0].event != EventPublished || b.said[0].body.Reference != SeatReference(ArtifactSeat, "m/tools@"+digestOf(blob)) {
t.Fatalf("Put = %q, %v", ref, err) t.Fatalf("Publish = %+v, %v; said %+v", got, err, b.said)
} }
var man struct { var man struct {
Layers []struct { Layers []struct {
Size int64 `json:"size"` Digest string `json:"digest"`
Size int64 `json:"size"`
} `json:"layers"` } `json:"layers"`
} }
if err := json.Unmarshal(store.manifests["m/tools:0.1.4"], &man); err != nil || len(man.Layers) != 1 || man.Layers[0].Size != int64(len(blob)) { if err := json.Unmarshal(store.manifests["m/tools:0.1.4"], &man); err != nil || len(man.Layers) != 1 || man.Layers[0].Size != int64(len(blob)) || man.Layers[0].Digest != digestOf(blob) {
t.Errorf("the version's manifest %s, want one layer of %d bytes", store.manifests["m/tools:0.1.4"], len(blob)) t.Errorf("the version's manifest %s, want one layer of %d bytes", store.manifests["m/tools:0.1.4"], len(blob))
} }
if _, err := o.Put(PublishRequest{Name: "m/tools", Version: "0.1.5", Kind: KindBundle, Digest: "sha256:" + strings.Repeat("e", 64), gone := "sha256:" + strings.Repeat("e", 64)
Parts: []Part{{Digest: "sha256:" + strings.Repeat("e", 64)}}}, map[string][]byte{}); err == nil { _, err = b.h.Publish(ArtifactSeat, "x.mesh-delivery-builder", PublishRequest{Name: "m/tools", Version: "0.1.5", Kind: KindBundle, Digest: gone, Transfer: "republish-m-0-1-5-tools"})
t.Error("a digest the store does not hold was tagged") if err == nil || !strings.Contains(err.Error(), "not stored: ") || !strings.Contains(err.Error(), gone) || store.manifests["m/tools:0.1.5"] != nil ||
len(b.said) != 2 || b.said[1].event != EventRefused || !b.findings["publish-refused m/tools@0.1.5"] {
t.Errorf("a digest not held: %v; said %+v; findings %v", err, b.said, b.findings)
} }
}) })
t.Run("mirror answers at once and says the copy's end on the seat's events", func(t *testing.T) { t.Run("mirror answers at once and says the copy's end on the seat's events", func(t *testing.T) {
@@ -219,6 +219,25 @@ func (o *OCI) Held(name, version, digest string) (string, bool, error) {
return name + "@" + d, true, nil return name + "@" + d, true, nil
} }
// Holds is whether the store holds what r's digest names under r's name: an image's manifest by its digest, or a bundle's
// or an archive's blob.
func (o *OCI) Holds(r PublishRequest) (bool, error) {
if r.Kind != KindImage && r.Kind != KindUpstream {
return o.Has(r.Name, r.Digest)
}
resp, _, err := o.do(http.MethodHead, "/v2/"+r.Name+"/manifests/"+r.Digest, "", nil, accepted...)
if err != nil {
return false, err
}
switch resp.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
}
return false, fmt.Errorf("the store answered %d for the manifest %s of %s", resp.StatusCode, r.Digest, r.Name)
}
// Versions are name's tags. // Versions are name's tags.
func (o *OCI) Versions(name string) ([]string, error) { func (o *OCI) Versions(name string) ([]string, error) {
resp, b, err := o.do(http.MethodGet, "/v2/"+name+"/tags/list", "", nil) resp, b, err := o.do(http.MethodGet, "/v2/"+name+"/tags/list", "", nil)
@@ -143,6 +143,9 @@ type Store interface {
// Held is the reference's path of name at version when the store already holds digest under it: a rebuild of a held // Held is the reference's path of name at version when the store already holds digest under it: a rebuild of a held
// fingerprint answers it and stores nothing. // fingerprint answers it and stores nothing.
Held(name, version, digest string) (reference string, ok bool, err error) Held(name, version, digest string) (reference string, ok bool, err error)
// Holds says whether the store holds what a publish's digest names under its name: an image's manifest, or a bundle's or
// an archive's blob. The store is the artifact store (A-50): `artifact-store://<name>/blobs/<digest>` is that blob.
Holds(r PublishRequest) (bool, error)
} }
// SeatReference is a reference in the form the seat answers (novox/hq ADR 0605 §9): the seat's name, then the path the // SeatReference is a reference in the form the seat answers (novox/hq ADR 0605 §9): the seat's name, then the path the
@@ -173,6 +176,20 @@ type Holder struct {
// Idle writes a transfer's idle key with its own time to live, once (parts.BusArm over IdleState); the receiver // Idle writes a transfer's idle key with its own time to live, once (parts.BusArm over IdleState); the receiver
// arms it on each open that leaves a transfer open. Nil arms nothing. // arms it on each open that leaves a transfer open. Nil arms nothing.
Idle func(id string, ttl time.Duration) error Idle func(id string, ttl time.Duration) error
// Logf says each wait of a publish with its time (stderr in the bundle); nil says nothing.
Logf func(string, ...any)
}
// timed runs f and logs its name and time when the holder logs.
func (h *Holder) timed(what string, f func() error) error {
if h.Logf == nil {
return f()
}
start := time.Now()
h.Logf("publish: %s starts", what)
err := f()
h.Logf("publish: %s ended after %s (%v)", what, time.Since(start).Round(time.Millisecond), err)
return err
} }
func (h *Holder) now() time.Time { func (h *Holder) now() time.Time {
@@ -249,23 +266,83 @@ func (h *Holder) Publish(seat, caller string, r PublishRequest) (PublishAnswer,
} }
// A rebuild of a held fingerprint answers the held version and stores nothing. // A rebuild of a held fingerprint answers the held version and stores nothing.
if r.Version != "" { if r.Version != "" {
if ref, ok, err := h.Store.Held(r.Name, r.Version, r.Digest); err != nil { var ref string
var ok bool
err := h.timed("the store's read of what it holds (the program's registry)", func() (err error) {
ref, ok, err = h.Store.Held(r.Name, r.Version, r.Digest)
return err
})
if err != nil {
return PublishAnswer{}, fmt.Errorf("publish: the store cannot say whether it holds %s@%s: %w", r.Name, r.Version, err) return PublishAnswer{}, fmt.Errorf("publish: the store cannot say whether it holds %s@%s: %w", r.Name, r.Version, err)
} else if ok { } else if ok {
return PublishAnswer{Transfer: r.Transfer, Ended: true}, h.Say(seat, EventPublished, h.eventOf(seat, r, SeatReference(seat, ref), "")) return PublishAnswer{Transfer: r.Transfer, Ended: true}, h.timed("the published event on the seat", func() error {
return h.Say(seat, EventPublished, h.eventOf(seat, r, SeatReference(seat, ref), ""))
})
} }
} }
if r.Version != "" && len(r.Parts) == 0 {
return h.tagHeld(seat, r)
}
wants := make([]parts.Want, 0, len(r.Parts)) wants := make([]parts.Want, 0, len(r.Parts))
for _, p := range r.Parts { for _, p := range r.Parts {
wants = append(wants, parts.Want{Digest: p.Digest, Size: p.Size}) wants = append(wants, parts.Want{Digest: p.Digest, Size: p.Size})
} }
got, err := h.receiver().Open(r.Transfer, caller, wants, Opening{PublishRequest: r, Seat: seat}) var got parts.Received
err := h.timed("the opening of the transfer on the bus (objects, transfer state, idle key)", func() (err error) {
got, err = h.receiver().Open(r.Transfer, caller, wants, Opening{PublishRequest: r, Seat: seat})
return err
})
if err != nil { if err != nil {
return answerOf(got), fmt.Errorf("publish: %w", err) return answerOf(got), fmt.Errorf("publish: %w", err)
} }
return answerOf(got), nil return answerOf(got), nil
} }
// tagHeld is a publish with a version, a digest and no parts (design 47 G A-417, A-73, A-50): what the store holds of that
// digest under the name (the build's untagged publish, the artifact store's blob) is checked against the builder's reservation
// and tagged with the version, and published is said; when the store holds none of it, the publish is refused as not stored,
// naming the digest. Nothing travels.
func (h *Holder) tagHeld(seat string, r PublishRequest) (PublishAnswer, error) {
var held bool
err := h.timed("the store's read of the digest it holds (the program's registry)", func() (err error) {
held, err = h.Store.Holds(r)
return err
})
if err != nil {
return PublishAnswer{}, fmt.Errorf("publish: the store cannot say whether it holds %s of %s: %w", r.Digest, r.Name, err)
}
refuse := func(reason string) (PublishAnswer, error) {
h.Finding("publish-refused", r.Name+"@"+r.Version, reason, true)
if err := h.Say(seat, EventRefused, h.eventOf(seat, r, "", reason)); err != nil {
return PublishAnswer{}, fmt.Errorf("publish: %s, and the refused is not said: %w", reason, err)
}
return PublishAnswer{Transfer: r.Transfer, Ended: true}, errors.New("publish: " + reason)
}
if !held {
return refuse(fmt.Sprintf("not stored: the store holds no %s of %s, so version %s names nothing", r.Digest, r.Name, r.Version))
}
ok, why, err := h.Builder.Recorded(r.Name, r.Version, r.Digest)
if err != nil {
return PublishAnswer{}, fmt.Errorf("publish: the reservation could not be read, so nothing is tagged: %w", err)
}
if !ok {
return refuse("the builder did not reserve it: " + why)
}
tag := r
tag.Parts = []Part{{Digest: r.Digest}}
var path string
if err := h.timed("the store's tag of the held digest (the program's registry)", func() (err error) {
path, err = h.Store.Put(tag, map[string][]byte{})
return err
}); err != nil {
return refuse("the store did not tag it: " + err.Error())
}
h.Finding("publish-refused", r.Name+"@"+r.Version, "", false)
return PublishAnswer{Transfer: r.Transfer, Ended: true}, h.timed("the published event on the seat", func() error {
return h.Say(seat, EventPublished, h.eventOf(seat, r, SeatReference(seat, path), ""))
})
}
// PublishPart is `publish-part`: one chunk, from the caller that opened the transfer alone. // PublishPart is `publish-part`: one chunk, from the caller that opened the transfer alone.
func (h *Holder) PublishPart(seat, caller string, r PartRequest) (PublishAnswer, error) { func (h *Holder) PublishPart(seat, caller string, r PartRequest) (PublishAnswer, error) {
got, err := h.receiver().Take(r.Transfer, caller, r.Digest, r.Sequence, r.Last, r.Data, func(t Transfer) error { got, err := h.receiver().Take(r.Transfer, caller, r.Digest, r.Sequence, r.Last, r.Data, func(t Transfer) error {
+2 -2
View File
@@ -1,6 +1,6 @@
#!/bin/sh #!/bin/sh
# The wallpaper (module feh, novox/hq ADR 0208). Owned by the mesh: replaced at every push. The # The wallpaper (module feh, novox/hq ADR 0208). Owned by the mesh: replaced at every delivery. The
# session's start runs it, and so may anything that wants the declared wallpaper back. The image is # session's start runs it, and so may anything that wants the declared wallpaper back. The image is
# the module's own, in ~/.local/share/feh/wallpapers. feh_set changes the wallpaper for a session # the module's own, in ~/.local/share/feh/wallpapers. feh_set changes the wallpaper for a session
# without touching this file. # without touching this file (the second single-module proof, 2026-11-17).
feh --no-fehbg --bg-fill "$HOME/.local/share/feh/wallpapers/default.jpg" feh --no-fehbg --bg-fill "$HOME/.local/share/feh/wallpapers/default.jpg"
+4 -1
View File
@@ -152,7 +152,10 @@ export function seatTools(gitea: GiteaClient, deps: SeatDeps): ToolDefinition[]
verb(GIT_VERBS.branch, "The seat git's branch: a branch's head, and whether a protection rule refuses a direct push and admits only merges.", verb(GIT_VERBS.branch, "The seat git's branch: a branch's head, and whether a protection rule refuses a direct push and admits only merges.",
async (args): Promise<BranchAnswer> => { async (args): Promise<BranchAnswer> => {
const { owner, repo, full } = repositoryOf(args.repository); const { owner, repo, full } = repositoryOf(args.repository);
const branch = String(args.branch ?? ""); // No branch named is the repository's default branch, read from the forge, as the verb file reads it (A-399).
let branch = String(args.branch ?? "");
if (branch === "") branch = String((await gitea.getRepo(owner, repo)).default_branch ?? "");
if (branch === "") throw new Error(`the forge names no default branch of ${full}`);
// The head first: a branch the forge does not hold is its 404, the kind not-found, never an unprotected branch. // The head first: a branch the forge does not hold is its 404, the kind not-found, never an unprotected branch.
const head = await gitea.branchHead(owner, repo, branch); const head = await gitea.branchHead(owner, repo, branch);
return { repository: full, branch, head, protected: pushRefused(await gitea.appliedProtection(owner, repo, branch)) }; return { repository: full, branch, head, protected: pushRefused(await gitea.appliedProtection(owner, repo, branch)) };
@@ -58,6 +58,7 @@ func main() {
// compiles that program and runs it shut off, answers its mark judged by the candidate at the tree's commit, and the // compiles that program and runs it shut off, answers its mark judged by the candidate at the tree's commit, and the
// program reached no key, no login, no name and no socket of the machine. // program reached no key, no login, no name and no socket of the machine.
func TestTheUnitLevelUnderTheWorkspaceRunnerRunsTheTreesProgramShutOff(t *testing.T) { func TestTheUnitLevelUnderTheWorkspaceRunnerRunsTheTreesProgramShutOff(t *testing.T) {
t.Parallel()
needTools(t, "the candidate's program shut off", "go", "git", "bwrap") needTools(t, "the candidate's program shut off", "go", "git", "bwrap")
home := t.TempDir() home := t.TempDir()
key, login := filepath.Join(home, ".ssh", "mesh_ed25519"), filepath.Join(home, "broker") key, login := filepath.Join(home, ".ssh", "mesh_ed25519"), filepath.Join(home, "broker")
@@ -195,6 +195,7 @@ func TestACandidateThatDoesNotCompileFailsTheStepWithTheCompilersFirstErrorAndRu
} }
func TestTheInstalledProgramSaysItJudged(t *testing.T) { // 1898 K1, ADR 0535 §5 func TestTheInstalledProgramSaysItJudged(t *testing.T) { // 1898 K1, ADR 0535 §5
t.Parallel()
m, states, ws, _ := workspaceFor(t, "slow") m, states, ws, _ := workspaceFor(t, "slow")
for _, runner := range []any{nil, RunnerInstalled} { for _, runner := range []any{nil, RunnerInstalled} {
args := map[string]any{"workspace": ws, "level": LevelUnit} args := map[string]any{"workspace": ws, "level": LevelUnit}
@@ -223,6 +224,7 @@ func TestAnUnknownRunnerIsRefused(t *testing.T) { // 1898 K1
} }
func TestTheShutOffProcessRunsTheStepItIsAsked(t *testing.T) { // 1898 K2: the candidate's side of the spec func TestTheShutOffProcessRunsTheStepItIsAsked(t *testing.T) { // 1898 K2: the candidate's side of the spec
t.Parallel()
m, _, ws, calls := workspaceFor(t, "slow") m, _, ws, calls := workspaceFor(t, "slow")
m.Inside = true m.Inside = true
a := m.answer(context.Background(), levelSpec{Step: stepUnit, Workspace: ws, Packages: []string{"./..."}, Bound: UnitBound}) a := m.answer(context.Background(), levelSpec{Step: stepUnit, Workspace: ws, Packages: []string{"./..."}, Bound: UnitBound})
@@ -24,70 +24,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/testbus" "git.novox.be/novox/mesh-sdk/go/testbus"
) )
// nodeEngineTree is the node-engine's repository as the checker places it beside this check (MESH_CHECK_BESIDE, the // copyHostTree copies the files of a source tree into dst, leaving out its .git.
// repository mesh-host), and with it every Go module of the other repositories placed there (mesh-sdk, mesh-contracts,
// mesh-delivery, mesh-networking, the catalogue's trunk) and of the catalogue under check, first, as candidates: the workspace lays those the
// node-engine's composer requires at the zero version, transitively (workspaceOf), so they resolve as the group lays
// them, wherever the composer lives. The test
// builds the node-engine's real composer command from that tree; it does not stand one in. With no beside directory it
// fails, and with the node-engine beside but no composer command in it it fails: a composer test that could not build its
// composer proves nothing. With a beside directory that holds no node-engine at all it skips aloud (NOT JUDGED), since a
// checker from before this release places none for the catalogue.
func nodeEngineTree(t *testing.T) (tree string, modules []string) {
t.Helper()
beside := os.Getenv("MESH_CHECK_BESIDE")
if beside == "" {
t.Fatal("NOT CHECKED: MESH_CHECK_BESIDE is not set, so the node-engine's repository (mesh-host) is not beside this check")
}
tree = filepath.Join(beside, "mesh-host")
if !dirExists(tree) {
// A checker from before novox/hq ADR 0593 places only mesh-sdk beside a catalogue check, so the node-engine's
// repository is absent, not wrong: said aloud, since nothing here could build a composer to judge.
t.Skipf("NOT JUDGED: the node-engine's repository (mesh-host) is not beside this check in MESH_CHECK_BESIDE=%s; the running checker places none for the catalogue, so the composer is judged only once it does", beside)
}
if _, err := os.Stat(filepath.Join(tree, "cmd", "mesh-compose")); err != nil {
t.Fatalf("NOT CHECKED: the node-engine's composer command is not beside this check in MESH_CHECK_BESIDE=%s: %v", beside, err)
}
entries, err := os.ReadDir(beside)
if err != nil {
t.Fatal(err)
}
var roots []string
for _, e := range entries {
if e.Name() == "mesh-host" || !dirExists(filepath.Join(beside, e.Name())) {
continue
}
root, err := filepath.EvalSymlinks(filepath.Join(beside, e.Name()))
if err != nil {
t.Fatal(err)
}
roots = append(roots, root)
}
// The catalogue under check, first: beside lies its trunk (issue 1295), whose modules a head's must win over.
catalogue, err := filepath.Abs(filepath.Join("..", "..", "..", ".."))
if err != nil {
t.Fatal(err)
}
for _, root := range append([]string{catalogue}, roots...) {
filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return nil
}
if d.IsDir() && (d.Name() == ".git" || d.Name() == "testdata" || d.Name() == "node_modules") {
return filepath.SkipDir
}
if !d.IsDir() && d.Name() == "go.mod" {
modules = append(modules, filepath.Dir(p))
}
return nil
})
}
return tree, modules
}
func dirExists(p string) bool { st, err := os.Stat(p); return err == nil && st.IsDir() }
// copyTree copies the files of a source tree into dst, leaving out its .git.
func copyHostTree(t *testing.T, src, dst string) { func copyHostTree(t *testing.T, src, dst string) {
t.Helper() t.Helper()
src, err := filepath.EvalSymlinks(src) src, err := filepath.EvalSymlinks(src)
@@ -130,7 +67,7 @@ func goEnvOf(t *testing.T, name string) string {
} }
// 1622/AC1, AC3 against the real bus (ADR 0593 §7, C9): a compose ask put on the Go build seat's check stream is pulled by // 1622/AC1, AC3 against the real bus (ADR 0593 §7, C9): a compose ask put on the Go build seat's check stream is pulled by
// the module started whole on that bus, which fetches the commit (the node-engine's repository beside the check, its Go module at the root), builds the composer command the ask names there with nothing fetched, fills the // the module started whole on that bus, which fetches the commit (a stand-in composer of this module's testdata, its Go module at the root), builds the composer command the ask names there with nothing fetched, fills the
// request's manifests from the indexer's state and runs the composer; the step key <node>.<ask id> passes with the // request's manifests from the indexer's state and runs the composer; the step key <node>.<ask id> passes with the
// composer's verdict. A second ask of the same commit runs the kept composer: fetched and built once. // composer's verdict. A second ask of the same commit runs the kept composer: fetched and built once.
func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) { func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) {
@@ -157,14 +94,11 @@ func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) {
m.Run = StepRunner(t.TempDir()) m.Run = StepRunner(t.TempDir())
m.GoModCache = goEnvOf(t, "GOMODCACHE") // the module cache this module's own build filled, read and not fetched m.GoModCache = goEnvOf(t, "GOMODCACHE") // the module cache this module's own build filled, read and not fetched
run, gocache := m.Run, goEnvOf(t, "GOCACHE") // this machine's build cache, so the composer's packages compile once run, gocache := m.Run, goEnvOf(t, "GOCACHE") // this machine's build cache, so the composer's packages compile once
hostTree, beside := nodeEngineTree(t) // The composer is a stand-in of this module's own testdata (design 47 G A-474: a module's tests read nothing outside it, and a
workFile := "" // pipeline's step places no other repository beside the tree): a program of the composer's contract that imports nothing.
hostTree := filepath.Join("testdata", "composer")
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran { m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
env = append(env, "GOCACHE="+gocache) return run(ctx, append(env, "GOCACHE="+gocache), name, args...)
if workFile != "" {
env = append(env, "GOWORK="+workFile, "GOFLAGS=-mod=readonly") // a workspace takes no -mod=mod
}
return run(ctx, env, name, args...)
} }
read, digests := catalogHolds(t, `{"module":"demo","version":"1"}`) read, digests := catalogHolds(t, `{"module":"demo","version":"1"}`)
m.ReadEntry = read m.ReadEntry = read
@@ -181,18 +115,6 @@ func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) {
return nil, err return nil, err
} }
copyHostTree(t, hostTree, tree) copyHostTree(t, hostTree, tree)
if len(beside) > 0 { // the group's other repositories, laid beside the node-engine's tree as one workspace
work, err := workspaceOf(tree, beside)
if err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(m.Work, ws, "go.work"), []byte(work), 0o600); err != nil {
return nil, err
}
workFile = filepath.Join(m.Work, ws, "go.work")
// Nothing is downloaded here: this level has no network. The check's own fetch downloaded this workspace's
// graph (downloadBeside), and a module it lacks fails the build naming it.
}
return map[string]any{"workspace": ws}, nil return map[string]any{"workspace": ws}, nil
} }
} }
@@ -203,7 +125,7 @@ func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) {
ask := func() sdkstep.Step { ask := func() sdkstep.Step {
a := composeAskOf(t, digests[0]) a := composeAskOf(t, digests[0])
a.Compose.Fetch.Repository, a.Compose.Dir, a.Compose.Package = "novox/mesh-host", catalogclient.RootPath, "github.com/novox/mesh-host/cmd/mesh-compose" a.Compose.Fetch.Repository, a.Compose.Dir, a.Compose.Package = "novox/mesh-catalog", catalogclient.RootPath, "standin/cmd/mesh-compose"
raw, err := sdkstep.EncodeStepAsk(a) raw, err := sdkstep.EncodeStepAsk(a)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -260,7 +260,7 @@ func (m *Module) digestHere(ctx context.Context, workspace, level string, names
namesHeader = []string{"node-names " + strings.Join(sortedNames, ",")} namesHeader = []string{"node-names " + strings.Join(sortedNames, ",")}
} }
dirs, err := m.goModDirs(ctx, tree) dirs, err := m.testDirs(ctx, tree)
if err != nil { if err != nil {
return DigestResult{}, err return DigestResult{}, err
} }
@@ -273,6 +273,7 @@ func TestAPackageWithTheReadsOutsideMarkerIsKeptOnlyByWhatItLists(t *testing.T)
} }
edit(t, tree, "docs/one.md", "one\n") edit(t, tree, "docs/one.md", "one\n")
edit(t, tree, "other.md", "other\n") edit(t, tree, "other.md", "other\n")
// Each marker is judged in a fixture of its own, side by side: the digests are go commands, and the table is the test's longest part.
for name, marker := range map[string]string{ for name, marker := range map[string]string{
"an empty marker": "# nothing listed\n\n", "an empty marker": "# nothing listed\n\n",
"a sentence": "the tests read ../docs\n", "a sentence": "the tests read ../docs\n",
@@ -282,19 +283,44 @@ func TestAPackageWithTheReadsOutsideMarkerIsKeptOnlyByWhatItLists(t *testing.T)
"a path that is not clean": "docs/./one.md\n", "a path that is not clean": "docs/./one.md\n",
"a good line and a bad line": "docs\nnope/*.md\n", "a good line and a bad line": "docs\nnope/*.md\n",
} { } {
edit(t, tree, "c/"+ReadsOutsideMarker, marker) t.Run(name, func(t *testing.T) {
first, second := digests(t, m, "unit"), digests(t, m, "unit") t.Parallel()
if len(first.Packages[c]) != 64 || first.Packages[c] == second.Packages[c] || first.Packages[c] == plain.Packages[c] { m, tree := digestFixture(t)
t.Errorf("%s: c's digest is stable: %s, %s, before %s", name, first.Packages[c], second.Packages[c], plain.Packages[c]) plain := digests(t, m, "unit")
} if err := os.MkdirAll(filepath.Join(tree, "docs"), 0o755); err != nil {
if strings.Join(first.ReadsOutside, ",") != c { t.Fatal(err)
t.Errorf("%s: the answer does not name the package that reads outside: %v", name, first.ReadsOutside) }
} edit(t, tree, "docs/one.md", "one\n")
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] { edit(t, tree, "other.md", "other\n")
t.Errorf("%s: a has no marker and its digest moved: %s, %s, before %s", name, first.Packages[a], second.Packages[a], plain.Packages[a]) edit(t, tree, "c/"+ReadsOutsideMarker, marker)
} first, second := digests(t, m, "unit"), digests(t, m, "unit")
if len(first.Packages[c]) != 64 || first.Packages[c] == second.Packages[c] || first.Packages[c] == plain.Packages[c] {
t.Errorf("%s: c's digest is stable: %s, %s, before %s", name, first.Packages[c], second.Packages[c], plain.Packages[c])
}
if strings.Join(first.ReadsOutside, ",") != c {
t.Errorf("%s: the answer does not name the package that reads outside: %v", name, first.ReadsOutside)
}
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] {
t.Errorf("%s: a has no marker and its digest moved: %s, %s, before %s", name, first.Packages[a], second.Packages[a], plain.Packages[a])
}
})
} }
}
// The marker that lists what the package's tests read, and the marker of what the package needs (issue 890/AC3): in a fixture of its
// own, so it runs beside the table above.
func TestAListedReadsOutsideMarkerKeepsTheDigestOfOnlyWhatItLists(t *testing.T) {
t.Parallel()
m, tree := digestFixture(t)
plain := digests(t, m, "unit")
c, a := "fixture/digest/c", "fixture/digest/a"
if err := os.MkdirAll(filepath.Join(tree, "docs"), 0o755); err != nil {
t.Fatal(err)
}
edit(t, tree, "docs/one.md", "one\n")
edit(t, tree, "other.md", "other\n")
// A list: a glob, a directory read whole, and a path whose absence is read. // A list: a glob, a directory read whole, and a path whose absence is read.
edit(t, tree, "c/"+ReadsOutsideMarker, "# what c's tests read\ndocs/*.md\ndocs\n?gone\n") edit(t, tree, "c/"+ReadsOutsideMarker, "# what c's tests read\ndocs/*.md\ndocs\n?gone\n")
before := digests(t, m, "unit") before := digests(t, m, "unit")
@@ -34,7 +34,7 @@ func TestTheGuardSaysToolchainReportedOnAChangeWithTheProgramDigest(t *testing.T
t.Fatal(err) t.Fatal(err)
} }
if len(got) != 1 || got[0].subject != "mesh.seat.node-build-go.event.toolchain-reported.ace" || got[0].body["node"] != "ace" || got[0].body["language"] != "go" || if len(got) != 1 || got[0].subject != "mesh.seat.node-build-go.event.toolchain-reported.ace" || got[0].body["node"] != "ace" || got[0].body["language"] != "go" ||
got[0].body["version"] != "go1.27.1" || got[0].body["recipe"] != m.Recipe.Version() || len(got[0].body["digest"]) != len("sha256:")+64 { got[0].body["version"] != "go1.27" || got[0].body["recipe"] != m.Recipe.Version() || len(got[0].body["digest"]) != len("sha256:")+64 {
t.Fatalf("events = %+v, want one toolchain-reported for ace, go with the program's digest", got) t.Fatalf("events = %+v, want one toolchain-reported for ace, go with the program's digest", got)
} }
if tc, _ := states.toolchain("ace.go"); tc.Digest != got[0].body["digest"] { if tc, _ := states.toolchain("ace.go"); tc.Digest != got[0].body["digest"] {
@@ -51,7 +51,7 @@ func TestTheGuardSaysToolchainReportedOnAChangeWithTheProgramDigest(t *testing.T
os.WriteFile(prog, []byte("another go program"), 0o755) os.WriteFile(prog, []byte("another go program"), 0o755)
m.Probe = func(context.Context) (string, error) { return "go1.27.2", nil } m.Probe = func(context.Context) (string, error) { return "go1.27.2", nil }
m.Check(context.Background()) m.Check(context.Background())
if len(got) != 2 || got[1].body["version"] != "go1.27.2" || got[1].body["digest"] == got[0].body["digest"] { if len(got) != 2 || got[1].body["version"] != "go1.27" || got[1].body["digest"] == got[0].body["digest"] {
t.Errorf("a new version said %+v, want a second event with another digest", got) t.Errorf("a new version said %+v, want a second event with another digest", got)
} }
} }
@@ -20,6 +20,8 @@ import (
"sync" "sync"
"time" "time"
"golang.org/x/mod/modfile"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep" sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
dforgekey "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/forgekey" dforgekey "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/forgekey"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/steprun" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/steprun"
@@ -106,6 +108,8 @@ func (m *Module) fetch(ctx context.Context, args map[string]any) (res any, err e
if err != nil { if err != nil {
return nil, err return nil, err
} }
// A fetch with no dir (the checker's check of a pull request, until G removes the checker; A-447e) fetches the tree
// whole: every go.mod of it.
siblings, err := siblingsArg(args["beside"]) siblings, err := siblingsArg(args["beside"])
if err != nil { if err != nil {
return nil, err return nil, err
@@ -165,6 +169,20 @@ func (m *Module) fetchInto(ctx context.Context, req sdkstep.FetchRequest, id str
if err != nil { if err != nil {
return nil, err return nil, err
} }
if dir := cleanDir(req.Dir); dir != "" {
// A fetch asked with a module's directory is of that Go module and what its go.mod replaces with a local path,
// never another Go module of the tree (design 47 G, A-447). Said beside the tree, so every later step lists the
// same modules.
if dirs, err = scopedDirs(tree, dir, dirs); err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(root, scopeMark), []byte(strings.Join(dirs, "\n")), 0o600); err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(root, moduleDirMark), []byte(dir), 0o600); err != nil {
return nil, err
}
}
res.Modules = dirs res.Modules = dirs
if base != "" && res.Merge == "" { if base != "" && res.Merge == "" {
return res, nil // a conflict: the tree is at the trunk, nothing of the head is built or tested return res, nil // a conflict: the tree is at the trunk, nothing of the head is built or tested
@@ -313,7 +331,13 @@ func (m *Module) shutOffLoad(ctx context.Context, env []string, dir, d string, j
if _, err := os.Stat(filepath.Join(dir, "vendor", "modules.txt")); err == nil { if _, err := os.Stat(filepath.Join(dir, "vendor", "modules.txt")); err == nil {
return nil // a directory that vendors is read from its vendor directory, never through the module cache return nil // a directory that vendors is read from its vendor directory, never through the module cache
} }
missing, err := ownFilesCover(dir, loaded) // The files the steps build with: a directory with a temporary modfile (an asked version, a pin) is built with it and
// the sums beside it, which hold the requirements of the versions taken (takeRequirementsOf).
files := dir
if modfile, ok := m.pinModfile(dir); ok {
files = filepath.Dir(modfile)
}
missing, err := ownFilesCover(files, loaded)
if err != nil { if err != nil {
return fmt.Errorf("the go.mod and go.sum of %s: %w", d, err) return fmt.Errorf("the go.mod and go.sum of %s: %w", d, err)
} }
@@ -324,7 +348,7 @@ func (m *Module) shutOffLoad(ctx context.Context, env []string, dir, d string, j
return nil return nil
} }
// ownFilesCover is what the go.mod and go.sum in dir lack for the modules its packages load, each as the line it lacks: // ownFilesCover is what the go.mod and go.sum in dir (a directory's own, or those of its temporary modfile) lack for the modules its packages load, each as the line it lacks:
// a mesh module (of the forge) required, the line it lacks naming the zero version (ADR 0573 §2; one named at a commit // a mesh module (of the forge) required, the line it lacks naming the zero version (ADR 0573 §2; one named at a commit
// is built against the asked version all the same, §4, and needs no lookup), every other required at the version the // is built against the asked version all the same, §4, and needs no lookup), every other required at the version the
// load selected, with its two go.sum lines (the module's and its go.mod's) for the version Go reads it at. A module // load selected, with its two go.sum lines (the module's and its go.mod's) for the version Go reads it at. A module
@@ -434,7 +458,11 @@ func (m *Module) downloadAndVerify(ctx context.Context, dir, d string) error {
} }
// goModDirs are the directories of the tree that hold a go.mod, relative to it, in order. // goModDirs are the directories of the tree that hold a go.mod, relative to it, in order.
// When the fetch was asked with a module's directory (scopeMark), the steps after it see only the directories it chose.
func (m *Module) goModDirs(ctx context.Context, tree string) ([]string, error) { func (m *Module) goModDirs(ctx context.Context, tree string) ([]string, error) {
if b, err := os.ReadFile(filepath.Join(filepath.Dir(tree), scopeMark)); err == nil {
return strings.Fields(string(b)), nil
}
r := m.Run(ctx, nil, "git", "-C", tree, "ls-files", "--", "go.mod", "*/go.mod") r := m.Run(ctx, nil, "git", "-C", tree, "ls-files", "--", "go.mod", "*/go.mod")
if r.Status != 0 || r.Err != "" { if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("git ls-files: %s", dforgekey.Said(r.Err+"\n"+r.Stderr)) return nil, fmt.Errorf("git ls-files: %s", dforgekey.Said(r.Err+"\n"+r.Stderr))
@@ -454,6 +482,27 @@ func (m *Module) goModDirs(ctx context.Context, tree string) ([]string, error) {
return dirs, nil return dirs, nil
} }
// testDirs are the directories the unit and integration steps and the digest take: the chosen go.mod directories at or
// under the module's own directory. A directory the module's go.mod replaces from outside its own stays for download and
// resolution alone; its tests are another module's (design 47 G, A-447c). A workspace fetched with no dir keeps every go.mod.
func (m *Module) testDirs(ctx context.Context, tree string) ([]string, error) {
all, err := m.goModDirs(ctx, tree)
if err != nil {
return nil, err
}
own, ok := moduleDirOf(tree)
if !ok || own == "." || own == "" {
return all, nil
}
out := []string{}
for _, d := range all {
if d == own || strings.HasPrefix(d, own+"/") {
out = append(out, d)
}
}
return out, nil
}
// inTestdata says a path lies under a directory named testdata, whose go.mod files are fixtures Go ignores, not modules // inTestdata says a path lies under a directory named testdata, whose go.mod files are fixtures Go ignores, not modules
// of the tree (issue 1810). // of the tree (issue 1810).
func inTestdata(path string) bool { func inTestdata(path string) bool {
@@ -608,3 +657,85 @@ func narrowed(tree string) bool {
_, err := os.Stat(filepath.Join(filepath.Dir(tree), narrowedMark)) _, err := os.Stat(filepath.Join(filepath.Dir(tree), narrowedMark))
return err == nil return err == nil
} }
// scopeMark is the file beside a workspace's tree naming the Go module directories a fetch asked with a module's
// directory chose, one to a line (empty: none).
const scopeMark = "gomodules"
// moduleDirMark is the file beside a workspace's tree naming the module directory the fetch was asked with: the one
// module.json a step compares generated files for (design 47 G, A-447c).
const moduleDirMark = "moduledir"
// moduleDirOf is the module directory the workspace of tree was fetched for, ok false for a workspace with none.
func moduleDirOf(tree string) (string, bool) {
b, err := os.ReadFile(filepath.Join(filepath.Dir(tree), moduleDirMark))
if err != nil {
return "", false
}
return string(b), true
}
// cleanDir is a request's module directory in the form goModDirs lists, "" for none.
func cleanDir(dir string) string {
if strings.TrimSpace(dir) == "" {
return ""
}
return strings.Trim(filepath.ToSlash(filepath.Clean(dir)), "/")
}
// scopedDirs are the go.mod directories of all that a module directory dir chooses: dir itself and each directory its
// go.mod replaces with a local path, followed through their own replaces (design 47 G, A-447), in order. A directory
// the tree holds no go.mod for is left out. A directory holding no go.mod chooses every go.mod under it, and one that
// chooses no Go module at all is an error naming it (A-447a).
func scopedDirs(tree, dir string, all []string) ([]string, error) {
have := map[string]bool{}
for _, d := range all {
have[d] = true
}
chosen := map[string]bool{}
todo := []string{dir}
if !have[dir] {
// A directory with no go.mod of its own (a module whose Go code lies in go/, a contracts module with a go.mod per
// seat) is every go.mod under it; the tree is already narrowed to the module's paths (A-447a).
todo = nil
for _, d := range all {
if dir == "." || strings.HasPrefix(d, dir+"/") {
todo = append(todo, d)
}
}
}
for len(todo) > 0 {
d := todo[0]
todo = todo[1:]
if chosen[d] || !have[d] {
continue
}
chosen[d] = true
file := filepath.Join(tree, filepath.FromSlash(d), "go.mod")
body, err := os.ReadFile(file)
if err != nil {
return nil, err
}
f, err := modfile.Parse(file, body, nil) // not ParseLax: that one ignores the replaces
if err != nil {
return nil, fmt.Errorf("%s: %w", file, err)
}
for _, r := range f.Replace {
if !modfile.IsDirectoryPath(r.New.Path) {
continue
}
target := filepath.ToSlash(filepath.Clean(filepath.Join(filepath.FromSlash(d), filepath.FromSlash(r.New.Path))))
todo = append(todo, target)
}
}
out := []string{}
for _, d := range all {
if chosen[d] {
out = append(out, d)
}
}
if len(out) == 0 {
return nil, fmt.Errorf("the module directory %q holds no Go module (no go.mod in it or under it)", dir)
}
return out, nil
}
@@ -61,6 +61,13 @@ func (f *fakeStep) run(ctx context.Context, env []string, name string, args ...s
os.MkdirAll(filepath.Dir(filepath.Join(args[i+1], name)), 0o755) os.MkdirAll(filepath.Dir(filepath.Join(args[i+1], name)), 0o755)
os.WriteFile(filepath.Join(args[i+1], name), []byte(text), 0o644) os.WriteFile(filepath.Join(args[i+1], name), []byte(text), 0o644)
} }
// Every go.mod the fake lists exists, since a fetch asked with a directory reads its go.mod (A-447).
for _, l := range strings.Fields(f.gomods) {
if _, err := os.Stat(filepath.Join(args[i+1], l)); err != nil && f.files[l] == "" {
os.MkdirAll(filepath.Dir(filepath.Join(args[i+1], l)), 0o755)
os.WriteFile(filepath.Join(args[i+1], l), []byte("module fake\n"), 0o644)
}
}
for _, v := range f.vendors { for _, v := range f.vendors {
os.MkdirAll(filepath.Join(args[i+1], v, "vendor"), 0o755) os.MkdirAll(filepath.Join(args[i+1], v, "vendor"), 0o755)
os.WriteFile(filepath.Join(args[i+1], v, "vendor", "modules.txt"), nil, 0o644) os.WriteFile(filepath.Join(args[i+1], v, "vendor", "modules.txt"), nil, 0o644)
@@ -91,7 +98,24 @@ func fetchModule(t *testing.T, f *fakeStep) (*Module, *keptStates) {
return m, states return m, states
} }
// fetchTool is the fetch tool as a build module is asked, always with its module's directory (A-447b): the root of the
// tree unless the args name one. rawFetchTool is the tool as it is.
func fetchTool(t *testing.T, m *Module) func(map[string]any) (any, error) { func fetchTool(t *testing.T, m *Module) func(map[string]any) (any, error) {
t.Helper()
raw := rawFetchTool(t, m)
return func(args map[string]any) (any, error) {
if _, ok := args["dir"]; !ok {
named := map[string]any{"dir": "."}
for k, v := range args {
named[k] = v
}
args = named
}
return raw(args)
}
}
func rawFetchTool(t *testing.T, m *Module) func(map[string]any) (any, error) {
t.Helper() t.Helper()
for _, tool := range m.Tools(m.StepTools()) { for _, tool := range m.Tools(m.StepTools()) {
if tool.Name == "build_go_fetch" { if tool.Name == "build_go_fetch" {
@@ -107,7 +131,9 @@ var fetchArgs = map[string]any{"repository": "novox/mesh-sdk", "commit": commitS
// 1485/AC1: git is asked through FetchCommit with the ssh remote, and go mod download and verify run in each go.mod // 1485/AC1: git is asked through FetchCommit with the ssh remote, and go mod download and verify run in each go.mod
// directory (not in one that vendors) with GOTOOLCHAIN=local, GOFLAGS=-mod=mod and the module's cache paths. // directory (not in one that vendors) with GOTOOLCHAIN=local, GOFLAGS=-mod=mod and the module's cache paths.
func TestFetchAsksGitForTheSshRemoteAndDownloadsEachModule(t *testing.T) { func TestFetchAsksGitForTheSshRemoteAndDownloadsEachModule(t *testing.T) {
f := &fakeStep{gomods: "go.mod\nsub/dir/go.mod\nvend/go.mod\n", vendors: []string{"vend"}} // The root go.mod replaces the other two by directory: a fetch asked with a module's directory chooses what it replaces.
f := &fakeStep{gomods: "go.mod\nsub/dir/go.mod\nvend/go.mod\n", vendors: []string{"vend"},
files: map[string]string{"go.mod": "module fake\nreplace a => ./sub/dir\nreplace b => ./vend\n"}}
m, states := fetchModule(t, f) m, states := fetchModule(t, f)
if _, err := fetchTool(t, m)(fetchArgs); err != nil { if _, err := fetchTool(t, m)(fetchArgs); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -270,7 +296,8 @@ func TestAVerifyThatFailsEndsTheStepNamingTheDirectoryAndTheLine(t *testing.T) {
return forgekey.Ran{} return forgekey.Ran{}
}} }}
m, states := fetchModule(t, f) m, states := fetchModule(t, f)
if _, err := fetchTool(t, m)(fetchArgs); err != nil { // Asked for the module of tools, as a build is for one module (A-447b).
if _, err := fetchTool(t, m)(map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA, "dir": "tools"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
st := states.waitEnd(t) st := states.waitEnd(t)
@@ -2,9 +2,13 @@ package main
import ( import (
"context" "context"
"os"
"path/filepath"
"reflect" "reflect"
"strings"
"testing" "testing"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey" "git.novox.be/novox/mesh-sdk/go/forgekey"
) )
@@ -25,3 +29,123 @@ func TestGoModDirsLeaveOutTheFixturesUnderTestdata(t *testing.T) {
t.Fatalf("go.mod directories = %v, want %v", got, want) t.Fatalf("go.mod directories = %v, want %v", got, want)
} }
} }
// A-447: a fetch asked with a module's directory chooses that Go module and the directories its go.mod replaces with a
// local path, through their own replaces; the later steps list the same. The takein module of the indexer is no module
// of the assigner's build (live, 2026-11-16).
func TestAFetchAskedWithADirectoryChoosesItsModuleAndWhatItReplacesAlone(t *testing.T) {
all := []string{".", "modules/assigner", "modules/indexer", "modules/indexer/takein", "modules/lib", "modules/lib/deep", "modules/other", "rootless/x/go", "seats/a", "seats/b"}
mods := map[string]string{
"modules/assigner": "module a\nreplace x => ../lib\nreplace (\n\ty => ../../outside\n\tz v1.0.0 => z v1.1.0\n)\n",
"modules/indexer": "module i\nreplace t => ./takein\n",
"modules/indexer/takein": "module t\n",
"modules/lib": "module l\nreplace d => ./deep\n",
"modules/lib/deep": "module d\n",
"modules/other": "module o\n",
".": "module r\n",
"seats/a": "module sa\nreplace q => ../q\n",
"seats/b": "module sb\n",
"rootless/x/go": "module rx\n",
}
for _, c := range []struct {
dir string
want []string
}{
{"modules/assigner", []string{"modules/assigner", "modules/lib", "modules/lib/deep"}},
{"modules/indexer", []string{"modules/indexer", "modules/indexer/takein"}},
{"modules/other/", []string{"modules/other"}},
{"modules/lib/deep/", []string{"modules/lib/deep"}},
{"seats", []string{"seats/a", "seats/b"}}, // a go.mod per seat, none in seats itself
{".", []string{"."}}, // the root holds a go.mod
{"rootless/x", []string{"rootless/x/go"}}, // Go code lies in go/
{"modules/none", nil}, // chooses no Go module: an error naming it
} {
tree := t.TempDir()
for d, body := range mods {
if err := os.MkdirAll(filepath.Join(tree, d), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(tree, d, "go.mod"), []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
got, err := scopedDirs(tree, cleanDir(c.dir), all)
if c.want == nil {
if err == nil || !strings.Contains(err.Error(), c.dir) {
t.Errorf("dir %q chose %v, %v; want an error naming it", c.dir, got, err)
}
continue
}
if err != nil || !reflect.DeepEqual(got, c.want) {
t.Errorf("dir %q chooses %v, %v; want %v", c.dir, got, err, c.want)
}
}
// What the steps after the fetch list is what the fetch chose, beside the tree.
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, scopeMark), []byte("modules/assigner\nmodules/lib"), 0o600); err != nil {
t.Fatal(err)
}
m := &Module{}
got, err := m.goModDirs(context.Background(), filepath.Join(root, "tree"))
if err != nil || !reflect.DeepEqual(got, []string{"modules/assigner", "modules/lib"}) {
t.Errorf("the later steps list %v, %v", got, err)
}
if cleanDir("") != "" || cleanDir(".") != "." {
t.Errorf("cleanDir of nothing = %q, of . = %q", cleanDir(""), cleanDir("."))
}
}
// A-447e: a fetch with no dir (the checker's check of a pull request, until G removes the checker) fetches the tree whole:
// it chooses every go.mod of it.
func TestAFetchWithNoDirectoryChoosesEveryGoModOfTheTree(t *testing.T) {
f := &fakeStep{gomods: "go.mod\nsub/go.mod\n"}
m, states := fetchModule(t, f)
if _, err := rawFetchTool(t, m)(map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA}); err != nil {
t.Fatal(err)
}
st := states.waitEnd(t)
if st.State != sdkstep.StepPassed || !strings.Contains(string(st.Result), `"modules":[".","sub"]`) {
t.Fatalf("a fetch with no dir ended %s: %q %s", st.State, st.Reason, st.Result)
}
}
// A-447c: the unit level of a module whose go.mod replaces ../lib tests the module alone; lib is downloaded and resolved
// for it, and its tests are another module's.
func TestTheUnitLevelTestsOnlyTheModulesOwnDirectoriesNotWhatItReplaces(t *testing.T) {
f := &fakeStep{}
m, _ := fetchModule(t, f)
root := filepath.Join(m.Work, "ws1")
tree := filepath.Join(root, "tree")
for d, body := range map[string]string{"mod": "module example.org/mod\nreplace example.org/lib => ../lib\n", "lib": "module example.org/lib\n"} {
if err := os.MkdirAll(filepath.Join(tree, d), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(tree, d, "go.mod"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
for name, body := range map[string]string{scopeMark: "lib\nmod", moduleDirMark: "mod"} {
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
dirs, err := m.testDirs(context.Background(), tree)
if err != nil || !reflect.DeepEqual(dirs, []string{"mod"}) {
t.Fatalf("the test directories = %v, %v; want mod alone", dirs, err)
}
if _, err := m.runLevel(context.Background(), tree, levelOpts{workspace: "ws1", level: LevelUnit}); err != nil {
t.Fatal(err)
}
for _, l := range f.lines() {
if strings.Contains(l, "/tree/lib test") {
t.Errorf("the module's unit level tested lib: %s", l)
}
}
// No dir asked: every go.mod of the tree stays.
if err := os.Remove(filepath.Join(root, moduleDirMark)); err != nil {
t.Fatal(err)
}
if dirs, _ = m.testDirs(context.Background(), tree); len(dirs) != 2 {
t.Errorf("a workspace with no module directory tests %v, want every go.mod", dirs)
}
}
@@ -149,8 +149,8 @@ func TestAnyInstalledGoIsHealthyNamedAndReported(t *testing.T) {
t.Errorf("health = %+v, want healthy naming %s", a, installed) t.Errorf("health = %+v, want healthy naming %s", a, installed)
} }
tc, ok := states.toolchain("ace.go") tc, ok := states.toolchain("ace.go")
if !ok || tc.Version != installed || tc.Recipe != m.Recipe.Version() { if !ok || tc.Version != sdkstep.LanguageVersion(installed) || tc.Recipe != m.Recipe.Version() {
t.Errorf("ace.go = %+v, %v; want version %s and recipe %s", tc, ok, installed, m.Recipe.Version()) t.Errorf("ace.go = %+v, %v; want version %s and recipe %s", tc, ok, sdkstep.LanguageVersion(installed), m.Recipe.Version())
} }
ran := false ran := false
run := func(context.Context, map[string]any) (any, error) { ran = true; return "done", nil } run := func(context.Context, map[string]any) (any, error) { ran = true; return "done", nil }
@@ -58,7 +58,7 @@ func (m *Module) integrationHere(ctx context.Context, workspace, tree string, pa
defer os.RemoveAll(run) defer os.RemoveAll(run)
began := m.clock() began := m.clock()
dirs, err := m.goModDirs(ctx, tree) dirs, err := m.testDirs(ctx, tree)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -186,6 +186,7 @@ func TestABuildRunsShutOffAndOnlyATrunkBuildWritesTheCaches(t *testing.T) {
// 1309-S3: the digest's go list runs shut off. // 1309-S3: the digest's go list runs shut off.
func TestTheDigestListsPackagesShutOff(t *testing.T) { func TestTheDigestListsPackagesShutOff(t *testing.T) {
t.Parallel()
m, _ := digestFixture(t) m, _ := digestFixture(t)
var seen []shutOff var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen) m.Run = throughBubblewrap(t, m.Run, &seen)
@@ -15,6 +15,7 @@ import (
// resolves no name, reaches no socket of the machine (a TCP port on its loopback, a unix socket) and leaves the shared // resolves no name, reaches no socket of the machine (a TCP port on its loopback, a unix socket) and leaves the shared
// caches unchanged. The probe is testdata/repos/probe. // caches unchanged. The probe is testdata/repos/probe.
func TestARunStepReachesNoKeyLoginNameOrSocketAndLeavesTheCachesUnchanged(t *testing.T) { func TestARunStepReachesNoKeyLoginNameOrSocketAndLeavesTheCachesUnchanged(t *testing.T) {
t.Parallel()
needTools(t, "the shut-off unit level", "go", "git", "bwrap") needTools(t, "the shut-off unit level", "go", "git", "bwrap")
tree := repository(t, "probe") tree := repository(t, "probe")
work, err := os.MkdirTemp("", "w") work, err := os.MkdirTemp("", "w")
@@ -0,0 +1,48 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// A directory built with a temporary modfile (an asked version, a pin) is judged by the modfile and the sums beside it, the
// files its steps build with, and not by its own go.mod and go.sum: the requirements of an asked version of a mesh package
// are in the modfile (takeRequirementsOf), and the directory's own files lack them.
func TestADirectoryWithAModfileIsJudgedByTheModfileAndItsSums(t *testing.T) {
const nats = "github.com/nats-io/nats.go"
for _, c := range []struct {
name, modfile, sums string
lacks bool
}{
{name: "the modfile names it with its sums", modfile: "require " + nats + " v1.54.0\n", sums: nats + " v1.54.0 h1:a=\n" + nats + " v1.54.0/go.mod h1:b=\n"},
{name: "the modfile lacks it", modfile: "", sums: "", lacks: true},
{name: "the modfile names it and the sums lack it", modfile: "require " + nats + " v1.54.0\n", sums: nats + " v1.54.0 h1:a=\n", lacks: true},
} {
t.Run(c.name, func(t *testing.T) {
m := &Module{Work: t.TempDir()}
dir := filepath.Join(m.Work, "ws", "tree", "mod")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
head := "module fixture/mod\n\ngo 1.26\n\n"
// The directory's own files lack the requirement, whatever the modfile holds.
os.WriteFile(filepath.Join(dir, "go.mod"), []byte(head), 0o644)
pins := m.pinsDir(dir)
os.MkdirAll(pins, 0o755)
os.WriteFile(filepath.Join(pins, "go.mod"), []byte(head+c.modfile), 0o644)
os.WriteFile(filepath.Join(pins, "go.sum"), []byte(c.sums), 0o644)
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
return forgekey.Ran{Stdout: loadedModuleMark + nats + " v1.54.0\n"}
}
err := m.shutOffLoad(context.Background(), nil, dir, "mod", true)
if c.lacks != (err != nil) || (err != nil && !strings.Contains(err.Error(), "lacking")) {
t.Errorf("shutOffLoad = %v, want a refusal naming what is lacking: %v", err, c.lacks)
}
})
}
}
@@ -87,8 +87,9 @@ func main() {
go func() { go func() {
<-stdio.Served() <-stdio.Served()
if err := findings.Start(ctx); err != nil { if err := findings.Start(ctx); err != nil {
logf("the finding writer: %v", err) // Said, and the module goes on serving: a finding writer that cannot hear its asks keeps its findings
os.Exit(1) // in its state alone, as the indexer does (lead answer A-394); its builds and steps never wait on it.
logf("the finding writer cannot start, its findings are kept in the state alone: %v", err)
} }
}() }()
} }
@@ -158,6 +158,7 @@ func sleepABit() { time.Sleep(30 * time.Millisecond) }
// The workspace is self-contained after the fetch: no alternates file, and git finds the commit with the mirror gone, // The workspace is self-contained after the fetch: no alternates file, and git finds the commit with the mirror gone,
// as a step shut off with the workspace alone needs. Real git, a local repository as the forge. // as a step shut off with the workspace alone needs. Real git, a local repository as the forge.
func TestTheWorkspaceIsSelfContainedAfterAFetchThroughTheMirror(t *testing.T) { func TestTheWorkspaceIsSelfContainedAfterAFetchThroughTheMirror(t *testing.T) {
t.Parallel()
if _, err := exec.LookPath("git"); err != nil { if _, err := exec.LookPath("git"); err != nil {
t.Skip("no git") t.Skip("no git")
} }
@@ -176,6 +177,10 @@ func TestTheWorkspaceIsSelfContainedAfterAFetchThroughTheMirror(t *testing.T) {
return strings.TrimSpace(string(out)) return strings.TrimSpace(string(out))
} }
git(src, "init", "-q") git(src, "init", "-q")
// A fetch is for one module (A-447b): the repository holds a module at its root.
if err := os.WriteFile(filepath.Join(src, "go.mod"), []byte("module example.org/x\n\ngo 1.26\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(src, "f.txt"), []byte("one\n"), 0o644); err != nil { if err := os.WriteFile(filepath.Join(src, "f.txt"), []byte("one\n"), 0o644); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -93,8 +93,8 @@ func TestStartWritesTheToolchainAndHealthIsHealthy(t *testing.T) {
break break
} }
} }
if tc.Version != "go1.27.1" || tc.Recipe != m.Recipe.Version() || tc.Language != "go" { if tc.Version != "go1.27" || tc.Recipe != m.Recipe.Version() || tc.Language != "go" {
t.Fatalf("ace.go = %+v, want go1.27.1 and recipe %s", tc, m.Recipe.Version()) t.Fatalf("ace.go = %+v, want go1.27 and recipe %s", tc, m.Recipe.Version())
} }
if a := m.Health(ctx); a.State != health.StateHealthy || !a.Healthy { if a := m.Health(ctx); a.State != health.StateHealthy || !a.Healthy {
t.Errorf("health = %+v, want healthy", a) t.Errorf("health = %+v, want healthy", a)
@@ -45,7 +45,7 @@ func TestAFetchWithNamedPathsNarrowsTheWorkspace(t *testing.T) {
if strings.Join(res.Modules, ",") != "lib" { if strings.Join(res.Modules, ",") != "lib" {
t.Errorf("the fetch answered the modules %v, want lib alone", res.Modules) t.Errorf("the fetch answered the modules %v, want lib alone", res.Modules)
} }
_, st = fetched(t, m, states, map[string]any{"repository": "novox/narrow", "commit": head, "paths": []any{"cmd"}}) _, st = fetched(t, m, states, map[string]any{"repository": "novox/narrow", "commit": head, "dir": "", "paths": []any{"cmd"}})
if st.State != sdkstep.StepFailed || !strings.Contains(st.Reason, "no module directory") { if st.State != sdkstep.StepFailed || !strings.Contains(st.Reason, "no module directory") {
t.Errorf("paths without a directory: %s %q", st.State, st.Reason) t.Errorf("paths without a directory: %s %q", st.State, st.Reason)
} }
@@ -21,6 +21,7 @@ import (
// build_go_fetch is asked that head as the package's version and build_go_build runs on the member, then both pass, the // build_go_fetch is asked that head as the package's version and build_go_build runs on the member, then both pass, the
// build with GOPROXY=off against the head: the older commit holds no Go package at all. // build with GOPROXY=off against the head: the older commit holds no Go package at all.
func TestRealFetchAndBuildOfAMemberAskedAPullHeadItsGoModDoesNotName(t *testing.T) { func TestRealFetchAndBuildOfAMemberAskedAPullHeadItsGoModDoesNotName(t *testing.T) {
t.Parallel()
// Not skipped when a tool is missing: a pass of the check must mean this test ran (the check prints no test log). // Not skipped when a tool is missing: a pass of the check must mean this test ran (the check prints no test log).
for _, tool := range []string{"go", "git"} { for _, tool := range []string{"go", "git"} {
if _, err := exec.LookPath(tool); err != nil { if _, err := exec.LookPath(tool); err != nil {
@@ -50,7 +50,15 @@ func newPinHarness(t *testing.T, gomod string, vendor bool, pulls string) (*pinH
for _, g := range strings.Fields(f.gomods) { for _, g := range strings.Fields(f.gomods) {
d := filepath.Join(args[i+1], filepath.Dir(g)) d := filepath.Join(args[i+1], filepath.Dir(g))
os.MkdirAll(d, 0o755) os.MkdirAll(d, 0o755)
os.WriteFile(filepath.Join(d, "go.mod"), []byte(gomod), 0o644) body := gomod
if g == "go.mod" { // the root replaces the other directories, so a fetch asked with it chooses them (A-447)
for _, o := range strings.Fields(f.gomods) {
if o != "go.mod" {
body += "\nreplace local/" + filepath.Dir(o) + " => ./" + filepath.Dir(o) + "\n"
}
}
}
os.WriteFile(filepath.Join(d, "go.mod"), []byte(body), 0o644)
os.WriteFile(filepath.Join(d, "go.sum"), []byte("sum\n"), 0o644) os.WriteFile(filepath.Join(d, "go.sum"), []byte("sum\n"), 0o644)
} }
if vendor { if vendor {
@@ -296,6 +304,7 @@ func TestARelativeReplaceStillResolvesInTheTemporaryModfile(t *testing.T) {
// the asked versions, else from the trunk's latest published version, and never looked for as the commit 000000000000 // the asked versions, else from the trunk's latest published version, and never looked for as the commit 000000000000
// among the pull request heads. // among the pull request heads.
func TestARequirementAtTheZeroVersionIsTakenFromTheAskOrTheTrunk(t *testing.T) { func TestARequirementAtTheZeroVersionIsTakenFromTheAskOrTheTrunk(t *testing.T) {
t.Parallel()
const sdk = "git.novox.be/novox/mesh-sdk/go" const sdk = "git.novox.be/novox/mesh-sdk/go"
zeroMod := "module fixture/member\n\ngo 1.26\n\nrequire (\n\t" + sdk + " v0.0.0-00010101000000-" + zeroShort + "\n)\n" zeroMod := "module fixture/member\n\ngo 1.26\n\nrequire (\n\t" + sdk + " v0.0.0-00010101000000-" + zeroShort + "\n)\n"
trunk := strings.Repeat("a", 40) trunk := strings.Repeat("a", 40)
@@ -171,6 +171,7 @@ func TestRealFetchMergesConflictsAndPlainlyFetches(t *testing.T) {
// sibling and passes with it; the sibling is a plain directory with no symlink in it; and a sibling that holds a // sibling and passes with it; the sibling is a plain directory with no symlink in it; and a sibling that holds a
// symlink fails the fetch naming it. // symlink fails the fetch naming it.
func TestRealTestsReadTheSiblingBesideTheTree(t *testing.T) { func TestRealTestsReadTheSiblingBesideTheTree(t *testing.T) {
t.Parallel()
needTools(t, "the shut-off unit level", "go", "git", "bwrap") // the tests run in bubblewrap (1309) needTools(t, "the shut-off unit level", "go", "git", "bwrap") // the tests run in bubblewrap (1309)
f := newForge(t) f := newForge(t)
files := map[string]string{} files := map[string]string{}
@@ -36,7 +36,7 @@ func (m *Module) run(ctx context.Context, fetch steprun.StepFunc, a sdkstep.Step
return nil, err return nil, err
} }
key := sdkstep.RunKey(job.Program, sdkstep.ToolchainUsed{Version: installed, Recipe: m.Recipe.Version()}, a.Packages) key := sdkstep.RunKey(job.Program, sdkstep.ToolchainUsed{Version: installed, Recipe: m.Recipe.Version()}, a.Packages)
fetchArgs := sdkstep.FetchRequest{Repository: job.Program.Repository, Commit: job.Program.Commit, Packages: a.Packages}.Args() fetchArgs := sdkstep.FetchRequest{Repository: job.Program.Repository, Commit: job.Program.Commit, Dir: job.Program.Dir, Packages: a.Packages}.Args()
exe, built, err := m.programOf(ctx, fetch, "programs", "checking program", key, fetchArgs, job.Program.Dir, job.Program.Package) exe, built, err := m.programOf(ctx, fetch, "programs", "checking program", key, fetchArgs, job.Program.Dir, job.Program.Package)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -118,6 +118,7 @@ func resolve(t *testing.T, addr string, a builds.BuiltArtifact) []byte {
// reference resolves in a real artifact store to bytes of that digest. The build is the real go's, shut off, of the // reference resolves in a real artifact store to bytes of that digest. The build is the real go's, shut off, of the
// fixture testdata/repos/build, with an archive beside its bundle. // fixture testdata/repos/build, with an archive beside its bundle.
func TestADoneBuildsArtifactsResolveInARealArtifactStore(t *testing.T) { func TestADoneBuildsArtifactsResolveInARealArtifactStore(t *testing.T) {
t.Parallel()
needTools(t, "the build", "go", "bwrap") needTools(t, "the build", "go", "bwrap")
work := t.TempDir() work := t.TempDir()
tree := filepath.Join(work, "ws1", "tree") tree := filepath.Join(work, "ws1", "tree")
@@ -109,18 +109,22 @@ func (m *Module) test(ctx context.Context, args map[string]any) (any, error) {
} }
// compareGenerated regenerates the generated Go files of each contracts module in the tree (every module.json in the tree, in // compareGenerated regenerates the generated Go files of each contracts module in the tree (every module.json in the tree, in
// any folder: ADR 0634 §8; CAT-BG5) and fails the level on any out of step, Why naming each path. // any folder: ADR 0634 §8; CAT-BG5; for a workspace fetched for a module's directory, that module's own alone) and fails the level on any out of step, Why naming each path.
func compareGenerated(tree string, res *TestResult) error { func compareGenerated(tree string, res *TestResult) error {
dirs, err := steprun.ModuleDirs(tree) dirs, err := steprun.ModuleDirs(tree)
if err != nil { if err != nil {
return err return err
} }
for _, dir := range dirs { for _, dir := range dirs {
// A step for a module compares its own module.json alone, never another module's of the repository (A-447c).
rel, _ := filepath.Rel(tree, dir)
if own, ok := moduleDirOf(tree); ok && filepath.ToSlash(rel) != own {
continue
}
paths, err := steprun.OutOfStep(dir, steprun.GeneratedRootGo) paths, err := steprun.OutOfStep(dir, steprun.GeneratedRootGo)
if err != nil { if err != nil {
return err return err
} }
rel, _ := filepath.Rel(tree, dir)
for _, p := range paths { for _, p := range paths {
res.OutOfStep = append(res.OutOfStep, filepath.ToSlash(filepath.Join(rel, p))) res.OutOfStep = append(res.OutOfStep, filepath.ToSlash(filepath.Join(rel, p)))
} }
@@ -252,7 +256,7 @@ type levelOpts struct {
// runLevel runs `go test -json` in each go.mod directory of the tree and judges the whole against the bounds. // runLevel runs `go test -json` in each go.mod directory of the tree and judges the whole against the bounds.
func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestResult, error) { func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestResult, error) {
dirs, err := m.goModDirs(ctx, tree) dirs, err := m.testDirs(ctx, tree)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -62,6 +62,7 @@ func repository(t *testing.T, name string) string {
// test_test.go read the output captured from them, and this one proves the capture is what the real go says today. It // test_test.go read the output captured from them, and this one proves the capture is what the real go says today. It
// starts go test, so it carries the tag. // starts go test, so it carries the tag.
func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) { func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
t.Parallel()
needTools(t, "the unit level on fixtures", "go", "git", "bwrap") needTools(t, "the unit level on fixtures", "go", "git", "bwrap")
want := map[string]struct { want := map[string]struct {
passed, noTests bool passed, noTests bool
@@ -104,6 +105,7 @@ func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
// 086 A-C3 against the real go (design 47 line 571 at b229f5c4): a tree holding an unformatted file passes its unit // 086 A-C3 against the real go (design 47 line 571 at b229f5c4): a tree holding an unformatted file passes its unit
// level, its two tests run, and the lint step's body fails naming the file. // level, its two tests run, and the lint step's body fails naming the file.
func TestAnUnformattedFileFailsTheRealLintStepAndNotTheUnitLevel(t *testing.T) { func TestAnUnformattedFileFailsTheRealLintStepAndNotTheUnitLevel(t *testing.T) {
t.Parallel()
needTools(t, "the unit level's gofmt and vet", "go", "git", "bwrap") needTools(t, "the unit level's gofmt and vet", "go", "git", "bwrap")
home := t.TempDir() home := t.TempDir()
m := &Module{Home: home, Work: t.TempDir(), Cache: t.TempDir()} m := &Module{Home: home, Work: t.TempDir(), Cache: t.TempDir()}
@@ -150,7 +152,7 @@ func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.
needTools(t, "the integration level", "go", "git", "initdb", "pg_ctl", "postgres", "bwrap") needTools(t, "the integration level", "go", "git", "initdb", "pg_ctl", "postgres", "bwrap")
tree := repository(t, "store") tree := repository(t, "store")
// A short work directory: a unix socket's path is bounded at 108 bytes. // A short work directory: a unix socket's path is bounded at 108 bytes.
work, err := os.MkdirTemp("/tmp", "w") work, err := os.MkdirTemp(os.TempDir(), "w")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -185,6 +187,7 @@ func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.
// 1487/AC2: given a fixture Go module, when it is built twice, both digests are equal and the archive holds one // 1487/AC2: given a fixture Go module, when it is built twice, both digests are equal and the archive holds one
// executable named as binaryName says. It runs the real go, so it carries the tag. // executable named as binaryName says. It runs the real go, so it carries the tag.
func TestBuildTwiceGivesOneDigestAndOneExecutable(t *testing.T) { func TestBuildTwiceGivesOneDigestAndOneExecutable(t *testing.T) {
t.Parallel()
needTools(t, "the build", "go", "bwrap") needTools(t, "the build", "go", "bwrap")
work := t.TempDir() work := t.TempDir()
tree := filepath.Join(work, "ws1", "tree") tree := filepath.Join(work, "ws1", "tree")
@@ -340,3 +340,36 @@ func TestADirectoryThatVendorsIsReadFromItsVendorDirectory(t *testing.T) {
t.Errorf("the last GOFLAGS of go test in a vendoring directory is %q", flags) t.Errorf("the last GOFLAGS of go test in a vendoring directory is %q", flags)
} }
} }
// A-447c: a workspace fetched for a module's directory compares the generated files of that module's own module.json
// alone, never another module's of the repository.
func TestUnitLevelComparesOnlyTheOwnModulesGeneratedFiles(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "empty")
tree := filepath.Join(m.Work, ws, "tree")
for _, folder := range []string{"domain", "other"} {
for p, text := range map[string]string{
"module.json": `{"module":"c","build":{"artifacts":[{"name":"go","kind":"package","language":"go","paths":["go"],"generated-from":"."}]}}`,
"seats/one-store/seat.json": `{"format":1,"seat":"one-store","scope":"mesh","grain":"seat-list","domain":"Module","decision":"ADR 1","verbs":[{"name":"query","input":{"type":"object"}}]}`,
"go/go.mod": "module example.org/c/go\n",
"go/stray.go": "package stray\n",
} {
full := filepath.Join(tree, folder, filepath.FromSlash(p))
_ = os.MkdirAll(filepath.Dir(full), 0o755)
if err := os.WriteFile(full, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
}
if err := os.WriteFile(filepath.Join(m.Work, ws, moduleDirMark), []byte("domain"), 0o600); err != nil {
t.Fatal(err)
}
res, _ := runUnit(t, m, states, ws)
if len(res.OutOfStep) == 0 {
t.Fatalf("nothing out of step: %+v", res)
}
for _, p := range res.OutOfStep {
if !strings.HasPrefix(p, "domain/") {
t.Errorf("out of step = %v, want the module's own files under domain/ alone", res.OutOfStep)
}
}
}
@@ -0,0 +1,44 @@
// Command mesh-compose is the stand-in composer the compose ask's test builds: a program of the contract the Go build module
// runs (`-request <file>`, the last line of its standard output a composer verdict of version 1), which judges nothing and
// answers each case of the request by its name, with no manifest and no machine. It imports nothing of the mesh.
package main
import (
"encoding/json"
"flag"
"fmt"
"os"
)
func main() {
file := flag.String("request", "", "the request file")
flag.Parse()
raw, err := os.ReadFile(*file)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
var req struct {
Cases []struct {
Name string `json:"name"`
} `json:"cases"`
}
if err := json.Unmarshal(raw, &req); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
type verdict struct {
Name string `json:"name"`
Manifests map[string]any `json:"manifests"`
}
out := struct {
V int `json:"v"`
SDK string `json:"sdk"`
Cases []verdict `json:"cases"`
}{V: 1, SDK: "stand-in"}
for _, c := range req.Cases {
out.Cases = append(out.Cases, verdict{Name: c.Name, Manifests: map[string]any{}})
}
line, _ := json.Marshal(out)
fmt.Println(string(line))
}
@@ -0,0 +1,3 @@
module standin
go 1.26
@@ -74,6 +74,8 @@ func putReporting(put steprun.Put, r *toolchainReporter, digest func(version str
} }
if eerr := r.Report(key, rev, t); eerr != nil && say != nil { if eerr := r.Report(key, rev, t); eerr != nil && say != nil {
say("toolchain-reported could not be said: %v", eerr) say("toolchain-reported could not be said: %v", eerr)
} else if eerr == nil && say != nil {
say("toolchain-reported said: %s %s (key %s, revision %d)", t.Language, t.Version, key, rev)
} }
return rev, nil return rev, nil
} }
@@ -53,7 +53,7 @@ func TestGoModVerifyRunsOncePerGoModAndGoSumDigest(t *testing.T) {
// Without bubblewrap the module cache is not read-only to a step: a pass is never reused. // Without bubblewrap the module cache is not read-only to a step: a pass is never reused.
func TestAPassOfVerifyIsNotReusedWhenStepsAreNotShutOff(t *testing.T) { func TestAPassOfVerifyIsNotReusedWhenStepsAreNotShutOff(t *testing.T) {
f := &fakeStep{gomods: "a/go.mod\nb/go.mod\n", files: map[string]string{"a/go.mod": "m\n", "b/go.mod": "m\n"}} f := &fakeStep{gomods: "a/go.mod\nb/go.mod\n", files: map[string]string{"a/go.mod": "module m\n", "b/go.mod": "module m\n"}}
m, states := fetchModule(t, f) // no Bubblewrap m, states := fetchModule(t, f) // no Bubblewrap
fetchOnce(t, m, states, fetchArgs) fetchOnce(t, m, states, fetchArgs)
fetchOnce(t, m, states, fetchArgs) fetchOnce(t, m, states, fetchArgs)
@@ -65,7 +65,7 @@ func TestAPassOfVerifyIsNotReusedWhenStepsAreNotShutOff(t *testing.T) {
// A verify that fails leaves no mark: the next fetch verifies again. // A verify that fails leaves no mark: the next fetch verifies again.
func TestAFailedVerifyLeavesNoMark(t *testing.T) { func TestAFailedVerifyLeavesNoMark(t *testing.T) {
fail := true fail := true
f := &fakeStep{gomods: "go.mod\n", files: map[string]string{"go.mod": "m\n"}} f := &fakeStep{gomods: "go.mod\n", files: map[string]string{"go.mod": "module m\n"}}
f.goRule = func(line string) forgekey.Ran { f.goRule = func(line string) forgekey.Ran {
if strings.HasSuffix(line, " mod verify") && fail { if strings.HasSuffix(line, " mod verify") && fail {
return forgekey.Ran{Status: 1, Stdout: "x v1: dir has been modified\n"} return forgekey.Ran{Status: 1, Stdout: "x v1: dir has been modified\n"}
@@ -88,7 +88,7 @@ func TestDownloadsRunSideBySideUpToTheBound(t *testing.T) {
running, most := 0, 0 running, most := 0, 0
release := make(chan struct{}) // closed once two run at once: they overlap however slowly the machine starts them release := make(chan struct{}) // closed once two run at once: they overlap however slowly the machine starts them
f := &fakeStep{gomods: "a/go.mod\nb/go.mod\nc/go.mod\nd/go.mod\n", files: map[string]string{ f := &fakeStep{gomods: "a/go.mod\nb/go.mod\nc/go.mod\nd/go.mod\n", files: map[string]string{
"a/go.mod": "a\n", "b/go.mod": "b\n", "c/go.mod": "c\n", "d/go.mod": "d\n"}} "a/go.mod": "module a\n", "b/go.mod": "module b\n", "c/go.mod": "module c\n", "d/go.mod": "module d\n"}}
f.goRule = func(line string) forgekey.Ran { f.goRule = func(line string) forgekey.Ran {
if strings.HasSuffix(line, " mod download") { if strings.HasSuffix(line, " mod download") {
mu.Lock() mu.Lock()
@@ -53,6 +53,7 @@ type downloadInfo struct {
Version string Version string
Sum string Sum string
GoModSum string GoModSum string
GoMod string // the go.mod of the version in the module cache
Error string Error string
} }
@@ -159,7 +160,10 @@ func (m *Module) takeVersion(ctx context.Context, root, dir string, tv *TakenVer
return fmt.Errorf("the asked version %s of %s: %w", tv.Asked, tv.Package, err) return fmt.Errorf("the asked version %s of %s: %w", tv.Asked, tv.Package, err)
} }
tv.Built, tv.Said = tv.Asked, "Go cannot fetch it: placed from the pull request's head, which has no checksum" tv.Built, tv.Said = tv.Asked, "Go cannot fetch it: placed from the pull request's head, which has no checksum"
return nil if _, err := os.Stat(filepath.Join(where, "go.mod")); err != nil {
return nil
}
return m.takeRequirementsOf(ctx, pinsDir, filepath.Join(where, "go.mod"))
} }
if tagged && compareVersions(tv.Named, info.Version) > 0 { if tagged && compareVersions(tv.Named, info.Version) > 0 {
tv.Said = fmt.Sprintf("go.mod names %s, newer than the group's tagged version %s, which is built: the group's check is of its tagged version (ADR 0607 §3)", tv.Named, info.Version) tv.Said = fmt.Sprintf("go.mod names %s, newer than the group's tagged version %s, which is built: the group's check is of its tagged version (ADR 0607 §3)", tv.Named, info.Version)
@@ -177,12 +181,21 @@ func (m *Module) takeVersion(ctx context.Context, root, dir string, tv *TakenVer
return fmt.Errorf("the asked version %s of %s could not be required: %s", info.Version, tv.Package, causeLine(r.Err+"\n"+r.Stderr)) return fmt.Errorf("the asked version %s of %s could not be required: %s", info.Version, tv.Package, causeLine(r.Err+"\n"+r.Stderr))
} }
// The checksums Go computed of the download, which go mod verify checks the module cache against. // The checksums Go computed of the download, which go mod verify checks the module cache against.
if err := appendSums(pinsDir, tv.Package, info); err != nil {
return err
}
// The requirements of that version go in too: the modfile names every module its packages load (ownFilesCover).
return m.takeRequirementsOf(ctx, pinsDir, info.GoMod)
}
// appendSums writes the checksums of one download (the module's and its go.mod's) into the go.sum in pinsDir.
func appendSums(pinsDir, module string, info downloadInfo) error {
sums := "" sums := ""
if info.Sum != "" { if info.Sum != "" {
sums += tv.Package + " " + info.Version + " " + info.Sum + "\n" sums += module + " " + info.Version + " " + info.Sum + "\n"
} }
if info.GoModSum != "" { if info.GoModSum != "" {
sums += tv.Package + " " + info.Version + "/go.mod " + info.GoModSum + "\n" sums += module + " " + info.Version + "/go.mod " + info.GoModSum + "\n"
} }
f, err := os.OpenFile(filepath.Join(pinsDir, "go.sum"), os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0o600) f, err := os.OpenFile(filepath.Join(pinsDir, "go.sum"), os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0o600)
if err != nil { if err != nil {
@@ -195,6 +208,46 @@ func (m *Module) takeVersion(ctx context.Context, root, dir string, tv *TakenVer
return err return err
} }
// takeRequirementsOf requires, in the modfile of pinsDir, each module the go.mod at gomodPath (the go.mod of a version just
// taken, from what the fetch downloaded) requires at a version the modfile lacks or names older, with the checksums of
// each: the graph of a package version taken into a modfile is the version's, not the one go.mod was written against (a
// module whose go.mod names mesh-sdk v0.1.13 and is built at a newer one loads what the newer requires). A module of the
// forge is left to its own rule (the zero version, a pin, an asked version).
func (m *Module) takeRequirementsOf(ctx context.Context, pinsDir, gomodPath string) error {
if gomodPath == "" {
return nil
}
raw, err := os.ReadFile(gomodPath)
if err != nil {
return fmt.Errorf("the go.mod of the version taken: %w", err)
}
have, err := os.ReadFile(filepath.Join(pinsDir, "go.mod"))
if err != nil {
return err
}
want, _ := goModRequirements(string(raw))
named, _ := goModRequirements(string(have))
for _, module := range slices.Sorted(maps.Keys(want)) {
version := want[module]
if strings.HasPrefix(module, forgePrefix) || (named[module] != "" && compareVersions(named[module], version) >= 0) {
continue
}
r := m.Run(ctx, m.goEnv(), "go", "-C", pinsDir, "mod", "download", "-json", module+"@"+version)
var info downloadInfo
_ = json.Unmarshal([]byte(r.Stdout), &info)
if r.Status != 0 || r.Err != "" || info.Version == "" || info.Error != "" {
return downloadFailure(strings.TrimSpace(info.Error+"\n"+r.Err+"\n"+r.Stderr), "downloading %s@%s, a requirement of the version taken", module, version)
}
if r := m.Run(ctx, m.goEnv(), "go", "-C", pinsDir, "mod", "edit", "-require="+module+"@"+info.Version); r.Status != 0 || r.Err != "" {
return fmt.Errorf("%s@%s, a requirement of the version taken, could not be required: %s", module, info.Version, causeLine(r.Err+"\n"+r.Stderr))
}
if err := appendSums(pinsDir, module, info); err != nil {
return err
}
}
return nil
}
// goModRequirements are the module paths a go.mod requires, with the version each names, and those it replaces. // goModRequirements are the module paths a go.mod requires, with the version each names, and those it replaces.
func goModRequirements(gomod string) (required map[string]string, replaced map[string]bool) { func goModRequirements(gomod string) (required map[string]string, replaced map[string]bool) {
required, replaced = map[string]string{}, map[string]bool{} required, replaced = map[string]string{}, map[string]bool{}
@@ -19,6 +19,7 @@ import (
// group's head Go cannot fetch is placed from the forge's pull request heads; a go.mod that does not require the package // group's head Go cannot fetch is placed from the forge's pull request heads; a go.mod that does not require the package
// takes nothing. // takes nothing.
func TestAnAskedVersionIsWrittenIntoTheModfileWhateverGoModNames(t *testing.T) { func TestAnAskedVersionIsWrittenIntoTheModfileWhateverGoModNames(t *testing.T) {
t.Parallel()
const sdk = "git.novox.be/novox/mesh-sdk/go" const sdk = "git.novox.be/novox/mesh-sdk/go"
trunk := strings.Repeat("a", 40) trunk := strings.Repeat("a", 40)
offHead := strings.Repeat("b", 40) // a commit no pull request's head carries offHead := strings.Repeat("b", 40) // a commit no pull request's head carries
@@ -30,10 +31,17 @@ func TestAnAskedVersionIsWrittenIntoTheModfileWhateverGoModNames(t *testing.T) {
forgeDown bool // the forge's listing of pull request heads fails too forgeDown bool // the forge's listing of pull request heads fails too
passing bool // the step ends failed as a transport fault, which the bus gives again (ADR 0664 §1) passing bool // the step ends failed as a transport fault, which the bus gives again (ADR 0664 §1)
downloads int // the downloads asked in all (0: not counted) downloads int // the downloads asked in all (0: not counted)
requires string // the go.mod of the version taken, its requirements to be taken too
}{ }{
{name: "a trunk commit newer than go.mod's", asked: trunk, {name: "a trunk commit newer than go.mod's", asked: trunk,
answer: `{"Path":"` + sdk + `","Version":"v0.1.15-0.20261101000000-aaaaaaaaaaaa","Sum":"h1:zip=","GoModSum":"h1:mod="}`, answer: `{"Path":"` + sdk + `","Version":"v0.1.15-0.20261101000000-aaaaaaaaaaaa","Sum":"h1:zip=","GoModSum":"h1:mod="}`,
built: "v0.1.15-0.20261101000000-aaaaaaaaaaaa", edit: "-require=" + sdk + "@v0.1.15-0.20261101000000-aaaaaaaaaaaa"}, built: "v0.1.15-0.20261101000000-aaaaaaaaaaaa", edit: "-require=" + sdk + "@v0.1.15-0.20261101000000-aaaaaaaaaaaa"},
// The requirements of the version taken go into the modfile with their sums, but a module of the forge is left to its
// own rule and one the modfile names at or above the version is left as it is.
{name: "a trunk commit whose go.mod requires more", asked: trunk,
answer: `{"Path":"` + sdk + `","Version":"v0.1.16-0.20261116170327-aaaaaaaaaaaa","Sum":"h1:zip=","GoModSum":"h1:mod=","GoMod":"GOMOD"}`,
built: "v0.1.16-0.20261116170327-aaaaaaaaaaaa", edit: "-require=" + sdk + "@v0.1.16-0.20261116170327-aaaaaaaaaaaa",
requires: "module " + sdk + "\n\ngo 1.26\n\nrequire (\n\tgithub.com/nats-io/nats.go v1.54.0\n\tgithub.com/klauspost/compress v1.20.0\n\tgit.novox.be/novox/mesh-contracts/go/x v0.1.0\n)\n"},
{name: "go.mod names a newer one", asked: trunk, {name: "go.mod names a newer one", asked: trunk,
answer: `{"Path":"` + sdk + `","Version":"v0.1.15-0.20261001000000-aaaaaaaaaaaa","Sum":"h1:zip="}`, answer: `{"Path":"` + sdk + `","Version":"v0.1.15-0.20261001000000-aaaaaaaaaaaa","Sum":"h1:zip="}`,
built: "v0.1.15-0.20261022120000-" + pinShort, said: "newer than the asked"}, built: "v0.1.15-0.20261022120000-" + pinShort, said: "newer than the asked"},
@@ -96,11 +104,20 @@ func TestAnAskedVersionIsWrittenIntoTheModfileWhateverGoModNames(t *testing.T) {
if name == "git" && strings.Contains(line, "ls-remote") && c.forgeDown { if name == "git" && strings.Contains(line, "ls-remote") && c.forgeDown {
return forgekey.Ran{Status: 128, Stderr: "fatal: unable to access the forge: Could not resolve host\n"} return forgekey.Ran{Status: 128, Stderr: "fatal: unable to access the forge: Could not resolve host\n"}
} }
if name == "go" && strings.Contains(line, " mod download -json github.com/") {
module, version, _ := strings.Cut(strings.Fields(line)[len(strings.Fields(line))-1], "@")
return forgekey.Ran{Stdout: `{"Path":"` + module + `","Version":"` + version + `","Sum":"h1:` + module + `zip=","GoModSum":"h1:` + module + `mod="}`}
}
if name == "go" && strings.Contains(line, " mod download -json "+sdk+"@") { if name == "go" && strings.Contains(line, " mod download -json "+sdk+"@") {
h.mu.Lock() h.mu.Lock()
h.cmds = append(h.cmds, cmdRecord{env: env, line: line}) h.cmds = append(h.cmds, cmdRecord{env: env, line: line})
h.mu.Unlock() h.mu.Unlock()
downloads++ downloads++
if c.requires != "" {
gomod := filepath.Join(t.TempDir(), "go.mod")
os.WriteFile(gomod, []byte(c.requires), 0o644)
return forgekey.Ran{Stdout: strings.ReplaceAll(c.answer, "GOMOD", gomod)}
}
if c.failing { if c.failing {
return forgekey.Ran{Status: 1, Stdout: c.answer} return forgekey.Ran{Status: 1, Stdout: c.answer}
} }
@@ -158,6 +175,18 @@ func TestAnAskedVersionIsWrittenIntoTheModfileWhateverGoModNames(t *testing.T) {
if placed := strings.Contains(string(body), "replace "+sdk+" => "); placed != c.failing { if placed := strings.Contains(string(body), "replace "+sdk+" => "); placed != c.failing {
t.Errorf("the modfile replaces mesh-sdk by a placed head: %v, want %v:\n%s", placed, c.failing, body) t.Errorf("the modfile replaces mesh-sdk by a placed head: %v, want %v:\n%s", placed, c.failing, body)
} }
if c.requires != "" {
for _, dep := range []string{"github.com/nats-io/nats.go@v1.54.0", "github.com/klauspost/compress@v1.20.0"} {
module, _, _ := strings.Cut(dep, "@")
if !h.ran("-require="+dep) || !strings.Contains(string(sums), module+" "+strings.TrimPrefix(dep, module+"@")+" h1:"+module+"zip=") ||
!strings.Contains(string(sums), module+" "+strings.TrimPrefix(dep, module+"@")+"/go.mod h1:"+module+"mod=") {
t.Errorf("the requirement %s of the version taken was not required with its sums:\n%s", dep, sums)
}
}
if h.ran("mesh-contracts/go/x") {
t.Error("a module of the forge required by the version taken was looked for as a requirement")
}
}
if _, other := h.m.pinModfile(filepath.Join(h.m.Work, res.Workspace, "tree", "other")); other { if _, other := h.m.pinModfile(filepath.Join(h.m.Work, res.Workspace, "tree", "other")); other {
t.Error("a go.mod that requires no asked package was handed a modfile") t.Error("a go.mod that requires no asked package was handed a modfile")
} }
@@ -34,7 +34,7 @@ func TestTheGuardSaysToolchainReportedOnAChangeWithTheProgramDigest(t *testing.T
t.Fatal(err) t.Fatal(err)
} }
if len(got) != 1 || got[0].subject != "mesh.seat.node-build-node.event.toolchain-reported.ace" || got[0].body["node"] != "ace" || got[0].body["language"] != "typescript" || if len(got) != 1 || got[0].subject != "mesh.seat.node-build-node.event.toolchain-reported.ace" || got[0].body["node"] != "ace" || got[0].body["language"] != "typescript" ||
got[0].body["version"] != "v26.10.0" || got[0].body["recipe"] != m.Recipe.Version() || len(got[0].body["digest"]) != len("sha256:")+64 { got[0].body["version"] != "v26.10" || got[0].body["recipe"] != m.Recipe.Version() || len(got[0].body["digest"]) != len("sha256:")+64 {
t.Fatalf("events = %+v, want one toolchain-reported for ace, go with the program's digest", got) t.Fatalf("events = %+v, want one toolchain-reported for ace, go with the program's digest", got)
} }
if tc, _ := states.toolchain("ace.typescript"); tc.Digest != got[0].body["digest"] { if tc, _ := states.toolchain("ace.typescript"); tc.Digest != got[0].body["digest"] {
@@ -51,7 +51,7 @@ func TestTheGuardSaysToolchainReportedOnAChangeWithTheProgramDigest(t *testing.T
os.WriteFile(prog, []byte("another node program"), 0o755) os.WriteFile(prog, []byte("another node program"), 0o755)
m.Probe = func(context.Context) (string, error) { return "v26.11.0", nil } m.Probe = func(context.Context) (string, error) { return "v26.11.0", nil }
m.Check(context.Background()) m.Check(context.Background())
if len(got) != 2 || got[1].body["version"] != "v26.11.0" || got[1].body["digest"] == got[0].body["digest"] { if len(got) != 2 || got[1].body["version"] != "v26.11" || got[1].body["digest"] == got[0].body["digest"] {
t.Errorf("a new version said %+v, want a second event with another digest", got) t.Errorf("a new version said %+v, want a second event with another digest", got)
} }
} }
@@ -8,6 +8,7 @@ import (
"strings" "strings"
"testing" "testing"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/health" "git.novox.be/novox/mesh-sdk/go/health"
) )
@@ -36,8 +37,8 @@ func TestAnyInstalledNodeIsHealthyNamedAndReported(t *testing.T) {
t.Errorf("health = %+v, want healthy naming %s", a, installed) t.Errorf("health = %+v, want healthy naming %s", a, installed)
} }
tc, ok := states.toolchain("ace.typescript") tc, ok := states.toolchain("ace.typescript")
if !ok || tc.Version != installed || tc.Recipe != m.Recipe.Version() { if !ok || tc.Version != sdkstep.LanguageVersion(installed) || tc.Recipe != m.Recipe.Version() {
t.Errorf("ace.typescript = %+v, %v; want version %s and recipe %s", tc, ok, installed, m.Recipe.Version()) t.Errorf("ace.typescript = %+v, %v; want version %s and recipe %s", tc, ok, sdkstep.LanguageVersion(installed), m.Recipe.Version())
} }
if err := m.Prepare(context.Background()); err != nil { if err := m.Prepare(context.Background()); err != nil {
t.Errorf("no step can run on node %s: %v", installed, err) t.Errorf("no step can run on node %s: %v", installed, err)
@@ -94,8 +94,8 @@ func TestStartWritesTheToolchainAndHealthIsHealthy(t *testing.T) {
break break
} }
} }
if tc.Version != "v26.10.0" || tc.Recipe != m.Recipe.Version() || tc.Language != "typescript" { if tc.Version != "v26.10" || tc.Recipe != m.Recipe.Version() || tc.Language != "typescript" {
t.Fatalf("ace.typescript = %+v, want v26.10.0 and recipe %s", tc, m.Recipe.Version()) t.Fatalf("ace.typescript = %+v, want v26.10 and recipe %s", tc, m.Recipe.Version())
} }
if a := m.Health(ctx); a.State != health.StateHealthy || !a.Healthy { if a := m.Health(ctx); a.State != health.StateHealthy || !a.Healthy {
t.Errorf("health = %+v, want healthy", a) t.Errorf("health = %+v, want healthy", a)
@@ -69,6 +69,10 @@ type DeliveryStore interface {
type ForgeReads interface { type ForgeReads interface {
OpenPulls() (gitseat.OpenPullsAnswer, error) OpenPulls() (gitseat.OpenPullsAnswer, error)
Pull(repository string, number int64) (gitseat.PullAnswer, error) Pull(repository string, number int64) (gitseat.PullAnswer, error)
// Branch is a repository branch's head (the seat git's verb branch).
Branch(repository, branch string) (gitseat.BranchAnswer, error)
// Contains is whether commit is in (or is) the commit in (the seat git's verb contains).
Contains(repository, commit, in string) (bool, error)
} }
// kvKept is the store over the runtime. // kvKept is the store over the runtime.
@@ -41,7 +41,10 @@ func (k *memKept) Delete(state, key string) error { delete(k.m[state], key); ret
type fakeForge struct { type fakeForge struct {
open []gitseat.OpenPull open []gitseat.OpenPull
pulls map[string]gitseat.Pull pulls map[string]gitseat.Pull
fail bool heads map[string]string // "<repository>@<branch>" -> head
// contains says whether commit is in `in`; nil means every commit is in every other (a straight line forward)
contains func(commit, in string) bool
fail bool
} }
func (f *fakeForge) OpenPulls() (gitseat.OpenPullsAnswer, error) { func (f *fakeForge) OpenPulls() (gitseat.OpenPullsAnswer, error) {
@@ -50,6 +53,25 @@ func (f *fakeForge) OpenPulls() (gitseat.OpenPullsAnswer, error) {
} }
return gitseat.OpenPullsAnswer{Pulls: f.open}, nil return gitseat.OpenPullsAnswer{Pulls: f.open}, nil
} }
func (f *fakeForge) Branch(repository, branch string) (gitseat.BranchAnswer, error) {
if f.fail {
return gitseat.BranchAnswer{}, errors.New("the forge is away")
}
head, ok := f.heads[repository+"@"+branch]
if !ok {
return gitseat.BranchAnswer{Error: "no such branch"}, nil
}
return gitseat.BranchAnswer{Branch: branch, Repository: repository, Head: head}, nil
}
func (f *fakeForge) Contains(repository, commit, in string) (bool, error) {
if f.fail {
return false, errors.New("the forge is away")
}
if f.contains == nil {
return true, nil
}
return f.contains(commit, in), nil
}
func (f *fakeForge) Pull(repository string, number int64) (gitseat.PullAnswer, error) { func (f *fakeForge) Pull(repository string, number int64) (gitseat.PullAnswer, error) {
p, ok := f.pulls[fmt.Sprintf("%s#%d", repository, number)] p, ok := f.pulls[fmt.Sprintf("%s#%d", repository, number)]
if !ok { if !ok {
@@ -44,7 +44,7 @@ type FindingWriter interface {
// generated contract of the seat that defines it (A-17). It is started once the runtime serves (A-40). // generated contract of the seat that defines it (A-17). It is started once the runtime serves (A-40).
func NewFindingWriter() (*views.HeldWriter[meshconditions.Finding, meshconditions.FindingCleared, meshconditions.FindingsStated, meshconditions.FindingsAskedEvent], error) { func NewFindingWriter() (*views.HeldWriter[meshconditions.Finding, meshconditions.FindingCleared, meshconditions.FindingsStated, meshconditions.FindingsAskedEvent], error) {
return views.NewStdioHeldWriter(views.WriterConfig[meshconditions.Finding, meshconditions.FindingCleared, meshconditions.FindingsStated, meshconditions.FindingsAskedEvent]{ return views.NewStdioHeldWriter(views.WriterConfig[meshconditions.Finding, meshconditions.FindingCleared, meshconditions.FindingsStated, meshconditions.FindingsAskedEvent]{
Asked: meshconditions.SubjectFindingsAsked, Module: Module, State: stdio.State("findings"), Kinds: []string{ApprovalCarriedOpen}}) Asked: meshconditions.SubjectFindingsAsked, Module: Module, State: stdio.State("findings"), Kinds: []string{ApprovalCarriedOpen, SourceUncompared}})
} }
// approvalID is the id of a pull request's finding: its repository and number, each escaped as a key's name is. // approvalID is the id of a pull request's finding: its repository and number, each escaped as a key's name is.
@@ -23,11 +23,14 @@ type Holder struct {
// Merges and Old are the old sections the switch reads (switch.go); Source is the gate's source (merges.go). // Merges and Old are the old sections the switch reads (switch.go); Source is the gate's source (merges.go).
Merges MergeStore Merges MergeStore
// Kept is the gate's states deliveries and groups, and Forge the seat git's reads (deliveries.go, §716). // Kept is the gate's states deliveries and groups, and Forge the seat git's reads (deliveries.go, §716).
Kept DeliveryStore Kept DeliveryStore
Forge ForgeReads Forge ForgeReads
dmu sync.Mutex // Catalog reads the seat mesh-catalog's registrations (sourcestart.go); startMu keeps one start of the sources at a time.
Old OldState Catalog Registered
Source SourceStore startMu sync.Mutex
dmu sync.Mutex
Old OldState
Source SourceStore
// Owed says what the gate owes the forge as the seat's event owed (owed.go). // Owed says what the gate owes the forge as the seat's event owed (owed.go).
Owed OwedSayer Owed OwedSayer
// Records keeps the delivery records on the gate's ledger (record.go, ADR 0656 §7); nil keeps none. // Records keeps the delivery records on the gate's ledger (record.go, ADR 0656 §7); nil keeps none.
@@ -32,7 +32,7 @@ func (l *listening) get() string { l.mu.Lock(); defer l.mu.Unlock(); return l.n
// The events this owner follows: the forge bridge's word on an act owed, and the step events the delivery records are built // The events this owner follows: the forge bridge's word on an act owed, and the step events the delivery records are built
// from (consumer.go). // from (consumer.go).
var followed = append(append([]string{owedEndedEvent}, stepPatterns()...), pullUpdatedEvent, pullMergedEvent, pullClosedEvent) var followed = append(append([]string{owedEndedEvent}, stepPatterns()...), pullUpdatedEvent, pullMergedEvent, pullClosedEvent, registrationChangedEvent, forgeMergedEvent)
func main() { func main() {
settingsFile := os.Getenv("MESH_DELIVERY_GATE_SETTINGS") settingsFile := os.Getenv("MESH_DELIVERY_GATE_SETTINGS")
@@ -68,6 +68,7 @@ func main() {
h.Old = kvOld{} h.Old = kvOld{}
h.Kept = kvKept{} h.Kept = kvKept{}
h.Forge = seatGit{ask: stdio.Ask} h.Forge = seatGit{ask: stdio.Ask}
h.Catalog = seatCatalog{ask: stdio.Ask}
h.Consumer = &StepConsumer{Records: h.Records, Families: stepFamilies, Now: h.Now, Logf: logf, Taken: h.TakenIn} h.Consumer = &StepConsumer{Records: h.Records, Families: stepFamilies, Now: h.Now, Logf: logf, Taken: h.TakenIn}
// The gate's one held writer of its own findings (design 47 G A-39, A-37, A-40): made at once, started once the runtime serves. // The gate's one held writer of its own findings (design 47 G A-39, A-37, A-40): made at once, started once the runtime serves.
fw, err := NewFindingWriter() fw, err := NewFindingWriter()
@@ -116,6 +117,14 @@ func run(h *Holder, l *listening) {
} }
l.set("listening") l.set("listening")
logf("[mesh-delivery] listening: %s", strings.Join(followed, ", ")) logf("[mesh-delivery] listening: %s", strings.Join(followed, ", "))
// A watched branch no source key holds starts from the forge, once at each start and at each registration change (A-401).
// then each head is compared with the forge's, and a merge made outside the gate is followed (A-464).
go func() {
startSourcesAtStart(h)
if _, err := h.FollowAtStart(); err != nil {
logf("[mesh-delivery] the sources were not all compared with the forge's heads: %v", err)
}
}()
// A missed pull event is put right from the forge's own record, once at each start (§716). // A missed pull event is put right from the forge's own record, once at each start (§716).
if err := h.ReadForge(); err != nil { if err := h.ReadForge(); err != nil {
logf("[mesh-delivery] the open pull requests were not read from the forge, they are read again at the next start: %v", err) logf("[mesh-delivery] the open pull requests were not read from the forge, they are read again at the next start: %v", err)
@@ -172,6 +181,13 @@ func take(h *Holder, e stdio.Envelope) error {
if e.Key == pullMergedEvent { if e.Key == pullMergedEvent {
return errors.Join(h.DeliveryMerged(e.Body), h.pullEnded(e.Body, "merged")) return errors.Join(h.DeliveryMerged(e.Body), h.pullEnded(e.Body, "merged"))
} }
if e.Key == registrationChangedEvent {
_, err := h.StartSources()
return err
}
if forgeMergedEvent != "" && stdio.TopicMatches(forgeMergedEvent, e.Key) {
return h.ForgeMerged(e.Body)
}
if e.Key == pullClosedEvent { if e.Key == pullClosedEvent {
return errors.Join(h.DeliveryClosed(e.Body), h.pullEnded(e.Body, "closed")) return errors.Join(h.DeliveryClosed(e.Body), h.pullEnded(e.Body, "closed"))
} }
@@ -134,13 +134,16 @@ func TestItClaimsTheDeliverySeatAndFollowsWhatItHandles(t *testing.T) {
} }
} }
} }
// From G's delivery the gate hears neither the bridge's merged and pushed messages nor the teller's answers nor the view's // From G's delivery the gate hears neither the bridge's pushed messages nor the teller's answers nor the view's reach
// reach (design 47 G §592, §593): it consumes the indexer's take-in.taken and the bridge's owed.ended, and the git events that clear // (design 47 G §592, §593); it hears the bridge's merged message alone, to follow every merge (A-464): it consumes the indexer's take-in.taken and the bridge's owed.ended, and the git events that clear
// an approval carried. // an approval carried.
if want := append(slices.Clone(followed), meshconditions.Seat+"."+meshconditions.EventFindingsAsked); !reflect.DeepEqual(m.Consumes, want) { if want := append(slices.Clone(followed), meshconditions.Seat+"."+meshconditions.EventFindingsAsked); !reflect.DeepEqual(m.Consumes, want) {
t.Fatalf("consumes %v, follows %v", m.Consumes, followed) t.Fatalf("consumes %v, follows %v", m.Consumes, followed)
} }
for _, c := range m.Consumes { for _, c := range m.Consumes {
if c == forgeMergedEvent {
continue // A-464: the source follows every merge, so the gate hears the forge bridge's merged message, and it alone
}
for _, gone := range []string{".forge.", "teller", "view", "forge-pull", "forge-status"} { for _, gone := range []string{".forge.", "teller", "view", "forge-pull", "forge-status"} {
if strings.Contains(c, gone) { if strings.Contains(c, gone) {
t.Errorf("the manifest still consumes %s, which G's gate does not hear (§431, §592, §593)", c) t.Errorf("the manifest still consumes %s, which G's gate does not hear (§431, §592, §593)", c)
@@ -181,7 +184,7 @@ func TestItClaimsTheDeliverySeatAndFollowsWhatItHandles(t *testing.T) {
} }
// It calls no controller verb and no forge verb: the forge's reads and writes are the bridge's (ADR 0437 §8; design 47 §592). // It calls no controller verb and no forge verb: the forge's reads and writes are the bridge's (ADR 0437 §8; design 47 §592).
// It uses operator-channel only to say its conditions (ADR 0437 §9, issue 1275): the teller asks no more (§431). // It uses operator-channel only to say its conditions (ADR 0437 §9, issue 1275): the teller asks no more (§431).
want := []string{"seat:git.open-pulls", "seat:git.pull", "seat:mesh-hand-acts.hand-act", WantsEntryVerb, EngineCompositionVerb} want := []string{"seat:git.branch", "seat:git.open-pulls", "seat:git.pull", "seat:mesh-catalog.registrations", "seat:mesh-hand-acts.hand-act", WantsEntryVerb, EngineCompositionVerb}
if !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) { if !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) {
t.Fatalf("uses %v: operator-channel, for its conditions alone", m.Uses) t.Fatalf("uses %v: operator-channel, for its conditions alone", m.Uses)
} }
@@ -0,0 +1,278 @@
package main
// The gate's source of a watched branch follows every merge into that branch, whoever made it (design 47 G A-464, A-464b, amending
// A-463): on the forge bridge's `merged` of a repository the source names, at the branch the source watches, the gate moves that
// repository's head to the merge and says source-moved, as a take-in moves it; at its start it compares each head with the
// forge's branch head, asked of the seat git, and moves every one behind it the same way, so a merge whose message was lost is
// followed too; gate_source_follow does the same for one repository now. A head the source does not name, or a branch it does
// not watch, is left: only a start or a registration change adds them (sourcestart.go).
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// forgeMergedEvent is the forge bridge's merged message of any repository, as the manifest consumes it.
var forgeMergedEvent = declared("forge.*.*.merged")
// moveHead moves the head the branch's source names for repository to head, and says source-moved with the key's revision. It
// answers false, and writes nothing, when the branch has no source key, the key does not name the repository or names the head
// already. group is the group that made the merge, empty for a merge no group of the gate made.
func (h *Holder) moveHead(branch, repository, head, group string) (moved bool, err error) {
if h.Source == nil || head == "" {
return false, nil
}
held, _, err := h.Source.Read(branch)
if err != nil || held == nil {
return false, err
}
name := ""
for r := range held.Heads {
if trunk.SameRepository(r, repository) {
name = r
}
}
if name == "" || held.Heads[name] == head {
return false, nil // not named, or no change: no revision and no source-moved (A-464c)
}
was := held.Heads[name]
// Only forward (A-464c): the head held must be contained in the merge, asked of the seat git.
if h.Forge == nil {
return false, errors.New("this gate reads no forge, so it cannot tell whether the merge is ahead of the head")
}
ahead, err := h.Forge.Contains(name, was, head)
if err != nil {
return false, fmt.Errorf("whether %.12s is in %.12s of %s cannot be read, so the head stays: %w", was, head, name, err)
}
if !ahead {
h.logf("[mesh-delivery] source.%s keeps %s at %.12s: the merge %.12s does not contain it (heard again, late or out of order), left without effect", branch, name, was, head)
return false, nil
}
rev, err := h.Source.Set(branch, func(prev *Source) (Source, bool) {
moved = false
if prev == nil || prev.Heads[name] != was {
return Source{}, false // another writer moved it first
}
next := Source{Group: group, Heads: map[string]string{}}
for r, c := range prev.Heads {
next.Heads[r] = c
}
next.Heads[name] = head
moved = true
return next, true
})
if err != nil {
return false, fmt.Errorf("source.%s cannot be moved: %w", branch, err)
}
if !moved || h.SaySeat == nil {
return moved, nil
}
at := h.Now().UTC()
return true, h.SaySeat(seat.SubjectSourceMoved(keys.EscapeName(branch)),
seat.SourceMovedEvent{Branch: branch, Group: group, Revision: int64(rev), At: &at},
"source-moved."+keys.EscapeName(branch)+"."+strconv.FormatUint(rev, 10))
}
// ForgeMerged takes one merged message of the forge bridge: the head of its repository in the source of the branch it landed on
// moves to the merge. A merge no delivery group made (the message names no group) is said so. A message that cannot be read, or
// names no branch or merge, is said and taken: heard again it would be refused again.
func (h *Holder) ForgeMerged(body json.RawMessage) error {
f, err := forgemsg.DecodeForge(body)
if err != nil {
h.logf("[mesh-delivery] a merged message of the forge bridge is not readable, the source does not follow it: %v", err)
return nil
}
if f.Kind != forgemsg.ForgeMerged {
return nil
}
if f.Base == "" || f.Merge == "" {
h.logf("[mesh-delivery] the merged message of %s names no branch or no merge, the source does not follow it", f.Repository)
return nil
}
moved, err := h.moveHead(f.Base, f.Repository, f.Merge, f.Group)
if err != nil {
return err
}
if moved {
if f.Group == "" { // A-464b: a merge whose message names no delivery group is one the gate did not make
h.logf("[mesh-delivery] source.%s follows the merge %.12s of %s: not made by a delivery group of the gate", f.Base, f.Merge, f.Repository)
} else {
h.logf("[mesh-delivery] source.%s follows the merge %.12s of %s, made by the delivery group %s", f.Base, f.Merge, f.Repository, f.Group)
}
}
return nil
}
// SourceUncompared is the finding of a start comparison that left a head uncompared (A-464e).
const SourceUncompared = "source-uncompared"
// FollowAtStart compares each head of every source key with the forge's head of that branch and moves every one behind it, at
// each start. It ends only when every head was compared (A-464e): a head the forge could not be asked for (the seat git not
// answering yet, as when its bundle starts in the same tool runner) is compared again after a growing pause of at most a
// minute, and the finding source-uncompared names the repositories and the last error while any stands. It answers the heads
// moved as "<repository>@<branch>".
func (h *Holder) FollowAtStart() ([]string, error) {
if h.Source == nil || h.Forge == nil {
return nil, errors.New("this gate keeps no source or reads no forge")
}
var moved []string
var only map[string]bool
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
got, left := h.followPass(only)
moved = append(moved, got...)
if len(left) == 0 {
h.clearUncompared()
h.logf("[mesh-delivery] the sources compared with the forge's heads at the start: moved %v", moved)
return moved, nil
}
only = map[string]bool{}
var names []string
var last error
for n, err := range left {
only[n] = true
names = append(names, n)
last = err
}
sort.Strings(names)
h.sayUncompared(names, last)
h.logf("[mesh-delivery] %d heads are not compared with the forge's yet (%s; last: %v); asking again in %s", len(names), strings.Join(names, ", "), last, wait)
startSleep(wait)
}
}
// followPass compares the heads named in only (every head when nil) once; it answers the heads moved and, by "<repository>@<branch>",
// each head it could not compare with why. A source it cannot list or read is one entry named for itself.
func (h *Holder) followPass(only map[string]bool) (moved []string, left map[string]error) {
left = map[string]error{}
branches, err := h.Source.Branches()
if err != nil {
left["the branches of the source"] = err
return nil, left
}
for _, branch := range branches {
held, _, err := h.Source.Read(branch)
if err != nil {
left["source."+branch] = err
continue
}
if held == nil {
continue
}
for repository, head := range held.Heads {
name := repository + "@" + branch
if only != nil && !only[name] {
continue
}
ok, err := h.followForge(branch, repository, head)
if err != nil {
left[name] = err
continue
}
if ok {
moved = append(moved, name)
}
}
}
return moved, left
}
func (h *Holder) sayUncompared(names []string, last error) {
if h.Findings == nil {
return
}
at := h.Now().UTC()
err := h.Findings.Write(context.Background(), meshconditions.Finding{Scope: FindingScope, ID: SourceUncompared, Kind: SourceUncompared,
Subject: meshconditions.Subject{Scope: FindingScope, ID: SourceUncompared},
Severity: meshconditions.FindingSeverityWarning, Resolver: "operator",
Summary: fmt.Sprintf("the gate's source is not compared with the forge's head for %s; the last error: %v", strings.Join(names, ", "), last),
Headline: "The gate's source is not compared with the forge",
Explanation: "At its start the gate compares each head of its source with the forge's head of the branch. A head it could not compare is asked again after a growing pause, and this clears once every head is compared.",
Needs: "the seat git answering; gate_source_follow moves one repository's head by hand.",
Source: SourceUncompared, Observer: Module, At: at, Since: at})
if err != nil {
h.logf("[mesh-delivery] the finding %s is not written: %v", SourceUncompared, err)
}
}
func (h *Holder) clearUncompared() {
if h.Findings == nil {
return
}
for _, f := range h.Findings.Held() {
if f.Scope == FindingScope && f.ID == SourceUncompared && f.Kind == SourceUncompared {
if err := h.Findings.Clear(context.Background(), FindingScope, SourceUncompared, SourceUncompared, "every head is compared with the forge's"); err != nil {
h.logf("[mesh-delivery] the finding %s is not cleared: %v", SourceUncompared, err)
}
}
}
}
// followForge moves one repository's head in a branch's source to the forge's head of the branch when they differ.
func (h *Holder) followForge(branch, repository, held string) (bool, error) {
a, err := h.Forge.Branch(repository, branch)
if err == nil && a.Error != "" {
err = errors.New(a.Error)
}
if err == nil && a.Head == "" {
err = errors.New("the forge named no head")
}
if err != nil {
return false, err
}
if sameCommit(held, a.Head) && len(held) == len(a.Head) {
return false, nil
}
moved, err := h.moveHead(branch, repository, a.Head, "")
if moved {
h.logf("[mesh-delivery] source.%s follows the forge: %s from %.12s to %.12s", branch, repository, held, a.Head)
}
return moved, err
}
// SourceFollow moves one repository's head to the forge's head of the branch by hand, after the hand-act log took it.
func (h *Holder) SourceFollow(branch, repository, why, by string, acts handActs) (map[string]any, error) {
if strings.TrimSpace(branch) == "" || strings.TrimSpace(repository) == "" || strings.TrimSpace(why) == "" {
return nil, errors.New("say the branch, the repository and why its head follows the forge")
}
if h.Source == nil || h.Forge == nil {
return nil, errors.New("this gate keeps no source or reads no forge")
}
if acts == nil {
return nil, errors.New("the hand-act log cannot be reached here, so nothing was moved")
}
held, _, err := h.Source.Read(branch)
if err != nil {
return nil, err
}
if held == nil {
return nil, fmt.Errorf("the gate keeps no source of the branch %s", branch)
}
name := ""
for r := range held.Heads {
if trunk.SameRepository(r, repository) {
name = r
}
}
if name == "" {
return nil, fmt.Errorf("the source of %s names no head of %s", branch, repository)
}
if _, err := acts.Record(meshhandactsclient.Request{What: "moved the head of " + repository + " in the source of the branch " + branch + " to the forge's head",
Why: why, Cause: "gate_source_follow", Verb: "source-follow"}); err != nil {
return nil, fmt.Errorf("the hand act was not recorded through mesh-hand-acts, so nothing was moved: %w", err)
}
moved, err := h.followForge(branch, name, held.Heads[name])
return map[string]any{"branch": branch, "repository": name, "moved": moved, "by": by}, err
}
@@ -0,0 +1,193 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Design 47 G A-464, A-464b: the forge bridge's merged message moves the head of its repository in the source of the branch it
// landed on and says source-moved; a merge naming no delivery group is said not made by one; a branch no source key holds, a
// repository the key does not name, and a head the key holds already move nothing.
func TestAMergedMessageMovesTheSourceOfItsBranchWhoeverMadeTheMerge(t *testing.T) {
old, merge := strings.Repeat("a", 40), strings.Repeat("b", 40)
body := func(repository, base, group string) []byte {
f := forgemsg.Forge{V: 1, Kind: forgemsg.ForgeMerged, Repository: repository, Base: base, Merge: merge, Head: merge, Number: 14, Root: strings.Repeat("d", 40)}
data, err := forgemsg.EncodeForge(f)
if err != nil {
t.Fatal(err)
}
return data
}
for _, c := range []struct {
name, repository, base string
moved bool
}{
{"watched branch, named repository", "novox/mesh-contracts", "main", true},
{"a branch no source key holds", "novox/mesh-contracts", "other", false},
{"a repository the key does not name", "novox/other", "main", false},
} {
t.Run(c.name, func(t *testing.T) {
src := &memSource{vals: map[string]Source{"main": {Group: "g", Heads: map[string]string{"novox/mesh-contracts": old}}}}
var said []string
var logged []string
h := &Holder{Now: time.Now, Source: src, Forge: &fakeForge{},
Logf: func(f string, a ...any) { logged = append(logged, f) },
SaySeat: func(subject string, _ any, id string) error { said = append(said, id); return nil }}
if err := h.ForgeMerged(body(c.repository, c.base, "")); err != nil {
t.Fatal(err)
}
got := src.vals["main"]
if c.moved != (got.Heads["novox/mesh-contracts"] == merge) || c.moved != (len(said) == 1) || (c.moved && got.Group != "") {
t.Fatalf("source %+v said %v: moved wanted %v", got, said, c.moved)
}
if c.moved && !strings.Contains(strings.Join(logged, "\n"), "not made by a delivery group of the gate") {
t.Errorf("a merge no group made is not said so: %v", logged)
}
// Heard again, the key names the head already: nothing is moved or said twice.
if err := h.ForgeMerged(body(c.repository, c.base, "")); err != nil || len(said) > 1 {
t.Errorf("heard twice: %v, said %v", err, said)
}
})
}
}
// Design 47 G A-464: at its start the gate compares each head with the forge's branch head and moves every one behind it; a head
// the forge cannot read is said and left; gate_source_follow does it for one repository after the hand-act log took it.
func TestTheStartAndTheToolMoveAHeadBehindTheForge(t *testing.T) {
old, now := strings.Repeat("a", 40), strings.Repeat("c", 40)
mk := func() (*Holder, *branchSource, *[]string) {
src := &branchSource{memSource{vals: map[string]Source{"main": {Heads: map[string]string{"o/f1": old, "o/f2": old, "o/f3": now}}}}}
var said []string
return &Holder{Now: time.Now, Source: src, Logf: func(string, ...any) {},
Forge: &fakeForge{heads: map[string]string{"o/f1@main": now, "o/f3@main": now}}, // o/f2 cannot be read
SaySeat: func(_ string, _ any, id string) error { said = append(said, id); return nil }}, src, &said
}
h, src, said := mk()
fw := &fakeFindings{}
h.Findings = fw
forge := h.Forge.(*fakeForge)
var pauses []time.Duration
var heldWhileWaiting int
prevSleep := startSleep
startSleep = func(d time.Duration) {
pauses = append(pauses, d)
heldWhileWaiting = len(fw.held) // the finding stands while a head is uncompared
forge.heads["o/f2@main"] = now // the seat git answers by the second pass
}
defer func() { startSleep = prevSleep }()
moved, err := h.FollowAtStart()
if err != nil || len(moved) != 2 || src.vals["main"].Heads["o/f1"] != now || src.vals["main"].Heads["o/f2"] != now || len(*said) != 2 {
t.Fatalf("start: moved %v err %v heads %v said %v: want o/f1 and, after the second pass, o/f2 moved", moved, err, src.vals["main"].Heads, *said)
}
if len(pauses) != 1 || heldWhileWaiting != 1 || len(fw.held) != 0 {
t.Fatalf("pauses %v, finding held while waiting %d and after %d: want one pause, the finding raised while a head stood uncompared and cleared after", pauses, heldWhileWaiting, len(fw.held))
}
h, src, said = mk()
acts := &handActsFake{refuse: errors.New("log down")}
if _, err := h.SourceFollow("main", "o/f1", "missed a merge", "me", acts); err == nil || src.vals["main"].Heads["o/f1"] != old {
t.Fatalf("moved without the log: %v", err)
}
acts.refuse = nil
if _, err := h.SourceFollow("main", "o/f1", "", "me", acts); err == nil {
t.Fatal("moved with no reason")
}
out, err := h.SourceFollow("main", "o/f1", "missed a merge", "me", acts)
if err != nil || out["moved"] != true || len(acts.took) != 1 || acts.took[0].Cause != "gate_source_follow" || src.vals["main"].Heads["o/f1"] != now {
t.Fatalf("tool: %v %v log %+v heads %v", out, err, acts.took, src.vals["main"].Heads)
}
}
// Design 47 G A-464: the gate takes the forge bridge's merged event of any repository, by the manifest's own pattern.
func TestTheGateTakesTheForgeBridgesMergedEventOfAnyRepository(t *testing.T) {
merge := strings.Repeat("b", 40)
data, err := forgemsg.EncodeForge(forgemsg.Forge{V: 1, Kind: forgemsg.ForgeMerged, Repository: "novox/mesh-contracts", Base: "main", Merge: merge, Head: merge, Number: 14, Root: strings.Repeat("d", 40)})
if err != nil {
t.Fatal(err)
}
src := &memSource{vals: map[string]Source{"main": {Heads: map[string]string{"novox/mesh-contracts": strings.Repeat("a", 40)}}}}
h := &Holder{Now: time.Now, Source: src, Forge: &fakeForge{}, Logf: func(string, ...any) {}}
key := strings.Replace(strings.Replace(forgeMergedEvent, "*", "novox", 1), "*", "mesh-contracts", 1)
if err := take(h, stdio.Envelope{Key: key, Body: data}); err != nil || src.vals["main"].Heads["novox/mesh-contracts"] != merge {
t.Fatalf("event %q: %v, source %+v", key, err, src.vals["main"])
}
}
// Design 47 G A-464c: a head moves only forward and only when it changes. A merged message for a commit that does not contain
// the held head (replayed, late, out of order) moves nothing and is said; a second message for the same head writes no
// revision; and a take-in.taken for a head a merged already moved writes none and says no source-moved.
func TestAHeadMovesOnlyForwardAndOnlyWhenItChanges(t *testing.T) {
oldC, newC := strings.Repeat("a", 40), strings.Repeat("b", 40)
msg := func(merge string) []byte {
data, err := forgemsg.EncodeForge(forgemsg.Forge{V: 1, Kind: forgemsg.ForgeMerged, Repository: "novox/app", Base: "main", Merge: merge, Head: merge, Number: 3, Root: strings.Repeat("d", 40)})
if err != nil {
t.Fatal(err)
}
return data
}
src := &memSource{vals: map[string]Source{"main": {Heads: map[string]string{"novox/app": newC}}}}
var said, logged []string
// newC contains oldC, not the other way round.
forge := &fakeForge{contains: func(commit, in string) bool { return commit == in || (commit == oldC && in == newC) }}
h := &Holder{Now: time.Now, Source: src, Forge: forge, Logf: func(f string, a ...any) { logged = append(logged, f) },
SaySeat: func(_ string, _ any, id string) error { said = append(said, id); return nil }}
if err := h.ForgeMerged(msg(oldC)); err != nil || src.vals["main"].Heads["novox/app"] != newC || src.rev != 0 || len(said) != 0 || len(logged) == 0 {
t.Fatalf("a replayed old merge: %v, source %+v, revisions %d, said %v, logged %v: want nothing moved and a log line", err, src.vals["main"], src.rev, said, logged)
}
if err := h.ForgeMerged(msg(newC)); err != nil || src.rev != 0 || len(said) != 0 {
t.Fatalf("a second message for the same head wrote revision %d, said %v", src.rev, said)
}
// A take-in.taken naming the head a merged moved already writes no revision and says nothing.
body := []byte(`{"v":1,"kind":"taken","key":"g.feat","what":"group","group":"feat","branch":"main","source":7,` +
`"merges":[{"repository":"novox/app","number":3,"merge":"` + newC + `"}],"entries":[]}`)
if err := h.moveSource(mustTaken(t, body)); err != nil || src.rev != 0 || len(said) != 0 {
t.Fatalf("take-in for a head already moved: %v, revision %d, said %v", err, src.rev, said)
}
// A real forward move still writes one revision.
src.vals["main"] = Source{Heads: map[string]string{"novox/app": oldC}}
if err := h.ForgeMerged(msg(newC)); err != nil || src.rev != 1 || len(said) != 1 {
t.Fatalf("a forward move: %v, revision %d, said %v", err, src.rev, said)
}
}
func mustTaken(t *testing.T, body []byte) meshcatalog.TakeInTaken {
t.Helper()
var v meshcatalog.TakeInTaken
if err := json.Unmarshal(body, &v); err != nil {
t.Fatal(err)
}
return v
}
// Design 47 G A-464d: a take-in.taken arriving after a later merge moved the head leaves the head where it is.
func TestATakeInTakenAfterALaterMergeLeavesTheHead(t *testing.T) {
oldC, newC := strings.Repeat("a", 40), strings.Repeat("b", 40)
src := &memSource{vals: map[string]Source{"main": {Heads: map[string]string{"novox/app": newC}}}}
var said []string
h := &Holder{Now: time.Now, Source: src, Logf: func(string, ...any) {},
Forge: &fakeForge{contains: func(commit, in string) bool { return commit == in || (commit == oldC && in == newC) }},
SaySeat: func(_ string, _ any, id string) error { said = append(said, id); return nil }}
body := []byte(`{"v":1,"kind":"taken","key":"g.feat","what":"group","group":"feat","branch":"main","source":7,` +
`"merges":[{"repository":"novox/app","number":3,"merge":"` + oldC + `"}],"entries":[]}`)
if err := h.moveSource(mustTaken(t, body)); err != nil || src.vals["main"].Heads["novox/app"] != newC || src.rev != 0 || len(said) != 0 {
t.Fatalf("take-in of an older merge: %v, source %+v, revision %d, said %v: want the head left at the later merge", err, src.vals["main"], src.rev, said)
}
}
// fakeFindings is the gate's finding writer in memory.
type fakeFindings struct{ held []meshconditions.Finding }
func (f *fakeFindings) Write(_ context.Context, x meshconditions.Finding) error {
f.held = append(f.held, x)
return nil
}
func (f *fakeFindings) Clear(_ context.Context, _, _, _, _ string) error { f.held = nil; return nil }
func (f *fakeFindings) Held() []meshconditions.Finding { return f.held }
@@ -0,0 +1,248 @@
package main
// A watched branch starts from the forge, as a clone does (design 47 G A-401, §430, §15): at its start and on each
// registration-changed of the seat mesh-catalog, the gate writes source.<branch> for every watched branch the registrations name,
// each registered repository at the head of that branch, asked of the seat git's verb branch, the group left empty, and says
// source-moved. A repository whose head cannot be read is left out and said. A head a key holds is never changed here (A-403): from
// then on only a take-in moves it (A-34).
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// registrationChangedEvent is the seat mesh-catalog's event, as the gate's manifest consumes it.
var registrationChangedEvent = declared(meshcatalog.EventRegistrationChanged)
// Registered is what the gate reads of the seat mesh-catalog: its registrations.
type Registered interface {
Registrations() (meshcatalog.RegistrationsAnswer, error)
}
// seatCatalog reads the seat mesh-catalog through the tool runner.
type seatCatalog struct{ ask asking }
func (s seatCatalog) Registrations() (meshcatalog.RegistrationsAnswer, error) {
var a meshcatalog.RegistrationsAnswer
raw, err := s.ask("seat:"+meshcatalog.Seat+"."+meshcatalog.VerbRegistrations, meshcatalog.RegistrationsInput{})
if err != nil {
return a, err
}
answer, said, ok := answerOf(raw)
if !ok {
return a, fmt.Errorf("registrations refused: %s", lastLine(said))
}
if err := json.Unmarshal(answer, &a); err != nil {
return a, fmt.Errorf("the answer of registrations is not readable: %w", err)
}
return a, nil
}
// Branch is the seat git's verb branch: a repository's branch head.
func (s seatGit) Branch(repository, branch string) (gitseat.BranchAnswer, error) {
var a gitseat.BranchAnswer
return a, s.call(gitseat.VerbBranch, gitseat.BranchInput{Repository: repository, Branch: branch}, &a)
}
// Contains is whether commit is the commit in or an ancestor of it.
func (s seatGit) Contains(repository, commit, in string) (bool, error) {
var a gitseat.ContainsAnswer
if err := s.call(gitseat.VerbContains, gitseat.ContainsInput{Repository: repository, Commit: commit, In: in}, &a); err != nil {
return false, err
}
if a.Error != "" {
return false, errors.New(a.Error)
}
return a.Contains, nil
}
// watchedRepositories are the repositories each watched branch is registered for, each once, sorted.
func watchedRepositories(regs []meshcatalog.Registration) map[string][]string {
seen := map[string]map[string]bool{}
for _, r := range regs {
if strings.TrimSpace(r.Origin) == "" {
continue
}
for _, b := range r.Branches {
if strings.TrimSpace(b) == "" {
continue
}
if seen[b] == nil {
seen[b] = map[string]bool{}
}
seen[b][r.Origin] = true
}
}
out := map[string][]string{}
for b, rs := range seen {
for r := range rs {
out[b] = append(out[b], r)
}
sort.Strings(out[b])
}
return out
}
// StartSources adds to the source key of every watched branch the registrations name each registered repository it lacks, and answers the
// branches it wrote. A registrations answer that cannot be read is an error (the event is offered again); a repository whose
// head cannot be read is said and left out.
func (h *Holder) StartSources() ([]string, error) {
started, _, err := h.startSources("")
return started, err
}
// startSleep waits between the start's passes; a test replaces it.
var startSleep = time.Sleep
// errRegistrationsUnread is a start pass's error when the registrations cannot be read.
var errRegistrationsUnread = errors.New("the registrations cannot be read")
// startSourcesAtStart runs the start pass once (A-442), and asks again, as switchToRecords does (2 s doubling to a minute),
// only while the registrations cannot be read: the seat mesh-catalog may serve after the gate does. A repository a pass
// leaves out is said by the pass, and added at the next registration change or by gate_source_start (A-401).
func startSourcesAtStart(h *Holder) {
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
started, leftOut, err := h.startSources("")
if err == nil || !errors.Is(err, errRegistrationsUnread) {
h.logf("[mesh-delivery] the sources started from the forge at the start: %v; left out %v; %v", started, leftOut, err)
return
}
h.logf("[mesh-delivery] the registrations cannot be read yet (%v); asking again in %s", err, wait)
startSleep(wait)
}
}
// startSources starts the branches no key holds, or only branch when it is named; it answers the branches written and the
// repositories left out as "<repository>@<branch>".
func (h *Holder) startSources(only string) (started, leftOut []string, err error) {
if h.Source == nil || h.Catalog == nil || h.Forge == nil {
return nil, nil, errors.New("this gate keeps no source, reads no registrations or reads no forge")
}
h.startMu.Lock()
defer h.startMu.Unlock()
regs, err := h.Catalog.Registrations()
if err != nil {
return nil, nil, fmt.Errorf("%w: %w", errRegistrationsUnread, err)
}
watched := watchedRepositories(regs.Registrations)
if only != "" {
if _, ok := watched[only]; !ok {
return nil, nil, fmt.Errorf("no registration names the branch %s", only)
}
}
names := make([]string, 0, len(watched))
for b := range watched {
if only == "" || b == only {
names = append(names, b)
}
}
sort.Strings(names)
var errs []error
for _, branch := range names {
held, _, err := h.Source.Read(branch)
if err != nil {
errs = append(errs, err)
continue
}
heads := map[string]string{}
for _, repository := range watched[branch] {
if held != nil {
if _, has := held.Heads[repository]; has {
continue // a head a key holds moves by a take-in alone (A-34)
}
}
a, err := h.Forge.Branch(repository, branch)
if err == nil && a.Error != "" {
err = errors.New(a.Error)
}
if err == nil && a.Head == "" {
err = errors.New("the forge named no head")
}
if err != nil {
leftOut = append(leftOut, repository+"@"+branch)
h.logf("[mesh-delivery] source.%s starts without %s: its head cannot be read (%v); it is added at the next registration change or by gate_source_start", branch, repository, err)
continue
}
heads[repository] = a.Head
}
if len(heads) == 0 {
continue
}
wrote := false
rev, err := h.Source.Set(branch, func(prev *Source) (Source, bool) {
next := Source{Heads: map[string]string{}}
if prev != nil {
next.Group = prev.Group
for r, c := range prev.Heads {
next.Heads[r] = c
}
}
wrote = false
for r, c := range heads {
if _, has := next.Heads[r]; !has { // a held head is never changed
next.Heads[r] = c
wrote = true
}
}
return next, wrote
})
if err != nil {
errs = append(errs, fmt.Errorf("source.%s cannot be started: %w", branch, err))
continue
}
if !wrote {
continue
}
started = append(started, branch)
if h.SaySeat == nil {
continue
}
at := h.Now().UTC()
if err := h.SaySeat(seat.SubjectSourceMoved(keys.EscapeName(branch)),
seat.SourceMovedEvent{Branch: branch, Revision: int64(rev), At: &at},
"source-moved."+keys.EscapeName(branch)+"."+strconv.FormatUint(rev, 10)); err != nil {
errs = append(errs, err)
}
}
return started, leftOut, errors.Join(errs...)
}
func (h *Holder) logf(format string, a ...any) {
if h.Logf != nil {
h.Logf(format, a...)
}
}
// SourceStart starts one watched branch by hand, after the hand-act log took it: the same as the start does.
func (h *Holder) SourceStart(branch, why, by string, acts handActs) (map[string]any, error) {
if strings.TrimSpace(branch) == "" || strings.TrimSpace(why) == "" {
return nil, errors.New("say the branch and why its source is started")
}
if h.Source == nil || h.Catalog == nil || h.Forge == nil {
return nil, errors.New("this gate keeps no source, reads no registrations or reads no forge")
}
if acts == nil {
return nil, errors.New("the hand-act log cannot be reached here, so nothing was started")
}
if _, err := acts.Record(meshhandactsclient.Request{What: "started the source of the branch " + branch,
Why: why, Cause: "gate_source_start", Verb: "source-start"}); err != nil {
return nil, fmt.Errorf("the hand act was not recorded through mesh-hand-acts, so nothing was started: %w", err)
}
started, leftOut, err := h.startSources(branch)
out := map[string]any{"branch": branch, "started": started, "left-out": leftOut, "by": by}
if err == nil && len(started) == 0 && len(leftOut) == 0 {
out["note"] = "the key names every registered repository already: only a take-in moves a head"
}
return out, err
}
@@ -79,5 +79,26 @@ func sourceTools(h *Holder, acts handActs) []stdio.Tool {
Run: func(a map[string]any) (any, error) { Run: func(a map[string]any) (any, error) {
return h.SourceAgain(strArg(a, "branch"), strArg(a, "why"), stdio.CallerOf(a), acts) return h.SourceAgain(strArg(a, "branch"), strArg(a, "why"), stdio.CallerOf(a), acts)
}}, }},
{Name: "gate_source_start",
Description: "Start the source of one watched branch the registrations name and no source key holds yet, from the forge: each " +
"registered repository at the branch's head, the group empty, then source-moved; recorded in the hand-act log of the seat " +
"mesh-hand-acts first. A key the gate holds is never rewritten.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"branch": str("the watched branch"), "why": str("why it is started by hand, in plain words")},
"required": []string{"branch", "why"}},
Run: func(a map[string]any) (any, error) {
return h.SourceStart(strArg(a, "branch"), strArg(a, "why"), stdio.CallerOf(a), acts)
}},
{Name: "gate_source_follow",
Description: "Move the head one watched branch's source names for one repository to the forge's head of that branch now, as " +
"the gate does when it hears a merge and at its start (design 47 G A-464); recorded in the hand-act log of the seat " +
"mesh-hand-acts first. Nothing is moved when the source already names the forge's head.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"branch": str("the watched branch"), "repository": str("the repository, owner/name"),
"why": str("why its head follows the forge by hand, in plain words")},
"required": []string{"branch", "repository", "why"}},
Run: func(a map[string]any) (any, error) {
return h.SourceFollow(strArg(a, "branch"), strArg(a, "repository"), strArg(a, "why"), stdio.CallerOf(a), acts)
}},
} }
} }
@@ -1,8 +1,13 @@
package main package main
import ( import (
"encoding/json"
"errors" "errors"
"fmt"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
stdio "git.novox.be/novox/mesh-sdk/go"
"reflect"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -103,3 +108,204 @@ func TestTheSourceAnswerNamesTheBranchItWasAskedAndStatusSaysEveryBranch(t *test
t.Fatalf("status says every branch's source: %+v", h.SourceAll()) t.Fatalf("status says every branch's source: %+v", h.SourceAll())
} }
} }
type fakeCatalog struct {
regs []meshcatalog.Registration
err error
}
func (f fakeCatalog) Registrations() (meshcatalog.RegistrationsAnswer, error) {
return meshcatalog.RegistrationsAnswer{Registrations: f.regs}, f.err
}
// Design 47 G A-401: a watched branch starts from the forge as a clone does: each registered repository at the branch's head,
// the group empty, source-moved said; a repository whose head cannot be read is left out; a key held is never rewritten.
func TestAWatchedBranchStartsFromTheForge(t *testing.T) {
regs := []meshcatalog.Registration{
{Module: "car-hunter", Origin: "o/car-hunter", Branches: []string{"main"}},
{Module: "mesh-a", Origin: "o/mesh", Branches: []string{"main"}},
{Module: "mesh-b", Origin: "o/mesh", Branches: []string{"main"}}, // the same repository twice: once
{Module: "f1", Origin: "o/f1-circuits", Branches: []string{"master"}},
{Module: "l2p", Origin: "o/link2pay", Branches: []string{"nox-mesh"}},
{Module: "ph", Origin: "o/photos", Branches: []string{"nox-mesh", "main"}},
}
forge := &fakeForge{heads: map[string]string{
"o/car-hunter@main": "c1", "o/mesh@main": "c2", "o/f1-circuits@master": "c3", "o/link2pay@nox-mesh": "c4",
"o/photos@main": "c6", // o/photos@nox-mesh cannot be read
}}
for _, c := range []struct {
name string
held map[string]Source
catalog fakeCatalog
only string
started []string
heads map[string]map[string]string
events int
wantErr bool
}{
{name: "every branch no key holds", catalog: fakeCatalog{regs: regs},
started: []string{"main", "master", "nox-mesh"}, events: 3,
heads: map[string]map[string]string{"main": {"o/car-hunter": "c1", "o/mesh": "c2", "o/photos": "c6"},
"master": {"o/f1-circuits": "c3"}, "nox-mesh": {"o/link2pay": "c4"}}},
{name: "a held key lacking repositories gets them, its heads unchanged", catalog: fakeCatalog{regs: regs},
held: map[string]Source{"main": {Group: "g", Heads: map[string]string{"o/mesh": "old"}}},
started: []string{"main", "master", "nox-mesh"}, events: 3,
heads: map[string]map[string]string{"main": {"o/mesh": "old", "o/car-hunter": "c1", "o/photos": "c6"}, "master": {"o/f1-circuits": "c3"}, "nox-mesh": {"o/link2pay": "c4"}}},
{name: "a held key lacking one repository", catalog: fakeCatalog{regs: regs[:2]},
held: map[string]Source{"main": {Group: "g", Heads: map[string]string{"o/mesh": "old"}}},
started: []string{"main"}, events: 1,
heads: map[string]map[string]string{"main": {"o/mesh": "old", "o/car-hunter": "c1"}}},
{name: "a held key naming all: nothing written or said", catalog: fakeCatalog{regs: regs[:2]},
held: map[string]Source{"main": {Group: "g", Heads: map[string]string{"o/mesh": "old", "o/car-hunter": "older"}}},
events: 0,
heads: map[string]map[string]string{"main": {"o/mesh": "old", "o/car-hunter": "older"}}},
{name: "by hand, one branch", catalog: fakeCatalog{regs: regs}, only: "master",
started: []string{"master"}, events: 1, heads: map[string]map[string]string{"master": {"o/f1-circuits": "c3"}}},
{name: "registrations unreadable", catalog: fakeCatalog{err: errors.New("away")}, wantErr: true},
{name: "no registration names the branch", catalog: fakeCatalog{regs: regs}, only: "nope", wantErr: true},
} {
t.Run(c.name, func(t *testing.T) {
src := &branchSource{memSource{vals: map[string]Source{}}}
for b, s := range c.held {
src.vals[b] = s
}
events := 0
var logged []string
h := &Holder{Now: time.Now, Source: src, Forge: forge, Catalog: c.catalog,
Logf: func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) },
SaySeat: func(subject string, body any, id string) error {
events++
if ev := body.(seat.SourceMovedEvent); ev.Group != "" {
t.Errorf("a source-moved for started heads names no group (A-404), got %q", ev.Group)
}
return nil
}}
started, _, err := h.startSources(c.only)
if (err != nil) != c.wantErr {
t.Fatalf("err %v", err)
}
if c.wantErr {
return
}
if !reflect.DeepEqual(started, c.started) || events != c.events {
t.Fatalf("started %v (%d events), want %v (%d)", started, events, c.started, c.events)
}
for b, want := range c.heads {
if got := src.vals[b].Heads; !reflect.DeepEqual(got, want) {
t.Errorf("source.%s heads %v, want %v", b, got, want)
}
}
for b, h := range c.held {
if src.vals[b].Group != h.Group {
t.Errorf("source.%s keeps the group %q a take-in wrote, got %q", b, h.Group, src.vals[b].Group)
}
}
if c.only == "" && c.held == nil && !strings.Contains(strings.Join(logged, "\n"), "o/photos") {
t.Errorf("the repository left out was not said: %v", logged)
}
})
}
}
// Design 47 G A-401: gate_source_start is recorded in the hand-act log first, and starts nothing when the log refuses it.
func TestSourceStartByHandIsRecordedFirst(t *testing.T) {
src := &branchSource{memSource{vals: map[string]Source{}}}
h := &Holder{Now: time.Now, Source: src, Forge: &fakeForge{heads: map[string]string{"o/f1@master": "c3"}},
Catalog: fakeCatalog{regs: []meshcatalog.Registration{{Module: "f1", Origin: "o/f1", Branches: []string{"master"}}}}}
acts := &handActsFake{refuse: errors.New("log down")}
if _, err := h.SourceStart("master", "newly watched", "me", acts); err == nil || len(src.vals) != 0 {
t.Fatalf("started without the log: %v %v", err, src.vals)
}
acts.refuse = nil
if _, err := h.SourceStart("master", "", "me", acts); err == nil {
t.Fatal("started with no reason")
}
if _, err := h.SourceStart("master", "newly watched", "me", acts); err != nil {
t.Fatal(err)
}
if len(acts.took) != 1 || acts.took[0].Cause != "gate_source_start" || src.vals["master"].Heads["o/f1"] != "c3" {
t.Fatalf("log %+v source %+v", acts.took, src.vals)
}
}
// Design 47 G A-401: the registration-changed branch of take and the start pass both write the new branch's key and say
// source-moved; removing either leaves the branch unstarted.
func TestRegistrationChangedAndTheStartPassStartABranch(t *testing.T) {
regs := []meshcatalog.Registration{{Module: "f1", Origin: "o/f1", Branches: []string{"master"}}}
for _, c := range []struct {
name string
run func(h *Holder) error
}{
{"registration-changed", func(h *Holder) error {
return take(h, stdio.Envelope{Key: registrationChangedEvent, Body: json.RawMessage(`{}`)})
}},
{"the start pass", func(h *Holder) error { startSourcesAtStart(h); return nil }},
} {
t.Run(c.name, func(t *testing.T) {
if registrationChangedEvent == "" {
t.Fatal("the manifest does not consume registration-changed")
}
src := &branchSource{memSource{vals: map[string]Source{}}}
var subjects []string
h := &Holder{Now: time.Now, Source: src, Forge: &fakeForge{heads: map[string]string{"o/f1@master": "c3"}},
Catalog: fakeCatalog{regs: regs}, Logf: func(string, ...any) {},
SaySeat: func(subject string, body any, id string) error { subjects = append(subjects, subject); return nil }}
if err := c.run(h); err != nil {
t.Fatal(err)
}
if src.vals["master"].Heads["o/f1"] != "c3" || len(subjects) != 1 || !strings.HasSuffix(subjects[0], ".event.source-moved.master") {
t.Fatalf("source %+v events %v", src.vals, subjects)
}
})
}
}
// Design 47 G A-442: the start pass runs once; it asks again, doubling its wait, only while the registrations cannot be read,
// and a repository left out is said, not waited for.
func TestTheStartAsksAgainOnlyWhileTheRegistrationsCannotBeRead(t *testing.T) {
regs := []meshcatalog.Registration{{Module: "f1", Origin: "o/f1", Branches: []string{"master"}}}
for _, c := range []struct {
name string
unreadFor int // passes the registrations cannot be read
forgeFail bool // the forge cannot name the head
waits []time.Duration
started bool
}{
{"registrations unread twice", 2, false, []time.Duration{2 * time.Second, 4 * time.Second}, true},
{"a repository left out is not waited for", 0, true, nil, false},
} {
t.Run(c.name, func(t *testing.T) {
var waits []time.Duration
old := startSleep
startSleep = func(d time.Duration) { waits = append(waits, d) }
defer func() { startSleep = old }()
src := &branchSource{memSource{vals: map[string]Source{}}}
forge := &fakeForge{fail: c.forgeFail, heads: map[string]string{"o/f1@master": "c3"}}
cat := &flakyCatalog{fakeCatalog: fakeCatalog{regs: regs}, failFor: c.unreadFor}
var said []string
h := &Holder{Now: time.Now, Source: src, Forge: forge, Catalog: cat,
Logf: func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }}
startSourcesAtStart(h)
if !reflect.DeepEqual(waits, c.waits) || (src.vals["master"].Heads["o/f1"] == "c3") != c.started {
t.Fatalf("waits %v source %+v", waits, src.vals)
}
if c.forgeFail && !strings.Contains(strings.Join(said, "\n"), "o/f1@master") {
t.Fatalf("the left-out repository is not said: %v", said)
}
})
}
}
// flakyCatalog cannot read the registrations for its first failFor asks.
type flakyCatalog struct {
fakeCatalog
failFor int
}
func (f *flakyCatalog) Registrations() (meshcatalog.RegistrationsAnswer, error) {
if f.failFor > 0 {
f.failFor--
return meshcatalog.RegistrationsAnswer{}, errors.New("not served yet")
}
return f.fakeCatalog.Registrations()
}
@@ -9,6 +9,7 @@ import (
mdc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshdeliveryclient" mdc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshdeliveryclient"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/keys" "git.novox.be/novox/mesh-sdk/go/keys"
) )
@@ -107,9 +108,43 @@ func (h *Holder) moveSource(t meshcatalog.TakeInTaken) error {
for _, m := range t.Merges { for _, m := range t.Merges {
heads[m.Repository] = m.Merge heads[m.Repository] = m.Merge
} }
// A head moves only forward (A-464d): a repository whose held head is not contained in the merge is left where it is.
if held, _, err := h.Source.Read(t.Branch); err != nil {
return err
} else if held != nil && h.Forge != nil {
for r, c := range heads {
for name, was := range held.Heads {
if !trunk.SameRepository(name, r) || was == c {
continue
}
ahead, err := h.Forge.Contains(name, was, c)
if err != nil {
return fmt.Errorf("whether %.12s is in %.12s of %s cannot be read, so the head stays: %w", was, c, name, err)
}
if !ahead {
h.logf("[mesh-delivery] source.%s keeps %s at %.12s: the take-in's merge %.12s does not contain it, left without effect", t.Branch, name, was, c)
delete(heads, r)
}
}
}
if len(heads) == 0 {
return nil
}
}
wrote := false
rev, err := h.Source.Set(t.Branch, func(prev *Source) (Source, bool) { rev, err := h.Source.Set(t.Branch, func(prev *Source) (Source, bool) {
wrote = false
next := Source{Group: t.Group, Heads: heads} next := Source{Group: t.Group, Heads: heads}
if prev != nil { if prev != nil {
same := true
for r, c := range heads {
if prev.Heads[r] != c {
same = false
}
}
if same {
return Source{}, false // a merged message moved these heads already: no revision (A-464c)
}
next.Heads = map[string]string{} next.Heads = map[string]string{}
for r, c := range prev.Heads { for r, c := range prev.Heads {
next.Heads[r] = c next.Heads[r] = c
@@ -118,12 +153,13 @@ func (h *Holder) moveSource(t meshcatalog.TakeInTaken) error {
next.Heads[r] = c next.Heads[r] = c
} }
} }
wrote = true
return next, true return next, true
}) })
if err != nil { if err != nil {
return fmt.Errorf("source.%s cannot be moved: %w", t.Branch, err) return fmt.Errorf("source.%s cannot be moved: %w", t.Branch, err)
} }
if h.SaySeat == nil { if h.SaySeat == nil || !wrote {
return nil return nil
} }
at := h.Now().UTC() at := h.Now().UTC()
@@ -84,7 +84,7 @@ func TestATakeInTakenOpensOneRecordAndMovesItsBranchsSource(t *testing.T) {
if s := src.vals["main"]; s.Group != "feat" || s.Heads["novox/app"] != merge { if s := src.vals["main"]; s.Group != "feat" || s.Heads["novox/app"] != merge {
t.Fatalf("source.main %+v", s) t.Fatalf("source.main %+v", s)
} }
if len(events) != 2 || !strings.HasSuffix(events[0].subject, ".event.source-moved.main") { if len(events) != 1 || !strings.HasSuffix(events[0].subject, ".event.source-moved.main") {
t.Fatalf("said %+v", events) t.Fatalf("said %+v", events)
} }
drop := json.RawMessage(`{"v":1,"kind":"taken","key":"m.00000000000000000002","what":"revision","dropped":true,"branch":"main","source":8,` + drop := json.RawMessage(`{"v":1,"kind":"taken","key":"m.00000000000000000002","what":"revision","dropped":true,"branch":"main","source":8,` +
@@ -92,7 +92,7 @@ func TestATakeInTakenOpensOneRecordAndMovesItsBranchsSource(t *testing.T) {
if err := c.Hear(stdio.Envelope{Key: "mesh-delivery-indexer.take-in.taken", Body: drop}); err != nil { if err := c.Hear(stdio.Envelope{Key: "mesh-delivery-indexer.take-in.taken", Body: drop}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if kept, _ := h.Records.Kept(); len(kept) != 1 || len(events) != 2 { if kept, _ := h.Records.Kept(); len(kept) != 1 || len(events) != 1 {
t.Fatalf("a hand drop opened %d records and said %d events", len(kept), len(events)) t.Fatalf("a hand drop opened %d records and said %d events", len(kept), len(events))
} }
} }
+7 -1
View File
@@ -22,6 +22,8 @@
"git.pull-updated", "git.pull-updated",
"git.pull-merged", "git.pull-merged",
"git.pull-closed", "git.pull-closed",
"mesh-catalog.registration-changed",
"mesh-delivery-forge-bridge.forge.*.*.merged",
"mesh-conditions.findings-asked" "mesh-conditions.findings-asked"
], ],
"emits": [ "emits": [
@@ -351,8 +353,10 @@
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg": "0.1" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg": "0.1"
}, },
"invokes": [ "invokes": [
"seat:git.branch",
"seat:git.open-pulls", "seat:git.open-pulls",
"seat:git.pull", "seat:git.pull",
"seat:mesh-catalog.registrations",
"seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-act",
"seat:mesh-wants.entry", "seat:mesh-wants.entry",
"seat:node-engine.composition" "seat:node-engine.composition"
@@ -379,7 +383,9 @@
"gate_record_repair", "gate_record_repair",
"gate_records", "gate_records",
"gate_source", "gate_source",
"gate_source_again" "gate_source_again",
"gate_source_follow",
"gate_source_start"
], ],
"config": { "config": {
"record-retention": { "record-retention": {
@@ -2,6 +2,19 @@
// its own, on its own bus account (`runs-as`), so that its account alone writes the hand-act log (`acts`, its module // its own, on its own bus account (`runs-as`), so that its account alone writes the hand-act log (`acts`, its module
// log: a stream that keeps every act in order) and `warrants`. It reads the router's record of each ask for a warrant; the controller's log is no longer read // log: a stream that keeps every act in order) and `warrants`. It reads the router's record of each ask for a warrant; the controller's log is no longer read
// (issue 774). stdout is the MCP channel; what this module says, it says on stderr. // (issue 774). stdout is the MCP channel; what this module says, it says on stderr.
// Its first change delivered through its own pipeline: lint, unit, integration and composition, then its want (design 47 G).
// Its second, with each step working on this module alone (principle 10).
// Its third, once the runner names this module's directory on every step.
// Its fourth, once the runner hears each step end and the assigner finds the node-engine.
// Its fifth, once each pipeline runs at its own pinned catalog revision.
// Its sixth, once the node-engine composes a pipeline's revision.
// Its seventh, once the node-engine may read that revision.
// Its eighth, once the tool that answers it lets the node-engine read it.
// Its ninth, once the node-engine reads each manifest digest in the form its state keys use.
// Its tenth, once the composed input travels in chunks.
// Its eleventh, once a machine whose node-engine is not heard running composes with the catalog's node-engine.
// Its twelfth, once the start branches are merged and taken in.
// Its thirteenth, once the assigner hears a composition run's end on the kept seat events.
package main package main
import ( import (
@@ -601,14 +601,14 @@ func TestAnAskIsRefusedInWordsToItsAsker(t *testing.T) {
} }
} }
all := strings.Join(refused, "\n") all := strings.Join(refused, "\n")
// A fourth offered ask is never refused (ADR 0666 §12): it is taken and the asker's oldest is held back. // A fourth offered ask is never refused and none is held back (design 47 G A-424).
for _, want := range []string{"d-1: the ask is refused: an ask that destroys lasts at most", "d-2: its words carry"} { for _, want := range []string{"d-1: the ask is refused: an ask that destroys lasts at most", "d-2: its words carry"} {
if !strings.Contains(all, want) { if !strings.Contains(all, want) {
t.Errorf("not refused: %q in\n%s", want, all) t.Errorf("not refused: %q in\n%s", want, all)
} }
} }
if strings.Contains(all, "d-6") || store.asks["mesh-delivery.d-6"].State != StateOpen || store.asks["mesh-delivery.d-3"].State != StateHeldBack { if strings.Contains(all, "d-6") || store.asks["mesh-delivery.d-6"].State != StateOpen || store.asks["mesh-delivery.d-3"].State != StateOpen {
t.Errorf("the fourth ask was refused or the oldest not held back: d-6 %s, d-3 %s", store.asks["mesh-delivery.d-6"].State, store.asks["mesh-delivery.d-3"].State) t.Errorf("the fourth ask was refused or the oldest held back: d-6 %s, d-3 %s", store.asks["mesh-delivery.d-6"].State, store.asks["mesh-delivery.d-3"].State)
} }
// A refused ask is kept as a closed record with its outcome and words (design 47 G §570), and `asks` lists it refused. // A refused ask is kept as a closed record with its outcome and words (design 47 G §570), and `asks` lists it refused.
if rec, kept := store.asks["mesh-delivery.d-1"]; !kept || rec.State != string(occ.OutcomeRefused) || rec.Warrant == nil || if rec, kept := store.asks["mesh-delivery.d-1"]; !kept || rec.State != string(occ.OutcomeRefused) || rec.Warrant == nil ||
+3 -11
View File
@@ -563,9 +563,9 @@ func (h *Holder) TakeAsk(verb, asker string, body json.RawMessage) error {
h.refuseAsk(asker, a, "its whole words carry "+refusal.String()+", which may not leave the mesh on any channel: withhold it") h.refuseAsk(asker, a, "its whole words carry "+refusal.String()+", which may not leave the mesh on any channel: withhold it")
return nil return nil
} }
// A fourth offered ask is taken, never refused: the asker's oldest offered one is held back once this one is recorded // Every ask is offered when it is raised: the channel keeps no count of an asker's open asks and holds none back for room
// (holdBackOldest). An ask with the same `about` replaces the open one, a held-back one too. // (design 47 G A-424). An ask with the same `about` replaces the open one, a held-back one of before too.
open, replaced := 0, "" replaced := ""
h.mu.Lock() h.mu.Lock()
for k, e := range h.asks { for k, e := range h.asks {
if e.rec.Asker != asker || !liveState(e.rec.State) { if e.rec.Asker != asker || !liveState(e.rec.State) {
@@ -573,14 +573,9 @@ func (h *Holder) TakeAsk(verb, asker string, body json.RawMessage) error {
} }
if a.About != "" && e.rec.Ask.About == a.About { if a.About != "" && e.rec.Ask.About == a.About {
replaced = k replaced = k
continue
}
if e.rec.State != StateHeldBack {
open++
} }
} }
h.mu.Unlock() h.mu.Unlock()
holdBack := open >= occ.MostOpen
rec := AskRecord{Asker: asker, Ask: a, State: StateOpen, Opened: now, Tickets: h.ticketsFor(a)} rec := AskRecord{Asker: asker, Ask: a, State: StateOpen, Opened: now, Tickets: h.ticketsFor(a)}
if len(rec.Tickets) == 0 { if len(rec.Tickets) == 0 {
// Held, not refused (ADR 0397): the ask breaks no rule, and the operator is asked when a channel can carry it. // Held, not refused (ADR 0397): the ask breaks no rule, and the operator is asked when a channel can carry it.
@@ -611,9 +606,6 @@ func (h *Holder) TakeAsk(verb, asker string, body json.RawMessage) error {
} else { } else {
h.keepAsk(key, rec, 0) h.keepAsk(key, rec, 0)
} }
if holdBack {
h.holdBackOldest(asker, key)
}
if rec.State == StateHeld { if rec.State == StateHeld {
h.Logf("[messenger] %s asks %q (%s): held, not shown: %s", asker, a.Headline, a.ID, rec.HeldWhy) h.Logf("[messenger] %s asks %q (%s): held, not shown: %s", asker, a.Headline, a.ID, rec.HeldWhy)
return nil return nil
+4 -3
View File
@@ -105,7 +105,8 @@ func TestAHeldAskExpiresAsAnyOpenAskDoes(t *testing.T) {
} }
} }
func TestAHeldAskCountsTowardTheOffersOfItsAskerAndTheOldestIsHeldBack(t *testing.T) { // Design 47 G A-424: four held asks of one asker are taken and held for want of a channel, and none is held back for room.
func TestFourHeldAsksOfOneAskerAreHeldAndNoneHeldBack(t *testing.T) {
h, _, _, c, store, w := newRouter(t) h, _, _, c, store, w := newRouter(t)
withoutVerifiedSender(h) withoutVerifiedSender(h)
for i := 1; i <= 4; i++ { for i := 1; i <= 4; i++ {
@@ -114,8 +115,8 @@ func TestAHeldAskCountsTowardTheOffersOfItsAskerAndTheOldestIsHeldBack(t *testin
submit(t, h, "mesh-delivery", a) submit(t, h, "mesh-delivery", a)
c.pass(time.Second) c.pass(time.Second)
} }
if got := w.of("mesh-delivery"); len(got) != 0 || store.asks["mesh-delivery.d-1"].State != StateHeldBack || store.asks["mesh-delivery.d-4"].State != StateHeld { if got := w.of("mesh-delivery"); len(got) != 0 || store.asks["mesh-delivery.d-1"].State != StateHeld || store.asks["mesh-delivery.d-4"].State != StateHeld {
t.Fatalf("%+v: a fourth is taken, never refused; the oldest is held back (%s)", got, store.asks["mesh-delivery.d-1"].State) t.Fatalf("%+v: a fourth is taken, never refused, and none is held back (d-1 %s)", got, store.asks["mesh-delivery.d-1"].State)
} }
} }
+32 -52
View File
@@ -5,14 +5,12 @@ import (
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel" oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
"sort" "sort"
"time" "time"
occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
) )
// An ask held back (novox/hq ADR 0666 §12, design 46 § 6): an asker has at most MostOpen asks offered at once; a fourth // An ask held back (novox/hq ADR 0666 §12, design 46 § 6) was one kept open behind newer asks of its asker, offered again when
// is offered and nothing is refused, so no ask is hidden: the asker's oldest offered ask is held back, its tickets // one of them ended. The channel holds none back any more (design 47 G A-424): every ask is offered when it is raised, and one
// withdrawn and its message edited to say it waits behind newer asks, and it is offered again, with new tickets, when // held back before the change is offered at the router's start (offerAllHeldBack) and by ask-again. `held` is another state, an
// one of the asker's offered asks ends. `held` is another state, an ask no ready channel can carry (ADR 0397). // ask no ready channel can carry (ADR 0397).
// StateHeldBack is an ask kept open behind newer asks of its asker. // StateHeldBack is an ask kept open behind newer asks of its asker.
const StateHeldBack = "held-back" const StateHeldBack = "held-back"
@@ -90,50 +88,35 @@ func (h *Holder) freshAsk(key string) (AskRecord, uint64, bool) {
return rec, rev, true return rec, rev, true
} }
// holdBackOldest holds back the asker's oldest offered ask other than except (the newest, just taken): its tickets // offerHeldBack offers the asker's held-back asks of before the change again (A-424): there is no room to wait for. The work
// are withdrawn, it stays open, and its own message is edited to say it waits. One held for want of a channel counts as offered // lock is held.
// and is held back the same way. The work lock is held. func (h *Holder) offerHeldBack(asker string) {
func (h *Holder) holdBackOldest(asker, except string) { for {
h.mu.Lock() h.mu.Lock()
key := h.pickLocked(asker, except, StateOpen, StateHeld) key := h.pickLocked(asker, "", StateHeldBack)
h.mu.Unlock() h.mu.Unlock()
if key == "" { if key == "" {
return return
}
h.offerOneHeldBack(key)
if after := h.askOf(key); after == nil || after.rec.State == StateHeldBack {
return // not offered (its record changed meanwhile): left for the next start rather than tried forever
}
} }
rec, rev, ok := h.freshAsk(key)
if !ok || (rec.State != StateOpen && rec.State != StateHeld) {
return
}
now := h.Now()
rec.State, rec.HeldBackSince, rec.Tickets, rec.HeldSince, rec.HeldWhy = StateHeldBack, now, nil, time.Time{}, ""
if !h.saveHeld(key, rec, rev) {
return
}
h.Logf("[messenger] %s's ask %q (%s) is held back behind its newer asks: it is offered again when one ends", asker, rec.Ask.Headline, rec.Ask.ID)
h.mu.Lock()
r := h.open[rec.Record]
h.mu.Unlock()
if r == nil || r.Origin != "ask" {
return // a condition's message is the newer ask's to show
}
h.mu.Lock()
r.Answered = "Waits behind newer questions of " + asker + ".\nIt is offered again, with its answers, when one of them ends."
h.mu.Unlock()
h.edit(r, "answered", false)
h.persist(r)
} }
// offerHeldBack offers the asker's oldest held-back ask again when a slot is free: new tickets, its message edited back // offerAllHeldBack offers every ask held back before the change, at each tick of the router (A-424). The work lock is held.
// to the question. The work lock is held. func (h *Holder) offerAllHeldBack() {
func (h *Holder) offerHeldBack(asker string) {
if h.offeredCount(asker) >= occ.MostOpen {
return
}
h.mu.Lock() h.mu.Lock()
key := h.pickLocked(asker, "", StateHeldBack) askers := map[string]bool{}
for _, e := range h.asks {
if e.rec.State == StateHeldBack {
askers[e.rec.Asker] = true
}
}
h.mu.Unlock() h.mu.Unlock()
if key != "" { for asker := range askers {
h.offerOneHeldBack(key) h.offerHeldBack(asker)
} }
} }
@@ -172,9 +155,9 @@ func (h *Holder) offerOneHeldBack(key string) {
h.showAsk(key, rec, now) h.showAsk(key, rec, now)
} }
// AskAgain is the seat operator-channel's verb ask-again (G §394, §450): offer a held-back ask now, holding back the // AskAgain is the seat operator-channel's verb ask-again (G §394, §450): it offers an ask held back before the channel stopped
// asker's oldest offered ask in its place when the asker already has the most offered. It answers the ask's state after it, // holding any back (A-424), and holds back nothing in its place. It answers the ask's state after it, or the error that says why
// or the error that says why it was not offered. // it was not offered.
func (h *Holder) AskAgain(key string) oc.AskAgainAnswer { func (h *Holder) AskAgain(key string) oc.AskAgainAnswer {
h.work.Lock() h.work.Lock()
defer h.work.Unlock() defer h.work.Unlock()
@@ -186,10 +169,7 @@ func (h *Holder) AskAgain(key string) oc.AskAgainAnswer {
case !ok: case !ok:
return oc.AskAgainAnswer{Error: fmt.Sprintf("this router holds no ask %q: its key is <asker>.<id>, as operator-channel.asks lists", key)} return oc.AskAgainAnswer{Error: fmt.Sprintf("this router holds no ask %q: its key is <asker>.<id>, as operator-channel.asks lists", key)}
case rec.State != StateHeldBack: case rec.State != StateHeldBack:
return oc.AskAgainAnswer{Error: fmt.Sprintf("the ask %s is %s, not held back: only a held-back ask is offered again", key, rec.State)} return oc.AskAgainAnswer{Error: fmt.Sprintf("the ask %s is %s, not held back: only a held-back ask is offered again, and the channel holds none back any more", key, rec.State)}
}
if h.offeredCount(rec.Asker) >= occ.MostOpen {
h.holdBackOldest(rec.Asker, key)
} }
h.offerOneHeldBack(key) h.offerOneHeldBack(key)
got, _, _ := h.freshAsk(key) got, _, _ := h.freshAsk(key)
@@ -9,100 +9,50 @@ import (
occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient" occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
) )
// ADR 0666 §12, design 46 §6 and test 71: an asker has at most three asks offered at once; a fourth is offered and the // Design 47 G A-424: the channel keeps no count of an asker's open asks and holds none back: every ask is offered when it is
// oldest is held back (its tickets withdrawn, its message edited, its record open), offered again with new tickets // raised, five of one asker at once too, and none is refused. An ask held back before the change is offered again at the
// when one of the three ends, offered now by messenger_ask_again; an ask with the same `about` replaces a held-back // router's next tick, with new tickets; ask-again offers one now and holds nothing back in its place.
// one, and a Decline ends an ask whose asker may ask again at once. func TestEveryAskIsOfferedAndOneHeldBackBeforeIsOfferedAgain(t *testing.T) {
func TestNoAskIsHiddenTheOldestOfFourIsHeldBackAndOfferedAgain(t *testing.T) { h, _, _, c, store, w := newRouter(t)
h, tg, _, c, store, w := newRouter(t)
link(h, store, c, "42", c.now().Add(-2*time.Hour)) link(h, store, c, "42", c.now().Add(-2*time.Hour))
ask := func(id, about string) { state := func(id string) string { return store.asks["mesh-delivery."+id].State }
for i, id := range []string{"d-1", "d-2", "d-3", "d-4", "d-5"} {
a := ackAsk(c, id) a := ackAsk(c, id)
a.About = about a.About = "about-" + string(rune('1'+i))
submit(t, h, "mesh-delivery", a) submit(t, h, "mesh-delivery", a)
c.pass(time.Second) c.pass(time.Second)
settle(h, c) settle(h, c)
} }
state := func(id string) string { return store.asks["mesh-delivery."+id].State } for _, id := range []string{"d-1", "d-2", "d-3", "d-4", "d-5"} {
for i, id := range []string{"d-1", "d-2", "d-3"} { if state(id) != StateOpen || len(store.asks["mesh-delivery."+id].Tickets) == 0 {
ask(id, "about-"+string(rune('1'+i))) t.Fatalf("%s is %s with %d tickets: every ask is offered", id, state(id), len(store.asks["mesh-delivery."+id].Tickets))
}
oldTicket := ""
for tk := range store.asks["mesh-delivery.d-1"].Tickets {
oldTicket = tk
}
ask("d-4", "about-4")
// M1: the fourth reaches the channel, the oldest is held back, none is refused.
if len(w.of("mesh-delivery")) != 0 || state("d-4") != StateOpen || state("d-1") != StateHeldBack {
t.Fatalf("refused %+v; d-4 %s, d-1 %s", w.of("mesh-delivery"), state("d-4"), state("d-1"))
}
if rec := store.asks["mesh-delivery.d-1"]; len(rec.Tickets) != 0 || rec.HeldBackSince.IsZero() {
t.Errorf("held back %+v: tickets withdrawn, since set", rec)
}
if m, ok := tg.edits["1"]; !ok || len(m.Actions) != 0 || !strings.Contains(m.Note, "Waits behind newer questions of mesh-delivery") {
t.Errorf("its message was not edited to say it waits: %v %+v", ok, m)
}
if _, err := h.Chosen("telegram", choiceOf(oldTicket, "42", c.now())); err == nil || !strings.Contains(err.Error(), "waits behind newer questions") || len(w.of("mesh-delivery")) != 0 {
t.Errorf("a tap of a withdrawn ticket: %v %+v", err, w.of("mesh-delivery"))
}
// M3: asks lists it with since when and the newer asks it waits behind; open includes it.
q, _ := AsksQueryOf(map[string]any{"key": "mesh-delivery.d-1"})
raw, _ := json.Marshal(q.Narrow(h.AsksNow()))
for _, want := range []string{`"state":"held-back"`, `"held-since":"`, `"behind":["mesh-delivery.d-2","mesh-delivery.d-3","mesh-delivery.d-4"]`} {
if !strings.Contains(string(raw), want) {
t.Errorf("asks lacks %s: %s", want, raw)
} }
} }
if q, _ = AsksQueryOf(nil); !strings.Contains(string(mustJSON(q.Narrow(h.AsksNow()))), "mesh-delivery.d-1") { // One held back before the change, as the store kept it.
t.Error("the open asks leave out the held-back one") key := "mesh-delivery.d-2"
} got, rev, _ := store.GetAsk(key)
// M5: an ask with the same about replaces a held-back one; the three offered stay three, so the oldest of them is held back. got.State, got.Tickets, got.HeldBackSince = StateHeldBack, nil, c.now()
ask("d-5", "about-1") rev, err := store.UpdateAsk(key, *got, rev)
if state("d-1") != string(occ.OutcomeReplaced) || state("d-2") != StateHeldBack || state("d-5") != StateOpen || state("d-3") != StateOpen { if err != nil {
t.Fatalf("after d-5 about the held-back ask's thing: d-1 %s d-2 %s d-3 %s d-5 %s", state("d-1"), state("d-2"), state("d-3"), state("d-5"))
}
// M2: when one of the offered ends, the oldest held back is offered again with new tickets and its message asks again.
if a := h.AskAgain("mesh-delivery.d-3"); !strings.Contains(a.Error, "not held back") {
t.Errorf("ask-again of an offered ask: %+v", a)
}
if err := h.TakeAsk("cancel", "mesh-delivery", []byte(`{"id":"d-3"}`)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
settle(h, c) h.keepAsk(key, *got, rev)
if rec := store.asks["mesh-delivery.d-2"]; rec.State != StateOpen || len(rec.Tickets) == 0 || !rec.HeldBackSince.IsZero() { if a := h.AskAgain("mesh-delivery.d-1"); !strings.Contains(a.Error, "not held back") {
t.Fatalf("d-2 after d-3 ended: %+v", rec) t.Errorf("ask-again of an offered ask: %+v", a)
}
if m := tg.edits["2"]; len(m.Actions) == 0 || strings.Contains(m.Note, "Waits behind") {
t.Errorf("its message does not ask again: %+v", m)
}
// M4: ask-again by hand offers a held-back ask now, holding back the oldest offered one; the seat operator-channel's ask-again (G §450).
ask("d-6", "about-6")
ask("d-7", "about-7")
held := ""
for _, id := range []string{"d-2", "d-4", "d-5", "d-6", "d-7"} {
if state(id) == StateHeldBack {
held = id
}
}
if held == "" {
t.Fatal("no ask is held back after two more")
} }
if a := h.AskAgain("mesh-delivery.d-none"); a.Error == "" || a.State != "" { if a := h.AskAgain("mesh-delivery.d-none"); a.Error == "" || a.State != "" {
t.Errorf("ask-again of an ask the router holds none of: %+v", a) t.Errorf("ask-again of an ask the router holds none of: %+v", a)
} }
if a := h.AskAgain("mesh-delivery." + held); a.Error != "" || a.State != string(StateOpen) || state(held) != StateOpen { h.Tick()
t.Fatalf("%s is %s; ask-again answered %+v", held, state(held), a) settle(h, c)
if rec := store.asks[key]; rec.State != StateOpen || len(rec.Tickets) == 0 || !rec.HeldBackSince.IsZero() {
t.Fatalf("the ask held back before the change, after a tick: %+v", rec)
} }
n := 0 for _, id := range []string{"d-1", "d-3", "d-4", "d-5"} {
for _, id := range []string{"d-2", "d-4", "d-5", "d-6", "d-7"} { if state(id) != StateOpen {
if state(id) == StateHeldBack { t.Errorf("%s is %s after the held-back one was offered: nothing is held back in its place", id, state(id))
n++
} }
} }
if n != 2 || state("d-5") != StateHeldBack {
t.Errorf("%d asks held back after ask-again (d-5 %s): the oldest offered one took its place", n, state("d-5"))
}
// A Decline ends an ask; its asker may ask again at once, and nothing is refused.
for _, wt := range w.of("mesh-delivery") { for _, wt := range w.of("mesh-delivery") {
if wt.Outcome == occ.OutcomeRefused { if wt.Outcome == occ.OutcomeRefused {
t.Errorf("refused: %+v", wt) t.Errorf("refused: %+v", wt)
@@ -778,6 +778,7 @@ func (h *Holder) Tick() {
h.mu.Unlock() h.mu.Unlock()
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) }) sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
h.sweepAsks(now) h.sweepAsks(now)
h.offerAllHeldBack() // held back before the channel stopped holding any back, offered at the first tick (design 47 G A-424)
h.releaseHeld() h.releaseHeld()
h.offerUnsent() h.offerUnsent()
h.breakGlass() h.breakGlass()
+4 -2
View File
@@ -197,13 +197,15 @@ func allTools() []stdio.Tool {
out = append(out, keeperHealthTool(StatusInState, HeldInState, jsz)) out = append(out, keeperHealthTool(StatusInState, HeldInState, jsz))
out = append(out, readTools(jsz, LiveMsgGet(LiveBus()))...) out = append(out, readTools(jsz, LiveMsgGet(LiveBus()))...)
rs := RequestsInState() rs := RequestsInState()
out = append(out, snapshotListTool(rs, bundleKeepDays())...) out = append(out, snapshotListTool(rs, bundleKeepDays)...)
out = append(out, plansTool(PlansInState)) out = append(out, plansTool(PlansInState))
return append(out, requestTools(rs)...) return append(out, requestTools(rs)...)
} }
// bundleKeepDays is the setting snapshot-keep-days as this bundle's configuration holds it (design 47 G A-112), its // bundleKeepDays is the setting snapshot-keep-days as this bundle's configuration holds it (design 47 G A-112), its
// default when the runtime gives none. // default when the runtime gives none. It is read when a tool asks, never while the tools are listed: a config read
// waits for the runtime's channel, which opens only once the bundle serves, so a read before Serve never returns and
// the bundle never answers initialize.
func bundleKeepDays() int { func bundleKeepDays() int {
if raw, err := stdio.Config("snapshot-keep-days"); err == nil { if raw, err := stdio.Config("snapshot-keep-days"); err == nil {
var n int var n int
+90 -1
View File
@@ -35,6 +35,7 @@ const (
consumerRemoved = "consumer-removed" consumerRemoved = "consumer-removed"
bucketKeysRemoved = "bucket-keys-removed" bucketKeysRemoved = "bucket-keys-removed"
streamRemoved = "stream-removed" streamRemoved = "stream-removed"
streamPurged = "stream-purged"
// maxKeysShown bounds the keys an answer lists; the count is always whole. // maxKeysShown bounds the keys an answer lists; the count is always whole.
maxKeysShown = 200 maxKeysShown = 200
) )
@@ -332,7 +333,6 @@ func RemoveStream(ctx context.Context, bus Bus, said Said, a map[string]any) (an
return out, nil return out, nil
} }
// removalTools are the module's three tools that change the bus; every other tool here reads.
// removalTools are the three removal tools. **They refuse an entry of the new form** (design 25 § 12.3; ADR 0595 §4 as // removalTools are the three removal tools. **They refuse an entry of the new form** (design 25 § 12.3; ADR 0595 §4 as
// its insight of 2026-11-03 has it): the mesh deletes a module's kept data only through the bus module's drop and // its insight of 2026-11-03 has it): the mesh deletes a module's kept data only through the bus module's drop and
// recreate, so a stream, a bucket or a consumer of a stream the keeper made (its state asked, read through newForm) is // recreate, so a stream, a bucket or a consumer of a stream the keeper made (its state asked, read through newForm) is
@@ -360,6 +360,11 @@ func removalTools(bus Bus, said Said, newForm heldEntries) []stdio.Tool {
Description: "Removes (deletes) one named JetStream stream whole, e.g. a stream the controller left: CONTROL, ASSIGNMENTS, one of its buckets as KV_<bucket> or a node's declaration files as OBJ_declaration-files-<node> (novox/hq ADR 0590 §6; a bucket's whole removal is this tool's, never nats_bucket_remove's). Needs `why`, kept in the audit trail; reads the stream first and answers its messages, bytes and consumers; refuses a stream that still has a consumer; a stream already gone is answered as such and is no error, so asking again is safe. `dry_run` only reads. The bus refuses any stream the module's credential is not granted.", Description: "Removes (deletes) one named JetStream stream whole, e.g. a stream the controller left: CONTROL, ASSIGNMENTS, one of its buckets as KV_<bucket> or a node's declaration files as OBJ_declaration-files-<node> (novox/hq ADR 0590 §6; a bucket's whole removal is this tool's, never nats_bucket_remove's). Needs `why`, kept in the audit trail; reads the stream first and answers its messages, bytes and consumers; refuses a stream that still has a consumer; a stream already gone is answered as such and is no error, so asking again is safe. `dry_run` only reads. The bus refuses any stream the module's credential is not granted.",
Input: map[string]any{"stream": str("the stream, e.g. ASSIGNMENTS"), "why": str("why it is removed, for the audit trail (required)"), "dry_run": map[string]any{"type": "boolean", "description": "read the stream, remove nothing"}}, Input: map[string]any{"stream": str("the stream, e.g. ASSIGNMENTS"), "why": str("why it is removed, for the audit trail (required)"), "dry_run": map[string]any{"type": "boolean", "description": "read the stream, remove nothing"}},
Run: run(RemoveStream)}, Run: run(RemoveStream)},
{Name: "nats_stream_purge",
Description: "Purges (removes) the messages of one named JetStream stream below a sequence, keeping that sequence and every later one: e.g. a build seat's work queue (SEAT_NODE_BUILD_GO) emptied of old asks nobody waits for, up to the first ask still wanted. Needs `why`, kept in the audit trail; reads the stream first and answers its first and last sequence and its messages before and after; the stream and its consumers stay; refuses a `below` past the stream's end (a whole purge is not this tool's). `dry_run` only reads. Read the sequences with nats_stream_messages. The bus refuses any stream the module's credential is not granted.",
Input: map[string]any{"stream": str("the stream, e.g. SEAT_NODE_BUILD_GO"), "below": num("the first sequence kept: every message below it is purged"),
"why": str("why they are purged, for the audit trail (required)"), "dry_run": map[string]any{"type": "boolean", "description": "read the stream, purge nothing"}},
Run: run(PurgeStream)},
{Name: "nats_bucket_remove", {Name: "nats_bucket_remove",
Description: "Removes (purges) keys from a key-value bucket: one node's keys (`node`, the keys starting `<node>.`) or the keys under a `prefix`, e.g. a full acts bucket of the node-engine. Needs `why`, kept in the audit trail; never the whole bucket. Counts the keys first and lists them (`dry_run` only counts; `expect` refuses when the count differs); removes only those keys, and their history. See the buckets and their keys with nats_buckets. The bus refuses any bucket the module's credential is not granted.", Description: "Removes (purges) keys from a key-value bucket: one node's keys (`node`, the keys starting `<node>.`) or the keys under a `prefix`, e.g. a full acts bucket of the node-engine. Needs `why`, kept in the audit trail; never the whole bucket. Counts the keys first and lists them (`dry_run` only counts; `expect` refuses when the count differs); removes only those keys, and their history. See the buckets and their keys with nats_buckets. The bus refuses any bucket the module's credential is not granted.",
Input: map[string]any{"bucket": str("the bucket as nats_buckets lists it, e.g. mesh-host_acts"), "node": str("a node whose keys (`<node>.`) go"), "prefix": str("key prefix instead of node"), Input: map[string]any{"bucket": str("the bucket as nats_buckets lists it, e.g. mesh-host_acts"), "node": str("a node whose keys (`<node>.`) go"), "prefix": str("key prefix instead of node"),
@@ -388,3 +393,87 @@ func refuseNewForm(newForm heldEntries, stream string) error {
} }
return nil return nil
} }
// StreamPurged is what nats_stream_purge answers and says.
type StreamPurged struct {
Stream string `json:"stream"`
Below uint64 `json:"below"`
Why string `json:"why"`
DryRun bool `json:"dry_run"`
First uint64 `json:"first_seq"`
Last uint64 `json:"last_seq"`
Before uint64 `json:"messages_before"`
After uint64 `json:"messages_after"`
Purged uint64 `json:"purged"`
Said string `json:"said"`
At time.Time `json:"at"`
}
// PurgeStream removes the messages of one named stream below a sequence, keeping that sequence and every later one,
// after reading the stream, and says so: a work queue whose old asks nobody waits for any more is emptied up to the
// first ask still wanted (issue 1952's removal tools, one more of their kind). The stream and its consumers stay.
func PurgeStream(ctx context.Context, bus Bus, said Said, a map[string]any) (any, error) {
w, err := why(a)
if err != nil {
return nil, err
}
stream := arg(a, "stream")
if !name.MatchString(stream) {
return nil, errors.New("stream is required, one stream's name (letters, digits, - and _): nothing was purged")
}
below, ok := a["below"].(float64)
if !ok || below < 2 || below != float64(uint64(below)) {
return nil, errors.New("below is required, the first sequence kept (a whole number above 1): nothing was purged")
}
dry, _ := a["dry_run"].(bool)
js, closeBus, err := bus(ctx)
if err != nil {
return nil, err
}
defer closeBus()
out := StreamPurged{Stream: stream, Below: uint64(below), Why: w, DryRun: dry, At: time.Now().UTC()}
sctx, cancel := step(ctx)
s, err := js.Stream(sctx, stream)
cancel()
if err != nil {
return nil, unanswered(err, fmt.Sprintf("the stream %s could not be read, so nothing was purged", stream))
}
sctx, cancel = step(ctx)
info, err := s.Info(sctx)
cancel()
if err != nil {
return nil, unanswered(err, fmt.Sprintf("the stream %s could not be read, so nothing was purged", stream))
}
out.First, out.Last, out.Before = info.State.FirstSeq, info.State.LastSeq, info.State.Msgs
if out.Below > out.Last+1 {
return nil, fmt.Errorf("%s ends at sequence %d: below %d would purge it whole, which is not this tool's; nothing was purged", stream, out.Last, out.Below)
}
if dry {
out.Said = fmt.Sprintf("%s holds %d messages, sequences %d to %d; those below %d would be purged", stream, out.Before, out.First, out.Last, out.Below)
return out, nil
}
sctx, cancel = step(ctx)
err = s.Purge(sctx, jetstream.WithPurgeSequence(out.Below))
cancel()
if err != nil {
return nil, unanswered(err, fmt.Sprintf("the messages of %s below %d were not purged", stream, out.Below))
}
sctx, cancel = step(ctx)
info, err = s.Info(sctx)
cancel()
if err != nil {
return nil, unanswered(err, fmt.Sprintf("the purge of %s below %d was accepted but could not be confirmed", stream, out.Below))
}
if info.State.FirstSeq < out.Below && info.State.Msgs > 0 {
return nil, fmt.Errorf("%s still starts at sequence %d after the purge below %d was accepted", stream, info.State.FirstSeq, out.Below)
}
out.After = info.State.Msgs
if out.Before > out.After {
out.Purged = out.Before - out.After
}
out.Said = fmt.Sprintf("%s: %d messages below sequence %d purged, %d kept", stream, out.Purged, out.Below, out.After)
if err := said(streamPurged, out); err != nil {
return nil, fmt.Errorf("the messages of %s below %d WERE purged (%s) but the audit event could not be said: %w", stream, out.Below, w, err)
}
return out, nil
}
+2 -2
View File
@@ -324,7 +324,7 @@ func heldRequest(store keeper.Store) func(id string) (*keeper.Request, error) {
} }
// snapshotListTool is nats_snapshots, which the tools bundle keeps beside the verbs the keeper's process answers. // snapshotListTool is nats_snapshots, which the tools bundle keeps beside the verbs the keeper's process answers.
func snapshotListTool(rs requestStore, keepDays int) []stdio.Tool { func snapshotListTool(rs requestStore, keepDays func() int) []stdio.Tool {
return []stdio.Tool{ return []stdio.Tool{
{Name: "nats_snapshots", {Name: "nats_snapshots",
Description: "The keeper's snapshots of kept data taken before a drop or recreate, from its state requests: each one's request, module, local name, stream, size, digest, time, whether it is protected (restored only in place, never sent), whether it is expired (past snapshot-keep-days, removed at the keeper's next request) and whether it is recorded (a snapshot of before milestone G holds no digest and is sent or removed by hand only).", Description: "The keeper's snapshots of kept data taken before a drop or recreate, from its state requests: each one's request, module, local name, stream, size, digest, time, whether it is protected (restored only in place, never sent), whether it is expired (past snapshot-keep-days, removed at the keeper's next request) and whether it is recorded (a snapshot of before milestone G holds no digest and is sent or removed by hand only).",
@@ -344,7 +344,7 @@ func snapshotListTool(rs requestStore, keepDays int) []stdio.Tool {
continue continue
} }
out = append(out, map[string]any{"request": r.ID, "module": r.Module, "local": r.Local, "stream": keeper.BusName(r.Module, r.Local), out = append(out, map[string]any{"request": r.ID, "module": r.Module, "local": r.Local, "stream": keeper.BusName(r.Module, r.Local),
"size": r.Size, "digest": r.Digest, "at": r.At, "protected": r.Protected, "expired": r.Expired(rs.Now(), keepDays), "recorded": r.Recorded()}) "size": r.Size, "digest": r.Digest, "at": r.At, "protected": r.Protected, "expired": r.Expired(rs.Now(), keepDays()), "recorded": r.Recorded()})
} }
return map[string]any{"snapshots": out}, nil return map[string]any{"snapshots": out}, nil
}}, }},
@@ -222,7 +222,7 @@ func TestTheKeepersProcessAnswersTheSnapshotVerbs(t *testing.T) {
t.Errorf("the hand acts %v: the caller of a call that named none is %s", acts, stdio.Unproven) t.Errorf("the hand acts %v: the caller of a call that named none is %s", acts, stdio.Unproven)
} }
// nats_snapshots lists every snapshot, the expired and the unrecorded said. // nats_snapshots lists every snapshot, the expired and the unrecorded said.
out, err = snapshotListTool(rs, 90)[0].Run(nil) out, err = snapshotListTool(rs, func() int { return 90 })[0].Run(nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+11 -1
View File
@@ -456,7 +456,8 @@ func (k *Keeper) statusWritten(s Status, err error) {
// watched branch's newest revision is applied (Heard), a module of the new form that sends gets its sent store, a drift // watched branch's newest revision is applied (Heard), a module of the new form that sends gets its sent store, a drift
// between branches is said once, and a module in no watched branch's newest revision has its plans key deleted, its // between branches is said once, and a module in no watched branch's newest revision has its plans key deleted, its
// entries then orphans. Before every watched branch has its first view it plans nothing, and says what it waits for in // entries then orphans. Before every watched branch has its first view it plans nothing, and says what it waits for in
// its status; the first plan checks what is held. // its status, and an answer listing no module before one has listed any is the same (none-yet); the first plan checks what
// is held.
func (k *Keeper) PlanCatalog() error { func (k *Keeper) PlanCatalog() error {
at := k.now() at := k.now()
view, err := k.Catalog() view, err := k.Catalog()
@@ -466,6 +467,15 @@ func (k *Keeper) PlanCatalog() error {
} }
k.mu.Lock() k.mu.Lock()
first := !k.viewed first := !k.viewed
// A catalog half with no module in it before any has been seen is a trunk with no revision yet (`none-yet`, design 47
// G A-367): not yet viewed, so nothing is planned, no plan deleted and no entry made an orphan for want of knowledge.
if first && len(view.Modules) == 0 {
k.mu.Unlock()
k.setStatus(func(s *Status) {
s.FirstView.Asked, s.FirstView.Error = at, "the catalog half lists no module yet (none-yet): nothing is planned or orphaned until a revision lists one"
})
return nil
}
k.viewed = true k.viewed = true
k.mu.Unlock() k.mu.Unlock()
var revision uint64 var revision uint64
+39 -3
View File
@@ -450,6 +450,9 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
if len(bus.created) != 0 { if len(bus.created) != 0 {
t.Fatalf("created %v before the first view", bus.created) t.Fatalf("created %v before the first view", bus.created)
} }
k.Catalog = func() (CatalogView, error) {
return CatalogView{Modules: []CatalogModule{{Accepted: Accepted{Module: "m", Digest: "d1", Revision: 3, BusState: []catalogue.BusStateEntry{drafts}}}}}, nil
}
if err := k.PlanCatalog(); err != nil { if err := k.PlanCatalog(); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -541,6 +544,37 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
t.Errorf("old sections %d: 8 MiB counted", k.Status().OldBytes) t.Errorf("old sections %d: 8 MiB counted", k.Status().OldBytes)
} }
}}, }},
// A-367: a trunk with no revision yet (none-yet) is not yet viewed: nothing is planned, deleted or orphaned.
{"a none-yet catalog half plans nothing and orphans nothing until a module is listed", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) {
k.Heard(Accepted{Module: "m", Revision: 9, BusState: []catalogue.BusStateEntry{drafts}})
k.Catalog = func() (CatalogView, error) {
return CatalogView{Modules: []CatalogModule{{Accepted: Accepted{Module: "m", Revision: 9, BusState: []catalogue.BusStateEntry{drafts}}}}}, nil
}
_ = k.PlanCatalog()
k.mu.Lock()
k.viewed = false
k.mu.Unlock()
k.Catalog = func() (CatalogView, error) { return CatalogView{}, nil }
if err := k.PlanCatalog(); err != nil {
t.Fatal(err)
}
if _, ok := k.planned("m"); !ok {
t.Error("m's plan is deleted by a none-yet view")
}
if a := k.Held()["m"]; a.Made["drafts"].Orphaned {
t.Errorf("held %+v: drafts orphaned by a none-yet view", a)
}
if k.Viewed() || !strings.Contains(k.Status().FirstView.Error, "none-yet") {
t.Errorf("viewed %v, status %+v: none-yet is not a first view and says so", k.Viewed(), k.Status().FirstView)
}
k.Catalog = func() (CatalogView, error) {
return CatalogView{Modules: []CatalogModule{{Accepted: Accepted{Module: "m", Revision: 10, BusState: []catalogue.BusStateEntry{drafts}}}}}, nil
}
_ = k.PlanCatalog()
if !k.Viewed() || k.Status().FirstView.Error != "" {
t.Error("the first revision with entries is viewed and plans as before")
}
}},
// §538: a module in no watched branch's newest revision has its plans key deleted, its entries then orphans. // §538: a module in no watched branch's newest revision has its plans key deleted, its entries then orphans.
{"a module no watched branch lists loses its plan, its entries orphans", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) { {"a module no watched branch lists loses its plan, its entries orphans", func(t *testing.T, k *Keeper, bus *fakeBus, out *said) {
k.Catalog = func() (CatalogView, error) { k.Catalog = func() (CatalogView, error) {
@@ -1240,9 +1274,11 @@ func TestTheKeeperMakesWhatAVerdictAsksForAndSaysIt(t *testing.T) {
} { } {
t.Run(row.name, func(t *testing.T) { t.Run(row.name, func(t *testing.T) {
k, bus, _, out := newKeeper(CatalogView{}) k, bus, _, out := newKeeper(CatalogView{})
if err := k.PlanCatalog(); err != nil { // The rows start after the first view: an empty answer is none-yet (A-367), so the view is marked answered here.
t.Fatal(err) k.mu.Lock()
} k.viewed = true
k.mu.Unlock()
k.setStatus(func(s *Status) { s.FirstView.Asked, s.FirstView.Answered = k.now(), k.now() })
row.run(t, k, bus, out) row.run(t, k, bus, out)
}) })
} }
+2
View File
@@ -57,6 +57,7 @@
"consumer-removed", "consumer-removed",
"bucket-keys-removed", "bucket-keys-removed",
"stream-removed", "stream-removed",
"stream-purged",
"bucket-expiry-set", "bucket-expiry-set",
"snapshot-restored" "snapshot-restored"
], ],
@@ -171,6 +172,7 @@
"nats_consumer_remove", "nats_consumer_remove",
"nats_bucket_remove", "nats_bucket_remove",
"nats_stream_remove", "nats_stream_remove",
"nats_stream_purge",
"nats_bucket_expiry", "nats_bucket_expiry",
"nats_history", "nats_history",
"nats_create", "nats_create",