Compare commits

..
Author SHA1 Message Date
jschoubben 22b70c82b0 Stand in the artifact store in the toolchain test's shelf: R4e's build modules require it (ADR 0513), and the group's composed check judges them at the catalogue's group head
mesh/repo-check head 22b70c82b0ab merged into trunk cdfc63cf102b beside mesh-catalog#457@896bd883 +1 more — pass: THE CHANGE ALTERS ITS OWN CHECK (go/merge…
mesh/merge-gate head 22b70c82b0ab merged into trunk cdfc63cf102b — pass: group feat/r4e composed: every machine composes with the change as it did without …
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792707174627079192 (novox/mesh-sdk at 7aacbc16 into main, done, …
mesh/delivery-group group feat/r4e closed: a member was closed before the group merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:989 | 22b70c82b0ab on 7aacbc162014 | unit 1675 run, 0 kept, 3 skipped | integration…
2026-10-23 00:03:06 +02:00
jschoubben 6aa8e869a3 Name no paths in mesh-sdk's own manifest until R4e is delivered: today's controller judges R4e's group and refuses the word (issue 1758, ADR 0499 §7, ADR 0503 §4)
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 6aa8e869a3b6 merged into trunk cdfc63cf102b beside mesh-catalog#457@896bd883 +1 more — fail: THE CHANGE ALTERS ITS OWN CHECK (go/merge…
mesh/delivery-group group feat/r4e rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#457: modules/gitea/go.mod names mesh-sdk 104533e8c0c9, which is neither on mesh-sdk's trunk nor the group's…
2026-10-22 23:57:26 +02:00
jschoubben 104533e8c0 S2: read a manifest for a merge's named tree as the indexer does
mesh/merge-gate head 104533e8c0c9 merged into trunk cdfc63cf102b — fail: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus step; 9 wait(…
mesh/repo-check head 104533e8c0c9 merged into trunk cdfc63cf102b — pass: THE CHANGE ALTERS ITS OWN CHECK (go/merge-check.sh): main's version judged it; the…
mesh/delivery-group group feat/r4e rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: modules/gitea.type-check: index.ts(263,29): error TS2345: Argument of type 'BranchProtection | null | undefined' is not assig…
catalogue.ParseManifestForInputs (issue 1751): a manifest falling back to a predefined key was narrowed by the indexer and not by a build module, two computations of one tree (novox/hq ADR 0499 clause 2).
2026-10-22 21:07:40 +02:00
jschoubben 259bfd0c2b S1: keep no package.json or go.mod outside the named tree in a narrowed build
A file outside the key changed what is built (novox/hq ADR 0499 clause 4); a build that needs one names it among its paths.
2026-10-22 21:07:22 +02:00
jschoubben 875c680d98 Merge remote-tracking branch 'origin/main' into feat/r4e-sdk
# Conflicts:
#	go/catalogue/state.go
#	go/catalogue/state_test.go
#	go/process/state.go
2026-10-22 20:37:49 +02:00
mesh-tools cdfc63cf10 Merge pull request 'Let a process watch, put, list and delete its own state's keys, as a tools bundle can (upgrade safety, issue 1840, ADR 0507 §8)' (#69) from feat/upgrade-safety into main 2026-10-22 18:34:24 +00:00
jschoubben d1af87a83a Leave a narrowed build only its named paths, manifest and locks, in one function for both build modules (issue 1758, ADR 0499 §4) 2026-10-22 18:31:07 +02:00
jschoubben e76a5a31a0 Read a manifest for its build's inputs without the published list of predefined keys (issue 1751, dev-r4e-builder's request)
The indexer and the builder hold no list, so claude-code, dunst and xterm were refused;
ParseManifestForInputs takes each fallback as its value declares it and judges no
config value, refusing everything else ParseManifest refuses.
2026-10-22 18:18:55 +02:00
jschoubben 8892a11768 Merge remote-tracking branch 'origin/main' into feat/upgrade-safety
mesh/merge-gate head 8892a1176800 merged into trunk 4fac7810a1e4 — pass: group feat/upgrade-safety composed: every machine composes with the change as it d…
mesh/repo-check head 8892a1176800 merged into trunk 4fac7810a1e4 beside mesh-catalog#450@86ba54b0 +1 more — pass: its merge-check.sh passed, each part in i…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792694058124013462 (novox/mesh-sdk at cdfc63cf into main, don…
mesh/delivery-group group feat/upgrade-safety merged: every member is merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 8892a1176800 on 4fac7810a1e4 | unit 3372 run, 0 kept, 7 skipped | integration…
2026-10-22 18:13:24 +02:00
mesh-tools 4fac7810a1 Merge pull request 'Keep a machine that does not resolve on the private network and in the names, so it is refused alone (issue 1833)' (#68) from fix/1833-one-machine-alone into main 2026-10-22 16:10:23 +00:00
jschoubben b07e741c41 Say in the controller seat's verbs what R4e removes and what the checker now does (R4e, dev-r4e-controller's request via the lead; ADR 0501 §5, ADR 0503 §3, ADR 0429 §18)
build, rebuild, forget-checks and delivery-plan stay listed, refused by name; replay
is a dry run only; delivery-check asks nothing; try is the checker's.
2026-10-22 18:07:37 +02:00
jschoubben 72a477eb1c Carry a candidate module's named paths, so the builder's fetch for a candidate names them (issue 1758, ADR 0499 §2, the builder lane's request) 2026-10-22 18:06:49 +02:00
jschoubben 8c3bfad86b Run the run step's isolation proof only where bubblewrap is, under its own tag (issue 1309, S19, the lead's decision)
The try's toolchain container has no bubblewrap; the checker runs the isolation tag on
the build nodes, and merge-check.sh still vets it.
2026-10-22 18:06:10 +02:00
jschoubben 9cb4d59093 Keep each artifact's named paths in the modules value, so the checker needs no manifest to see a change touch them (issue 1758, ADR 0499, lead decision L8) 2026-10-22 18:00:24 +02:00
jschoubben fd2ae83431 Name the error of a fetch whose lock waits for an unpublished package (issue 1587, ADR 0500 §6)
waits for: is the prefix the builder turns into a queued key with Waits; it is not
one of the errors held until the fingerprint changes.
2026-10-22 18:00:02 +02:00
jschoubben 4dd967e62f Let a module that builds no artifact have a done key that can be delivered (issue 1613, the lead's request for the builder lane)
A done key needs its manifest digest, no longer an artifact; Referenced answers ok
for it, since no artifact lacks a reference.
2026-10-22 17:59:55 +02:00
jschoubben ab2ca6e3a7 Push a done build's bundle or archive to the artifact store from the build module, in one shared client (issue 1587, ADR 0498 §1-§2)
buildstep.PushBlob uploads by digest unless the store holds it, waits out a store
refusing connections as mesh-build-image does, and names the store's own words;
BlobReference is the form the key and the controller check.
2026-10-22 17:59:14 +02:00
jschoubben a90f523edc Name the TypeScript package's own files as lib's paths, so a Go-only change moves no TypeScript module (issue 1758, S6, ADR 0499 §6) 2026-10-22 17:47:26 +02:00
jschoubben fc7c4e97e6 Stop committing the Go dependencies, which go.sum already pins (issue 1304, S8)
go/vendor's 810 files go and .gitignore names vendor/, as merge-check.sh already
said; every dependency resolves by go.sum and go mod verify fails by name.
2026-10-22 17:47:06 +02:00
jschoubben f7cd91e49d Accept a modules value with the new fingerprint alone, and say mesh/check is the required status (issue 1851, S7 and S16, ADR 0453 §6, ADR 0503 §3 §5)
FP and None stay readable for build-agent's path until R4g; the gate's status
comment and the checks verb now name mesh/check as what a main requires.
2026-10-22 17:46:56 +02:00
jschoubben 3b3cabd2b0 Prove a run step reaches no key, login, name or socket and leaves the caches unchanged (issue 1309, S19, ADR 0448 check) 2026-10-22 17:46:10 +02:00
jschoubben f8b8f3a666 issue 1840/S2: take a narrowed watch's current values as a set, since the bus hands them in the order of their last revision 2026-10-22 17:45:58 +02:00
jschoubben 1f4ff20488 Keep the shared caches unchanged by a candidate's step, under a throwaway writable layer (issue 1309, S18, ADR 0448)
A trunk build alone writes them; any other step writes into an overlay that dies with it.
2026-10-22 17:45:35 +02:00
jschoubben 39a2144cd6 Run a step's command shut off in bubblewrap, so a change's code reaches no key, login or network (issue 1309, S17, ADR 0448)
buildstep.Isolation and Isolated: its own namespaces with only a loopback, an
environment of exactly what the step is given, the system and toolchain read-only,
the workspace writable, and no home, key, bus login, runtime directory or name service.
2026-10-22 17:45:21 +02:00
jschoubben 2aab86c5cb Delete the copied list of the controller's readable states and the reads check built on it (issue 1790, S12, ADR 0495 §4-§5)
catalogue.ControllerMirrors and StateReadsNothingDeclares go, and the catalogue pass
no longer judges a read against the owner's manifest: an owner's announcement and its
own announcement test hold what it keeps.
2026-10-22 17:43:49 +02:00
jschoubben 33387ac3e5 Give every module the announcement test, on an exported test bus (issue 1790, S11, ADR 0495 §5)
announcetest.Check starts the module against a bus of the test's own and holds its
announcement equal to what its code declares; testbus leaves internal/ so a module's
own integration tests can start it.
2026-10-22 17:43:07 +02:00
jschoubben 57d2f31f05 Let a module's code announce the states it keeps, in Go and TypeScript (issue 1790, S10, ADR 0495 §2 §5)
process.Conn.Keeps, stdio.Keeps and the TypeScript keeps publish the announcement on
the module's own subject; afterwards a state the code did not announce is refused at
the call, so announcement and code cannot drift. The package goes to 0.1.15 for keeps.
2026-10-22 17:42:24 +02:00
jschoubben bb8b3bb43b Define how a module announces the states it keeps, so readers are granted from the owner's word (issue 1790, S9, ADR 0495 §2-§3)
announce.States on the module's own event subject mesh.mod.<m>.event.states, kept by
the controller in mesh-controller_states-announced by module.
2026-10-22 17:34:35 +02:00
jschoubben d13bc6ecc6 Let a key wait queued for a package version, so it is not failed before the package is published (issue 1587, S15, ADR 0500 §6)
waits names <name>@<version> on a queued key only; an older reader reads it as queued.
2026-10-22 17:31:42 +02:00
jschoubben ad1f6e6ac2 Name the build errors in one place, and which are not built again until the source changes (issue 1587, S14, ADR 0498 §2 §4 §5)
not stored, not pinned, version taken and refused kind (moved here from the builder);
FailedUntilChanged keeps an unpinned upstream or a taken version from rebuilding in a
loop, while a failed push is built again.
2026-10-22 17:31:40 +02:00
jschoubben 97895851f8 Check each done artifact's reference against its kind, and say when a key can be delivered (issue 1587, S13, ADR 0498 §1 §5 §6 §8)
A bundle or archive names its artifact-store blob, an image or upstream copy the
registry's reference to its digest (L1), a package name@version; Referenced names
the artifacts without one, since the controller delivers only from a key with all.
2026-10-22 17:31:40 +02:00
jschoubben d14a6d301c Prove on real git that a Go-only commit leaves mesh-sdk's package tree equal (issue 1758, S5, ADR 0499 §2 check) 2026-10-22 17:30:25 +02:00
jschoubben 7cf73ed76b Hand a build's fetch the named paths in a shared request, so no build module reads them from a manifest (issue 1758, ADR 0499 §4, ADR 0495 §1)
buildstep.FetchRequest is what the builder writes and the build modules read: the
repository, commit, ref, base and, for a module naming its paths, its directory and
those paths from the indexer's kept inputs.
2026-10-22 17:29:58 +02:00
jschoubben d32743499c Narrow the merge preview's tree of a module that names its paths (issue 1758, S4, ADR 0499 §2)
ModuleTrees reads each named path's object with one git ls-tree and answers their
named tree, so the preview gives the value the indexer gives; a named path absent at
the commit fails by name rather than pinning a source that does not exist.
2026-10-22 17:29:17 +02:00
jschoubben 8d65cb3016 Carry each named path's git object beside a module's other inputs (issue 1758, S3, ADR 0499 §2-§3)
Forge.Paths, PreviewModule.Paths, KeptInputs2.Paths and buildstep.ModuleTree.Paths,
each optional and read as absent by older readers; a kept or previewed tree that its
named paths do not compute to is refused, so a recompute gives the merge's value.
2026-10-22 17:27:58 +02:00
jschoubben 0c8d0f73fa Compute a module's narrowed tree in one function, so every caller gives the merge's value (issue 1758, S2, ADR 0499 §2)
delivery.NamedTree hashes the manifest digest with each named path's git object;
catalogue.Manifest.NamedPaths gives the union of the paths, only when every artifact
names some. The named: prefix keeps the value apart from a git tree id.
2026-10-22 17:27:02 +02:00
jschoubben dfc2c858d9 Share the launcher rule of a TypeScript bundle, so builder and controller name it alike (R4e, the lead's request for dev-r4e-controller)
The rule mesh-build-node and build-agent each keep their own copy of: x.js is
started by x.serve.mjs beside it; other languages get none (ADR 0193).
2026-10-22 17:25:21 +02:00
jschoubben 06fe1ee8b3 Read the manifest word paths, so a package names the files it is built from (issue 1758, S1, ADR 0499 §1)
A package or a non-Go bundle may name its build source; a Go program, an archive, an
image and an upstream artifact refuse the word by name, as do a path leaving the
module's directory, the whole directory and a pattern.
2026-10-22 17:25:02 +02:00
jschoubben bd7731f358 Count a machine that does not resolve and holds a place as on the network for its name and the mesh's sources too (issue 1833)
mesh/repo-check head bd7731f3588c merged into trunk 5652312f09d2 beside mesh-controller#437@28a278c1 — pass: its merge-check.sh passed, each part in its to…
mesh/merge-gate head bd7731f3588c merged into trunk 5652312f09d2 — pass: group fix/1833-one-machine-alone composed: every machine composes with the change …
mesh/delivery delivered
mesh/delivery-group group fix/1833-one-machine-alone merged: every member is merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | bd7731f3588c on 4fac7810a1e4 | unit 1634 run, 0 kept, 3 skipped | integration…
The tunnel kept such a machine while the names and the packet filter's sources from the mesh, read from their own
unchecked resolve, dropped it when that pass refused it too (a seat claimed twice): every other machine was sent the
hub as a peer without its name or its address. onTheNetwork keeps it by the same rule as network().
2026-10-22 15:26:48 +02:00
jschoubben c154bdedd3 Keep a machine that does not resolve on the private network at its place, so one machine's modules refuse only its own declaration (issue 1833)
A machine whose own set could not be resolved was left off the private network; when it was the hub, the network
could not be composed and every machine's declaration failed. Its placement (key and address) is its last good place
and does not change when its modules fail, so it stays on the network and only its own declaration is refused. A
refused machine with no place is left off, and a network that cannot be composed from what remains still fails.
2026-10-22 15:03:17 +02:00
jschoubben 5ea6c215a4 issue 1840/S1: let a process delete a key, as a tools bundle can 2026-10-22 14:36:24 +02:00
jschoubben 2b8ada5eb9 Merge remote-tracking branch 'origin/main' into feat/upgrade-safety 2026-10-22 14:24:43 +02:00
mesh-tools 5652312f09 Merge pull request 'Make the route proxy's bus helpers public in mesh-sdk (issue 1633, ADR 0480 §4)' (#67) from feat/r4d into main 2026-10-22 12:19:16 +00:00
jschoubben 076e1f7947 issue 1840/S2: let a process watch a state, current values first then each change, as a tools bundle can 2026-10-22 14:16:59 +02:00
jschoubben 26517ac00b issue 1840/S1: let a process put a key whole and list a state's keys, as a tools bundle can 2026-10-22 14:16:13 +02:00
jschoubben 637c33b7f8 issue 1840/S1c: name declared-packages among the controller's readable states, so a manifest may read it 2026-10-22 14:15:33 +02:00
jschoubben c61212eba7 Make the membership, its subject, stream, bare address and the pin public, so the route proxy leaves the controller's module (issue 1633)
mesh/delivery delivered
mesh/merge-gate head c61212eba77c merged into trunk c6e669f71e8e — pass: group feat/r4d composed: every machine composes with the change as it did without …
mesh/repo-check head c61212eba77c merged into trunk c6e669f71e8e — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery-group group feat/r4d merged: every member is merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | c61212eba77c on 5652312f09d2 | unit 1629 run, 0 kept, 3 skipped | integration…
2026-10-22 09:51:01 +02:00
mesh-tools c6e669f71e Merge pull request 'The checker runs beside today's check: the SDK part (layouts, shared build-run parts, merge and module trees)' (#66) from feat/the-checker into main 2026-10-21 21:46:39 +00:00
jschoubben 580da61a91 Name plans-building among the controller's readable states, for the checker's read (issue 1687, ADR 0483 clause 2)
mesh/repo-check head 580da61a9107 merged into trunk 07a65927c362 beside mesh-catalog#423@12ff1cfb +1 more — pass: its merge-check.sh passed, each part in i…
mesh/merge-gate head 580da61a9107 merged into trunk 07a65927c362 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker merged: every member is merged
2026-10-21 22:44:52 +02:00
jschoubben 78364e318b Put the module trees of a merge into go/buildstep for both fetch steps, and let a preview name the conflicts of a head that does not merge (issues 1675, 1676, 1445)
mesh/repo-check head 78364e318bdc merged into trunk 07a65927c362 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery-group group feat/the-checker rejected: composed together: 1 problem(s) the change brings; the first: mesh-delivery-checker reads "mesh-controller…
mesh/merge-gate head 78364e318bdc merged into trunk 07a65927c362 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 21:21:48 +02:00
jschoubben afbe371003 Merge main (mesh-sdk #62, issue 1756) into feat/r4c5-sdk 2026-10-21 20:59:47 +02:00
jschoubben 9aaa5c80ac Accept the gate's recheck status mesh/repo-check among the owed contexts, so a pinned recheck is not refused (issue 1673) 2026-10-21 19:35:14 +02:00
mesh-tools 07a65927c3 Merge pull request 'Count an item's text as a placement of a config value (issue 1658)' (#62) from fix/1658-item-placement into main 2026-10-21 17:34:36 +00:00
jschoubben 92fdc8c86a Parse the beside catalogue once per test binary, so mesh-sdk's unit level fits its bound (issue 1756)
mesh/merge-gate head 92fdc8c86a17 merged into trunk 725590e06e75 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 92fdc8c86a17 merged into trunk 725590e06e75 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 19:26:30 +02:00
jschoubben 0730ef6092 Keep the unit level cheap: the merge's real-git tests start processes, so they are integration tests; the places tests wait 10 ms, not 50 (issue 895) 2026-10-21 19:26:02 +02:00
jschoubben 4495a16608 Put the merge of a head onto a trunk commit into go/buildstep, so both build modules' fetch steps call one function and none copies the controller's (issue 1674) 2026-10-21 19:22:33 +02:00
jschoubben b944d07158 Add the checker's owed bucket, the mesh/check status context and the previews layout, and correct the checker header: the checker decides the status, the bridge sets it (issue 1673) 2026-10-21 19:21:22 +02:00
jschoubben 6d0c2076a3 Give the checker's verdict a skipped count, a composition and the over-bound levels, so the line never hides skips or claims a composition not run (issue 1672) 2026-10-21 19:19:14 +02:00
jschoubben 4375ed8b2c Move the builder's shared build-run parts into go/delivery/buildrun, so the checker shares them and no module keeps a copy (issue 1671) 2026-10-21 19:18:11 +02:00
jschoubben c0bbff3b4b Merge remote-tracking branch 'origin/main' into fix/1658-item-placement
mesh/merge-gate head c0bbff3b4bb2 merged into trunk 725590e06e75 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head c0bbff3b4bb2 merged into trunk 725590e06e75 — fail: merge-check.sh (typescript) passed; its go/merge-check.sh failed: make them cheap …
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:17:26 +02:00
jschoubben c8e35097e8 Count an item's text as a placement of a config value, so a key only an item says is not refused (issue 1658)
mesh/merge-gate waits for its group's composed check
mesh/repo-check head c8e35097e861 merged into trunk d4deecdb380b beside mesh-controller#420@858b06a9 +1 more — pass: its merge-check.sh passed, each part i…
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 12:54:08 +02:00
912 changed files with 6942 additions and 497324 deletions
+1
View File
@@ -1,2 +1,3 @@
node_modules/
dist/
vendor/
+126
View File
@@ -0,0 +1,126 @@
// Package announce is how a module says on the bus which states it keeps and which of them others may read (novox/hq
// ADR 0495 clauses 2 and 3; issue 1790): the subject it says it on, the value it says, and where the controller keeps
// the last one said by each module. What a module announced is the truth of what it keeps; a reader is granted a read
// from it, never from a list or from the owner's manifest.
//
// A module announces from its code (process.Conn.Keeps, stdio.Keeps, the TypeScript SDK's keeps), never from its
// manifest, which only the core reads (clause 1). A module that keeps no state on the bus announces nothing.
package announce
import (
"encoding/json"
"fmt"
"regexp"
"sort"
)
// Version is the version of the value this package writes.
const Version = 1
// Event is the event type of an announcement, under the module's own namespace.
const Event = "states"
// Subject is where module announces what it keeps: its own event subject, mesh.mod.<module>.event.states, which its
// account may already publish to and the events stream keeps.
func Subject(module string) string { return "mesh.mod." + module + ".event." + Event }
// SubjectAll is every module's announcement, what the controller consumes.
const SubjectAll = "mesh.mod.*.event." + Event
// KeptBucket is the controller's bucket of the last announcement of each module, by Key.
const KeptBucket = "mesh-controller_states-announced"
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// Key is module's key in KeptBucket: the module itself, which must be one plain name, since a module's name is part of
// its buckets' names (catalogue.StateProblems).
func Key(module string) (string, error) {
if !name.MatchString(module) {
return "", fmt.Errorf("announce: %q is not a module that keeps state: lower-case letters, digits and hyphens", module)
}
return module, nil
}
// State is one state a module keeps, by its local name, and whether others may read it. Others never write it: a
// state another module writes is the owner's own code's business.
type State struct {
Name string `json:"name"`
Readable bool `json:"readable,omitempty"`
}
// States is an announcement: the module, the machine whose instance said it, and every state the module keeps, sorted
// by name. An empty list says the module keeps nothing.
type States struct {
V int `json:"v"`
Module string `json:"module"`
Node string `json:"node,omitempty"`
States []State `json:"states"`
}
// Keeps says whether the module announced keeping the state.
func (s States) Keeps(state string) bool {
for _, k := range s.States {
if k.Name == state {
return true
}
}
return false
}
// Readable says whether the module announced the state readable by others.
func (s States) Readable(state string) bool {
for _, k := range s.States {
if k.Name == state {
return k.Readable
}
}
return false
}
func (s *States) check() error {
if s.Module == "" {
return fmt.Errorf("announce: an announcement names no module")
}
if _, err := Key(s.Module); err != nil {
return err
}
seen := map[string]bool{}
for _, k := range s.States {
if !name.MatchString(k.Name) {
return fmt.Errorf("announce: %s announces the state %q, and a state is named locally: lower-case letters, digits and hyphens", s.Module, k.Name)
}
if seen[k.Name] {
return fmt.Errorf("announce: %s announces the state %q twice", s.Module, k.Name)
}
seen[k.Name] = true
}
return nil
}
// Encode is s as JSON at Version, its states sorted; it refuses a module or a state no bucket could be named by and a
// state announced twice, naming it.
func Encode(s States) ([]byte, error) {
if err := s.check(); err != nil {
return nil, err
}
s.V = Version
s.States = append([]State{}, s.States...)
sort.Slice(s.States, func(i, j int) bool { return s.States[i].Name < s.States[j].Name })
return json.Marshal(s)
}
// Decode reads an announcement; it refuses another version and what Encode refuses.
func Decode(data []byte) (States, error) {
var s States
if err := json.Unmarshal(data, &s); err != nil {
return States{}, fmt.Errorf("announce: the announcement is not JSON: %w", err)
}
if s.V != Version {
return States{}, fmt.Errorf("announce: the announcement is version %d, and this reads version %d", s.V, Version)
}
if err := s.check(); err != nil {
return States{}, err
}
sort.Slice(s.States, func(i, j int) bool { return s.States[i].Name < s.States[j].Name })
return s, nil
}
+71
View File
@@ -0,0 +1,71 @@
package announce
import (
"strings"
"testing"
)
// A module says on the bus which states it keeps and which of them others may read (novox/hq ADR 0495 clause 2), on its
// own event subject, in a versioned value the controller keeps by module.
func TestAnAnnouncementRoundTripsSortedAndSaysWhatIsReadable(t *testing.T) {
data, err := Encode(States{Module: "mesh-controller", Node: "novox", States: []State{
{Name: "plans-building", Readable: true}, {Name: "assignments", Readable: true}, {Name: "own"}}})
if err != nil {
t.Fatal(err)
}
back, err := Decode(data)
if err != nil {
t.Fatal(err)
}
if back.V != Version || back.Module != "mesh-controller" || back.Node != "novox" {
t.Fatalf("decoded %+v", back)
}
var names []string
for _, s := range back.States {
names = append(names, s.Name)
}
if strings.Join(names, ",") != "assignments,own,plans-building" {
t.Fatalf("the states are not sorted: %v", names)
}
if !back.Readable("plans-building") || back.Readable("own") || back.Readable("absent") {
t.Fatal("Readable does not say what was announced readable")
}
if !back.Keeps("own") || back.Keeps("absent") {
t.Fatal("Keeps does not say what was announced")
}
if Subject("mesh-controller") != "mesh.mod.mesh-controller.event.states" || SubjectAll != "mesh.mod.*.event.states" {
t.Fatalf("the subjects are %q and %q", Subject("mesh-controller"), SubjectAll)
}
if KeptBucket != "mesh-controller_states-announced" {
t.Fatalf("the kept bucket is %q", KeptBucket)
}
if k, err := Key("mesh-delivery-builder"); err != nil || k != "mesh-delivery-builder" {
t.Fatalf("Key = %q, %v", k, err)
}
if nothing, err := Encode(States{Module: "quiet"}); err != nil || !strings.Contains(string(nothing), `"states":[]`) {
t.Fatalf("a module keeping nothing: %s, %v", nothing, err)
}
}
func TestAnAnnouncementRefusesWhatNoBucketCouldBe(t *testing.T) {
for _, c := range []struct {
s States
says string
}{
{States{Module: ""}, "no module"},
{States{Module: "a.b"}, `"a.b"`},
{States{Module: "a", States: []State{{Name: "x_y"}}}, `"x_y"`},
{States{Module: "a", States: []State{{Name: ""}}}, `""`},
{States{Module: "a", States: []State{{Name: "x"}, {Name: "x", Readable: true}}}, "twice"},
} {
if _, err := Encode(c.s); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%+v: %v, wanted a refusal saying %s", c.s, err, c.says)
}
}
if _, err := Decode([]byte(`{"v":2,"module":"a","states":[]}`)); err == nil {
t.Error("an unknown version was read")
}
if _, err := Key("a.b"); err == nil {
t.Error("a module with a dot became a key")
}
}
+91
View File
@@ -0,0 +1,91 @@
// Package announcetest is the test each module runs with its own checks to hold that it announces on the bus every
// state it keeps (novox/hq ADR 0495 clause 5; issue 1790): it starts the module against a bus of the test's own and
// compares what the module announced with what its code declares. With the bind guard of process.Conn.Keeps,
// stdio.Keeps and the TypeScript keeps, which refuses a state the code did not announce, announcement and code agree.
//
// It replaces the merge gate's check of one module's reads against another's manifest (clause 5); a test using it
// carries `//go:build integration`, since it starts a bus.
package announcetest
import (
"context"
"fmt"
"reflect"
"sort"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// Wait is how long Check waits for the module's announcement once it started it.
var Wait = 10 * time.Second
// Check starts a bus of the test's own, with a stream taking module's events, calls start with its address (start
// runs the module, or its part that connects, against it, and answers how to stop it), waits for the module's
// announcement and fails the test unless it is module's and names exactly want (in any order), readable as want says.
// It answers the announcement.
func Check(t testing.TB, module string, want []announce.State, start func(url string) (stop func())) announce.States {
t.Helper()
url := testbus.URL(t)
nc, err := nats.Connect(url)
if err != nil {
t.Fatalf("announcetest: connecting to the test bus: %v", err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), Wait)
defer cancel()
stream, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "ANNOUNCETEST", Subjects: []string{"mesh.mod." + module + ".event.>"}})
if err != nil {
t.Fatalf("announcetest: making the events stream: %v", err)
}
stop := start(url)
if stop != nil {
defer stop()
}
got, err := waitFor(ctx, stream, module)
if err != nil {
t.Fatalf("announcetest: %v", err)
}
if problem := differs(got, module, want); problem != "" {
t.Fatalf("announcetest: %s", problem)
}
return got
}
// waitFor reads the stream until module's announcement is in it.
func waitFor(ctx context.Context, stream jetstream.Stream, module string) (announce.States, error) {
for {
msg, err := stream.GetLastMsgForSubject(ctx, announce.Subject(module))
if err == nil {
return announce.Decode(msg.Data)
}
select {
case <-ctx.Done():
return announce.States{}, fmt.Errorf("%s announced nothing on %s within %s", module, announce.Subject(module), Wait)
case <-time.After(50 * time.Millisecond):
}
}
}
// differs says how got is not module's announcement of want, or "".
func differs(got announce.States, module string, want []announce.State) string {
if got.Module != module {
return fmt.Sprintf("the announcement names the module %q, not %q", got.Module, module)
}
w := append([]announce.State{}, want...)
sort.Slice(w, func(i, j int) bool { return w[i].Name < w[j].Name })
g := append([]announce.State{}, got.States...)
if !reflect.DeepEqual(g, w) && !(len(g) == 0 && len(w) == 0) {
return fmt.Sprintf("%s announced %v, and its code declares %v", module, g, w)
}
return ""
}
@@ -0,0 +1,29 @@
//go:build integration
package announcetest
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/process"
)
// 1790/S11: a module started against the test bus announces through process.Conn.Keeps, and Check reads that
// announcement back against what the code declares.
func TestCheckReadsTheModulesAnnouncementOffARealBus(t *testing.T) {
declared := []announce.State{{Name: "accounts", Readable: true}, {Name: "drafts"}}
got := Check(t, "mesh-mail", declared, func(url string) func() {
conn, err := process.Connect(process.Account{URL: url, Module: "mesh-mail", Node: "novox", User: "u", Password: "p"})
if err != nil {
t.Fatal(err)
}
if err := conn.Keeps(declared...); err != nil {
t.Fatal(err)
}
return conn.Close
})
if got.Node != "novox" {
t.Fatalf("announced from %q", got.Node)
}
}
+32
View File
@@ -0,0 +1,32 @@
package announcetest
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// The comparison is exact up to order: a state missing, one extra, a readability that differs or another module's
// announcement each fail, naming what differs.
func TestAnAnnouncementDiffersFromTheCodeByNameAndReadability(t *testing.T) {
got := announce.States{Module: "m", States: []announce.State{{Name: "a", Readable: true}, {Name: "b"}}}
if p := differs(got, "m", []announce.State{{Name: "b"}, {Name: "a", Readable: true}}); p != "" {
t.Fatalf("an equal announcement differs: %s", p)
}
for _, want := range [][]announce.State{
{{Name: "a", Readable: true}},
{{Name: "a", Readable: true}, {Name: "b"}, {Name: "c"}},
{{Name: "a"}, {Name: "b"}},
} {
if p := differs(got, "m", want); p == "" || !strings.Contains(p, "declares") {
t.Errorf("want %v: %q", want, p)
}
}
if p := differs(got, "other", nil); !strings.Contains(p, `"other"`) {
t.Errorf("another module's announcement: %q", p)
}
if p := differs(announce.States{Module: "m"}, "m", nil); p != "" {
t.Errorf("a module keeping nothing: %q", p)
}
}
+170
View File
@@ -0,0 +1,170 @@
package buildstep
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"syscall"
"time"
)
// RegistryWait is how long a push waits for an artifact store that refuses connections, longer than its nightly
// collection holds it still (about a minute and a half, issue 457); RegistryFirstPause is the first pause, doubling
// to RegistryMostPause. Variables so a test shortens them.
var (
RegistryWait = 5 * time.Minute
RegistryFirstPause = time.Second
RegistryMostPause = 10 * time.Second
)
// BlobReference is where a pushed bundle or archive is, as a done key names it and the controller delivers from
// (novox/hq ADR 0498 clause 1): artifact-store://<module>/<artifact>/blobs/<digest>, the store's address left out
// (issue 102).
func BlobReference(module, artifact, digest string) string {
return "artifact-store://" + module + "/" + artifact + "/blobs/" + digest
}
// PushBlob puts body into the artifact store at registry (host:port) as a blob of repository (<module>/<artifact>) by
// its digest (`sha256:` and hex), from the build module's own process, after the build step ended (novox/hq ADR 0498
// clauses 1 and 2): a blob the store already holds is done without an upload; otherwise it is uploaded (POST
// blobs/uploads/, then PUT with the digest). A store refusing connections is waited out up to RegistryWait; any other
// refusal fails, naming the store's status and its own words, for the caller's `not stored:` error. Bytes that are not
// their digest are refused before anything is sent.
func PushBlob(ctx context.Context, client *http.Client, registry, repository, digest string, body []byte) error {
sum := sha256.Sum256(body)
if got := "sha256:" + hex.EncodeToString(sum[:]); got != digest {
return fmt.Errorf("the blob is %s, not the digest %s it was to be pushed as", got, digest)
}
if client == nil {
client = http.DefaultClient
}
base := "http://" + registry + "/v2/" + repository + "/blobs/"
var present bool
if err := waitForStore(ctx, registry, func() error {
var err error
present, err = blobPresent(ctx, client, base+digest)
return err
}); err != nil {
return err
}
if present {
return nil
}
var location string
if err := waitForStore(ctx, registry, func() error {
var err error
location, err = startUpload(ctx, client, base+"uploads/")
return err
}); err != nil {
return err
}
target, err := url.Parse(base)
if err != nil {
return err
}
loc, err := target.Parse(location)
if err != nil {
return fmt.Errorf("the store answered the upload's location %q, which is not a URL: %w", location, err)
}
q := loc.Query()
q.Set("digest", digest)
loc.RawQuery = q.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodPut, loc.String(), bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("putting %s of %s: %w", digest, repository, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
return storeSaid(resp, "putting "+digest+" of "+repository)
}
return nil
}
func blobPresent(ctx context.Context, client *http.Client, at string) (bool, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodHead, at, nil)
if err != nil {
return false, err
}
resp, err := client.Do(req)
if err != nil {
return false, err
}
resp.Body.Close()
switch resp.StatusCode {
case http.StatusOK:
return true, nil
case http.StatusNotFound:
return false, nil
}
return false, fmt.Errorf("asking the store for %s: it answered %s", at, resp.Status)
}
func startUpload(ctx context.Context, client *http.Client, at string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, at, nil)
if err != nil {
return "", err
}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusAccepted {
return "", storeSaid(resp, "starting an upload at "+at)
}
location := resp.Header.Get("Location")
if location == "" {
return "", fmt.Errorf("starting an upload at %s: the store named no location", at)
}
return location, nil
}
// storeSaid is the store's refusal: its status and the first of its own words.
func storeSaid(resp *http.Response, what string) error {
words, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
return fmt.Errorf("%s: the store answered %s: %s", what, resp.Status, strings.TrimSpace(string(words)))
}
// refused is whether an error is a connection refused.
func refused(err error) bool {
return err != nil && (errors.Is(err, syscall.ECONNREFUSED) || strings.Contains(err.Error(), "connection refused"))
}
// waitForStore runs try, and while it fails because the store refuses connections, tries again with a growing pause
// until RegistryWait has passed. A refusal is the one error waited for: nothing was sent, so trying again does nothing
// twice.
func waitForStore(ctx context.Context, registry string, try func() error) error {
err := try()
if !refused(err) {
return err
}
started, pause := time.Now(), RegistryFirstPause
for {
left := RegistryWait - time.Since(started)
if left <= 0 {
return fmt.Errorf("the artifact store at %s refused every connection for %s, longer than its nightly collection holds it still, so it is down, not paused: %w",
registry, time.Since(started).Round(time.Millisecond), err)
}
select {
case <-ctx.Done():
return fmt.Errorf("stopped while waiting for the artifact store at %s: %w (last: %v)", registry, ctx.Err(), err)
case <-time.After(min(pause, left)):
}
pause = min(pause*2, RegistryMostPause)
if err = try(); !refused(err) {
return err
}
}
}
+151
View File
@@ -0,0 +1,151 @@
package buildstep
import (
"context"
"crypto/sha256"
"encoding/hex"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func digestOfBytes(b []byte) string {
h := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(h[:])
}
// registry is an artifact store as far as a blob push needs one: HEAD a blob, start an upload, PUT it with its digest.
type registry struct {
mu sync.Mutex
blobs map[string][]byte
calls []string
put int // the status a PUT answers, 201 when 0
}
func (r *registry) ServeHTTP(w http.ResponseWriter, req *http.Request) {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, req.Method+" "+req.URL.Path)
switch {
case req.Method == http.MethodHead && strings.Contains(req.URL.Path, "/blobs/sha256:"):
if _, ok := r.blobs[req.URL.Path[strings.LastIndex(req.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case req.Method == http.MethodPost && strings.HasSuffix(req.URL.Path, "/blobs/uploads/"):
w.Header().Set("Location", strings.TrimSuffix(req.URL.Path, "/")+"/u1?state=x")
w.WriteHeader(http.StatusAccepted)
case req.Method == http.MethodPut && strings.Contains(req.URL.Path, "/blobs/uploads/u1"):
if r.put != 0 {
w.WriteHeader(r.put)
_, _ = io.WriteString(w, `{"errors":[{"code":"DIGEST_INVALID","message":"provided digest did not match uploaded content"}]}`)
return
}
body, _ := io.ReadAll(req.Body)
d := req.URL.Query().Get("digest")
if req.URL.Query().Get("state") != "x" || d != digestOfBytes(body) {
w.WriteHeader(http.StatusBadRequest)
return
}
r.blobs[d] = body
w.WriteHeader(http.StatusCreated)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
}
// A done build's bundle or archive is pushed to the artifact store by its build module (novox/hq ADR 0498 clauses 1
// and 2): a blob the store already holds is done with no upload; one it lacks is uploaded by digest; its reference is
// the artifact-store form the builder's key and the controller check.
func TestABlobIsPushedOnceByItsDigest(t *testing.T) {
reg := &registry{blobs: map[string][]byte{}}
srv := httptest.NewServer(reg)
defer srv.Close()
addr := strings.TrimPrefix(srv.URL, "http://")
body := []byte("the packed bundle")
d := digestOfBytes(body)
if err := PushBlob(context.Background(), srv.Client(), addr, "mail/code", d, body); err != nil {
t.Fatal(err)
}
if string(reg.blobs[d]) != string(body) {
t.Fatalf("the store holds %q", reg.blobs[d])
}
reg.calls = nil
if err := PushBlob(context.Background(), srv.Client(), addr, "mail/code", d, body); err != nil {
t.Fatal(err)
}
if len(reg.calls) != 1 || !strings.HasPrefix(reg.calls[0], "HEAD ") {
t.Fatalf("a blob the store holds was uploaded again: %v", reg.calls)
}
ref := BlobReference("mail", "code", d)
if ref != "artifact-store://mail/code/blobs/"+d {
t.Fatalf("BlobReference = %q", ref)
}
b := delivery.BuilderBuild{State: delivery.BuildDone, Manifest: strings.Repeat("4", 64),
Artifacts: []delivery.BuiltArtifact{{Name: "code", Kind: "bundle", Digest: d, Reference: ref}}}
if _, err := delivery.EncodeBuilderBuild(b); err != nil {
t.Fatalf("the key refuses the reference BlobReference gives: %v", err)
}
}
// A store that refuses the upload fails the push naming the store's own words; bytes that are not the digest are
// refused before anything is sent.
func TestAPushTheStoreRefusesSaysTheStoresWords(t *testing.T) {
reg := &registry{blobs: map[string][]byte{}, put: http.StatusBadRequest}
srv := httptest.NewServer(reg)
defer srv.Close()
addr := strings.TrimPrefix(srv.URL, "http://")
body := []byte("x")
err := PushBlob(context.Background(), srv.Client(), addr, "mail/code", digestOfBytes(body), body)
if err == nil || !strings.Contains(err.Error(), "provided digest did not match") || !strings.Contains(err.Error(), "400") {
t.Fatalf("a refused upload said %v", err)
}
reg.calls = nil
err = PushBlob(context.Background(), srv.Client(), addr, "mail/code", digestOfBytes([]byte("other")), body)
if err == nil || len(reg.calls) != 0 {
t.Fatalf("bytes that are not their digest: %v, calls %v", err, reg.calls)
}
}
// A store refusing connections (its nightly collection holds it still) is waited out, up to RegistryWait (issue 457).
func TestAPushWaitsOutAStoreRefusingConnections(t *testing.T) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
addr := l.Addr().String()
l.Close() // refusing now
reg := &registry{blobs: map[string][]byte{}}
go func() {
time.Sleep(300 * time.Millisecond)
l2, err := net.Listen("tcp", addr)
if err != nil {
return
}
_ = http.Serve(l2, reg)
}()
beforeWait, beforePause := RegistryWait, RegistryFirstPause
RegistryWait, RegistryFirstPause = 3*time.Second, 100*time.Millisecond
defer func() { RegistryWait, RegistryFirstPause = beforeWait, beforePause }()
body := []byte("late")
if err := PushBlob(context.Background(), http.DefaultClient, addr, "mail/code", digestOfBytes(body), body); err != nil {
t.Fatalf("a store that came back was not waited for: %v", err)
}
l3, _ := net.Listen("tcp", "127.0.0.1:0")
gone := l3.Addr().String()
l3.Close()
RegistryWait = 300 * time.Millisecond
err = PushBlob(context.Background(), http.DefaultClient, gone, "mail/code", digestOfBytes(body), body)
if err == nil || !strings.Contains(err.Error(), "refused every connection") {
t.Fatalf("a store that stayed down said %v", err)
}
}
+175
View File
@@ -0,0 +1,175 @@
package buildstep
import (
"context"
"fmt"
"os"
"path"
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// Bubblewrap is the program a run step runs in (novox/hq ADR 0448): a build module declares its package, which needs
// unprivileged user namespaces (issue 1308).
const Bubblewrap = "bwrap"
// Isolation is how a run step's command is shut off (novox/hq ADR 0448; issue 1309, the lead's decision L2 of R4e: one
// runner in the SDK for every build module). The fetch step holds the forge key and the network and runs none of the
// change's code; the run step runs the change's build and tests in bubblewrap with:
//
// - no network: its own network namespace, with only a loopback, so a test may start a bus or a store of its own but
// resolves no name and reaches no socket of the machine;
// - none of the account: not its home, not the forge key, not a bus login, not the runtime directory (/run), since
// none of them is bound; HOME is a directory in the step's own /tmp unless Env names one;
// - an environment of exactly Env and the runner's env: nothing of the build module's own passes;
// - the system's programs and libraries read-only, and of /etc only what a program needs to run (users, libraries,
// certificates, time zone), with a hosts file naming only the loopback;
// - the Workspace and Writable directories read-write, the ReadOnly ones (the toolchain, the modules the fetch step
// downloaded) read-only;
// - the shared warm Caches read-only under a throwaway writable layer (S18), or read-write for a trunk build alone
// (WriteCaches), so a candidate's code never changes what a later build reads.
type Isolation struct {
Workspace string
Writable []string
ReadOnly []string
Caches []string
WriteCaches bool
Env []string
// Hosts is the file bound as /etc/hosts; Isolated writes one naming only the loopback.
Hosts string
}
// etcNeeded is what of /etc a program in the step may read: who the users are, where the libraries are, the
// certificates, the time zone. Not resolv.conf, not hosts: the step resolves no name.
var etcNeeded = []string{"/etc/passwd", "/etc/group", "/etc/nsswitch.conf", "/etc/ld.so.cache", "/etc/ld.so.conf",
"/etc/ld.so.conf.d", "/etc/ssl", "/etc/ca-certificates", "/etc/pki", "/etc/localtime", "/etc/os-release", "/etc/alternatives"}
// systemArgs binds the system's programs and libraries read-only: /usr, and /bin, /sbin, /lib, /lib64 as the symlinks
// they are on a merged-usr system, or bound when they are directories. A variable so a unit test fixes it.
var systemArgs = func() []string {
args := []string{"--ro-bind", "/usr", "/usr"}
for _, d := range []string{"/bin", "/sbin", "/lib", "/lib64"} {
info, err := os.Lstat(d)
switch {
case err != nil:
case info.Mode()&os.ModeSymlink != 0:
if target, err := os.Readlink(d); err == nil {
args = append(args, "--symlink", target, d)
}
case info.IsDir():
args = append(args, "--ro-bind", d, d)
}
}
for _, e := range etcNeeded {
args = append(args, "--ro-bind-try", e, e)
}
return args
}
// stepHome is the HOME of a step that is given none: inside its own /tmp, gone with it.
const stepHome = "/tmp/home"
func checkDir(what, dir string) error {
if dir == "" || !strings.HasPrefix(dir, "/") || path.Clean(dir) != dir || dir == "/" {
return fmt.Errorf("buildstep: the %s %q is not an absolute, clean directory other than the root", what, dir)
}
return nil
}
// Command is the bubblewrap command line that runs name with args shut off as iso says, env added to iso.Env. It
// refuses a directory that is not absolute and clean, or is the root, and an environment word without "=".
func (iso Isolation) Command(name string, args ...string) ([]string, error) {
return iso.command(nil, name, args...)
}
func (iso Isolation) command(env []string, name string, args ...string) ([]string, error) {
if err := checkDir("workspace", iso.Workspace); err != nil {
return nil, err
}
for _, group := range []struct {
what string
dirs []string
}{{"writable directory", iso.Writable}, {"read-only directory", iso.ReadOnly}, {"cache", iso.Caches}} {
for _, d := range group.dirs {
if err := checkDir(group.what, d); err != nil {
return nil, err
}
}
}
words := append(append([]string{}, iso.Env...), env...)
home := false
for _, w := range words {
k, _, ok := strings.Cut(w, "=")
if !ok || k == "" {
return nil, fmt.Errorf("buildstep: the environment word %q is not NAME=value", w)
}
home = home || k == "HOME"
}
argv := []string{"--unshare-all", "--die-with-parent", "--new-session", "--clearenv",
"--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp", "--dir", stepHome}
argv = append(argv, systemArgs()...)
for _, d := range iso.ReadOnly {
argv = append(argv, "--ro-bind", d, d)
}
argv = append(argv, cacheArgs(iso.Caches, iso.WriteCaches)...)
argv = append(argv, "--bind", iso.Workspace, iso.Workspace)
for _, d := range iso.Writable {
argv = append(argv, "--bind", d, d)
}
if iso.Hosts != "" {
argv = append(argv, "--ro-bind", iso.Hosts, "/etc/hosts")
}
for _, w := range words {
k, v, _ := strings.Cut(w, "=")
argv = append(argv, "--setenv", k, v)
}
if !home {
argv = append(argv, "--setenv", "HOME", stepHome)
}
argv = append(argv, "--chdir", iso.Workspace, "--", name)
return append(argv, args...), nil
}
// cacheArgs binds the shared warm caches (S18): for a trunk build read-write, so the next build is warm; for any other
// read-only beneath an overlay whose writes go to a tmpfs of the step's own, so the step writes as a build does and
// nothing it wrote outlives it (bubblewrap's --tmp-overlay, an overlay mount in the step's user namespace).
func cacheArgs(caches []string, write bool) []string {
var argv []string
for _, c := range caches {
if write {
argv = append(argv, "--bind", c, c)
continue
}
argv = append(argv, "--overlay-src", c, "--tmp-overlay", c)
}
return argv
}
// loopbackHosts is the hosts file of a step: the loopback and nothing else.
const loopbackHosts = "127.0.0.1 localhost\n::1 localhost\n"
// Isolated is run with every command shut off as iso says (Command), run's env going inside the step: the build modules'
// run steps call their test, build and digest commands through it, never run itself. Each command gets a hosts file of
// its own, naming only the loopback, removed when it ends.
func Isolated(run forgekey.Runner, iso Isolation) forgekey.Runner {
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
dir, err := os.MkdirTemp("", "step-hosts-")
if err != nil {
return forgekey.Ran{Status: 1, Err: "the step's hosts file: " + err.Error()}
}
defer os.RemoveAll(dir)
hosts := filepath.Join(dir, "hosts")
if err := os.WriteFile(hosts, []byte(loopbackHosts), 0o644); err != nil {
return forgekey.Ran{Status: 1, Err: "the step's hosts file: " + err.Error()}
}
iso := iso
iso.Hosts = hosts
argv, err := iso.command(env, name, args...)
if err != nil {
return forgekey.Ran{Status: 1, Err: err.Error()}
}
return run(ctx, nil, Bubblewrap, argv...)
}
}
+30
View File
@@ -0,0 +1,30 @@
package buildstep
import "testing"
// The shared warm caches are read under a throwaway writable layer (novox/hq ADR 0448; issue 1309): a candidate's step
// writes into them as a build does, and nothing it writes outlives it or reaches a later build; only a trunk build
// writes them.
func TestACandidatesStepWritesTheCachesOnlyIntoAThrowawayLayer(t *testing.T) {
fixedSystem(t)
iso := Isolation{Workspace: "/w", Caches: []string{"/var/cache/go-build", "/var/cache/npm"}}
argv, err := iso.Command("go", "test")
if err != nil {
t.Fatal(err)
}
for _, c := range iso.Caches {
if !holds(argv, "--overlay-src", c, "--tmp-overlay", c) {
t.Errorf("the cache %s is not under a throwaway layer: %v", c, argv)
}
if holds(argv, "--bind", c, c) {
t.Errorf("a candidate's step writes the cache %s: %v", c, argv)
}
}
iso.WriteCaches = true
argv, _ = iso.Command("go", "build")
for _, c := range iso.Caches {
if !holds(argv, "--bind", c, c) || holds(argv, "--tmp-overlay", c) {
t.Errorf("a trunk build does not write the cache %s: %v", c, argv)
}
}
}
+66
View File
@@ -0,0 +1,66 @@
//go:build integration && isolation
package buildstep
import (
"context"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// 1309/S19, the issue's proof: a run step, in bubblewrap as Isolated runs it, reads no key and no login, resolves no
// name, reaches no TCP or unix socket of the machine, finds no home of the account, and leaves the shared cache it
// wrote unchanged; the workspace it writes keeps what it wrote. It needs bubblewrap and unprivileged user namespaces
// (issue 1308) on the machine that runs it, and fails naming them when they are not there.
func TestARunStepReachesNothingAndLeavesTheCachesUnchanged(t *testing.T) {
if _, err := exec.LookPath(Bubblewrap); err != nil {
t.Fatalf("bubblewrap is not installed here, so the isolation of a run step cannot be proved (issue 1309): %v", err)
}
root := t.TempDir()
account := filepath.Join(root, "account")
key, login := filepath.Join(account, ".ssh", "id_ed25519"), filepath.Join(account, "bus-login.json")
writeFile(t, filepath.Dir(key), filepath.Base(key), "PRIVATE KEY")
writeFile(t, account, "bus-login.json", `{"password":"secret"}`)
cacheDir := filepath.Join(root, "cache")
writeFile(t, cacheDir, "warm", "warm\n")
workspace := filepath.Join(root, "work", "tree")
if err := os.MkdirAll(workspace, 0o755); err != nil {
t.Fatal(err)
}
probe := filepath.Join(workspace, "probe")
if out, err := exec.Command("go", "build", "-o", probe, "./testdata/isolated-probe").CombinedOutput(); err != nil {
t.Fatalf("building the probe: %v\n%s", err, out)
}
tcp, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer tcp.Close()
socket := filepath.Join(account, "bus.sock")
unix, err := net.Listen("unix", socket)
if err != nil {
t.Fatal(err)
}
defer unix.Close()
run := Isolated(forgekey.ExecRunnerFor(0), Isolation{Workspace: workspace, Caches: []string{cacheDir}})
r := run(context.Background(), []string{"PATH=/usr/bin:/bin"}, probe, key, login, tcp.Addr().String(), socket, "localhost.localdomain.invalid", filepath.Join(cacheDir, "warm"))
if r.Status != 0 || r.Err != "" || strings.TrimSpace(r.Stdout) != "shut off" {
t.Fatalf("the run step was not shut off (status %d, %s):\n%s%s", r.Status, r.Err, r.Stdout, r.Stderr)
}
if b, _ := os.ReadFile(filepath.Join(cacheDir, "warm")); string(b) != "warm\n" {
t.Fatalf("the step changed the shared cache: %q", b)
}
// The workspace is the step's own: what it writes there stays.
r = run(context.Background(), nil, "/bin/sh", "-c", "echo built > out")
if b, err := os.ReadFile(filepath.Join(workspace, "out")); r.Status != 0 || err != nil || string(b) != "built\n" {
t.Fatalf("the step's write to its workspace did not stay (status %d, %s%s): %q %v", r.Status, r.Stderr, r.Err, b, err)
}
}
+115
View File
@@ -0,0 +1,115 @@
package buildstep
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// pairs says whether argv holds the words of want one after another.
func holds(argv []string, want ...string) bool {
for i := 0; i+len(want) <= len(argv); i++ {
match := true
for j, w := range want {
if argv[i+j] != w {
match = false
break
}
}
if match {
return true
}
}
return false
}
func fixedSystem(t *testing.T) {
t.Helper()
before := systemArgs
systemArgs = func() []string { return []string{"--ro-bind", "/usr", "/usr", "--symlink", "usr/bin", "/bin"} }
t.Cleanup(func() { systemArgs = before })
}
// A run step's command runs in bubblewrap (novox/hq ADR 0448; issue 1309): no network, no namespace shared, an
// environment of exactly what the step is given, the workspace the one tree it writes, the toolchain read-only, and
// nothing else of the machine's: not the account's home, not the forge key or a bus login, not the runtime directory,
// and no name service.
func TestARunStepRunsShutOffInBubblewrap(t *testing.T) {
fixedSystem(t)
iso := Isolation{Workspace: "/work/ab12/tree", ReadOnly: []string{"/usr/lib/go"}, Writable: []string{"/work/ab12/out"},
Env: []string{"GOFLAGS=-mod=mod", "PATH=/usr/bin"}, Hosts: "/work/ab12/hosts"}
argv, err := iso.Command("go", "test", "./...")
if err != nil {
t.Fatal(err)
}
for _, want := range [][]string{
{"--unshare-all"}, {"--die-with-parent"}, {"--new-session"}, {"--clearenv"},
{"--proc", "/proc"}, {"--dev", "/dev"}, {"--tmpfs", "/tmp"},
{"--ro-bind", "/usr", "/usr"}, {"--symlink", "usr/bin", "/bin"},
{"--ro-bind", "/usr/lib/go", "/usr/lib/go"},
{"--bind", "/work/ab12/tree", "/work/ab12/tree"}, {"--bind", "/work/ab12/out", "/work/ab12/out"},
{"--ro-bind", "/work/ab12/hosts", "/etc/hosts"},
{"--setenv", "GOFLAGS", "-mod=mod"}, {"--setenv", "PATH", "/usr/bin"}, {"--setenv", "HOME", "/tmp/home"},
{"--chdir", "/work/ab12/tree"},
{"--", "go", "test", "./..."},
} {
if !holds(argv, want...) {
t.Errorf("the command has no %v: %v", want, argv)
}
}
if holds(argv, "--share-net") || strings.Contains(strings.Join(argv, " "), "resolv.conf") || strings.Contains(strings.Join(argv, " "), "/run") {
t.Errorf("the command reaches the network, the name service or the runtime directory: %v", argv)
}
// The workspace is mounted after /tmp's tmpfs, so a workspace under /tmp is not hidden by it.
if strings.Index(strings.Join(argv, "\x00"), "--tmpfs\x00/tmp") > strings.Index(strings.Join(argv, "\x00"), "--bind\x00/work/ab12/tree") {
t.Errorf("the tmpfs of /tmp comes after the workspace: %v", argv)
}
// A HOME the step is given is its own.
iso.Env = append(iso.Env, "HOME=/work/ab12/home")
argv, _ = iso.Command("true")
if holds(argv, "--setenv", "HOME", "/tmp/home") || !holds(argv, "--setenv", "HOME", "/work/ab12/home") {
t.Errorf("the step's own HOME was replaced: %v", argv)
}
}
func TestAnIsolationNamesNoPathItCannotPin(t *testing.T) {
for _, iso := range []Isolation{
{Workspace: ""},
{Workspace: "tree"},
{Workspace: "/"},
{Workspace: "/w", ReadOnly: []string{"relative"}},
{Workspace: "/w", Writable: []string{"/"}},
{Workspace: "/w", Caches: []string{"/a/../b"}},
{Workspace: "/w", Env: []string{"NOEQUALS"}},
} {
if _, err := iso.Command("true"); err == nil {
t.Errorf("%+v was accepted", iso)
}
}
}
// Isolated runs every command through bwrap, with the step's environment inside and a hosts file that names only the
// loopback, removed after.
func TestIsolatedRunsEachCommandThroughBubblewrap(t *testing.T) {
fixedSystem(t)
var ran []string
var hosts string
inner := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
ran = append([]string{name}, args...)
for i, a := range args {
if a == "/etc/hosts" && i > 0 {
hosts = args[i-1]
}
}
return forgekey.Ran{Stdout: "ok"}
}
r := Isolated(inner, Isolation{Workspace: "/w"})(context.Background(), []string{"A=1"}, "go", "build")
if r.Stdout != "ok" || ran[0] != Bubblewrap || !holds(ran, "--setenv", "A", "1") || !holds(ran, "--", "go", "build") {
t.Fatalf("ran %v: %+v", ran, r)
}
if hosts == "" {
t.Fatalf("no hosts file was bound: %v", ran)
}
}
+24
View File
@@ -0,0 +1,24 @@
package buildstep
import "strings"
// LauncherSuffix is what a TypeScript entrypoint's launcher is called beside it: index.js is started as
// index.serve.mjs (novox/hq ADR 0193), an ES module by its own extension whatever the bundle's package.json says.
const LauncherSuffix = ".serve.mjs"
// Launchers are the launchers a bundle's build writes, by entrypoint, relative to the bundle's root: for TypeScript
// every entrypoint `x.js` is started by `x.serve.mjs` beside it; an entrypoint that is not JavaScript and every other
// language, whose build is already executable, get none. The one rule mesh-build-node, build-agent and the controller
// share (R4e), so the name a build writes is the name a reader expects.
func Launchers(language string, entrypoints []string) map[string]string {
out := map[string]string{}
if language != "typescript" {
return out
}
for _, entry := range entrypoints {
if strings.HasSuffix(entry, ".js") {
out[entry] = strings.TrimSuffix(entry, ".js") + LauncherSuffix
}
}
return out
}
+22
View File
@@ -0,0 +1,22 @@
package buildstep
import (
"reflect"
"testing"
)
// A TypeScript bundle's entrypoint x.js is started as x.serve.mjs beside it (novox/hq ADR 0193), the rule mesh-build-node
// and build-agent both apply; an entrypoint that is not .js gets none, and another language gets none.
func TestATypeScriptEntrypointIsStartedByTheLauncherBesideIt(t *testing.T) {
got := Launchers("typescript", []string{"index.js", "tools/x.js", "data.json"})
want := map[string]string{"index.js": "index.serve.mjs", "tools/x.js": "tools/x.serve.mjs"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Launchers(typescript) = %v, want %v", got, want)
}
if got := Launchers("go", []string{"index.js"}); len(got) != 0 {
t.Fatalf("a Go bundle got launchers: %v", got)
}
if got := Launchers("typescript", nil); len(got) != 0 {
t.Fatalf("a bundle with no entrypoint got launchers: %v", got)
}
}
+112
View File
@@ -0,0 +1,112 @@
package buildstep
import (
"context"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// MergeResult is what MergeInto made of a head and a trunk commit: the trunk commit (Base) and the head, whole, and the
// merge commit and the paths the merge changes against the trunk commit; or, when the head does not merge cleanly, the
// files in conflict, with Merge and Changed empty.
type MergeResult struct {
Base, Head, Merge string
Conflicts []string
Changed []string
}
// MergeInto makes, in the workspace dir that FetchCommit left at commit, the merge of commit into the trunk commit
// base, as the forge's default merge would (a merge commit, never a fast-forward), and leaves the workspace at the merge
// (novox/hq ADR 0484 clause 3, ADR 0323). It is the one merge the build modules' fetch steps make, with the identity
// and flags of the controller's check merge (mesh-controller internal/builder/check_merge.go): a name nobody owns, no
// signature, no hook, nothing of the machine's git configuration that changes a merge. Nothing is pushed.
//
// A merge needs both histories, so it makes the fetch whole (the shallow clone FetchCommit made is deepened) and
// fetches base by its sha. A head that does not merge cleanly is an answer, not an error: the files in conflict, the
// merge undone and the workspace left at base. A head already in base answers a Merge equal to Base and no changed
// paths. An error is that the merge could not be tried: a base or commit that is not a full sha, no base, a workspace
// not at commit, or a fetch or git command that failed. remote and the options are those of FetchCommit; the runner's
// bound must allow a whole fetch.
func MergeInto(ctx context.Context, run forgekey.Runner, dir, remote, base, commit string, opts ...Option) (MergeResult, error) {
var o fetchOptions
for _, opt := range opts {
opt(&o)
}
res := MergeResult{Base: base, Head: commit}
if base == "" {
return res, fmt.Errorf("buildstep: no base: a merge needs the trunk commit it is made onto")
}
if err := checkFetch(o, dir, remote, base, ""); err != nil {
return res, err
}
if err := checkFetch(o, dir, remote, commit, ""); err != nil {
return res, err
}
env := []string{"GIT_TERMINAL_PROMPT=0"}
git := func(args ...string) (string, error) {
full := append([]string{"-c", "credential.helper=", "-C", dir}, args...)
r := run(ctx, env, "git", full...)
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("git %s: %s", args[0], firstLine(r.Err+" "+r.Stderr))
}
return r.Stdout, nil
}
at, err := git("rev-parse", "HEAD")
if err != nil {
return res, err
}
if head := strings.TrimSpace(at); head != commit {
return res, fmt.Errorf("buildstep: the workspace is at %s, not at the commit %s", head, commit)
}
shallow, err := git("rev-parse", "--is-shallow-repository")
if err != nil {
return res, err
}
fetch := []string{"fetch", "--no-tags", "origin", base}
if strings.TrimSpace(shallow) == "true" {
fetch = []string{"fetch", "--no-tags", "--unshallow", "origin", base}
}
if _, err := git(fetch...); err != nil {
return res, fmt.Errorf("the trunk commit %s: %w", base, err)
}
if _, err := git("checkout", "-q", "--detach", base); err != nil {
return res, fmt.Errorf("the trunk commit %s cannot be checked out: %w", base, err)
}
message := fmt.Sprintf("Check the merge of %s into %s (made for a check, never pushed)", short(commit), short(base))
args := append(append([]string{}, mergeIdentity...), "merge", "--no-ff", "--no-edit", "--no-verify", "--quiet", "-m", message, commit)
if _, err := git(args...); err != nil {
out, _ := git("diff", "--name-only", "--diff-filter=U")
res.Conflicts = strings.Fields(out)
_, _ = git("merge", "--abort")
if len(res.Conflicts) == 0 {
return res, fmt.Errorf("the head %s could not be merged into the trunk commit %s: %w", short(commit), short(base), err)
}
return res, nil
}
merge, err := git("rev-parse", "HEAD")
if err != nil {
return res, fmt.Errorf("the merge cannot be read: %w", err)
}
res.Merge = strings.TrimSpace(merge)
changed, err := git("diff", "--name-only", "--no-renames", base, res.Merge)
if err != nil {
return res, fmt.Errorf("the paths the merge changes cannot be read: %w", err)
}
res.Changed = strings.Fields(changed)
return res, nil
}
// mergeIdentity is who a check's merge is made as: it is never pushed, so it names nobody's account. No hook of the
// clone's or the build machine's git configuration runs, and no setting there changes the merge (the controller's
// check merge, internal/builder/check_merge.go).
var mergeIdentity = []string{"-c", "user.name=mesh merge check", "-c", "user.email=merge-check@mesh.invalid",
"-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "-c", "merge.verifySignatures=false"}
func short(commit string) string {
if len(commit) > 12 {
return commit[:12]
}
return commit
}
+172
View File
@@ -0,0 +1,172 @@
//go:build integration
package buildstep
import (
"context"
"os"
"os/exec"
"path/filepath"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// The merge on real git repositories starts git processes, so it is an integration test (the unit level keeps its bound,
// issue 895); the unit tests of merge_test.go prove the commands and the refusals with a fake runner.
func realGit(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.name=t", "-c", "user.email=t@example.invalid", "-c", "commit.gpgsign=false"}, args...)...)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
func writeFile(t *testing.T, dir, name, content string) {
t.Helper()
p := filepath.Join(dir, name)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// mergeRig is a local bare repository standing in for the forge, a work repository that pushes to it, and an empty
// workspace: the fetch step's shape (FetchCommit of the head into the workspace, then MergeInto).
type mergeRig struct {
t *testing.T
bare, work, ws string
fork, trunk string // the commit both histories share, and the trunk's head
remote string
}
func newMergeRig(t *testing.T) *mergeRig {
root := t.TempDir()
r := &mergeRig{t: t, bare: filepath.Join(root, "r.git"), work: filepath.Join(root, "work"), ws: filepath.Join(root, "ws")}
for _, d := range []string{r.bare, r.work, r.ws} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
r.remote = "file://" + r.bare
realGit(t, r.bare, "init", "-q", "--bare")
realGit(t, r.work, "init", "-q", "-b", "main")
writeFile(t, r.work, "a.txt", "one\ntwo\nthree\n")
writeFile(t, r.work, "modules/m/module.json", "{}\n")
realGit(t, r.work, "add", ".")
realGit(t, r.work, "commit", "-q", "-m", "fork point")
r.fork = realGit(t, r.work, "rev-parse", "HEAD")
return r
}
// head makes a commit on a branch off the fork point, pushes it as the pull request's head ref and answers its sha.
func (r *mergeRig) head(files map[string]string) string {
realGit(r.t, r.work, "checkout", "-q", "-B", "pr", r.fork)
for n, c := range files {
writeFile(r.t, r.work, n, c)
}
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "head")
h := realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "push", "-q", r.bare, "HEAD:refs/pull/1/head")
return h
}
// advance moves the trunk past the fork point with a commit changing files, pushes it and answers its sha.
func (r *mergeRig) advance(files map[string]string) string {
realGit(r.t, r.work, "checkout", "-q", "-B", "main", r.fork)
for n, c := range files {
writeFile(r.t, r.work, n, c)
}
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "trunk moves")
r.trunk = realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "push", "-q", "-f", r.bare, "HEAD:refs/heads/main")
return r.trunk
}
func (r *mergeRig) fetch(head string) {
if err := FetchCommit(context.Background(), forgekey.ExecRunner, r.ws, r.remote, head, "refs/pull/1/head", AllowFileRemote()); err != nil {
r.t.Fatal(err)
}
}
// 1674/AC1: a clean merge answers the merge commit and the changed files, and leaves the workspace at the merge, which
// holds both sides' changes.
func TestMergeIntoACleanMergeAnswersTheMergeCommitAndTheChangedFiles(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"})
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil {
t.Fatal(err)
}
if res.Base != base || res.Head != head || len(res.Merge) != 40 || res.Merge == base || res.Merge == head || len(res.Conflicts) != 0 {
t.Fatalf("result = %+v", res)
}
if !reflect.DeepEqual(res.Changed, []string{"b.txt"}) {
t.Errorf("changed = %v, want only what the head adds to the trunk: b.txt", res.Changed)
}
if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != res.Merge {
t.Errorf("the workspace is at %s, not at the merge %s", at, res.Merge)
}
for f, want := range map[string]string{"b.txt": "head\n", "a.txt": "one\ntwo\nthree\nfour\n"} {
if got, _ := os.ReadFile(filepath.Join(r.ws, f)); string(got) != want {
t.Errorf("%s = %q, want %q", f, got, want)
}
}
if parents := strings.Fields(realGit(t, r.ws, "rev-list", "--parents", "-n", "1", res.Merge)); len(parents) != 3 || parents[1] != base || parents[2] != head {
t.Errorf("the merge commit's parents = %v, want the trunk commit and the head", parents)
}
if author := realGit(t, r.ws, "log", "-1", "--format=%an <%ae>", res.Merge); author != "mesh merge check <merge-check@mesh.invalid>" {
t.Errorf("the merge is made as %q", author)
}
}
// 1674/AC2: a conflict is an answer: the files in conflict, no merge commit, and the workspace left at the trunk commit.
func TestMergeIntoAConflictAnswersTheFilesAndLeavesTheTreeAtTheTrunk(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"a.txt": "one\nHEAD\nthree\n", "c.txt": "c\n"})
base := r.advance(map[string]string{"a.txt": "one\nTRUNK\nthree\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil {
t.Fatalf("a conflict is an answer, not an error: %v", err)
}
if !reflect.DeepEqual(res.Conflicts, []string{"a.txt"}) || res.Merge != "" || len(res.Changed) != 0 || res.Base != base || res.Head != head {
t.Fatalf("result = %+v", res)
}
if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != base {
t.Errorf("the workspace is at %s, not at the trunk commit %s", at, base)
}
if got, _ := os.ReadFile(filepath.Join(r.ws, "a.txt")); string(got) != "one\nTRUNK\nthree\n" {
t.Errorf("a.txt = %q, want the trunk's", got)
}
if st := realGit(t, r.ws, "status", "--porcelain"); st != "" {
t.Errorf("the tree is not clean after the aborted merge: %q", st)
}
if _, err := os.Stat(filepath.Join(r.ws, "c.txt")); err == nil {
t.Error("a file of the head is in a tree left at the trunk")
}
}
// A head already in the trunk merges into nothing: the merge is the trunk commit and no path changes.
func TestMergeIntoAHeadAlreadyInTheTrunkChangesNothing(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"})
r.fork = head // the trunk continues from the head
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil || res.Merge != base || len(res.Changed) != 0 || len(res.Conflicts) != 0 {
t.Errorf("result = %+v, %v", res, err)
}
}
+74
View File
@@ -0,0 +1,74 @@
package buildstep
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// 1674/AC3: no base is refused before anything runs; so is a base or a commit that is not a full sha, a workspace that
// is not at the commit, and a remote that is not ssh://.
func TestMergeIntoRefusesWhatIsNotAMerge(t *testing.T) {
f := &fakeGit{}
for name, c := range map[string]struct{ remote, base, commit string }{
"no base": {"ssh://mesh-forge/o/r.git", "", sha},
"a short base": {"ssh://mesh-forge/o/r.git", "abc123", sha},
"a branch as base": {"ssh://mesh-forge/o/r.git", "main", sha},
"a short commit": {"ssh://mesh-forge/o/r.git", sha, "abc"},
"an https remote": {"https://forge.example/o/r.git", sha, sha},
} {
_, err := MergeInto(context.Background(), f.run, "/w", c.remote, c.base, c.commit)
if err == nil {
t.Errorf("%s accepted", name)
}
}
if len(f.calls) != 0 {
t.Errorf("git ran for a refused merge: %v", f.calls)
}
g := &fakeGit{rule: func(l string) forgekey.Ran {
if strings.Contains(l, "rev-parse HEAD") {
return forgekey.Ran{Stdout: strings.Repeat("f", 40) + "\n"}
}
return forgekey.Ran{}
}}
_, err := MergeInto(context.Background(), g.run, "/w", "ssh://mesh-forge/o/r.git", strings.Repeat("a", 40), sha)
if err == nil || !strings.Contains(err.Error(), "not at the commit") || len(g.calls) != 1 {
t.Errorf("a workspace at another commit: %v after %d commands", err, len(g.calls))
}
}
// 1674/AC4: the merge is made with the check merge's identity and flags, hooks off, and fetches with no credential.
func TestMergeIntoRunsTheCheckMergesCommands(t *testing.T) {
base := strings.Repeat("b", 40)
f := &fakeGit{rule: func(l string) forgekey.Ran {
switch {
case strings.HasSuffix(l, "rev-parse HEAD"):
return forgekey.Ran{Stdout: sha + "\n"}
case strings.Contains(l, "--is-shallow-repository"):
return forgekey.Ran{Stdout: "true\n"}
}
return forgekey.Ran{}
}}
if _, err := MergeInto(context.Background(), f.run, "/w", "ssh://mesh-forge/o/r.git", base, sha); err != nil {
t.Fatal(err)
}
all := strings.Join(f.calls, "\n")
for _, want := range []string{
pre + "fetch --no-tags --unshallow origin " + base,
pre + "checkout -q --detach " + base,
"user.name=mesh merge check", "user.email=merge-check@mesh.invalid", "commit.gpgsign=false", "core.hooksPath=/dev/null",
"merge --no-ff --no-edit --no-verify --quiet -m",
pre + "diff --name-only --no-renames " + base,
} {
if !strings.Contains(all, want) {
t.Errorf("the commands lack %q:\n%s", want, all)
}
}
for _, c := range f.calls {
if strings.Contains(c, "https://") || !strings.Contains(c, "credential.helper= ") {
t.Errorf("command %q carries a credential or https", c)
}
}
}
+71
View File
@@ -0,0 +1,71 @@
package buildstep
import (
"fmt"
"os"
"path"
"path/filepath"
"strings"
)
// narrowKept are the files of a module's directory a narrowed build keeps beside its named paths: its manifest and
// its lock files, each an input of the module's key (ModuleTree's Manifest and Locks). package.json and go.mod are
// not kept: neither is in the named tree, so a build that needs one names it among its artifact's paths, which puts
// it in the key (novox/hq ADR 0499 clause 4).
var narrowKept = append([]string{"module.json"}, lockFiles...)
// Narrow leaves in the fetched tree only what a build of an artifact that names its paths may see (novox/hq ADR 0499
// clause 4): r.Dir's named paths, its manifest and lock files, and .git; everything else is removed, so a file outside
// them cannot change what is built without changing its key. A named path absent at the tree is an error naming it,
// and nothing is removed then. With no paths in r it changes nothing. Both build modules' fetch steps call it.
func Narrow(tree string, r FetchRequest) error {
if len(r.Paths) == 0 {
return nil
}
keep := map[string]bool{".git": true}
for _, p := range r.Paths {
rel := path.Join(r.Dir, p)
if strings.HasPrefix(rel, "../") || rel == ".." || path.IsAbs(p) {
return fmt.Errorf("the named path %q leaves the module's directory", p)
}
if _, err := os.Lstat(filepath.Join(tree, filepath.FromSlash(rel))); err != nil {
return fmt.Errorf("the named path %q is not in the fetched tree: %w", p, err)
}
keep[rel] = true
}
for _, f := range narrowKept {
keep[path.Join(r.Dir, f)] = true
}
return prune(tree, "", keep)
}
// prune removes from dir (rel within the tree) every entry that is neither kept nor on the way to one.
func prune(root, rel string, keep map[string]bool) error {
entries, err := os.ReadDir(filepath.Join(root, filepath.FromSlash(rel)))
if err != nil {
return err
}
for _, e := range entries {
p := path.Join(rel, e.Name())
if keep[p] {
continue
}
ancestor := false
for k := range keep {
if strings.HasPrefix(k, p+"/") {
ancestor = true
break
}
}
if ancestor && e.IsDir() {
if err := prune(root, p, keep); err != nil {
return err
}
continue
}
if err := os.RemoveAll(filepath.Join(root, filepath.FromSlash(p))); err != nil {
return err
}
}
return nil
}
+82
View File
@@ -0,0 +1,82 @@
package buildstep
import (
"os"
"path/filepath"
"sort"
"strings"
"testing"
)
func filesUnder(t *testing.T, root string) []string {
t.Helper()
var out []string
_ = filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
if err == nil && !info.IsDir() {
rel, _ := filepath.Rel(root, p)
out = append(out, filepath.ToSlash(rel))
}
return nil
})
sort.Strings(out)
return out
}
// A build of an artifact that names its paths sees only those paths, its manifest and its lock files (novox/hq ADR 0499
// clause 4): Narrow removes everything else of the fetched tree but .git, package.json and go.mod among it, since
// neither is in the named tree nor among the lock files; with no paths it changes nothing.
func TestNarrowLeavesOnlyTheNamedPathsTheManifestAndTheLocks(t *testing.T) {
tree := t.TempDir()
for _, f := range []string{".git/HEAD", "module.json", "package.json", "package-lock.json", "go.sum", "src/index.ts",
"src/deep/a.ts", "test/a.test.ts", "tsconfig.json", "go/delivery/x.go", "README.md", "other/module.json"} {
writeTreeFile(t, tree, f, "x")
}
if err := Narrow(tree, FetchRequest{}); err != nil {
t.Fatal(err)
}
if got := len(filesUnder(t, tree)); got != 12 {
t.Fatalf("a request without paths changed the tree: %d files", got)
}
if err := Narrow(tree, FetchRequest{Dir: ".", Paths: []string{"src", "test", "tsconfig.json"}}); err != nil {
t.Fatal(err)
}
want := ".git/HEAD,go.sum,module.json,package-lock.json,src/deep/a.ts,src/index.ts,test/a.test.ts,tsconfig.json"
if got := strings.Join(filesUnder(t, tree), ","); got != want {
t.Fatalf("the narrowed tree holds %s\nwant %s", got, want)
}
}
func TestNarrowKeepsAModuleDirectorysPathsAndFailsOnAnAbsentOne(t *testing.T) {
tree := t.TempDir()
for _, f := range []string{".git/config", "modules/a/module.json", "modules/a/go.mod", "modules/a/src/x.ts",
"modules/a/notes.md", "modules/b/module.json", "root.txt"} {
writeTreeFile(t, tree, f, "x")
}
err := Narrow(tree, FetchRequest{Dir: "modules/a", Paths: []string{"src", "gone"}})
if err == nil || !strings.Contains(err.Error(), `"gone"`) {
t.Fatalf("an absent named path gave %v", err)
}
if len(filesUnder(t, tree)) != 7 {
t.Fatal("a refused narrowing removed files")
}
if err := Narrow(tree, FetchRequest{Dir: "modules/a", Paths: []string{"src"}}); err != nil {
t.Fatal(err)
}
if got := strings.Join(filesUnder(t, tree), ","); got != ".git/config,modules/a/module.json,modules/a/src/x.ts" {
t.Fatalf("the narrowed tree holds %s", got)
}
}
// A build that needs package.json or go.mod names it among its paths, and so keys on it (ADR 0499 clause 4).
func TestNarrowKeepsAManifestOfTheLanguageOnlyWhenNamed(t *testing.T) {
tree := t.TempDir()
for _, f := range []string{"module.json", "package.json", "go.mod", "src/a.ts"} {
writeTreeFile(t, tree, f, "x")
}
if err := Narrow(tree, FetchRequest{Dir: ".", Paths: []string{"src", "package.json"}}); err != nil {
t.Fatal(err)
}
if got := strings.Join(filesUnder(t, tree), ","); got != "module.json,package.json,src/a.ts" {
t.Fatalf("the narrowed tree holds %s", got)
}
}
+115
View File
@@ -0,0 +1,115 @@
package buildstep
import (
"fmt"
"path"
"regexp"
"strings"
)
var (
requestCommitOK = regexp.MustCompile(`^[0-9a-f]{40}$`)
requestRepositoryOK = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`)
)
// FetchRequest is what a build module's fetch tool is asked: the repository and the full commit, the pull request's
// ref when only its head carries the commit, the trunk commit to merge onto for a merge preview, and, for a build of a
// module whose every artifact names its paths, that module's directory and its named paths (novox/hq ADR 0499 clause 4).
//
// **The paths come from the builder, not from a manifest** (ADR 0495 clause 1; the lead's decision L8 of R4e): the
// builder fills Dir and Paths from the indexer's Module value (Path, and KeptInputs2.Paths' keys), and the build module
// gives the build and test steps a workspace holding those paths, the manifest and its lock files. A request without
// them builds from the module's directory, as before.
type FetchRequest struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Ref string `json:"ref,omitempty"`
Base string `json:"base,omitempty"`
Dir string `json:"dir,omitempty"`
Paths []string `json:"paths,omitempty"`
}
// FetchRequestProperties are the JSON schema properties of FetchRequest's dir and paths, for a fetch tool's Input.
func FetchRequestProperties() map[string]any {
return map[string]any{
"dir": map[string]any{"type": "string", "description": "the module's directory in the repository (\".\" for its root), with paths"},
"paths": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "the paths the module names, relative to dir, from the indexer's kept inputs: the build sees only these, the manifest and its lock files"},
}
}
// Args is r as a tool call's arguments.
func (r FetchRequest) Args() map[string]any {
args := map[string]any{"repository": r.Repository, "commit": r.Commit}
for k, v := range map[string]string{"ref": r.Ref, "base": r.Base, "dir": r.Dir} {
if v != "" {
args[k] = v
}
}
if len(r.Paths) > 0 {
args["paths"] = append([]string(nil), r.Paths...)
}
return args
}
// ParseFetchRequest reads a fetch tool's arguments; arguments it does not know (a build module's own) are left to the
// caller. It refuses a repository that is not <owner>/<name>, a commit or a base that is not 40 hex digits, paths with no
// module directory or a directory with no path, a directory leaving the repository and a path leaving the directory,
// naming each.
func ParseFetchRequest(args map[string]any) (FetchRequest, error) {
var r FetchRequest
r.Repository, _ = args["repository"].(string)
r.Commit, _ = args["commit"].(string)
r.Ref, _ = args["ref"].(string)
r.Base, _ = args["base"].(string)
r.Dir, _ = args["dir"].(string)
switch v := args["paths"].(type) {
case nil:
case []string:
r.Paths = append([]string(nil), v...)
case []any:
for _, p := range v {
s, ok := p.(string)
if !ok {
return FetchRequest{}, fmt.Errorf("paths is a list of strings")
}
r.Paths = append(r.Paths, s)
}
default:
return FetchRequest{}, fmt.Errorf("paths is a list of strings")
}
return r, r.Check()
}
// Check is what ParseFetchRequest refuses.
func (r FetchRequest) Check() error {
if !requestRepositoryOK.MatchString(r.Repository) || strings.Contains(r.Repository, "..") {
return fmt.Errorf("%q is not a repository, <owner>/<name>", r.Repository)
}
if !requestCommitOK.MatchString(r.Commit) {
return fmt.Errorf("%q is not a full commit (40 hex digits)", r.Commit)
}
if r.Base != "" && !requestCommitOK.MatchString(r.Base) {
return fmt.Errorf("%q is not a trunk commit (40 hex digits)", r.Base)
}
switch {
case len(r.Paths) > 0 && r.Dir == "":
return fmt.Errorf("the request names paths and names no module directory (dir) they are relative to")
case len(r.Paths) == 0 && r.Dir != "":
return fmt.Errorf("the request names a module directory and no path")
}
if r.Dir != "" && (outside(r.Dir) || strings.HasPrefix(r.Dir, "/")) {
return fmt.Errorf("the module directory %q leaves the repository", r.Dir)
}
for _, p := range r.Paths {
if p == "" || strings.HasPrefix(p, "/") || outside(p) || path.Clean(p) == "." {
return fmt.Errorf("the path %q leaves the module's directory or is not a path in it", p)
}
}
return nil
}
func outside(p string) bool {
c := path.Clean(p)
return c == ".." || strings.HasPrefix(c, "../")
}
+57
View File
@@ -0,0 +1,57 @@
package buildstep
import (
"reflect"
"strings"
"testing"
)
// A build's fetch is handed the paths of a module that names them (novox/hq ADR 0499 clause 4): the builder fills them
// from the indexer's kept inputs and the build module reads them from the call, never from the manifest (ADR 0495
// clause 1). The request survives the trip through a tool call's arguments.
func TestAFetchRequestCarriesTheNamedPathsThroughTheToolCall(t *testing.T) {
req := FetchRequest{Repository: "novox/mesh-sdk", Commit: strings.Repeat("a", 40), Ref: "refs/pull/7/head",
Dir: ".", Paths: []string{"package.json", "src"}}
args := req.Args()
// A tool call's arguments arrive as JSON: lists of any.
args["paths"] = []any{"package.json", "src"}
got, err := ParseFetchRequest(args)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, req) {
t.Fatalf("got %+v, want %+v", got, req)
}
plain, err := ParseFetchRequest(map[string]any{"repository": "novox/x", "commit": strings.Repeat("b", 40),
"base": strings.Repeat("c", 40), "beside": []any{}})
if err != nil || plain.Paths != nil || plain.Base != strings.Repeat("c", 40) {
t.Fatalf("a request without paths: %+v, %v", plain, err)
}
ok := map[string]any{"repository": "novox/x", "commit": strings.Repeat("b", 40)}
for _, c := range []struct {
set map[string]any
says string
}{
{map[string]any{"repository": "novox"}, "is not a repository"},
{map[string]any{"repository": "novox/../x"}, "is not a repository"},
{map[string]any{"commit": "abc"}, "is not a full commit"},
{map[string]any{"base": "abc"}, "is not a trunk commit"},
{map[string]any{"paths": []any{"src"}}, "names no module directory"},
{map[string]any{"dir": "modules/a", "paths": []any{"../b"}}, "leaves the module's directory"},
{map[string]any{"dir": "../a", "paths": []any{"src"}}, "leaves the repository"},
{map[string]any{"dir": "modules/a", "paths": []any{3}}, "paths is a list of strings"},
{map[string]any{"dir": "modules/a"}, "names a module directory and no path"},
} {
args := map[string]any{}
for k, v := range ok {
args[k] = v
}
for k, v := range c.set {
args[k] = v
}
if _, err := ParseFetchRequest(args); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%v: %v, wanted a refusal saying %q", c.set, err, c.says)
}
}
}
+49
View File
@@ -0,0 +1,49 @@
// The probe a run step's isolation is proved with (novox/hq ADR 0448; issue 1309, S19): run inside the step, it says
// on stdout every way it reached what a step must not, and "shut off" when it reached none. Its arguments: the files
// it must not read (the forge key, a bus login), a TCP address and a unix socket of the machine it must not reach, a
// name it must not resolve, and a cache file it writes, which must be unchanged after the step.
package main
import (
"fmt"
"net"
"os"
"time"
)
func main() {
if len(os.Args) != 7 {
fmt.Println("usage: probe <key> <login> <tcp address> <unix socket> <name> <cache file>")
os.Exit(2)
}
key, login, tcp, socket, name, cache := os.Args[1], os.Args[2], os.Args[3], os.Args[4], os.Args[5], os.Args[6]
reached := 0
say := func(f string, a ...any) { reached++; fmt.Printf(f+"\n", a...) }
for _, f := range []string{key, login} {
if _, err := os.ReadFile(f); err == nil {
say("read %s", f)
}
}
if c, err := net.DialTimeout("tcp", tcp, 2*time.Second); err == nil {
c.Close()
say("reached %s", tcp)
}
if c, err := net.DialTimeout("unix", socket, 2*time.Second); err == nil {
c.Close()
say("reached %s", socket)
}
if addrs, err := net.LookupHost(name); err == nil {
say("resolved %s to %v", name, addrs)
}
if home, err := os.UserHomeDir(); err == nil {
if entries, err := os.ReadDir(home); err == nil && len(entries) > 0 {
say("the home %s holds %d entries", home, len(entries))
}
}
if err := os.WriteFile(cache, []byte("written by the step\n"), 0o644); err != nil {
say("could not write the cache under its throwaway layer: %v", err)
}
if reached == 0 {
fmt.Println("shut off")
}
}
+123
View File
@@ -0,0 +1,123 @@
package buildstep
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"path"
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// ModuleTree is what a merge holds of one module directory: its tree id, the sha256 of its module.json and the sha256
// of each lock file it has (go.sum, package-lock.json), from which the checker and the indexer take the module's inputs
// (novox/hq ADR 0484 clause 3).
type ModuleTree struct {
Tree string `json:"tree"`
Manifest string `json:"manifest"`
Locks map[string]string `json:"locks,omitempty"`
// Paths are, of a module whose every artifact names its paths, each named path's git object id at the commit; Tree
// is then their delivery.NamedTree with Manifest (novox/hq ADR 0499 clause 2). Absent otherwise.
Paths map[string]string `json:"paths,omitempty"`
}
var lockFiles = []string{"go.sum", "package-lock.json"}
// ModuleTrees are the trees, in commit, of every directory of the workspace dir that holds a module.json, by directory
// (the repository's root is "."); for a module whose every artifact names its paths, the named tree of those paths and
// their objects (novox/hq ADR 0499 clause 2). The workspace is at commit, so the manifests and the lock files are read from it. env is
// the environment git runs with (nil for none). Both build modules' fetch steps answer it for a merge.
func ModuleTrees(ctx context.Context, run forgekey.Runner, env []string, dir, commit string) (map[string]ModuleTree, error) {
r := run(ctx, env, "git", "-C", dir, "ls-files", "--", "module.json", "*/module.json")
if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
}
trees := map[string]ModuleTree{}
for _, l := range strings.Split(r.Stdout, "\n") {
if l = strings.TrimSpace(l); l == "" {
continue
}
mdir := filepath.ToSlash(filepath.Dir(l))
spec := commit + ":" + mdir
if mdir == "." {
spec = commit + "^{tree}"
}
t := run(ctx, env, "git", "-C", dir, "rev-parse", spec)
if t.Status != 0 || t.Err != "" {
return nil, fmt.Errorf("git rev-parse %s: %s", spec, firstLine(t.Err+" "+t.Stderr))
}
mt := ModuleTree{Tree: strings.TrimSpace(t.Stdout)}
raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(l)))
if err != nil {
return nil, err
}
mt.Manifest = sha256Hex(raw)
if err := narrow(ctx, run, env, dir, commit, mdir, raw, &mt); err != nil {
return nil, err
}
for _, lock := range lockFiles {
if raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(mdir), lock)); err == nil {
if mt.Locks == nil {
mt.Locks = map[string]string{}
}
mt.Locks[lock] = sha256Hex(raw)
}
}
trees[mdir] = mt
}
return trees, nil
}
// narrow sets mt's Paths and its named Tree when the module's manifest names paths on every artifact (novox/hq ADR 0499
// clause 2): one `git ls-tree -z` gives each named path's object at commit. The merge preview reads the manifest here, as
// the indexer does, with catalogue.ParseManifestForInputs (issue 1751), and no other build step reads paths from one (ADR 0495 clause 1). A manifest that does not parse
// is not narrowed: the indexer gives it none. A named path absent at commit fails, naming the module and the path.
func narrow(ctx context.Context, run forgekey.Runner, env []string, dir, commit, mdir string, raw []byte, mt *ModuleTree) error {
m, err := catalogue.ParseManifestForInputs(raw)
if err != nil {
return nil
}
paths, every := m.NamedPaths()
if !every {
return nil
}
full := make([]string, len(paths))
for i, p := range paths {
full[i] = path.Join(mdir, p)
}
r := run(ctx, env, "git", append([]string{"-C", dir, "ls-tree", "-z", commit, "--"}, full...)...)
if r.Status != 0 || r.Err != "" {
return fmt.Errorf("git ls-tree %s of the module %s: %s", commit, mdir, firstLine(r.Err+" "+r.Stderr))
}
objects := map[string]string{}
for _, l := range strings.Split(r.Stdout, "\x00") {
meta, name, ok := strings.Cut(l, "\t")
fields := strings.Fields(meta)
if !ok || len(fields) != 3 {
continue
}
objects[name] = fields[2]
}
mt.Paths = map[string]string{}
for i, p := range paths {
object := objects[full[i]]
if object == "" {
return fmt.Errorf("the module %s names the path %q, which is not in %s", mdir, p, commit)
}
mt.Paths[p] = object
}
named, err := delivery.NamedTree(mt.Manifest, mt.Paths)
if err != nil {
return fmt.Errorf("the module %s: %w", mdir, err)
}
mt.Tree = named
return nil
}
func sha256Hex(b []byte) string { h := sha256.Sum256(b); return hex.EncodeToString(h[:]) }
+63
View File
@@ -0,0 +1,63 @@
//go:build integration
package buildstep
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// On a real repository shaped like mesh-sdk (novox/hq ADR 0499 clauses 2 and 6), a commit touching only the Go package
// leaves the TypeScript package's tree value equal, and a commit touching its source changes it; a module that names no
// path keeps its directory's tree id.
func TestAGoOnlyCommitLeavesTheNamedTreeEqualOnRealGit(t *testing.T) {
repo := t.TempDir()
realGit(t, repo, "init", "-q", "-b", "main")
writeFile(t, repo, "module.json", `{"module":"mesh-sdk","version":"1","build":{"artifacts":[`+
`{"name":"lib","kind":"package","language":"typescript","paths":["src","test","package.json","package-lock.json","tsconfig.json"]}]}}`)
writeFile(t, repo, "src/index.ts", "export const a = 1\n")
writeFile(t, repo, "test/a.test.ts", "// a\n")
writeFile(t, repo, "package.json", "{}\n")
writeFile(t, repo, "package-lock.json", "{}\n")
writeFile(t, repo, "tsconfig.json", "{}\n")
writeFile(t, repo, "go/delivery/x.go", "package delivery\n")
writeFile(t, repo, "other/module.json", `{"module":"other","version":"1"}`)
writeFile(t, repo, "other/a.txt", "a\n")
commit := func(msg string) string {
realGit(t, repo, "add", "-A")
realGit(t, repo, "commit", "-q", "-m", msg)
return realGit(t, repo, "rev-parse", "HEAD")
}
trees := func(c string) map[string]ModuleTree {
got, err := ModuleTrees(context.Background(), forgekey.ExecRunner, nil, repo, c)
if err != nil {
t.Fatal(err)
}
return got
}
first := trees(commit("first"))
if !strings.HasPrefix(first["."].Tree, "named:") || len(first["."].Paths) != 5 {
t.Fatalf("the root module was not narrowed: %+v", first["."])
}
if want := realGit(t, repo, "rev-parse", "HEAD:other"); first["other"].Tree != want || first["other"].Paths != nil {
t.Fatalf("a module naming no path: %+v, want its directory's tree %s", first["other"], want)
}
writeFile(t, repo, "go/delivery/x.go", "package delivery\n\nconst B = 2\n")
goOnly := trees(commit("go only"))
if goOnly["."].Tree != first["."].Tree {
t.Fatalf("a Go-only commit changed the TypeScript package's tree: %s, was %s", goOnly["."].Tree, first["."].Tree)
}
writeFile(t, repo, "src/index.ts", "export const a = 2\n")
src := trees(commit("src"))
if src["."].Tree == first["."].Tree {
t.Fatal("a commit to src/ left the TypeScript package's tree equal")
}
if src["."].Paths["src"] != realGit(t, repo, "rev-parse", "HEAD:src") {
t.Fatalf("the object of src is %s, git says %s", src["."].Paths["src"], realGit(t, repo, "rev-parse", "HEAD:src"))
}
}
+113
View File
@@ -0,0 +1,113 @@
package buildstep
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// A module whose every artifact names its paths takes its tree from them (novox/hq ADR 0499 clause 2): ModuleTrees asks
// git for each named path's object at the commit, keeps them in Paths and answers their delivery.NamedTree as Tree; a
// module naming none keeps its directory's tree, as before.
func TestModuleTreesNarrowsAModuleWhoseArtifactsNameTheirPaths(t *testing.T) {
root := t.TempDir()
sdk := `{"module":"mesh-sdk","version":"1","build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript","paths":["src","package.json"]}]}}`
writeTreeFile(t, root, "module.json", sdk)
writeTreeFile(t, root, "modules/b/module.json", `{"module":"b","version":"1"}`)
merge := strings.Repeat("c", 40)
var lsTree []string
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
switch args[2] {
case "ls-files":
return forgekey.Ran{Stdout: "module.json\nmodules/b/module.json\n"}
case "rev-parse":
return forgekey.Ran{Stdout: "tree-of-" + args[3] + "\n"}
case "ls-tree":
lsTree = append(lsTree, strings.Join(args[4:], " "))
return forgekey.Ran{Stdout: "100644 blob " + strings.Repeat("b", 40) + "\tpackage.json\x00" +
"040000 tree " + strings.Repeat("a", 40) + "\tsrc\x00"}
}
return forgekey.Ran{Status: 1, Stderr: "unexpected " + args[2]}
}
got, err := ModuleTrees(context.Background(), run, nil, root, merge)
if err != nil {
t.Fatal(err)
}
paths := map[string]string{"src": strings.Repeat("a", 40), "package.json": strings.Repeat("b", 40)}
want, _ := delivery.NamedTree(hexOf(sdk), paths)
if g := got["."]; g.Tree != want || len(g.Paths) != 2 || g.Paths["src"] != paths["src"] || g.Paths["package.json"] != paths["package.json"] {
t.Fatalf("the root module = %+v, want tree %s and paths %v", g, want, paths)
}
if len(lsTree) != 1 || lsTree[0] != merge+" -- package.json src" {
t.Fatalf("git ls-tree ran as %v", lsTree)
}
if g := got["modules/b"]; g.Tree != "tree-of-"+merge+":modules/b" || g.Paths != nil {
t.Fatalf("a module naming no path was narrowed: %+v", g)
}
}
// A named path absent at the commit fails the answer, naming the module and the path: a tree computed without it would
// pin a build source that does not exist.
func TestModuleTreesFailsNamingANamedPathAbsentAtTheCommit(t *testing.T) {
root := t.TempDir()
writeTreeFile(t, root, "modules/a/module.json", `{"module":"a","version":"1","build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript","paths":["src","gone"]}]}}`)
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
switch args[2] {
case "ls-files":
return forgekey.Ran{Stdout: "modules/a/module.json\n"}
case "rev-parse":
return forgekey.Ran{Stdout: "t\n"}
case "ls-tree":
if strings.Join(args[4:], " ") != strings.Repeat("c", 40)+" -- modules/a/gone modules/a/src" {
return forgekey.Ran{Status: 1, Stderr: "wrong arguments " + strings.Join(args[3:], " ")}
}
return forgekey.Ran{Stdout: "040000 tree " + strings.Repeat("a", 40) + "\tmodules/a/src\x00"}
}
return forgekey.Ran{Status: 1}
}
_, err := ModuleTrees(context.Background(), run, nil, root, strings.Repeat("c", 40))
if err == nil || !strings.Contains(err.Error(), `"gone"`) || !strings.Contains(err.Error(), "modules/a") {
t.Fatalf("an absent named path gave %v", err)
}
}
// A manifest whose config value falls back to a predefined key is narrowed here as the indexer narrows it: both read it
// with catalogue.ParseManifestForInputs (issue 1751), so a build module holding no published list computes the same
// tree (ADR 0499 clause 2).
func TestModuleTreesNarrowsAManifestFallingBackToAPredefinedKey(t *testing.T) {
was := catalogue.ReplacePublished(nil) // the build modules' case: no published list
defer catalogue.ReplacePublished(was)
root := t.TempDir()
raw := `{"module":"xterm","version":"1","config":{"size":{"type":"number","description":"The text size.","predefined":"font-size","default":12,"runtime":true}},` +
`"build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript","paths":["src"]}]}}`
if _, err := catalogue.ParseManifest([]byte(raw)); err == nil {
t.Fatal("ParseManifest reads the manifest with no list; the test proves nothing")
}
if _, err := catalogue.ParseManifestForInputs([]byte(raw)); err != nil {
t.Fatalf("the indexer's read refuses the manifest: %v", err)
}
writeTreeFile(t, root, "module.json", raw)
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
switch args[2] {
case "ls-files":
return forgekey.Ran{Stdout: "module.json\n"}
case "rev-parse":
return forgekey.Ran{Stdout: "whole\n"}
case "ls-tree":
return forgekey.Ran{Stdout: "040000 tree " + strings.Repeat("a", 40) + "\tsrc\x00"}
}
return forgekey.Ran{Status: 1}
}
got, err := ModuleTrees(context.Background(), run, nil, root, strings.Repeat("c", 40))
if err != nil {
t.Fatal(err)
}
want, _ := delivery.NamedTree(hexOf(raw), map[string]string{"src": strings.Repeat("a", 40)})
if g := got["."]; g.Tree != want || g.Paths["src"] == "" {
t.Fatalf("the module = %+v, want the named tree %s", g, want)
}
}
+93
View File
@@ -0,0 +1,93 @@
package buildstep
import (
"context"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
func hexOf(s string) string { h := sha256.Sum256([]byte(s)); return hex.EncodeToString(h[:]) }
func writeTreeFile(t *testing.T, root, name, content string) {
t.Helper()
p := filepath.Join(root, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// ModuleTrees answers, for each directory holding a module.json (the root as "."), the tree id git gives, the sha256 of
// module.json and of each lock file present, and reads them from the workspace.
func TestModuleTreesAnswersTreeManifestAndLocksPerModuleDirectory(t *testing.T) {
root := t.TempDir()
writeTreeFile(t, root, "module.json", `{"root":1}`)
writeTreeFile(t, root, "go.sum", "rootsum")
writeTreeFile(t, root, "modules/a/module.json", `{"a":1}`)
writeTreeFile(t, root, "modules/a/go.sum", "asum")
writeTreeFile(t, root, "modules/a/package-lock.json", "alock")
writeTreeFile(t, root, "modules/b/module.json", `{"b":1}`)
merge := strings.Repeat("c", 40)
var calls []string
var envs [][]string
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
calls = append(calls, name+" "+strings.Join(args[2:], " "))
envs = append(envs, env)
switch args[2] {
case "ls-files":
return forgekey.Ran{Stdout: "module.json\nmodules/a/module.json\nmodules/b/module.json\n"}
case "rev-parse":
return forgekey.Ran{Stdout: "tree-of-" + args[3] + "\n"}
}
return forgekey.Ran{Status: 1, Stderr: "unexpected"}
}
got, err := ModuleTrees(context.Background(), run, []string{"X=1"}, root, merge)
if err != nil {
t.Fatal(err)
}
want := map[string]ModuleTree{
".": {Tree: "tree-of-" + merge + "^{tree}", Manifest: hexOf(`{"root":1}`), Locks: map[string]string{"go.sum": hexOf("rootsum")}},
"modules/a": {Tree: "tree-of-" + merge + ":modules/a", Manifest: hexOf(`{"a":1}`), Locks: map[string]string{"go.sum": hexOf("asum"), "package-lock.json": hexOf("alock")}},
"modules/b": {Tree: "tree-of-" + merge + ":modules/b", Manifest: hexOf(`{"b":1}`)},
}
if !reflect.DeepEqual(got, want) {
t.Errorf("trees = %+v\nwant %+v", got, want)
}
if got["modules/b"].Locks != nil {
t.Error("a module with no lock file carries a lock map")
}
for _, e := range envs {
if len(e) != 1 || e[0] != "X=1" {
t.Errorf("git ran with env %v, want the one given", e)
}
}
}
func TestModuleTreesFailsNamingTheGitCommandOrTheUnreadableManifest(t *testing.T) {
root := t.TempDir()
fail := func(on string) forgekey.Runner {
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if args[2] == on {
return forgekey.Ran{Status: 128, Stderr: "fatal: no\n"}
}
return forgekey.Ran{Stdout: "modules/a/module.json\n"}
}
}
for _, on := range []string{"ls-files", "rev-parse"} {
if _, err := ModuleTrees(context.Background(), fail(on), nil, root, sha); err == nil || !strings.Contains(err.Error(), on) || !strings.Contains(err.Error(), "fatal: no") {
t.Errorf("%s failing: %v", on, err)
}
}
if _, err := ModuleTrees(context.Background(), fail("none"), nil, root, sha); err == nil {
t.Error("a module.json that is not in the workspace was read")
}
}
+66
View File
@@ -394,6 +394,72 @@ func (b *Build) problems(module string) []string {
"%s: %q mirrors %q, which names no tag or digest", module, a.Name, a.From))
}
problems = append(problems, upstreamFileProblems(module, a)...)
problems = append(problems, pathsProblems(module, a)...)
}
return problems
}
// NamedPaths are the paths a module's fingerprint takes its tree from (novox/hq ADR 0499 clause 2): the union of the
// paths its artifacts name, cleaned and sorted, and every true only when the module builds something and every artifact
// names paths; otherwise none, and the module keeps its directory's tree.
//
// **Read by the core alone** (ADR 0495 clause 1, the lead's decision L8 of R4e): the indexer and its merge preview
// (buildstep.ModuleTrees) call it. A build module's fetch step is handed the paths by the builder, from the indexer's
// kept inputs (delivery.KeptInputs2.Paths), and reads no manifest for them.
func (m Manifest) NamedPaths() (paths []string, every bool) {
if m.Build == nil || len(m.Build.Artifacts) == 0 {
return nil, false
}
seen := map[string]bool{}
for _, a := range m.Build.Artifacts {
if len(a.Paths) == 0 {
return nil, false
}
for _, p := range a.Paths {
seen[path.Clean(p)] = true
}
}
for p := range seen {
paths = append(paths, p)
}
sort.Strings(paths)
return paths, true
}
// pathsProblems is what is wrong with an artifact's `paths` (novox/hq ADR 0499 clause 1): named on a kind whose build
// source is derived or named otherwise, or a path that leaves the module's directory, is its whole directory, or is a
// pattern.
func pathsProblems(module string, a Artifact) []string {
if a.Paths == nil {
return nil
}
switch {
case a.Kind == ArtifactBundle && a.Language == "go":
return []string{fmt.Sprintf("%s: %q is a Go program and names paths: its build source is derived from the "+
"package it compiles (novox/hq ADR 0267 clause 1, ADR 0499 clause 1)", module, a.Name)}
case a.Kind != ArtifactBundle && a.Kind != ArtifactPackage:
return []string{fmt.Sprintf("%s: %q is a %q and names paths; only a package or a bundle in a language other "+
"than Go names the paths it is built from (novox/hq ADR 0499 clause 1)", module, a.Name, a.Kind)}
case len(a.Paths) == 0:
return []string{fmt.Sprintf("%s: %q names no path in paths; leave the word out to build from the module's "+
"directory", module, a.Name)}
}
var problems []string
for _, p := range a.Paths {
clean := path.Clean(p)
switch {
case strings.TrimSpace(p) == "":
problems = append(problems, fmt.Sprintf("%s: %q names an empty path", module, a.Name))
case strings.HasPrefix(p, "/") || clean == ".." || strings.HasPrefix(clean, "../"):
problems = append(problems, fmt.Sprintf("%s: %q names the path %q, which leaves the module's directory",
module, a.Name, p))
case clean == ".":
problems = append(problems, fmt.Sprintf("%s: %q names the module's whole directory (%q) as a path; leave "+
"the word out for that", module, a.Name, p))
case strings.ContainsAny(p, "*?[]{}\\"):
problems = append(problems, fmt.Sprintf("%s: %q names %q, which is a pattern; paths names files and "+
"directories", module, a.Name, p))
}
}
return problems
}
+15 -31
View File
@@ -22,16 +22,12 @@ func catalogueRoot(t *testing.T) string {
}
func TestEveryCatalogueManifestParses(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
named, routed := 0, 0
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(p)), err)
t.Errorf("%s: %v", filepath.Base(filepath.Dir(r.path)), err)
continue
}
for _, l := range m.Listens {
@@ -54,16 +50,12 @@ func TestEveryCatalogueManifestParses(t *testing.T) {
// definition names a domain or a public address the mesh acts on, and every value that must for now
// carries its reason (novox/hq ADR 0155, issue 134). The list it prints is the one that shrinks.
func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
var named []string
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Errorf("%s: %v", p, err)
t.Errorf("%s: %v", r.path, err)
continue
}
named = append(named, InstallationProblems(m)...)
@@ -78,15 +70,11 @@ func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
// backup line by hand, every directory a container writes is declared, and every irreplaceable item is
// backed up — so a store added is a store backed up, without a list of stores to keep in step.
func TestEveryCatalogueModuleDeclaresItsData(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
shelf := Shelf{}
granting := 0
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
continue // TestEveryCatalogueManifestParses says why
}
@@ -108,16 +96,12 @@ func TestEveryCatalogueModuleDeclaresItsData(t *testing.T) {
// overflow fails here, in the pull request that introduces it, rather than on the provider's machine
// the first time a real machine's name meets the module's (issue 263).
func TestEveryCatalogueIdentityFitsWhatItRequires(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
shelf := Shelf{}
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Fatalf("%s: %v", p, err)
t.Fatalf("%s: %v", r.path, err)
}
shelf[m.Module] = m
}
+44
View File
@@ -0,0 +1,44 @@
package catalogue
import (
"path/filepath"
"sync"
"testing"
)
// readManifest is one manifest of the catalogue as ReadTreeManifest read it.
type readManifest struct {
path string // the module.json
manifest Manifest
err error
}
var (
readOnce sync.Once
readAll []readManifest
readRoot string
)
// catalogueManifests is every manifest of the catalogue (catalogueRoot), read through ReadTreeManifest once
// per test binary and not once per test: the checks over the real catalogue each read and parsed every
// manifest again, and together took 1.5 s of mesh-sdk's unit bound of 2 s (issue 1756). What it returns is
// shared and read-only: a test that changes a manifest copies it first.
func catalogueManifests(t *testing.T) (string, []readManifest) {
t.Helper()
root := catalogueRoot(t)
readOnce.Do(func() {
readRoot = root
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return
}
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
readAll = append(readAll, readManifest{path: p, manifest: m, err: err})
}
})
if len(readAll) == 0 {
t.Fatalf("no manifests under %s", root)
}
return readRoot, readAll
}
+2 -1
View File
@@ -42,7 +42,8 @@ func deliveryVerbs() []Verb {
"paths": "the files it changes, comma-separated, from the repository's root",
"base": "the branch it merges into (default main)"}, []string{"repository", "paths"})},
{Name: "checks", Description: "What the mesh's checks said of a pull request's head or of one commit: each " +
"of the commit's mesh statuses (mesh/merge-gate, mesh/repo-check, mesh/delivery, …) with its state, " +
"of the commit's mesh statuses (mesh/check, the one a main requires from R4e, hq's mesh/repo-check, " +
"mesh/delivery, …) with its state, " +
"description and when it was set; the merge check's full verdict as the controller said it — each " +
"layer's summary, the machine that ran it, when, its build id and its report; and whether the branch's " +
"protection would let it merge, every required status being success.",
+11
View File
@@ -1191,6 +1191,17 @@ type Artifact struct {
// (novox/hq ADR 0193). Absent for nearly every bundle, which is then one file per entrypoint.
External []string `json:"external,omitempty"`
// Paths are the files and directories, relative to the module's directory, a package or a bundle in a language
// other than Go is built from (novox/hq ADR 0499 clause 1): with the module's manifest they are its build source,
// and a module whose every artifact names them takes its fingerprint's tree from them alone (clause 2). Absent means
// the module's directory, as before.
//
// **Named, because the build source of a package sharing a repository cannot be derived.** A Go program's is
// (ADR 0267 clause 1), an archive's `from` names it and an image's is the context its manifest names, so each of those
// refuses the word. A path stays inside the module's directory and is no pattern: a build source that could reach
// out of it, or grow with a directory listing, would be one no fingerprint pins.
Paths []string `json:"paths,omitempty"`
// Env is what a tools bundle is given on a machine (novox/hq ADR 0192): words and their values,
// paths and constants composed with ${dir:…} and ${port:…} exactly as a container's environment
// is, never a secret's content. The node's runtime hands it to this bundle and to no other.
+4 -10
View File
@@ -4,8 +4,6 @@ import (
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/internal/beside"
)
// A bind mount the module never declared is refused where it is written (novox/hq 04-ISSUES/026,
@@ -71,14 +69,10 @@ func TestTheRuntimeSocketIsGrantedByTheCapabilityAndNotOtherwise(t *testing.T) {
// **Every manifest in the catalogue (internal/beside) passes**, so the rule is not one the catalogue
// is already breaking.
func TestEveryCatalogueManifestDeclaresWhatItMounts(t *testing.T) {
files, _ := filepath.Glob(filepath.Join(beside.Catalogue(t), "*", "module.json"))
if len(files) == 0 {
t.Fatal("no manifests in the catalogue, so this proved nothing")
}
root := beside.Dir(t, "mesh-catalog") // once: it copies the captured tree
for _, file := range files {
if _, err := ReadTreeManifest(root, filepath.Dir(file)); err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(file)), err)
_, files := catalogueManifests(t)
for _, f := range files {
if f.err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(f.path)), f.err)
}
}
}
+29
View File
@@ -0,0 +1,29 @@
package catalogue
import (
"os"
"testing"
)
// mesh-sdk's own manifest is judged by the controller that runs when it is checked, and that controller reads it
// strictly: a manifest word ships one release ahead of its first user. R4e's group is judged by today's controller
// (novox/hq ADR 0503 §4), which does not know `paths` (ADR 0499), so mesh-sdk's lib names no paths until a release
// after R4e's delivery adds them back (ADR 0499 §6, issue 1758).
func TestOwnManifestUsesNoWordTheRunningControllerRefuses(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("mesh-sdk's module.json does not parse: %v", err)
}
if m.Build == nil || len(m.Build.Artifacts) == 0 {
t.Fatal("mesh-sdk's module.json declares no artifact")
}
for _, a := range m.Build.Artifacts {
if a.Paths != nil {
t.Errorf("artifact %q names paths %v: the running controller refuses the word until R4e is delivered", a.Name, a.Paths)
}
}
}
+67
View File
@@ -0,0 +1,67 @@
package catalogue
import (
"strings"
"testing"
)
// An artifact names the paths it is built from with `paths` (novox/hq ADR 0499 clause 1): a package, or a bundle in a
// language other than Go, may; a Go program's artifact, an archive, an image and an upstream artifact may not, and the
// refusal names the artifact and its kind. Each path stays inside the module's directory: no `..`, no absolute path,
// no pattern.
func TestAnArtifactNamesThePathsItIsBuiltFrom(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"mesh-sdk","version":"1","slug":"sdk","build":{"artifacts":[
{"name":"lib","kind":"package","language":"typescript",
"paths":["src","test","package.json","package-lock.json","tsconfig.json"]}]},"resources":[]}`))
if err != nil {
t.Fatalf("a package naming its paths was refused: %v", err)
}
if got := strings.Join(m.Build.Artifacts[0].Paths, ","); got != "src,test,package.json,package-lock.json,tsconfig.json" {
t.Fatalf("the paths were not kept as written: %q", got)
}
if _, err := ParseManifest([]byte(`{"module":"a","version":"1","build":{"artifacts":[
{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["index.js"],"loads":["index.js"],"paths":["src","package.json"]}]}}`)); err != nil {
t.Fatalf("a TypeScript bundle naming its paths was refused: %v", err)
}
for _, c := range []struct{ artifact, says string }{
{`{"name":"x","kind":"bundle","language":"go","system":"linux-amd64","paths":["cmd"]}`, `"x" is a Go program`},
{`{"name":"x","kind":"archive","from":"files","paths":["files"]}`, `"x" is a "archive"`},
{`{"name":"x","kind":"image","from":"Dockerfile","paths":["src"]}`, `"x" is a "image"`},
{`{"name":"x","kind":"upstream","from":"nats:2.10","paths":["src"]}`, `"x" is a "upstream"`},
{`{"name":"x","kind":"package","language":"typescript","paths":["../other"]}`, "leaves the module's directory"},
{`{"name":"x","kind":"package","language":"typescript","paths":["src/../../x"]}`, "leaves the module's directory"},
{`{"name":"x","kind":"package","language":"typescript","paths":["/etc"]}`, "leaves the module's directory"},
{`{"name":"x","kind":"package","language":"typescript","paths":["src/*.ts"]}`, "is a pattern"},
{`{"name":"x","kind":"package","language":"typescript","paths":["src/[ab]"]}`, "is a pattern"},
{`{"name":"x","kind":"package","language":"typescript","paths":[""]}`, "names an empty path"},
{`{"name":"x","kind":"package","language":"typescript","paths":["."]}`, "names the module's whole directory"},
{`{"name":"x","kind":"package","language":"typescript","paths":[]}`, "names no path"},
} {
_, err := ParseManifest([]byte(`{"module":"a","version":"1","build":{"artifacts":[` + c.artifact + `]}}`))
if err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v, wanted a refusal saying %q", c.artifact, err, c.says)
}
}
}
// The paths a module's tree is taken from are the union its artifacts name, cleaned and sorted, and only when every
// artifact names some (novox/hq ADR 0499 clause 2); a module with an artifact that names none, or with none at all,
// keeps its directory's tree.
func TestAModulesNamedPathsAreTheUnionWhenEveryArtifactNamesThem(t *testing.T) {
m := Manifest{Module: "a", Build: &Build{Artifacts: []Artifact{
{Name: "lib", Kind: ArtifactPackage, Paths: []string{"src/", "package.json"}},
{Name: "tools", Kind: ArtifactBundle, Paths: []string{"./src", "tools"}},
}}}
paths, every := m.NamedPaths()
if !every || strings.Join(paths, ",") != "package.json,src,tools" {
t.Fatalf("NamedPaths = %v, %v; want [package.json src tools], true", paths, every)
}
m.Build.Artifacts = append(m.Build.Artifacts, Artifact{Name: "img", Kind: ArtifactImage, From: "Dockerfile"})
if paths, every := m.NamedPaths(); every || paths != nil {
t.Fatalf("a module with an artifact naming no path gave %v, %v", paths, every)
}
if paths, every := (Manifest{Module: "b"}).NamedPaths(); every || paths != nil {
t.Fatalf("a module that builds nothing gave %v, %v", paths, every)
}
}
+41
View File
@@ -0,0 +1,41 @@
package catalogue
import "encoding/json"
// ParseManifestForInputs reads a manifest for the inputs of its build and its fingerprint: the indexer's and the
// builder's read (novox/hq issue 1751). It is never a judge of config values. A value falling back to a predefined key
// (ADR 0348 §7) is judged against the list the mesh published when this process holds one (PublishedPredefined);
// where it holds none, as in the indexer and the builder, each key a value names is taken as the value declares it (its
// type, trusted when it is), since the list is read where a manifest is judged: the merge gate from the tree's root,
// the controller from what it published. No input of a fingerprint or a build comes from the list. Everything else
// ParseManifest refuses is refused here too.
func ParseManifestForInputs(raw []byte) (Manifest, error) {
list := PublishedPredefined()
if list == nil {
list = fallbacksOf(raw)
}
return parseManifest(raw, list)
}
// fallbacksOf is the list of the predefined keys a manifest's config values fall back to, each as the value declares
// it. A manifest that is not JSON gives an empty list, and the parser says what is wrong with it.
func fallbacksOf(raw []byte) PredefinedList {
var m struct {
Config map[string]struct {
Type string `json:"type"`
Description string `json:"description"`
Predefined string `json:"predefined"`
Trusted bool `json:"trusted"`
} `json:"config"`
}
list := PredefinedList{}
if json.Unmarshal(raw, &m) != nil {
return list
}
for _, d := range m.Config {
if d.Predefined != "" {
list[d.Predefined] = PredefinedKey{Type: d.Type, Description: d.Description, Trusted: d.Trusted}
}
}
return list
}
+40
View File
@@ -0,0 +1,40 @@
package catalogue
import (
"os"
"strings"
"testing"
)
// The indexer and the builder hold no published list of predefined keys; a manifest whose config values fall back to
// one (claude-code, dunst, xterm at mesh-catalog 835bf22b) is still read for its build's and fingerprint's inputs
// (issue 1751).
func TestParseManifestForInputsReadsAValueFallingBackToAPredefinedKey(t *testing.T) { // issue 1751
was := ReplacePublished(nil) // the indexer's and the builder's case: no published list
defer ReplacePublished(was)
for _, module := range []string{"claude-code", "dunst", "xterm"} {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-" + module + ".module.json")
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil {
t.Fatalf("%s: ParseManifest reads it with no list; the test proves nothing", module)
}
if m, err := ParseManifestForInputs(raw); err != nil || m.Module != module {
t.Errorf("%s: %v", module, err)
}
}
}
func TestParseManifestForInputsStillRefusesWhatTheParserRefuses(t *testing.T) { // issue 1751
was := ReplacePublished(nil)
defer ReplacePublished(was)
raw, _ := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
bad := strings.Replace(string(raw), `"predefined": "font-size"`, `"predefined": "font-size", "trusted": "yes"`, 1)
if bad == string(raw) {
t.Fatal("the fixture no longer names font-size; the test proves nothing")
}
if _, err := ParseManifestForInputs([]byte(bad)); err == nil {
t.Error("a manifest the parser refuses was read")
}
}
-7
View File
@@ -423,13 +423,6 @@ func CatalogueProblems(shelf Shelf) []string {
"operator's account, which every agent runs as (novox/hq ADR 0259)", module, why))
}
}
// A read of a module's state that module does not keep (novox/hq ADR 0201) — said only where the
// owner is on the shelf, as a consumer may be installed before its emitter.
var manifests []Manifest
for _, module := range shelfOrder(shelf) {
manifests = append(manifests, shelf[module])
}
problems = append(problems, StateReadsNothingDeclares(manifests)...)
sort.Strings(problems)
return problems
}
-41
View File
@@ -126,44 +126,3 @@ func StateProblems(m Manifest) []string {
}
return problems
}
// ControllerMirrors are the controller's record mirrors a module may read, each `mesh-controller.<name>`
// (novox/hq ADR 0289, design 54 §5, design 57 F3). **The controller keeps them, not its manifest's `state`**:
// they are asserted with the controller's own buckets, with options of their own, and written by the
// controller's account alone (the writers table); declared as a module's state, they would be asserted a
// second way and granted to the module's own principal as well. Named here so a read of one is a read of
// kept state. The bus's list is broker.ControllerMirrors; a test keeps the two agreeing.
var ControllerMirrors = []string{"assignments", "declared-config", "predefined", "builds", "rollouts", "checks-running"}
// StateReadsNothingDeclares is every read across a catalogue whose owner is present and declares no
// such state. An absent owner says nothing — a module may be installed long before the one whose
// state it reads, as a consumer may before its emitter (design 32 §1).
func StateReadsNothingDeclares(manifests []Manifest) []string {
declared := map[string]map[string]bool{}
for _, m := range manifests {
own := map[string]bool{}
for _, s := range m.State {
own[s.Name] = true
}
declared[m.Module] = own
}
if own, present := declared[ControllerSeatName]; present {
for _, name := range ControllerMirrors {
own[name] = true
}
}
var problems []string
for _, m := range manifests {
for _, r := range m.Reads {
module, local, err := ReadState(r)
if err != nil {
continue
}
if own, present := declared[module]; present && !own[local] {
problems = append(problems, fmt.Sprintf(
"%s reads %q, and %s keeps no state called %q", m.Module, r, module, local))
}
}
}
return problems
}
+8 -38
View File
@@ -52,59 +52,29 @@ func TestAStateNameIsLocalAndOneToken(t *testing.T) {
}
}
// A read whose owner is present must name a state that owner keeps; an absent owner says nothing,
// because a module may be installed before the one whose state it reads.
func TestAReadNamesStateItsOwnerKeeps(t *testing.T) {
owner := Manifest{Module: "licence-manager", State: []StateDeclaration{{Name: "bindings"}}}
good := Manifest{Module: "claude-code", Reads: []string{"licence-manager.bindings", "absent.anything"}}
bad := Manifest{Module: "other", Reads: []string{"licence-manager.tokens"}}
if p := StateReadsNothingDeclares([]Manifest{owner, good}); len(p) != 0 {
t.Fatalf("a read of declared state was refused: %v", p)
}
p := StateReadsNothingDeclares([]Manifest{owner, bad})
if len(p) != 1 || !strings.Contains(p[0], `licence-manager keeps no state called "tokens"`) {
t.Fatalf("a read of state nobody keeps was not named: %v", p)
}
}
// **The controller's record mirrors are kept state** (design 54 §5, design 57 F3): a module may read
// `mesh-controller.assignments` and `mesh-controller.declared-config` although the controller's manifest
// declares no `state`; a read of anything else of the controller's is still named.
func TestAReadOfTheControllersMirrorsIsAReadOfKeptState(t *testing.T) {
controller := Manifest{Module: ControllerSeatName}
reader := Manifest{Module: "mesh-config", Reads: []string{"mesh-controller.assignments", "mesh-controller.declared-config"}}
if p := StateReadsNothingDeclares([]Manifest{controller, reader}); len(p) != 0 {
t.Fatalf("a read of the controller's mirrors was refused: %v", p)
}
other := Manifest{Module: "other", Reads: []string{"mesh-controller.calls"}}
if p := StateReadsNothingDeclares([]Manifest{controller, other}); len(p) != 1 {
t.Fatalf("a read of the controller's own calls was not named: %v", p)
}
}
// **Across the whole catalogue**: every state name is local, and every read whose owner is present
// names state that owner keeps.
// **Across the whole catalogue**: every state name is local and every read names a module's state.
func TestEveryManifestsStateIsLocalAndEveryReadIsKept(t *testing.T) {
manifests := theCatalogue(t)
var problems []string
for _, m := range manifests {
problems = append(problems, StateProblems(m)...)
}
problems = append(problems, StateReadsNothingDeclares(manifests)...)
if len(problems) > 0 {
t.Fatalf("the catalogue's state is not what ADR 0201 says:\n %s", strings.Join(problems, "\n "))
}
}
// `module check` says it too: the cross-catalogue pass names a read nothing on the shelf keeps.
func TestTheCataloguePassNamesAReadItsOwnerDoesNotKeep(t *testing.T) {
// The catalogue pass no longer judges one module's reads against another's manifest (novox/hq ADR 0495 clauses 1 and 5,
// issue 1790): what an owner keeps is what it announces on the bus, and each module's own announcement test holds that.
func TestTheCataloguePassJudgesNoReadAgainstTheOwnersManifest(t *testing.T) {
shelf := Shelf{
"licence-manager": {Module: "licence-manager", State: []StateDeclaration{{Name: "bindings"}}},
"claude-code": {Module: "claude-code", Reads: []string{"licence-manager.tokens"}},
}
problems := CatalogueProblems(shelf)
if len(problems) != 1 || !strings.Contains(problems[0], `keeps no state called "tokens"`) {
t.Fatalf("the catalogue pass said %v", problems)
for _, p := range CatalogueProblems(shelf) {
if strings.Contains(p, "keeps no state") {
t.Fatalf("the catalogue pass judged a read against the owner's manifest: %s", p)
}
}
}
@@ -0,0 +1,249 @@
{
"module": "claude-code",
"version": "1",
"moved": true,
"slug": "agent",
"capabilities": [
"package-manager"
],
"requires": [
"mcp-endpoint",
{
"name": "gitea-read",
"optional": true
}
],
"binds": {
"mcp-endpoint": "${dir:state}/mcp-endpoint.json"
},
"secrets": {
"gitea-read": "${dir:state}/gitea-read.secret"
},
"uses": [
"operator-channel"
],
"state": [
"servers",
"holdings",
"config",
"proposals",
"questions",
"memory",
"waits"
],
"reads": [
"claude-licence-manager.bindings"
],
"tools": [
"claude_code_status",
"claude_code_render",
"claude_code_guard",
"claude_code_pull",
"claude_code_grant",
"claude_code_add_api_key",
"claude_code_registrations",
"claude_code_memory",
"claude_code_mcp_list",
"claude_code_mcp_register",
"claude_code_mcp_unregister",
"claude_code_skill_list",
"claude_code_skill_register",
"claude_code_skill_unregister",
"claude_code_agent_list",
"claude_code_agent_register",
"claude_code_agent_unregister",
"claude_code_command_list",
"claude_code_command_register",
"claude_code_command_unregister",
"claude_code_hook_list",
"claude_code_hook_register",
"claude_code_hook_unregister",
"claude_code_output_style_list",
"claude_code_output_style_register",
"claude_code_output_style_unregister",
"claude_code_instructions_list",
"claude_code_instructions_register",
"claude_code_instructions_unregister",
"claude_code_instructions_propose",
"claude_code_proposals",
"claude_code_ask",
"claude_code_answer",
"claude_code_wait",
"claude_code_settings_get",
"claude_code_settings_set",
"claude_code_settings_clear",
"claude_code_permission_add",
"claude_code_permission_remove",
"claude_code_config_list",
"claude_code_config_show",
"claude_code_config_status",
"claude_code_config_import",
"claude_code_home_show",
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge",
"claude_code_judged"
],
"data": {
"own": [
{
"id": "agent-home",
"path": "${dir:agent-home}",
"class": "valuable",
"why": "the operator's agent's own files: its memory, history and projects"
}
]
},
"config": {
"role": {
"type": "string",
"description": "What this machine is for, in a sentence, written into the instructions every Claude Code session on it reads; empty says nothing.",
"trusted": true,
"default": ""
},
"mcp_servers": {
"type": "object",
"description": "The tool servers every session on this machine is given beside the mesh's own, by name, each as Claude Code's managed MCP configuration takes it; empty adds none.",
"trusted": true,
"default": {}
},
"managed_settings": {
"type": "object",
"description": "Keys of Claude Code's managed settings every session on this machine obeys, such as spinnerTipsEnabled, laid over the mesh's own; empty adds none. Falls back to agent-managed-settings, the mesh's or this node's.",
"trusted": true,
"predefined": "agent-managed-settings",
"default": {}
},
"instructions": {
"type": "object",
"description": "The instructions every session on this machine reads, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"output_styles": {
"type": "object",
"description": "The output styles offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"skills": {
"type": "object",
"description": "The skills offered to every session on this machine, by name, each its files by path, a text as one line or a list of its lines; a skill without its SKILL.md is left out; empty adds none.",
"trusted": true,
"default": {}
},
"commands": {
"type": "object",
"description": "The slash commands offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"agents": {
"type": "object",
"description": "The agents offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
}
},
"resources": [
{
"id": "package",
"type": "package",
"package": "claude-code"
},
{
"id": "managed",
"type": "directory",
"path": "/etc/claude-code",
"mode": "0755",
"unknown": "program"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/claude-agent",
"mode": "0755",
"template": "files/claude-agent"
},
{
"id": "agent-account-name",
"type": "file",
"path": "/etc/claude-code/agent-account",
"mode": "0644",
"template": "files/agent-account"
},
{
"id": "agent-account",
"type": "user",
"name": "${machine:agent-account}",
"home": "${machine:agent-home}",
"root": "${machine:agent-root}"
},
{
"id": "agent-home",
"type": "directory",
"path": "${machine:agent-home}/.claude",
"mode": "0700",
"owner": "${machine:agent-account}",
"unknown": "module"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "${machine:account}",
"place": ".",
"unknown": "program"
},
{
"id": "facts",
"type": "file",
"path": "${dir:state}/facts.json",
"mode": "0600",
"owner": "${machine:account}",
"template": "files/facts.json"
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.json",
"mode": "0600",
"owner": "${machine:account}",
"takes-config": ["role", "mcp_servers", "managed_settings", "instructions", "output_styles", "skills", "commands", "agents"],
"merge": "json",
"template": "files/settings.json"
}
],
"items": [
{
"collection": "shell:zsh:normal",
"template": "files/zsh-normal"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/claude-code",
"binary": "claude-code",
"loads": [
"claude-code"
],
"env": {
"MESH_CLAUDE_CODE_STATE": "${dir:state}",
"MESH_CLAUDE_CODE_FACTS": "${dir:state}/facts.json",
"MESH_CLAUDE_CODE_SETTINGS": "${dir:state}/settings.json"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,123 @@
{
"module": "dunst",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"claims": [
{
"name": "node-notifier",
"scope": "node",
"serves": [
"send",
"history"
]
}
],
"tools": [
"dunst_pause",
"dunst_resume",
"dunst_close_all",
"dunst_rules",
"dunst_count",
"dunst_reload"
],
"config": {
"font-size": {
"type": "number",
"description": "The size of a notification's text, in points of Inter: 10 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 10
},
"width": {
"type": "number",
"description": "The width of a notification, in pixels: 250 fits a title of about forty characters at 10 points, and a larger font needs a wider notification.",
"minimum": 50,
"maximum": 4000,
"default": 250
}
},
"resources": [
{
"id": "package",
"type": "package",
"package": "dunst"
},
{
"id": "client",
"type": "package",
"package": "libnotify"
},
{
"id": "configuration-dir",
"type": "directory",
"path": "${machine:account-home}/.config/dunst",
"owner": "${machine:account}",
"mode": "0755",
"unknown": "move-out"
},
{
"id": "dropins",
"type": "directory",
"path": "${machine:account-home}/.config/dunst/dunstrc.d",
"owner": "${machine:account}",
"mode": "0755",
"unknown": "move-out",
"knows": ["50-slack.conf"]
},
{
"id": "configuration",
"trusted": false,
"type": "file",
"path": "${machine:account-home}/.config/dunst/dunstrc",
"owner": "${machine:account}",
"mode": "0644",
"template": "files/dunstrc"
},
{
"id": "notifier",
"type": "service",
"unit": "dunst.service",
"scope": "user",
"user": "${machine:account}",
"reload-on": [
"configuration"
]
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/dunst-tools",
"binary": "dunst-tools",
"loads": [
"dunst-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
},
"emits": [
"action-chosen",
"started",
"paused"
],
"items": [
{
"collection": "contribution:node-display-session:config",
"name": "display-session",
"template": "files/i3/50-dunst.conf"
}
]
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
+3 -2
View File
@@ -10,7 +10,8 @@ import (
// the provision, and does not find out at its first build. The manifests are the catalogue's, as they are.
// builderNode is a machine that can run a build module: the package manager and user namespaces it asks of
// the machine, and the forge account (mesh-scope, provided by the forge) stood in for.
// the machine, and the forge account (mesh-scope, provided by the forge) and the artifact store the build
// modules publish to (ADR 0513, provided by distribution) stood in for.
func builderNode() Node {
return Node{Name: "builder", Site: "house",
Capabilities: map[string]bool{"package-manager": true, "user-namespaces": true}}
@@ -19,7 +20,7 @@ func builderNode() Node {
func toolchainShelf(t *testing.T) map[string]Manifest {
t.Helper()
m := map[string]Manifest{
"forge": {Module: "forge", Version: "1", Provides: []Offer{{Name: "forge-account"}, {Name: "npm-package-registry"}}},
"forge": {Module: "forge", Version: "1", Provides: []Offer{{Name: "forge-account"}, {Name: "npm-package-registry"}, {Name: "artifact-store"}}},
}
for _, name := range []string{"go", "nodejs", "mesh-build-go", "mesh-build-node", "dev-tools"} {
m[name] = catalogueManifest(t, name)
+22 -56
View File
@@ -147,24 +147,16 @@ var ControllerVerbs = []Verb{
}, nil, "data")},
// The verbs the delivery's owner asks with (novox/hq ADR 0239): the planner's answer, a group's order and
// composed check, and the start and end of a walk. Callable by a person too; mesh-delivery is who needs them.
{Name: "delivery-plan", Description: "The delivery plan of a diffset, as the planner computes it and keeping " +
"nothing (novox/hq ADR 0239): what it moves and builds after them in tiers, what each machine receives and " +
"what waits there, the steps that are not an ordinary send, and whether the gate would run.",
Input: schema(map[string]string{"repository": "owner/repository", "head": "the commit at hand",
"base": "the branch it merges into (default main)",
"paths": "the files it changes, comma-separated, from the repository's root",
"module-dirs": "the directories holding a module.json at the head, comma-separated, as the forge says them",
"removed": "the files among paths it deletes, comma-separated"}, []string{"repository", "paths"})},
{Name: "delivery-plan", Description: "Removed in R4e (ADR 0501 §5 / ADR 0503 §3): refused by name, naming its replacement",
Input: schema(nil, nil)},
{Name: "delivery-order", Description: "A delivery group's order (novox/hq ADR 0419): its members in the order " +
"they are delivered, every pair and why — built by, verb before caller, version skew, engine before " +
"controller — read from the module graph and the manifests' change alone; no `after:` line is read. The " +
"cycle, with every module and edge of it, is the one refusal.",
Input: schema(map[string]string{"members": "the members, as JSON: [{id, repository, base, head, number, paths, " +
"module_dirs, module_dirs_said, removed}]"}, []string{"members"})},
{Name: "delivery-check", Description: "Ask the build seat for a delivery group's composed check (novox/hq ADR " +
"0239): every member's head laid over the mesh in turn and judged as one future state. Answers the ask's id; " +
"the verdict is said as `checked` with the group's id. With one member and no group, that head is checked " +
"again as its pull request is.",
{Name: "delivery-check", Description: "Answers that the checker (mesh-delivery-checker) judges a group (novox/hq " +
"ADR 0503 §3); asks nothing. Its inputs stay, since hq's repository check still names it (ADR 0513 §4).",
Input: schema(map[string]string{"group": "the group's id (none for one head checked again)",
"members": "the members, as JSON, as delivery-order takes them",
"ready": "\"true\" when the pull request is marked ready: its check is queued before every other check (novox/hq ADR 0406 §3)"},
@@ -378,45 +370,33 @@ var ControllerVerbs = []Verb{
{Name: "clear", Description: "Cancel every waiting ask in the build queue — and with dead, every dead one too — each " +
"recorded failed, cancelled by hand. Never touches one in flight.",
Input: schema(map[string]string{"dead": "\"true\" to cancel the dead asks as well"}, nil, "dead")},
{Name: "rebuild", Description: "Ask a module's current source again under a new id — the branch it follows — or, given a " +
"build's id, that build's repository, path and ref. A module a plan holds unbuilt or failed joins that plan. Answers the new id.",
Input: schema(map[string]string{"what": "a module's name, or a build's id"}, []string{"what"})},
{Name: "replay", Description: "Ask a recorded build's repository and path again at the commit it built, under a new id. " +
"A dry run unless register: nothing recorded or registered. Registering is refused when a newer build of the module " +
"is registered — it would roll the older commit out (novox/hq issue 207) — unless older says so.",
Input: schema(map[string]string{
"id": "the build's id",
"register": "\"true\" to register what it builds",
"older": "\"true\", with register: even though a newer build of the module is registered",
}, []string{"id"}, "register", "older")},
{Name: "rebuild", Description: "Removed in R4e (ADR 0501 §5 / ADR 0503 §3): refused by name, naming its replacement",
Input: schema(nil, nil)},
{Name: "replay", Description: "Ask a recorded build's repository and path again at the commit it built, under a new id: " +
"a dry run only, nothing recorded or registered (novox/hq ADR 0429 §18: a fingerprint's key is built again by the " +
"builder, never registered by hand).",
Input: schema(map[string]string{"id": "the build's id"}, []string{"id"})},
// novox/hq issue 686: a builder ran a branch's store tests for the first time in its pull request's merge check.
{Name: "try", Description: "Try a pushed branch or commit before its pull request (novox/hq issue 686): the build " +
"seat merges it into the repository's trunk and runs the repository's own merge-check.sh on the merge, in the " +
"toolchain containers and against the throwaway store and bus a pull request's merge check gets. Answers at " +
{Name: "try", Description: "Try a pushed branch or commit before its pull request (novox/hq issue 686): the " +
"checker (mesh-delivery-checker) merges the head onto the repository's trunk and runs its levels on the merge, " +
"as it judges a pull request (ADR 0503 §3); no merge-check.sh is run and build-agent is not asked. Answers at " +
"once with an id; asked again with that id it waits up to 20 s and answers pass, fail or error, how long it " +
"took and the lines naming what failed — or that it is still waiting or running, to ask again. Sets no " +
"status on the forge, registers and builds nothing, runs no merge gate, counts as a check in the build queue " +
"(after deliveries and after every pull request's check, however long it waits) and is bounded as one. One " +
"try of a commit at a time — a second ask answers the first's id — and at most four waiting or running in " +
"all: a fifth is refused. Only for a repository of the mesh; the ref must be on the forge.",
"status on the forge, registers and builds nothing for delivery. One try of a commit at a time — a second " +
"ask answers the first's id. Only for a repository of the mesh; the ref must be on the forge.",
Input: schema(map[string]string{
"repository": "the repository on the forge, owner/name; with ref",
"ref": "the branch or commit to try, as pushed to the forge",
"base": "the branch it is merged into (the repository's default branch when none is named). The " +
"merge-check.sh that runs is that branch's, never the ref's own: the same trust as a pull request's base",
"base": "the branch it is merged into (the repository's default branch when none is named)",
"tests": "only the tests this `go test -run` pattern names, where the repository's merge-check.sh reads " +
"MESH_CHECK_TESTS; a script that does not runs whole",
"store-only": "\"true\": only the repository's store tests, where its merge-check.sh reads " +
"MESH_CHECK_STORE_ONLY",
"id": "instead of a repository: the id a try answered, to read what came of it",
"store-only": "\"true\": only the integration level",
"id": "instead of a repository: the id a try answered, to read what came of it",
}, nil, "store-only"),
Replaces: []string{"sh merge-check.sh", "docker run postgres"}},
// novox/hq ADR 0350 §5: asked by the delivery's owner when the forge says a pull request closed unmerged.
{Name: "forget-checks", Description: "Let go of every build a pull request's merge checks made for its merge in " +
"the artifact store's scratch namespace (novox/hq ADR 0350 §5): asked when it closed unmerged; a merged one's go " +
"when the walk answering its merge ends, and any check build goes after a week.",
Input: schema(map[string]string{"owner": "the repository's owner", "repo": "the repository",
"number": "the pull request's number"}, []string{"owner", "repo", "number"})},
{Name: "forget-checks", Description: "Removed in R4e (ADR 0501 §5 / ADR 0503 §3): refused by name, naming its replacement",
Input: schema(nil, nil)},
{Name: "kill", Description: "End a build where it runs: the machine that took it stops its commands and containers and " +
"announces it failed, killed by hand — settled, never handed to another machine.",
Input: schema(map[string]string{"id": "the build's id"}, []string{"id"})},
@@ -595,22 +575,8 @@ var ControllerVerbs = []Verb{
"machine": "one machine (optional)",
"retired": "\"true\": only what is retired",
}, nil, "retired")},
// Three shapes, as the command has them: one repository, every module standing on a base (on), or
// every module behind its source (behind). One per call; a second is refused as passed over.
{Name: "build", Description: "Have the build machine build a repository — or, instead, every module that " +
"stands on a base (on), or every module behind its source (behind). Answers at once with each build's id: " +
"`builds` with an id follows it line by line, and a module is registered when its outcome comes.",
Input: schema(map[string]string{
"repository": "the repository's URL, or its path on the forge holding the git seat (owner/name); " +
"needed unless on or behind is given",
"path": "with repository: the module's directory inside it (optional)",
"ref": "with repository: the branch, tag or commit to build (optional)",
"on": "instead of a repository: a module's name. Every module the mesh holds that was built against " +
"its artifacts is built again from its own source, bases among them asked first — the rebuild a " +
"changed base needs, which behind does not see because their sources did not move",
"behind": "\"true\", instead of a repository: build every module the mesh holds whose source has moved " +
"past the commit it was built from, bases asked first",
}, nil, "behind")},
{Name: "build", Description: "Removed in R4e (ADR 0501 §5 / ADR 0503 §3): refused by name, naming its replacement",
Input: schema(nil, nil)},
// What the records say the registries may keep, asked on demand (novox/hq ADR 0251, to-be 51).
{Name: "artifacts", Description: "Every artifact the mesh recorded making that lives in its artifact store, " +
"each with its repository, digest, kind (image or archive) and state: kept — and why: a definition names " +
+20 -3
View File
@@ -74,7 +74,10 @@ func (c *composer) overlayRange() string {
}
// network builds the private network over the machines that resolved the module for it. A machine is on it because it
// was given the module, or because it is joining with a key and an address (novox/hq ADR 0169).
// was given the module, or because it is joining with a key and an address (novox/hq ADR 0169), or because its own set
// could not be resolved and it holds a place (a key and an address): that place is its last good one on the network, and
// it does not change because its modules failed, so every other machine keeps its path to it -- the hub above all -- and
// only its own declaration is refused (issue 1833). A machine refused with no place is left off.
func (c *composer) network(on map[string]bool, refused map[string]string) (*overlay.Generator, error) {
tunnels := c.in.Tunnels
carried := c.in.CarriedPeers
@@ -85,9 +88,15 @@ func (c *composer) network(on map[string]bool, refused map[string]string) (*over
var nodes []overlay.Node
for _, node := range c.in.sorted() {
p := node.Place
if !on[node.Name] && !(isJoining[node.Name] && p.Key != "" && p.Address != "") {
placed := p.Key != "" && p.Address != ""
_, wasRefused := refused[node.Name]
if !on[node.Name] && !((isJoining[node.Name] || wasRefused) && placed) {
continue
}
if !on[node.Name] && wasRefused && !isJoining[node.Name] {
c.say("%s could not be resolved and keeps its place on the private network at %s, so the others keep their path to it",
node.Name, p.Address)
}
n := overlay.Node{
Name: node.Name, Key: p.Key, Endpoint: p.Endpoint,
Site: p.Site, Hub: p.Hub, Address: p.Address,
@@ -233,7 +242,9 @@ func (c *composer) whereEveryoneIs() (map[string]string, error) {
}
// onTheNetwork is every placed machine that resolves the private network: has an address AND runs what puts it there
// (novox/hq issue 079). A machine that does not resolve is not counted, and said.
// (novox/hq issue 079). A machine that does not resolve keeps its place when it holds one (a key and an address), by the
// same rule as network(), so its name and its address stay where the tunnel to it does (issue 1833); one without is not
// counted, and said.
func (c *composer) onTheNetwork() []Node {
var out []Node
for _, n := range c.in.sorted() {
@@ -247,6 +258,12 @@ func (c *composer) onTheNetwork() []Node {
catalogue.Node{Name: n.Name, Site: n.Place.Site, Capabilities: n.Capabilities},
catalogue.World{Unchecked: true, Holdings: c.in.Holdings})
if err != nil {
if n.Place.Key != "" {
c.say("%s could not be resolved and keeps its place on the private network at %s, so the others keep their path to it",
n.Name, n.Place.Address)
out = append(out, n)
continue
}
c.say("%s is not counted as on the network: it does not resolve: %v", n.Name, err)
continue
}
+8 -3
View File
@@ -159,9 +159,11 @@ func TestANodeNotOnThePrivateNetworkGetsNoAddressAndNoNetwork(t *testing.T) {
}
// 1360/AC3: a node whose resolution is refused is said with today's words: the network missing its hub names the
// machine that could not be resolved, which is almost always why.
// machine that could not be resolved, which is almost always why. The hub here holds no complete place (no key): a refused
// machine with one keeps it on the network (issue 1833), so only one without is missing from it.
func TestARefusedNodeIsSaidWhenTheNetworkCannotBeComposed(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "anchor").Place.Key = ""
one := catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
two := catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-one"], in.Catalogue["rival-two"] = one, two
@@ -210,9 +212,10 @@ func TestEachFieldOfTheInputReachesTheGenerator(t *testing.T) {
if len(base) != 2 {
t.Fatalf("the base mesh has %v", base)
}
t.Run("Catalogue: without the module nobody is on the network", func(t *testing.T) {
t.Run("Catalogue: without the module offering it nobody is on the network", func(t *testing.T) {
in := aMesh(t)
in.Catalogue = map[string]catalogue.Manifest{}
// The module stays known, so every machine still resolves: a machine that does not resolve keeps its place (issue 1833).
in.Catalogue = map[string]catalogue.Manifest{overlay.Name: {Module: overlay.Name, Version: "1"}}
if got := addressesOf(in); len(got) != 0 {
t.Errorf("got %v", got)
}
@@ -290,6 +293,8 @@ func TestEachFieldOfTheInputReachesTheGenerator(t *testing.T) {
}
laptop := nodeOf(t, &in, "laptop")
laptop.Assigned = append(laptop.Assigned, "holder-one", "holder-two")
// Without a key: a machine that does not resolve and holds a complete place keeps it on the network (issue 1833).
laptop.Place.Key = ""
if got := addressesOf(in); got["laptop"] != "" {
t.Fatalf("a machine whose set claims one seat twice is on the network: %v", got)
}
+3 -1
View File
@@ -387,9 +387,11 @@ func TestAnotherNodesKeptBuildDoesNotChangeTheRestOfTheMeshOrWhoIsOnTheNetwork(t
func overlayName() string { return overlay.Name }
// Round 1, finding 6: the nodes the composition did not count as on the network are returned, said, not lost.
// Round 1, finding 6: the nodes the composition did not count as on the network are returned, said, not lost. laptop holds no
// complete place (no key): one that does keeps it on the network (issue 1833) and is counted.
func TestANodeNotCountedAsOnTheNetworkIsSaidInTheResult(t *testing.T) {
in := aMesh(t)
nodeOf(t, &in, "laptop").Place.Key = ""
in.Catalogue["rival-one"] = catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-two"] = catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
laptop := nodeOf(t, &in, "laptop")
+167
View File
@@ -0,0 +1,167 @@
package compose
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// fourOnTheNetwork is aMesh and two more machines on the private network at the site `there`: `ace`, dialled at
// ace.example:51820, at 10.77.0.3, and `box`, behind no endpoint, at 10.77.0.4. A module `office` requires
// `document-server`, which nothing in the catalogue provides: the shape of issue 1833 (nextcloud on the hub wanting a
// document server nobody offered).
func fourOnTheNetwork(t *testing.T) Input {
t.Helper()
in := aMesh(t)
in.Catalogue["office"] = catalogue.Manifest{Module: "office", Version: "1", Requires: []string{"document-server"}}
caps := map[string]bool{"container-runtime": true, "wireguard": true, "systemd": true}
in.Nodes = append(in.Nodes,
Node{Name: "ace", Place: Place{Key: keyOf(t, "ace"), Endpoint: "ace.example:51820", Site: "there", Address: "10.77.0.3"},
Assigned: []string{overlay.Name}, Capabilities: caps},
Node{Name: "box", Place: Place{Key: keyOf(t, "box"), Site: "there", Address: "10.77.0.4"},
Assigned: []string{overlay.Name}, Capabilities: caps})
return in
}
// unresolvableBy makes one machine's own set fail to resolve: it is also given `office`, whose requirement nothing provides.
func unresolvableBy(t *testing.T, in *Input, name string) {
t.Helper()
n := nodeOf(t, in, name)
n.Assigned = append(n.Assigned, "office")
}
// networkOf is the generator of an Input's private network, failing the test when it is not composed.
func networkOf(t *testing.T, in Input) *overlay.Generator {
t.Helper()
gens, err := Generators(in)
if err != nil {
t.Fatalf("the private network was not composed: %v", err)
}
return gens[overlay.Name].(*overlay.Generator)
}
func placesOf(g *overlay.Generator) map[string]overlay.Node {
out := map[string]overlay.Node{}
for _, n := range g.Nodes() {
out[n.Name] = n
}
return out
}
// 1833/AC1: four machines, the hub's own set does not resolve; the other three compose their declarations, and the hub alone is
// refused, with an error that is its own set failing and names what could not be resolved.
func TestOneMachineThatDoesNotResolveIsRefusedAlone(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
for _, node := range []string{"laptop", "ace", "box"} {
d, problems, err := Compose(in, node)
if err != nil || len(problems) != 0 {
t.Errorf("%s does not compose because another machine does not resolve: %v %v", node, err, problems)
continue
}
if !has(resourceIDs(d), "file:mesh-wireguard.overlay-config") {
t.Errorf("%s is given no private network: %v", node, resourceIDs(d))
}
}
_, _, err := Compose(in, "anchor")
if err == nil || !IsUnresolvable(err) {
t.Fatalf("the machine that does not resolve is not refused as its own set failing: %v", err)
}
if !strings.Contains(err.Error(), `nothing provides "document-server", wanted by office`) {
t.Errorf("the refusal does not name what could not be resolved: %v", err)
}
}
// 1833/AC2: the private network is composed with every machine on it: the hub that does not resolve keeps its place (key,
// address, endpoint, site, hub role), so every other machine keeps its path through it.
func TestAMachineThatDoesNotResolveKeepsItsPlaceOnTheNetwork(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
got := placesOf(networkOf(t, in))
if len(got) != 4 {
t.Fatalf("the network is over %v, want all four machines", got)
}
hub := got["anchor"]
if !hub.Hub || hub.Key != keyOf(t, "anchor") || hub.Address != "10.77.0.1" || hub.Endpoint != "anchor.example:51820" || hub.Site != "here" {
t.Errorf("the hub did not keep its place: %+v", hub)
}
nodes, graph, err := Graph(in)
if err != nil || len(nodes) != 4 {
t.Fatalf("the graph is not shown with every machine: %v %v", nodes, err)
}
for _, node := range []string{"ace", "box"} {
through := false
for _, p := range graph[node] {
through = through || (p.Name == "anchor" && p.Endpoint == "anchor.example:51820")
}
if !through {
t.Errorf("%s has no path through the hub: %+v", node, graph[node])
}
}
}
// 1833/AC3: a machine that does not resolve and was never placed (a new machine: no key, no address) is left off the network,
// which is composed from the rest -- here with the hub that does not resolve on its last place -- and that machine alone is refused.
func TestANewMachineThatDoesNotResolveIsLeftOffAndRefusedAlone(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
in.Nodes = append(in.Nodes, Node{Name: "fresh", Place: Place{Site: "there"},
Assigned: []string{overlay.Name, "office"}, Capabilities: map[string]bool{"wireguard": true, "systemd": true}})
got := placesOf(networkOf(t, in))
if _, on := got["fresh"]; on || len(got) != 4 {
t.Fatalf("the network is over %v, want the four placed machines without fresh", got)
}
if _, _, err := Compose(in, "fresh"); err == nil || !IsUnresolvable(err) {
t.Errorf("the new machine is not refused as its own set failing: %v", err)
}
if _, problems, err := Compose(in, "box"); err != nil || len(problems) != 0 {
t.Errorf("another machine does not compose: %v %v", err, problems)
}
}
// 1833/AC4: a network that truly cannot be composed from what remains still fails: the hub does not resolve and has no
// complete place (no key), so nothing is the hub; the refusal is today's, naming the machine that could not be resolved.
func TestANetworkThatCannotBeComposedFromWhatRemainsStillFails(t *testing.T) {
in := fourOnTheNetwork(t)
unresolvableBy(t, &in, "anchor")
nodeOf(t, &in, "anchor").Place.Key = ""
_, err := Generators(in)
if err == nil {
t.Fatal("a network without a hub was composed")
}
for _, want := range []string{overlay.ErrNoHub.Error(), "and 1 node(s) could not be resolved at all, which is probably why:", " anchor: "} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q:\n%v", want, err)
}
}
if _, _, err := Compose(in, "laptop"); err == nil {
t.Error("a declaration was composed over a network that cannot be")
}
}
// 1833/AC2, a failure the composition's unchecked pass refuses too (two modules claiming one seat on the hub, the 1360/AC3
// shape): the hub keeps its place in what every other machine is sent besides the tunnel -- its internal name and its address
// among the sources "from the mesh" -- and another machine still composes.
func TestAMachineThatDoesNotResolveKeepsItsNameAndItsAddressInTheMesh(t *testing.T) {
in := fourOnTheNetwork(t)
in.Catalogue["rival-one"] = catalogue.Manifest{Module: "rival-one", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
in.Catalogue["rival-two"] = catalogue.Manifest{Module: "rival-two", Version: "1", Claims: []catalogue.Claim{{Name: "the-seat", Scope: catalogue.ScopeNode}}}
hub := nodeOf(t, &in, "anchor")
hub.Assigned = append(hub.Assigned, "rival-one", "rival-two")
if _, err := Resolve(in, "anchor"); !IsUnresolvable(err) {
t.Fatalf("the hub is meant not to resolve: %v", err)
}
c := newComposer(in)
if got := c.namesInTheMesh()["anchor."+c.suffix()]; got != "10.77.0.1" {
t.Errorf("the hub's name is not in the mesh's names: %v", c.namesInTheMesh())
}
if got := c.onThePrivateNetwork(); !slices.Contains(got, "10.77.0.1") || len(got) != 4 {
t.Errorf("the sources from the mesh are %v, want all four machines with the hub", got)
}
if _, problems, err := Compose(in, "ace"); err != nil || len(problems) != 0 {
t.Errorf("another machine does not compose: %v %v", err, problems)
}
}
+109 -8
View File
@@ -63,8 +63,18 @@ func ParseBuilderBuildKey(key string) (module, fingerprint string, err error) {
return module, fingerprint, nil
}
// BuiltArtifact is one artifact of a done build: its name in the manifest, its kind, its content digest and, for an
// image, the reference it was pushed to.
// BuiltArtifact is one artifact of a done build: its name in the manifest, its kind, its content digest (`sha256:` and
// 64 hex digits) and the reference that names where it is (novox/hq ADR 0498 clauses 1, 5 and 6; the lead's decision
// L1 of R4e), by kind:
//
// bundle, archive artifact-store://<module>/<artifact>/blobs/<digest> the blob pushed to the artifact store
// image, upstream <registry>/<module>/<artifact>@<digest> the image registry's reference, which
// artifact-store://<module>/<artifact>@<digest> catalogue.Recorded keeps without its address
// package <name>@<version> the version published from the trunk
//
// <artifact> is Name and <digest> is Digest. A reference is empty for a candidate's image or package, which is never
// pushed or published (ADR 0498 clause 6), and in a key done before R4e; the controller delivers only from a key whose
// every artifact names one (Referenced, clause 8). A kind left empty (a key written before kinds) is not checked.
type BuiltArtifact struct {
Name string `json:"name"`
Kind string `json:"kind,omitempty"`
@@ -72,6 +82,83 @@ type BuiltArtifact struct {
Reference string `json:"reference,omitempty"`
}
const artifactStore = "artifact-store://"
// referenceProblem says what is wrong with a's reference for its kind, or "".
func (a BuiltArtifact) referenceProblem() string {
r := a.Reference
if r == "" || a.Kind == "" {
return ""
}
switch a.Kind {
case "bundle", "archive":
path, ok := strings.CutPrefix(r, artifactStore)
module, rest, _ := strings.Cut(path, "/")
if !ok || module == "" || rest != a.Name+"/blobs/"+a.Digest {
return "is not artifact-store://<module>/" + a.Name + "/blobs/" + a.Digest
}
case "image", "upstream":
path, digest, ok := strings.Cut(r, "@")
if !ok || digest != a.Digest {
return "does not name its digest " + a.Digest
}
var host, module, name string
if rest, kept := strings.CutPrefix(path, artifactStore); kept {
module, name, _ = strings.Cut(rest, "/")
host = "kept"
} else {
parts := strings.Split(path, "/")
if len(parts) == 3 {
host, module, name = parts[0], parts[1], parts[2]
}
}
if host == "" || !(host == "kept" || host == "localhost" || strings.ContainsAny(host, ".:")) || module == "" || name != a.Name {
return "is not <registry>/<module>/" + a.Name + "@" + a.Digest
}
case "package":
at := strings.LastIndex(r, "@")
if at <= 0 || at == len(r)-1 || r[:at] == "@" {
return "is not <name>@<version>"
}
}
return ""
}
// Referenced says whether b is done and every artifact names where it is, which the controller delivers from (novox/hq
// ADR 0498 clause 8), and names the artifacts that name no reference. A done key with no artifact (a module that builds
// none, issue 1613) is referenced.
func (b BuilderBuild) Referenced() (ok bool, missing []string) {
for _, a := range b.Artifacts {
if a.Reference == "" {
missing = append(missing, a.Name)
}
}
return b.State == BuildDone && len(missing) == 0, missing
}
// The prefixes of a failed key's error that a build module answers (novox/hq ADR 0498 clauses 2, 4 and 5; ADR 0478
// clause 2), each followed by the artifact and the words of what refused it.
const (
// NotStored: the push of a bundle, an archive or an upstream copy to the artifact store failed; built again.
NotStored = "not stored: "
// NotPinned: an upstream artifact names no digest.
NotPinned = "not pinned: "
// VersionTaken: the package registry holds the package's version with another packed file.
VersionTaken = "version taken: "
// RefusedKind: no build module builds this kind of artifact in its language.
RefusedKind = "refused kind: "
// WaitsFor: a fetch found the module's lock naming a package version the registry does not hold yet; the builder
// writes the key queued with Waits (novox/hq ADR 0500 clause 6), and it is built once the package is published.
WaitsFor = "waits for: "
)
// FailedUntilChanged says whether a failed key's error is one that is not built again until the module's fingerprint
// changes (not pinned, version taken, refused kind: the same source fails the same way); every other failed key, a
// push that was not stored included, is built again as a new revision (ADR 0429 clause 18).
func FailedUntilChanged(err string) bool {
return strings.HasPrefix(err, NotPinned) || strings.HasPrefix(err, VersionTaken) || strings.HasPrefix(err, RefusedKind)
}
// ToolchainUsed is what a build used for one language (ADR 0405 clauses 2 and 7): the installed toolchain's version and
// the build module's recipe version.
type ToolchainUsed struct {
@@ -82,7 +169,8 @@ type ToolchainUsed struct {
// BuilderBuild is the value of a builder build key: its State (queued, building, done or failed), By the builder
// instance that holds it, Step what it is doing or did last, Manifest the digest of the module's manifest (the key of
// the indexer's manifests bucket), Artifacts when done, Toolchains by language when it built or is building (a done
// build names them, so two builders that differ are seen: ADR 0405 clause 2), and Error when failed.
// build names them, so two builders that differ are seen: ADR 0405 clause 2), Error when failed, and Waits when queued for a
// package version.
type BuilderBuild struct {
V int `json:"v"`
State string `json:"state"`
@@ -92,6 +180,10 @@ type BuilderBuild struct {
Artifacts []BuiltArtifact `json:"artifacts,omitempty"`
Toolchains map[string]ToolchainUsed `json:"toolchains,omitempty"`
Error string `json:"error,omitempty"`
// Waits is, on a queued key only, the package version `<name>@<version>` the module's lock names and the registry
// does not yet hold: the key is not claimed until that package's trunk key is done and published (novox/hq ADR 0500
// clause 6). A reader that does not know the field reads the key as queued.
Waits string `json:"waits,omitempty"`
}
func (b BuilderBuild) check() error {
@@ -120,6 +212,17 @@ func (b BuilderBuild) check() error {
if a.Digest == "" {
return fmt.Errorf("delivery: the artifact %q of the build has no digest", a.Name)
}
if p := a.referenceProblem(); p != "" {
return fmt.Errorf("delivery: the reference %q of the %s %q %s", a.Reference, a.Kind, a.Name, p)
}
}
if b.Waits != "" {
if b.State != BuildQueued {
return fmt.Errorf("delivery: a %s build waits for %q; only a queued key waits for a package", b.State, b.Waits)
}
if p := (BuiltArtifact{Kind: "package", Reference: b.Waits}).referenceProblem(); p != "" {
return fmt.Errorf("delivery: the package a key waits for, %q, %s", b.Waits, p)
}
}
switch b.State {
case BuildQueued, BuildBuilding:
@@ -127,9 +230,7 @@ func (b BuilderBuild) check() error {
return fmt.Errorf("delivery: a %s build carries no artifact and no error", b.State)
}
case BuildDone:
if len(b.Artifacts) == 0 {
return fmt.Errorf("delivery: a done build names no artifact")
}
// A module that builds no artifact gets a done key too, and is delivered from it (issue 1613).
if b.Manifest == "" {
return fmt.Errorf("delivery: a done build names no manifest digest")
}
@@ -144,8 +245,8 @@ func (b BuilderBuild) check() error {
return nil
}
// EncodeBuilderBuild is b as JSON at Version; it refuses a build its state does not allow (done without artifacts or a
// manifest digest, failed without an error, an artifact without a name or digest, a toolchain without a version or a
// EncodeBuilderBuild is b as JSON at Version; it refuses a build its state does not allow (done without a manifest
// digest, failed without an error, an artifact without a name or digest, a toolchain without a version or a
// recipe version).
func EncodeBuilderBuild(b BuilderBuild) ([]byte, error) {
if err := b.check(); err != nil {
+143
View File
@@ -0,0 +1,143 @@
package delivery
import (
"strings"
"testing"
)
func artifactOf(name, kind, digest, reference string) BuiltArtifact {
return BuiltArtifact{Name: name, Kind: kind, Digest: digest, Reference: reference}
}
// A done key names where each artifact is (novox/hq ADR 0498 clauses 1, 5 and 6; the lead's decision L1 of R4e): a
// bundle or an archive by its artifact-store blob, an image or an upstream copy by the registry's reference to its
// digest (or the kept artifact-store form), a package by name and version. A reference of the wrong form for its kind,
// or naming another digest or artifact, is refused, naming the artifact.
func TestADoneKeyNamesEachArtifactsReferenceInTheFormOfItsKind(t *testing.T) {
d := "sha256:" + digestOf('6')
good := []BuiltArtifact{
artifactOf("code", "bundle", d, "artifact-store://mail/code/blobs/"+d),
artifactOf("files", "archive", d, "artifact-store://mail/files/blobs/"+d),
artifactOf("server", "image", d, "10.0.0.1:5000/mail/server@"+d),
artifactOf("server", "image", d, "artifact-store://mail/server@"+d),
artifactOf("nats", "upstream", d, "registry.example/nats/nats@"+d),
artifactOf("lib", "package", d, "@novox/mesh-sdk@0.1.15"),
artifactOf("lib", "package", d, "left-pad@1.3.0"),
artifactOf("old", "", d, "anything at all"),
}
for _, a := range good {
b := doneBuild()
b.Artifacts = []BuiltArtifact{a}
if _, err := EncodeBuilderBuild(b); err != nil {
t.Errorf("%+v was refused: %v", a, err)
}
}
other := "sha256:" + digestOf('7')
bad := []BuiltArtifact{
artifactOf("code", "bundle", d, "artifact-store://mail/code/blobs/"+other),
artifactOf("code", "bundle", d, "artifact-store://mail/other/blobs/"+d),
artifactOf("code", "bundle", d, "10.0.0.1:5000/mail/code@"+d),
artifactOf("server", "image", d, "10.0.0.1:5000/mail@"+d),
artifactOf("server", "image", d, "10.0.0.1:5000/mail/server@"+other),
artifactOf("server", "image", d, "mail/server@"+d),
artifactOf("server", "image", d, "artifact-store://mail/server/blobs/"+d),
artifactOf("lib", "package", d, "@novox/mesh-sdk"),
artifactOf("lib", "package", d, "0.1.15"),
}
for _, a := range bad {
b := doneBuild()
b.Artifacts = []BuiltArtifact{a}
if _, err := EncodeBuilderBuild(b); err == nil || !strings.Contains(err.Error(), `"`+a.Name+`"`) {
t.Errorf("%+v: %v, wanted a refusal naming the artifact", a, err)
}
}
}
// The controller delivers only from a done key whose every artifact names a reference (ADR 0498 clause 8): Referenced
// says so and names the artifacts without one.
func TestReferencedNamesTheArtifactsWithoutAReference(t *testing.T) {
d := "sha256:" + digestOf('6')
b := doneBuild()
b.Artifacts = []BuiltArtifact{
artifactOf("code", "bundle", d, "artifact-store://mail/code/blobs/"+d),
artifactOf("server", "image", d, ""),
}
if ok, missing := b.Referenced(); ok || len(missing) != 1 || missing[0] != "server" {
t.Fatalf("Referenced = %v, %v; want false, [server]", ok, missing)
}
b.Artifacts[1].Reference = "10.0.0.1:5000/mail/server@" + d
if ok, missing := b.Referenced(); !ok || len(missing) != 0 {
t.Fatalf("Referenced = %v, %v; want true", ok, missing)
}
if ok, _ := (BuilderBuild{State: BuildQueued}).Referenced(); ok {
t.Fatal("a queued key is referenced")
}
}
// The errors a build module answers open with a fixed prefix (ADR 0498 clauses 2, 4 and 5; ADR 0478 clause 2), and three
// of them are not built again until the module's fingerprint changes; a failed push is.
func TestOnlyAPushFailureIsBuiltAgainBeforeTheFingerprintChanges(t *testing.T) {
for _, c := range []struct {
err string
again bool
}{
{NotStored + "code: the store said 500", true},
{NotPinned + "nats: upstream names no digest", false},
{VersionTaken + "@novox/mesh-sdk@0.1.15 holds another file", false},
{RefusedKind + "a package in go", false},
{"go test: 2 failed", true},
} {
if got := FailedUntilChanged(c.err); got == c.again {
t.Errorf("FailedUntilChanged(%q) = %v", c.err, got)
}
}
if NotStored != "not stored: " || NotPinned != "not pinned: " || VersionTaken != "version taken: " || RefusedKind != "refused kind: " {
t.Fatal("a prefix changed its words")
}
}
// A key whose module's lock names a version of the group's package not yet published waits, queued, saying which
// (ADR 0500 clause 6): an older reader reads it as queued, and only a queued key waits.
func TestAKeyWaitsQueuedForAPackageVersion(t *testing.T) {
b := BuilderBuild{State: BuildQueued, Manifest: digestOf('4'), Waits: "@novox/mesh-sdk@0.1.15"}
data, err := EncodeBuilderBuild(b)
if err != nil {
t.Fatal(err)
}
back, err := DecodeBuilderBuild(data)
if err != nil || back.Waits != b.Waits || back.State != BuildQueued {
t.Fatalf("decoded %+v, %v", back, err)
}
building := BuilderBuild{State: BuildBuilding, By: "x", Waits: "@novox/mesh-sdk@0.1.15"}
if _, err := EncodeBuilderBuild(building); err == nil {
t.Fatal("a building key that waits was accepted")
}
if _, err := EncodeBuilderBuild(BuilderBuild{State: BuildQueued, Waits: "mesh-sdk"}); err == nil {
t.Fatal("a wait naming no version was accepted")
}
}
// A module that builds no artifact gets a done key too (issue 1613), and it is delivered: Referenced is ok for a done
// key with no artifact, since no artifact lacks a reference; a queued one with none still is not.
func TestADoneKeyWithNoArtifactIsReferenced(t *testing.T) {
b := BuilderBuild{State: BuildDone, Manifest: digestOf('4')}
data, err := EncodeBuilderBuild(b)
if err != nil {
t.Fatalf("a done key with no artifact was refused: %v", err)
}
back, err := DecodeBuilderBuild(data)
if err != nil {
t.Fatal(err)
}
if ok, missing := back.Referenced(); !ok || len(missing) != 0 {
t.Fatalf("Referenced = %v, %v; want true", ok, missing)
}
}
// A Node fetch that finds the lock naming a package version the registry does not hold fails with WaitsFor; the
// builder then writes the key queued with Waits, and the error is not one that waits for the fingerprint to change.
func TestWaitsForIsAPrefixThatIsBuiltAgain(t *testing.T) {
if WaitsFor != "waits for: " || FailedUntilChanged(WaitsFor+"@novox/mesh-sdk@0.1.15") {
t.Fatalf("WaitsFor = %q", WaitsFor)
}
}
+4 -5
View File
@@ -15,7 +15,7 @@ func doneBuild() BuilderBuild {
Manifest: digestOf('4'),
Artifacts: []BuiltArtifact{
{Name: "code", Kind: "bundle", Digest: "sha256:" + digestOf('5')},
{Name: "server", Kind: "image", Digest: "sha256:" + digestOf('6'), Reference: "registry.example/mail@sha256:" + digestOf('6')},
{Name: "server", Kind: "image", Digest: "sha256:" + digestOf('6'), Reference: "registry.example/mail/server@sha256:" + digestOf('6')},
},
Toolchains: map[string]ToolchainUsed{
"go": {Version: "1.26.0", Recipe: "3"},
@@ -83,14 +83,13 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
}
}
// 1358/AC3: done without artifacts or without a manifest digest, failed without an error, an artifact without a digest
// 1358/AC3: done without a manifest digest, failed without an error, an artifact without a digest
// and an unknown state are refused.
func TestBuilderBuildChecked(t *testing.T) {
cases := map[string]struct {
edit func(*BuilderBuild)
words []string
}{
"done without artifacts": {func(b *BuilderBuild) { b.Artifacts = nil }, []string{"artifact"}},
"done without a manifest": {func(b *BuilderBuild) { b.Manifest = "" }, []string{"manifest"}},
"a malformed manifest": {func(b *BuilderBuild) { b.Manifest = "abc" }, []string{"abc"}},
"an artifact without a digest": {func(b *BuilderBuild) { b.Artifacts[1].Digest = "" }, []string{"digest", "server"}},
@@ -125,8 +124,8 @@ func TestBuilderBuildChecked(t *testing.T) {
t.Error("a queued build with artifacts accepted")
}
// decode refuses what encode refuses
if _, err := DecodeBuilderBuild([]byte(`{"v":1,"state":"done","manifest":"` + digestOf('4') + `"}`)); err == nil {
t.Error("decoded a done build without artifacts")
if _, err := DecodeBuilderBuild([]byte(`{"v":1,"state":"done"}`)); err == nil {
t.Error("decoded a done build without a manifest digest")
}
}
+88
View File
@@ -0,0 +1,88 @@
package buildrun
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ChooseNodes picks up to want build nodes for a language (ADR 0478 clause 3): the nodes whose toolchains key reports the
// version the nodes agree on, the ones with room under their build capacity first (places), then the fewest lanes the
// part has running or chosen there, a tie broken by the node's name. A place is reserved on each node chosen, in the
// same step as the choice, so builds choosing at once do not choose the same full node while another has room; the
// caller takes the place (Places.Acquire) or gives it up (Places.Cancel). A node at its capacity is still chosen when
// the others are too (fewer than want have room): its lane waits for a place. A node qualifies only when qualifies is true
// of it (online and its build module healthy; nil means every node does). It answers the toolchain all the reporting nodes
// agree on, qualifying or not, as the indexer computes it. When nothing is reported, or the nodes report different
// versions (so the indexer agreed none), it fails naming the language and why; when no reporting node qualifies it fails
// naming the nodes. A node whose build module refuses on its toolchain refuses the step, and the build fails naming it.
func ChooseNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, delivery.Toolchain2, error) {
tc, none := delivery.AgreedToolchain(entries)
if none != "" {
return nil, delivery.Toolchain2{}, fmt.Errorf("no %s build node qualifies: %s", language, none)
}
nodes, err := choose(language, entries, places, qualifies, want)
return nodes, tc, err
}
// ChooseAnyNodes is ChooseNodes for artifacts that need no toolchain, such as an archive (ADR 0478 clause 1): the
// build module's nodes agree on nothing, so any node of the build module that qualifies is chosen, by capacity and load only.
func ChooseAnyNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
return choose(language, entries, places, qualifies, want)
}
func choose(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
if places == nil {
places = &Places{}
}
nodes := make([]string, 0, len(entries))
var reporting []string
seen := map[string]bool{}
for _, e := range entries {
if seen[e.Node] {
continue
}
seen[e.Node] = true
reporting = append(reporting, e.Node)
if places.CapacityOf(e.Node) > 0 && (qualifies == nil || qualifies(e.Node)) { // a capacity of 0 keeps the part off a node
nodes = append(nodes, e.Node)
}
}
if len(nodes) == 0 {
sort.Strings(reporting)
return nil, fmt.Errorf("no %s build node qualifies: %s report the toolchain, and none is online with a healthy build module and a build capacity", language, strings.Join(reporting, ", "))
}
return places.choose(nodes, want), nil
}
// DigestsDifferPrefix opens the error of a build whose nodes built an artifact to different digests (ADR 0478 clause 4):
// such a key is not built again as a new revision, since the same source on the same toolchains gave two answers and
// building again settles nothing; it waits for another fingerprint.
const DigestsDifferPrefix = "digests differ:"
// Agree is the one digest every node built an artifact to (ADR 0478 clause 4), or an error naming each node and its
// digest when they differ. One node's digest agrees with itself.
func Agree(artifact string, byNode map[string]string) (string, error) {
nodes := make([]string, 0, len(byNode))
for n := range byNode {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var first string
for i, n := range nodes {
if i == 0 {
first = byNode[n]
continue
}
if byNode[n] != first {
msg := fmt.Sprintf("%s the nodes built %s to different digests:", DigestsDifferPrefix, artifact)
for _, n := range nodes {
msg += fmt.Sprintf(" %s %s;", n, byNode[n])
}
return "", fmt.Errorf("%s", msg[:len(msg)-1])
}
}
return first, nil
}
+99
View File
@@ -0,0 +1,99 @@
package buildrun
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
func hex64(c byte) string { return strings.Repeat(string(c), 64) }
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
func TestChooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { // 1607/AC4, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "novox", "ace", "g14", "shanks")
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 3, "ace": 1, "g14": 1, "shanks": 0}), nil, 2)
if err != nil {
t.Fatal(err)
}
if len(nodes) != 2 || nodes[0] != "shanks" || nodes[1] != "ace" {
t.Errorf("nodes = %v, want [shanks ace]: fewest steps first, a tie by name", nodes)
}
if tc.Version != "go1.27.1" || tc.Recipe != "1" || tc.Language != "go" {
t.Errorf("toolchain = %+v", tc)
}
}
func TestChooseWithOneNodeAnswersOneNode(t *testing.T) { // 1607/AC4
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "novox"), nil, nil, 2)
if err != nil || len(nodes) != 1 || nodes[0] != "novox" {
t.Errorf("nodes = %v, err %v, want [novox]", nodes, err)
}
}
func TestChooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies(t *testing.T) { // 1607/AC4
_, _, err := ChooseNodes("typescript", nil, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "typescript") || !strings.Contains(err.Error(), delivery.ToolchainNoReport) {
t.Errorf("no report: error = %v, want it to name typescript and %q", err, delivery.ToolchainNoReport)
}
entries := append(tcs("go", "go1.27.1", "novox"), tcs("go", "go1.26.0", "ace")...)
_, _, err = ChooseNodes("go", entries, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), "go1.26.0") || !strings.Contains(err.Error(), "go") {
t.Errorf("disagreeing nodes: error = %v, want both versions named", err)
}
}
func TestAgreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer(t *testing.T) { // 1607/AC4, ADR 0478 clause 4
d, err := Agree("tool", map[string]string{"ace": digest, "novox": digest})
if err != nil || d != digest {
t.Errorf("Agree = %q, %v, want %q", d, err, digest)
}
_, err = Agree("tool", map[string]string{"ace": digest, "novox": "sha256:" + hex64('9')})
if err == nil || !strings.Contains(err.Error(), "ace") || !strings.Contains(err.Error(), "novox") || !strings.Contains(err.Error(), digest) || !strings.Contains(err.Error(), hex64('9')) || !strings.Contains(err.Error(), "tool") {
t.Errorf("differing digests: error = %v, want the artifact, both nodes and both digests", err)
}
if d, err := Agree("tool", map[string]string{"ace": digest}); err != nil || d != digest {
t.Errorf("one node: %q, %v", d, err)
}
}
func TestChooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs(t *testing.T) { // 1607 review 3, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "ace", "g14", "novox")
online := func(n string) bool { return n != "g14" } // a sleeping laptop
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 5}), online, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27.1" {
t.Errorf("nodes = %v, toolchain %+v, err %v, want [ace novox] with g14 left out although it runs nothing", nodes, tc, err)
}
_, tc, err = ChooseNodes("go", entries, nil, func(string) bool { return false }, 2)
if err == nil || !strings.Contains(err.Error(), "online") || !strings.Contains(err.Error(), "g14") || !strings.Contains(err.Error(), "go") {
t.Errorf("none online: error = %v, want it to name the nodes that report and that none is online", err)
}
if tc.Version != "go1.27.1" {
t.Errorf("none online: the agreed toolchain %+v is still answered", tc)
}
}
// placesRunning is Places with these steps counted as running, for a test of the choice.
func placesRunning(running map[string]int) *Places {
return &Places{running: running, changed: make(chan struct{})}
}
func TestChoosePrefersNodesBelowTheirCapacity(t *testing.T) { // round 3
entries := tcs("go", "go1.27.1", "ace", "novox", "shanks")
// shanks has capacity 3 with 2 running: room; ace has the default 1 with 1 running: none; novox has none running.
p := placesRunning(map[string]int{"shanks": 2, "ace": 1})
p.Capacity = map[string]int{"shanks": 3}
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "novox" || nodes[1] != "shanks" {
t.Errorf("nodes = %v, err %v, want [novox shanks]: the nodes with room, ace at capacity left out", nodes, err)
}
}
+5
View File
@@ -0,0 +1,5 @@
// Package buildrun holds the parts every module that runs steps on the build modules shares: the caller, the steps
// watch and the runner of a long step (ADR 0469), the sets of presence, toolchains and healthy build modules, the
// places a node offers by capacity, and the choice of build nodes (ADR 0478). The builder and the checker of the
// delivery use them; they moved here from the builder's own package so neither keeps a copy (issue 1671, ADR 0484).
package buildrun
+56
View File
@@ -0,0 +1,56 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HaveSet holds what each node's node-engine says of each module on it (node-engine_have, key `<node>.<module>`): the
// builder reads it to know whether a node's build module is running and healthy, since a build module whose installed
// toolchain is not the one its recipe builds with is unhealthy (mesh-build-go build_go_health) and its steps would
// refuse (ADR 0478 clause 3). A module the node-engine has written no key of is not healthy. Safe for use from the watch
// and from a build at once.
type HaveSet struct {
mu sync.Mutex
healthy map[string]bool // by `<node>.<module>`
}
// Put takes a change of the have bucket in and answers whether a module's healthy state changed. A node's own key
// (`<node>.node`) and a value that is not a have are left out.
func (s *HaveSet) Put(key string, value []byte, deleted bool) bool {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
return false
}
ok := false
if !deleted {
h, err := delivery.DecodeHave(value)
if err != nil {
return false
}
ok = h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
k := node + "." + module
s.mu.Lock()
defer s.mu.Unlock()
if s.healthy[k] == ok {
return false
}
if s.healthy == nil {
s.healthy = map[string]bool{}
}
if ok {
s.healthy[k] = true
} else {
delete(s.healthy, k)
}
return true
}
// Healthy is whether the module on the node was last seen running and healthy.
func (s *HaveSet) Healthy(node, module string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.healthy[node+"."+module]
}
+7
View File
@@ -0,0 +1,7 @@
package buildrun
// The build modules the mesh runs, by the language each builds (ADR 0469).
const (
GoBuildModule = "mesh-build-go"
NodeBuildModule = "mesh-build-node"
)
+171
View File
@@ -0,0 +1,171 @@
package buildrun
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"sync"
)
// DefaultCapacity is the build capacity of a node that has no setting: one build step at a time.
const DefaultCapacity = 1
// ParseCapacity reads a capacity setting: `node=steps` pairs separated by commas, such as
// `shanks=3,ace=2`. A node without a pair has DefaultCapacity. A pair with no node, a count that is not a whole number of
// at least 0, and a node named twice are refused. A count of 0 keeps the part off the node: it is never chosen.
func ParseCapacity(s string) (map[string]int, error) {
out := map[string]int{}
for _, pair := range strings.Split(s, ",") {
pair = strings.TrimSpace(pair)
if pair == "" {
continue
}
node, count, ok := strings.Cut(pair, "=")
node, count = strings.TrimSpace(node), strings.TrimSpace(count)
n, err := strconv.Atoi(count)
if !ok || node == "" || err != nil || n < 0 {
return nil, fmt.Errorf("the capacity %q is not <node>=<steps at once, 0 or more>", pair)
}
if _, dup := out[node]; dup {
return nil, fmt.Errorf("the capacity names %s twice", node)
}
out[node] = n
}
return out, nil
}
// Places counts the build steps a part runs on each node and holds a lane back while its node is at its
// capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build
// module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by
// reserving a place on it at the moment of the choice (ChooseNodes), so that builds choosing at once see each other's
// choices and none waits on a full node while another has room. A lane waits for its place by watching the ends of the
// other lanes, not by a timer: ending a lane wakes the waiting ones.
type Places struct {
// Capacity is the number of steps at once by node; a node not in it has DefaultCapacity.
Capacity map[string]int
mu sync.Mutex
running map[string]int // lanes holding a place
reserved map[string]int // lanes chosen for the node that do not hold a place yet
changed chan struct{} // closed, and replaced, each time a lane ends
}
// CapacityOf is how many steps the part runs on the node at once: DefaultCapacity for a node the setting does not
// name, and 0 for one it names so, where the part never builds.
func (p *Places) CapacityOf(node string) int {
if n, ok := p.Capacity[node]; ok && n >= 0 {
return n
}
return DefaultCapacity
}
// Running is the steps in flight now, by node.
func (p *Places) Running() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.running))
for n, c := range p.running {
if c > 0 {
out[n] = c
}
}
return out
}
// Reserved is the lanes chosen for a node that hold no place yet, by node.
func (p *Places) Reserved() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.reserved))
for n, c := range p.reserved {
if c > 0 {
out[n] = c
}
}
return out
}
func (p *Places) init() {
if p.running == nil {
p.running = map[string]int{}
}
if p.reserved == nil {
p.reserved = map[string]int{}
}
if p.changed == nil {
p.changed = make(chan struct{})
}
}
// Acquire takes a place on the node, waiting while the node is at its capacity, and answers the function that gives it
// back. A place reserved for the node by a choice is the one taken. It returns an error when ctx ends first, and then
// gives up the reservation too.
func (p *Places) Acquire(ctx context.Context, node string) (release func(), err error) {
for {
p.mu.Lock()
p.init()
if p.running[node] < p.CapacityOf(node) {
p.running[node]++
if p.reserved[node] > 0 {
p.reserved[node]--
}
p.mu.Unlock()
var once sync.Once
return func() {
once.Do(func() {
p.mu.Lock()
p.running[node]--
close(p.changed)
p.changed = make(chan struct{})
p.mu.Unlock()
})
}, nil
}
wait := p.changed
p.mu.Unlock()
select {
case <-wait:
case <-ctx.Done():
p.Cancel(node)
return nil, fmt.Errorf("waiting for a place on %s: %w", node, ctx.Err())
}
}
}
// Cancel gives up a place reserved for the node that will not be taken.
func (p *Places) Cancel(node string) {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
if p.reserved[node] > 0 {
p.reserved[node]--
}
}
// choose orders the nodes (those with room first, counting the lanes running and the lanes reserved, then the fewest of
// them, then by name), keeps the first want and reserves a place on each, all at once.
func (p *Places) choose(nodes []string, want int) []string {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
load := func(n string) int { return p.running[n] + p.reserved[n] }
sort.SliceStable(nodes, func(i, j int) bool {
ri, rj := load(nodes[i]) < p.CapacityOf(nodes[i]), load(nodes[j]) < p.CapacityOf(nodes[j])
if ri != rj {
return ri
}
if load(nodes[i]) != load(nodes[j]) {
return load(nodes[i]) < load(nodes[j])
}
return nodes[i] < nodes[j]
})
if len(nodes) > want {
nodes = nodes[:want]
}
for _, n := range nodes {
p.reserved[n]++
}
return nodes
}
+126
View File
@@ -0,0 +1,126 @@
package buildrun
import (
"context"
"strings"
"testing"
"time"
)
func TestParseCapacityReadsNodeEqualsCountPairs(t *testing.T) {
c, err := ParseCapacity(" shanks=3, ace=2 ")
if err != nil || c["shanks"] != 3 || c["ace"] != 2 || len(c) != 2 {
t.Errorf("capacity = %v, %v", c, err)
}
if c, err := ParseCapacity(""); err != nil || len(c) != 0 {
t.Errorf("empty: %v, %v", c, err)
}
for _, bad := range []string{"shanks", "shanks=x", "=2", "a=1,a=2", "shanks=-1"} {
if _, err := ParseCapacity(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
func acquireAsync(p *Places, node string) chan func() {
got := make(chan func(), 1)
go func() {
release, err := p.Acquire(context.Background(), node)
if err != nil {
close(got)
return
}
got <- release
}()
return got
}
func TestThreeStepsOnANodeOfCapacityTwoRunTwoAtOnceAndTheThirdStartsWhenOneEnds(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"ace": 2}}
a, b := acquireAsync(p, "ace"), acquireAsync(p, "ace")
ra, rb := <-a, <-b
third := acquireAsync(p, "ace")
select {
case <-third:
t.Fatal("a third step started on a node of capacity 2 with two running")
case <-time.After(10 * time.Millisecond):
}
if p.Running()["ace"] != 2 {
t.Errorf("running = %v, want 2 on ace", p.Running())
}
ra()
select {
case r := <-third:
r()
case <-time.After(2 * time.Second):
t.Fatal("the third step did not start when one ended")
}
rb()
}
func TestANodeWithNoSettingRunsOneStepAtATime(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"shanks": 3}}
first := <-acquireAsync(p, "g14")
second := acquireAsync(p, "g14")
select {
case <-second:
t.Fatal("two steps on a node without a setting")
case <-time.After(10 * time.Millisecond):
}
first()
r := <-second
r()
if p.CapacityOf("g14") != 1 || p.CapacityOf("shanks") != 3 {
t.Errorf("capacities = %d, %d", p.CapacityOf("g14"), p.CapacityOf("shanks"))
}
}
func TestAWaitingStepStopsWhenItsContextEnds(t *testing.T) { // round 3
p := &Places{}
r := <-acquireAsync(p, "ace")
defer r()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { _, err := p.Acquire(ctx, "ace"); done <- err }()
cancel()
select {
case err := <-done:
if err == nil {
t.Error("Acquire returned no error for a cancelled wait")
}
case <-time.After(2 * time.Second):
t.Fatal("a cancelled wait did not return")
}
}
// The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node.
func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661
c, err := ParseCapacity("shanks=3,ace=2,g14=0,novox=0")
if err != nil || c["g14"] != 0 || c["novox"] != 0 || c["shanks"] != 3 || len(c) != 4 {
t.Fatalf("capacity = %v, %v", c, err)
}
p := &Places{Capacity: c}
if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity {
t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other"))
}
if _, err := ParseCapacity("shanks=-1"); err == nil {
t.Error("a negative capacity accepted")
}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other")
for i := 0; i < 6; i++ {
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if n == "g14" || n == "novox" {
t.Fatalf("choice %d: %v includes a node of capacity 0", i, nodes)
}
}
}
// Only nodes of capacity 0 report: nothing qualifies, and the build says why.
_, _, err = ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2)
if err == nil || !strings.Contains(err.Error(), "capacity") {
t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err)
}
}
+53
View File
@@ -0,0 +1,53 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// PresenceSet holds which nodes are online, as the watchman writes it (mesh-delivery-watchman_presence, key `<node>`, the
// node-engine's connection): a node asleep or off the bus is offline and its build module cannot answer. A node the
// watchman has no key of is not online. Safe for use from the watch and from a build at once.
type PresenceSet struct {
mu sync.Mutex
online map[string]bool
}
// Put takes a change of the presence bucket in and answers whether a node's online state changed. A part's key and a
// value that is not a presence are left out.
func (s *PresenceSet) Put(key string, value []byte, deleted bool) bool {
node, part, _, err := delivery.ParsePresenceKey(key)
if err != nil || part != "" {
return false
}
on := false
if !deleted {
p, err := delivery.DecodePresence(value)
if err != nil {
return false
}
on = p.Online
}
s.mu.Lock()
defer s.mu.Unlock()
if s.online[node] == on {
return false
}
if s.online == nil {
s.online = map[string]bool{}
}
if on {
s.online[node] = true
} else {
delete(s.online, node)
}
return true
}
// Online is whether the watchman last saw the node online.
func (s *PresenceSet) Online(node string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.online[node]
}
+43
View File
@@ -0,0 +1,43 @@
package buildrun
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
func TestPresenceSetKnowsOnlyNodesTheWatchmanSawOnline(t *testing.T) { // 1607 review 3
var s PresenceSet
if s.Online("ace") {
t.Error("a node the watchman never wrote is online")
}
if !s.Put("ace", presenceValue(t, true), false) || !s.Online("ace") {
t.Error("ace online was not taken in")
}
if s.Put("ace", presenceValue(t, true), false) {
t.Error("the same presence again counted as a change")
}
if !s.Put("ace", presenceValue(t, false), false) || s.Online("ace") {
t.Error("ace going offline was not taken in")
}
s.Put("g14", presenceValue(t, true), false)
if !s.Put("g14", nil, true) || s.Online("g14") {
t.Error("a deleted key is still online")
}
// A part's instance (part.<part>.<instance>) is not a node.
if s.Put("part.mesh-delivery-builder.novox", presenceValue(t, true), false) || s.Online("part") {
t.Error("a part's key was taken for a node")
}
if s.Put("novox", []byte("junk"), false) {
t.Error("junk counted")
}
}
+67
View File
@@ -0,0 +1,67 @@
package buildrun
import (
"context"
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Caller calls a tool of a module on one named machine (mesh-sdk process.Conn.CallOn; ADR 0469 clause 3).
type Caller interface {
CallOn(node, key string, body any) (json.RawMessage, error)
}
// Steps waits for the end of a build module's long step: the value of its key `<node>.<id>` in the module's steps state
// (ADR 0469 clause 2) once its state is not running. It is a watch of that key, not a poll; it returns when the step
// ends or ctx does.
type Steps interface {
Wait(ctx context.Context, module, key string) (delivery.Step, error)
}
// Runner runs a build module's long steps: the call answers at once with the step's key, and the result is read from that
// key when the step ends (ADR 0469 clause 2).
type Runner struct {
Caller Caller
Steps Steps
// Bound is how long a step may run before the builder stops waiting for it.
Bound time.Duration
}
// Run calls tool of the build module on node with args, waits for the step's end and answers its result. A step that
// failed or was refused is an error naming the node, the state and the reason.
func (r Runner) Run(ctx context.Context, module, node, tool string, args map[string]any) (json.RawMessage, error) {
raw, err := r.Caller.CallOn(node, module+"."+tool, args)
if err != nil {
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
var answer struct {
Step string `json:"step"`
Key string `json:"key"`
State string `json:"state"`
}
if err := json.Unmarshal(raw, &answer); err != nil {
return nil, fmt.Errorf("%s on %s answered what is not a step: %w", tool, node, err)
}
if answer.Key == "" {
return nil, fmt.Errorf("%s on %s answered no step key: %s", tool, node, raw)
}
if n, _, err := delivery.ParseStepKey(answer.Key); err != nil || n != node {
return nil, fmt.Errorf("%s on %s answered the step key %q, which is not a key of that node", tool, node, answer.Key)
}
wait, cancel := context.WithTimeoutCause(ctx, r.Bound, fmt.Errorf("the step %s did not end within %s", answer.Key, r.Bound))
defer cancel()
step, err := r.Steps.Wait(wait, module, answer.Key)
if err != nil {
if cause := context.Cause(wait); cause != nil && ctx.Err() == nil {
err = cause
}
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
if step.State != delivery.StepPassed {
return nil, fmt.Errorf("%s on %s: the step %s %s: %s", tool, node, answer.Key, step.State, step.Reason)
}
return step.Result, nil
}
+116
View File
@@ -0,0 +1,116 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
func TestRunCallsTheToolOnTheNodeAndReadsTheStepsKey(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", delivery.StepPassed, "", `{"workspace":"w1","commit":"c"}`)}}
res, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", map[string]any{"repository": "novox/r", "commit": "c"})
if err != nil {
t.Fatal(err)
}
if len(caller.calls) != 1 || caller.calls[0].node != "ace" || caller.calls[0].key != "mesh-build-go.build_go_fetch" {
t.Errorf("calls = %+v, want one on ace for mesh-build-go.build_go_fetch", caller.calls)
}
var fr struct{ Workspace string }
if err := json.Unmarshal(res, &fr); err != nil || fr.Workspace != "w1" {
t.Errorf("result = %s (%v), want the step's result", res, err)
}
}
func TestRunFailsOnAFailedOrRefusedStepNamingItsReason(t *testing.T) { // 1607/AC5
for _, state := range []string{delivery.StepFailed, delivery.StepRefused} {
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", state, "go is go1.26.0, the recipe builds with go1.27.1", "")}}
_, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), state) || !strings.Contains(err.Error(), "ace") {
t.Errorf("%s: error = %v, want the node, the state and the reason", state, err)
}
}
}
func TestRunStopsWaitingAtTheBound(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
_, err := Runner{Caller: caller, Steps: &fakeSteps{}, Bound: 20 * time.Millisecond}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "ace.abc1") || !strings.Contains(err.Error(), "did not end") {
t.Errorf("error = %v, want the step key and that it did not end within its bound", err)
}
}
func TestRunRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode(t *testing.T) { // 1607/AC5
for _, answer := range []string{`{"state":"running"}`, `{"step":"abc1","key":"novox.abc1","state":"running"}`, `not json`} {
_, err := Runner{Caller: &fakeCaller{answer: answer}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if err == nil {
t.Errorf("answer %s: no error", answer)
}
}
}
func TestRunReturnsTheCallsOwnError(t *testing.T) { // 1607/AC5
boom := errors.New("no responders")
_, err := Runner{Caller: &fakeCaller{err: boom}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if !errors.Is(err, boom) || !strings.Contains(err.Error(), "ace") {
t.Errorf("error = %v, want the call's error naming the node", err)
}
}
+64
View File
@@ -0,0 +1,64 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// LanguageOf is the language each build module's toolchains state holds (the indexer's ToolchainModules, reversed).
var LanguageOf = map[string]string{GoBuildModule: "go", NodeBuildModule: "typescript"}
// ToolchainSet holds what the build modules' toolchains buckets hold: each node's entry of a language, as the watches
// put them in. Safe for use from the watches and from a build at once.
type ToolchainSet struct {
mu sync.Mutex
by map[string]map[string]delivery.Toolchain // language -> node -> entry
}
// Put takes a change of a build module's toolchains bucket in: a put of a node's entry (key <node>.<language>) or its
// delete. It answers whether the language's entries changed (a node's version or recipe, or a node appearing or going).
// A key or a value that is not an entry of the module's language is left out.
func (s *ToolchainSet) Put(module, key string, value []byte, deleted bool) bool {
language, ok := LanguageOf[module]
if !ok {
return false
}
node, keyLanguage, err := delivery.ParseToolchainKey(key)
if err != nil || keyLanguage != language {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if deleted {
if _, had := s.by[language][node]; !had {
return false
}
delete(s.by[language], node)
return true
}
tc, err := delivery.DecodeToolchain(value)
if err != nil || tc.Language != language {
return false
}
if s.by == nil {
s.by = map[string]map[string]delivery.Toolchain{}
}
if s.by[language] == nil {
s.by[language] = map[string]delivery.Toolchain{}
}
cur, had := s.by[language][node]
s.by[language][node] = tc
return !had || cur.Version != tc.Version || cur.Recipe != tc.Recipe
}
// Entries are the nodes' entries of a language, in no particular order.
func (s *ToolchainSet) Entries(language string) []delivery.NodeToolchain {
s.mu.Lock()
defer s.mu.Unlock()
var out []delivery.NodeToolchain
for node, tc := range s.by[language] {
out = append(out, delivery.NodeToolchain{Node: node, Toolchain: tc})
}
return out
}
+51
View File
@@ -0,0 +1,51 @@
package buildrun
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
func TestToolchainSetKeepsEachNodesEntryByLanguageAndSaysWhenOneChanged(t *testing.T) { // 1607/AC6
var s ToolchainSet
if !s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("a first entry did not count as a change")
}
if s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("the same entry again counted as a change")
}
s.Put(GoBuildModule, "ace.go", tcValue(t, "go", "go1.27.1"), false)
s.Put(NodeBuildModule, "ace.typescript", tcValue(t, "typescript", "tsc5"), false)
if got := s.Entries("go"); len(got) != 2 {
t.Errorf("go entries = %+v, want two", got)
}
if got := s.Entries("typescript"); len(got) != 1 || got[0].Node != "ace" || got[0].Version != "tsc5" {
t.Errorf("typescript entries = %+v", got)
}
if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 {
t.Error("a delete did not remove the node's entry")
}
}
func TestToolchainSetLeavesOutWhatIsNotAToolchainOfTheModulesLanguage(t *testing.T) { // 1607/AC6
var s ToolchainSet
if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) {
t.Error("a typescript toolchain in the go build module's bucket counted")
}
if s.Put(GoBuildModule, "novox.go", []byte("not json"), false) || s.Put("other-module", "novox.go", tcValue(t, "go", "x"), false) {
t.Error("garbage or an unknown module counted")
}
if len(s.Entries("go")) != 0 {
t.Error("an entry was kept")
}
}
+40
View File
@@ -0,0 +1,40 @@
package delivery
import (
"strings"
"testing"
)
// A candidate's module carries the git objects of its named paths at the preview (novox/hq ADR 0499 clause 2), so the
// builder's fetch request for a candidate names them; an older candidate decodes without them, and a path that is not
// clean and relative, or names no object, is refused.
func TestACandidatesModuleCarriesItsNamedPaths(t *testing.T) {
c := sampleCandidate()
paths := map[string]string{"src": strings.Repeat("a", 40), "package.json": strings.Repeat("b", 40)}
c.Pulls[0].Fps = map[string]CandidateModule{"mesh-mail": {FP2: SourcePrefix2 + strings.Repeat("1", 64),
Manifest: strings.Repeat("2", 64), Paths: paths}}
c.FPSet = FingerprintSet(c.Pulls)
data, err := EncodeCandidate(c)
if err != nil {
t.Fatal(err)
}
back, err := DecodeCandidate(data)
if err != nil || back.Pulls[0].Fps["mesh-mail"].Paths["src"] != paths["src"] {
t.Fatalf("decoded %+v, %v", back.Pulls[0].Fps, err)
}
old := sampleCandidate()
old.Pulls[0].Fps = map[string]CandidateModule{"mesh-mail": {FP2: SourcePrefix2 + strings.Repeat("1", 64)}}
old.FPSet = FingerprintSet(old.Pulls)
if data, err := EncodeCandidate(old); err != nil {
t.Fatal(err)
} else if b, err := DecodeCandidate(data); err != nil || b.Pulls[0].Fps["mesh-mail"].Paths != nil {
t.Fatalf("an older candidate: %+v, %v", b, err)
}
for _, bad := range []map[string]string{{"../x": "a"}, {"/src": "a"}, {"": "a"}, {"src": ""}} {
c.Pulls[0].Fps = map[string]CandidateModule{"mesh-mail": {FP2: SourcePrefix2 + strings.Repeat("1", 64), Paths: bad}}
c.FPSet = FingerprintSet(c.Pulls)
if _, err := EncodeCandidate(c); err == nil {
t.Errorf("%v was accepted", bad)
}
}
}
+65 -4
View File
@@ -9,9 +9,11 @@ import (
)
// The checker's keys (novox/hq ADR 0447 clauses 1 to 3, ADR 0444, ADR 0446 clause 5, ADR 0453 clause 4): the verdicts of
// its checks and the passing verdicts it keeps per package and level. The checker writes them from R4c, dark; the
// forge bridge writes the status `mesh/check` from a verdict (unrequired until R4f), the gate reads it from R4f, and
// the view and the watchman read them (a running check carries a pulse).
// its checks and the passing verdicts it keeps per package and level. The checker writes them from R4c, dark, and
// writes the status `mesh/check` it decides as an owed act in its own bucket (CheckerOwedBucket, the gate's owed shape);
// the forge bridge, the only part that calls the forge, performs that act (unrequired until R4f), once per revision of
// the key (ADR 0484 clause 2). The gate reads the status from R4f, and the view and the watchman read the verdicts (a
// running check carries a pulse).
//
// **The verdict's key is the group and the heads, not the fingerprints** (the lead's limit for R4a). Research 063
// section 01 drew `verdicts.<g>.check.<fpset>`; ADR 0447 clause 1 judges a pull request's head merged onto its
@@ -88,6 +90,25 @@ func ParseCheckerVerdictKey(key string) (group, headsHash string, err error) {
type LevelResult struct {
Run int `json:"run"`
Kept int `json:"kept"`
// Skipped is the tests the run skipped, so a pass never hides them (issue 1251, ADR 0484 clause 7).
Skipped int `json:"skipped,omitempty"`
}
// The states of a Composition (ADR 0484 clause 6): the composition check is the assigner's, so until it delivers a
// verdict says CompositionNotRun, and a line never says the composition passed on that.
const (
CompositionNotRun = "not-run"
CompositionPass = "pass"
CompositionFail = "fail"
)
// Composition is what the check of the machines' composition made of a verdict: its State (not-run, pass or fail), the
// Words of a refusal, the mesh-sdk commit it composed with and the machines composed (ADR 0482 clause 9, ADR 0484 clause 6).
type Composition struct {
State string `json:"state"`
Words string `json:"words,omitempty"`
SDK string `json:"sdk,omitempty"`
Machines []string `json:"machines,omitempty"`
}
// Took is the time of each part of a check in integer milliseconds (ADR 0447 clause 3): waiting in the queue, cloning,
@@ -123,6 +144,33 @@ type Verdict struct {
Took Took `json:"took,omitzero"`
Machines []string `json:"machines,omitempty"`
SDK string `json:"sdk,omitempty"`
// Composition is the composition check's result; absent in a value written before it (ADR 0484 clause 6).
Composition *Composition `json:"composition,omitempty"`
// OverBound names the levels whose running went over their bound; it does not change the State (ADR 0453 clause 9).
OverBound []string `json:"over-bound,omitempty"`
}
func (c *Composition) check() error {
if c == nil {
return nil
}
switch c.State {
case CompositionNotRun:
if c.Words != "" || c.SDK != "" || len(c.Machines) > 0 {
return fmt.Errorf("delivery: the verdict's composition is not run and says nothing of words, the mesh-sdk commit or machines")
}
case CompositionPass:
if c.SDK == "" {
return fmt.Errorf("delivery: the verdict's composition passed and names no mesh-sdk commit")
}
case CompositionFail:
if c.Words == "" {
return fmt.Errorf("delivery: the verdict's composition failed and gives no words")
}
default:
return fmt.Errorf("delivery: the verdict's composition state %q is not %s, %s or %s", c.State, CompositionNotRun, CompositionPass, CompositionFail)
}
return nil
}
func (v Verdict) check() error {
@@ -141,10 +189,23 @@ func (v Verdict) check() error {
if err := checkLevel(level); err != nil {
return err
}
if r.Run < 0 || r.Kept < 0 {
if r.Run < 0 || r.Kept < 0 || r.Skipped < 0 {
return fmt.Errorf("delivery: the verdict's %s level counts a negative number of tests", level)
}
}
if err := v.Composition.check(); err != nil {
return err
}
seen := map[string]bool{}
for _, level := range v.OverBound {
if err := checkLevel(level); err != nil {
return fmt.Errorf("delivery: the verdict's over-bound names: %w", err)
}
if seen[level] {
return fmt.Errorf("delivery: the verdict's over-bound names the level %s twice", level)
}
seen[level] = true
}
for part, ms := range map[string]int64{"queue": v.Took.Queue, "clone": v.Took.Clone, "unit": v.Took.Unit,
"integration-setup": v.Took.IntegrationSetup, "integration-run": v.Took.IntegrationRun, "composition": v.Took.Composition} {
if ms < 0 {
+370
View File
@@ -1,6 +1,7 @@
package delivery
import (
"encoding/json"
"errors"
"strings"
"testing"
@@ -220,3 +221,372 @@ func TestCheckerVersions(t *testing.T) {
}
}
}
// oldVerdictJSON is a verdict as the SDK before issue 1672 wrote it (copied from EncodeVerdict of passVerdict's shape
// at the commit before this change): no skipped count, no composition, no over-bound names.
const oldVerdictJSON = `{"v":1,"group":"feat/x","heads":"4b5491557853","trunk":{"novox/mesh-control":"bbbb","novox/mesh-sdk":"aaaa"},"state":"pass","levels":{"integration":{"run":3,"kept":40},"unit":{"run":12,"kept":301}},"took":{"queue":1200,"clone":800,"unit":950,"integration-setup":2100,"integration-run":7400,"composition":3300},"machines":["ace","g14","novox","shanks"],"sdk":"v0.1.14-0.20261019091011-aa68b67389c9"}`
// oldVerdict is the reader of today's shape: the struct as it stood before the new fields.
type oldVerdict struct {
V int `json:"v"`
Group string `json:"group"`
Heads string `json:"heads"`
State string `json:"state"`
Levels map[string]struct{ Run, Kept int }
Machines []string
SDK string
}
// 1672/AC1: the skipped count, the composition and the over-bound names round trip and appear in the bytes.
func TestVerdictCarriesSkippedCompositionAndOverBound(t *testing.T) {
v := passVerdict()
v.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4}
v.Composition = &Composition{State: CompositionPass, SDK: v.SDK, Machines: []string{"ace", "novox"}}
v.OverBound = []string{LevelIntegration}
data, err := EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
for _, field := range []string{`"skipped":4`, `"composition":{"state":"pass"`, `"over-bound":["integration"]`} {
if !strings.Contains(string(data), field) {
t.Errorf("the value lacks %s: %s", field, data)
}
}
got, err := DecodeVerdict(data)
v.V = Version
if err != nil || !SameVerdict(got, v) || got.Levels[LevelUnit].Skipped != 4 || got.Composition == nil || got.Composition.State != CompositionPass {
t.Errorf("round trip = %+v, %v", got, err)
}
notRun := passVerdict()
notRun.Composition = &Composition{State: CompositionNotRun}
if _, err := EncodeVerdict(notRun); err != nil {
t.Errorf("a composition not yet run refused: %v", err)
}
failed := passVerdict()
failed.State, failed.Failed = VerdictFail, FailedFirst{Machine: "ace", Words: "x"}
failed.Composition = &Composition{State: CompositionFail, Words: "path /etc/x is owned by two modules", SDK: failed.SDK, Machines: []string{"ace"}}
if _, err := EncodeVerdict(failed); err != nil {
t.Errorf("a failed composition refused: %v", err)
}
}
// 1672/AC1: an invalid new value is refused, naming the field.
func TestVerdictNewFieldsChecked(t *testing.T) {
cases := map[string]struct {
edit func(*Verdict)
words []string
}{
"a negative skipped count": {func(v *Verdict) { v.Levels[LevelUnit] = LevelResult{Run: 1, Skipped: -1} }, []string{"unit", "negative"}},
"an unknown composition": {func(v *Verdict) { v.Composition = &Composition{State: "maybe"} }, []string{"composition", "maybe"}},
"a composition with no state": {func(v *Verdict) { v.Composition = &Composition{} }, []string{"composition"}},
"a failed composition without words": {func(v *Verdict) { v.Composition = &Composition{State: CompositionFail} }, []string{"composition", "words"}},
"a passed composition without the sdk": {func(v *Verdict) { v.Composition = &Composition{State: CompositionPass, Machines: []string{"ace"}} }, []string{"composition", "sdk"}},
"a not-run composition naming machines": {func(v *Verdict) { v.Composition = &Composition{State: CompositionNotRun, Machines: []string{"ace"}} }, []string{"composition", "not run"}},
"an unknown over-bound level": {func(v *Verdict) { v.OverBound = []string{"e2e"} }, []string{"e2e"}},
"an over-bound level named twice": {func(v *Verdict) { v.OverBound = []string{LevelUnit, LevelUnit} }, []string{"over-bound", "unit"}},
}
for name, c := range cases {
v := passVerdict()
c.edit(&v)
_, err := EncodeVerdict(v)
if err == nil {
t.Errorf("%s accepted", name)
continue
}
for _, w := range c.words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
}
// 1672/AC2: a value written by the SDK before this change decodes with the new reader, with the new fields empty, and
// the new SDK writes the same bytes for it; a value with the new fields decodes under a reader of today's shape.
func TestVerdictOldAndNewReadersAgree(t *testing.T) {
v, err := DecodeVerdict([]byte(oldVerdictJSON))
if err != nil {
t.Fatalf("the new reader of an old value: %v", err)
}
if v.Composition != nil || v.OverBound != nil || v.Levels[LevelUnit].Skipped != 0 || v.Levels[LevelUnit].Run != 12 {
t.Errorf("an old value grew fields: %+v", v)
}
again, err := EncodeVerdict(v)
if err != nil || string(again) != oldVerdictJSON {
t.Errorf("the new SDK writes %s, %v for an old value, want today's bytes", again, err)
}
n := passVerdict()
n.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4}
n.Composition = &Composition{State: CompositionNotRun}
n.OverBound = []string{LevelUnit}
data, err := EncodeVerdict(n)
if err != nil {
t.Fatal(err)
}
var ov oldVerdict
if err := json.Unmarshal(data, &ov); err != nil || ov.Group != "feat/x" || ov.State != VerdictPass || ov.Levels[LevelUnit].Run != 12 || ov.SDK == "" {
t.Errorf("the old reader of a new value: %+v, %v", ov, err)
}
}
func sampleCheckerOwed() Owed {
return Owed{Act: OwedStatus, Context: StatusContextCheck, State: "success", Target: "https://mesh.example/delivery/check/36",
Text: "pass 4b5491557853 on aaaa: unit 12 run 301 kept, integration 3 run 40 kept; composition: not yet run"}
}
// 1673/AC1: the checker's owed bucket is the checker module plus _owed, and its key is the gate's owed key with the act
// status=mesh/check.
func TestCheckerOwedKeyAndBucket(t *testing.T) {
if CheckerOwedBucket != "mesh-delivery-checker_owed" || CheckerOwedHistory != 5 || PreviewsBucket != "mesh-delivery-checker_previews" || PreviewsHistory != 5 {
t.Errorf("buckets %s %d %s %d", CheckerOwedBucket, CheckerOwedHistory, PreviewsBucket, PreviewsHistory)
}
key, err := CheckerOwedKey("novox/mesh-sdk", 36, "aaaa")
if err != nil || key != "novox/mesh-sdk.36.aaaa.status=mesh/check" {
t.Fatalf("key = %q, %v", key, err)
}
repo, number, head, err := ParseCheckerOwedKey(key)
if err != nil || repo != "novox/mesh-sdk" || number != 36 || head != "aaaa" {
t.Errorf("parse = %q %d %q %v", repo, number, head, err)
}
other, _ := OwedKey("novox/mesh-sdk", 36, "aaaa", "status=mesh/merge-gate")
for _, bad := range []string{other, "novox/mesh-sdk.36.aaaa.view", "novox/mesh-sdk.36.aaaa.note", "novox/mesh-sdk.36.aaaa.comment.1", ""} {
if _, _, _, err := ParseCheckerOwedKey(bad); err == nil {
t.Errorf("checker owed key %q accepted", bad)
}
}
if _, err := CheckerOwedKey("nope", 36, "aaaa"); err == nil {
t.Error("a bad repository accepted")
}
}
// 1673/AC2: an owed value with the context mesh/check is accepted and an unknown context is refused, in the act and in the value.
func TestOwedAcceptsMeshCheckAndRefusesAnUnknownContext(t *testing.T) {
for _, ctx := range []string{StatusContextCheck, StatusContextMergeGate} {
if kind, got, err := ParseOwedAct("status=" + ctx); err != nil || kind != OwedStatus || got != ctx {
t.Errorf("ParseOwedAct(status=%s) = %q %q %v", ctx, kind, got, err)
}
}
for _, bad := range []string{"status=mesh/other", "status=check", "status=mesh/Check"} {
if _, err := OwedKey("novox/mesh-sdk", 36, "aaaa", bad); err == nil || !strings.Contains(err.Error(), bad[len("status="):]) {
t.Errorf("owed act %q: %v, want a refusal naming the context", bad, err)
}
}
data, err := EncodeCheckerOwed(sampleCheckerOwed())
if err != nil {
t.Fatal(err)
}
if got, err := DecodeCheckerOwed(data); err != nil || !SameOwed(got, sampleCheckerOwed()) && got.Context != StatusContextCheck {
t.Errorf("round trip = %+v, %v", got, err)
}
unknown := sampleCheckerOwed()
unknown.Context = "mesh/other"
if _, err := EncodeOwed(unknown); err == nil || !strings.Contains(err.Error(), "mesh/other") {
t.Errorf("an owed status of the context mesh/other: %v", err)
}
// The checker's bucket holds only its own status.
gate := sampleCheckerOwed()
gate.Context = StatusContextMergeGate
for name, o := range map[string]Owed{"the gate's context": gate, "a view": {Act: OwedView, Text: "x"}, "a note": {Act: OwedNote, Text: "x", Ref: "mesh-plan"}} {
if _, err := EncodeCheckerOwed(o); err == nil {
t.Errorf("the checker's bucket takes %s", name)
}
}
for _, state := range []string{"pending", "success", "failure"} {
o := sampleCheckerOwed()
o.State = state
if _, err := EncodeCheckerOwed(o); err != nil {
t.Errorf("state %s: %v", state, err)
}
}
}
func samplePreview() Preview {
return Preview{Repository: "novox/mesh-delivery", Head: "aaaa", Trunk: "bbbb", Merge: "cccc",
Modules: map[string]PreviewModule{
"modules/mesh-delivery-checker": {Tree: "t1", Manifest: strings.Repeat("1", 64), Locks: map[string]string{"go.sum": strings.Repeat("2", 64)}},
"modules/mesh-delivery-view": {Tree: "t2", Manifest: strings.Repeat("3", 64)},
}}
}
// 1673/AC3: the previews key is <repository>.<head>.<trunk> and parses back.
func TestPreviewKey(t *testing.T) {
key, err := PreviewKey("novox/mesh-delivery", "aaaa", "bbbb")
if err != nil || key != "novox/mesh-delivery.aaaa.bbbb" {
t.Fatalf("key = %q, %v", key, err)
}
repo, head, trunk, err := ParsePreviewKey(key)
if err != nil || repo != "novox/mesh-delivery" || head != "aaaa" || trunk != "bbbb" {
t.Errorf("parse = %q %q %q %v", repo, head, trunk, err)
}
for _, bad := range [][3]string{{"nope", "aaaa", "bbbb"}, {"novox/x", "", "bbbb"}, {"novox/x", "aaaa", ""}, {"novox/x", "a.b", "bbbb"}, {"novox/x", "aaaa", "b b"}} {
if _, err := PreviewKey(bad[0], bad[1], bad[2]); err == nil {
t.Errorf("PreviewKey%v accepted", bad)
}
}
for _, bad := range []string{"", "novox/x.aaaa", "novox/x.aaaa.bbbb.cccc", "nope.aaaa.bbbb", "novox/x..bbbb", "novox/x.aaaa."} {
if _, _, _, err := ParsePreviewKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") {
t.Errorf("ParsePreviewKey(%q) = %v, want a refusal naming it", bad, err)
}
}
}
// 1673/AC4: a preview round trips, and an invalid one is refused naming the field.
func TestPreviewRoundTripAndRefusals(t *testing.T) {
data, err := EncodePreview(samplePreview())
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(string(data), `{"v":1,`) {
t.Errorf("at %s", data)
}
got, err := DecodePreview(data)
want := samplePreview()
want.V = Version
if err != nil || !SamePreview(got, want) {
t.Errorf("round trip = %+v, %v", got, err)
}
if _, err := EncodePreview(Preview{Repository: "novox/x", Head: "aaaa", Trunk: "bbbb", Merge: "cccc"}); err != nil {
t.Errorf("a preview of a head that changes no module refused: %v", err)
}
cases := map[string]struct {
edit func(*Preview)
words []string
}{
"no repository": {func(p *Preview) { p.Repository = "" }, []string{"repository"}},
"a bad repository": {func(p *Preview) { p.Repository = "nope" }, []string{"nope"}},
"no head": {func(p *Preview) { p.Head = "" }, []string{"head"}},
"no trunk": {func(p *Preview) { p.Trunk = "" }, []string{"trunk"}},
"no merge": {func(p *Preview) { p.Merge = "" }, []string{"merge"}},
"a module with no tree": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Manifest: strings.Repeat("3", 64)}
}, []string{"modules/mesh-delivery-view", "tree"}},
"a bad manifest digest": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Tree: "t2", Manifest: "xyz"}
}, []string{"modules/mesh-delivery-view", "xyz"}},
"an empty directory": {func(p *Preview) { p.Modules[""] = PreviewModule{Tree: "t", Manifest: strings.Repeat("3", 64)} }, []string{"directory"}},
"a lock with no hash": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Tree: "t2", Manifest: strings.Repeat("3", 64), Locks: map[string]string{"go.sum": ""}}
}, []string{"modules/mesh-delivery-view", "go.sum"}},
}
for name, c := range cases {
p := samplePreview()
c.edit(&p)
_, err := EncodePreview(p)
if err == nil {
t.Errorf("%s accepted", name)
continue
}
for _, w := range c.words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
if _, err := DecodePreview([]byte(`{"v":2,"repository":"novox/x"}`)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("a preview of version 2: %v", err)
}
if _, err := DecodePreview([]byte(`{"v":1,"repository":"novox/x","head":"a","trunk":"b"}`)); err == nil {
t.Error("decoded a preview with no merge")
}
}
// 1673/AC5: a reader of today's checker shapes ignores a field it does not know, and the checker's owed value is the
// gate's shape, so the gate's reader reads it (as indexer2_test.go does for the indexer's values).
func TestCheckerOwedIsReadableByTheGatesReaderAndAnOldOwedReaderIgnoresNewFields(t *testing.T) {
data, err := EncodeCheckerOwed(sampleCheckerOwed())
if err != nil {
t.Fatal(err)
}
var old struct {
V int `json:"v"`
Act string `json:"act"`
Context string `json:"context"`
State string `json:"state"`
}
if err := json.Unmarshal(data, &old); err != nil || old.Act != OwedStatus || old.Context != "mesh/check" {
t.Errorf("the old reader: %+v %v", old, err)
}
withNew := strings.Replace(string(data), `{"v":1,`, `{"v":1,"a-field-of-tomorrow":7,`, 1)
if _, err := DecodeOwed([]byte(withNew)); err != nil {
t.Errorf("an owed value with a field of tomorrow: %v", err)
}
p, _ := EncodePreview(samplePreview())
withNew = strings.Replace(string(p), `{"v":1,`, `{"v":1,"a-field-of-tomorrow":7,`, 1)
if _, err := DecodePreview([]byte(withNew)); err != nil {
t.Errorf("a preview with a field of tomorrow: %v", err)
}
}
// 1673/AC2 (review): the gate's recheck owes a pending status=mesh/repo-check (mesh-catalog mesh-delivery-gate owed.go,
// checks.go ctxRepo); its key and value round trip, and the checker's bucket still refuses it.
func TestTheGatesRecheckStatusRoundTripsAndTheCheckersBucketRefusesIt(t *testing.T) {
key, err := OwedKey("novox/mesh-sdk", 36, "aaaa", "status=mesh/repo-check")
if err != nil || key != "novox/mesh-sdk.36.aaaa.status=mesh/repo-check" {
t.Fatalf("key = %q, %v", key, err)
}
if _, _, _, act, err := ParseOwedKey(key); err != nil || act != "status=mesh/repo-check" {
t.Errorf("parse = %q, %v", act, err)
}
o := Owed{Act: OwedStatus, Context: StatusContextRepoCheck, State: "pending", Target: "https://mesh.example/delivery/36", Text: "checking again"}
data, err := EncodeOwed(o)
if err != nil {
t.Fatal(err)
}
if got, err := DecodeOwed(data); err != nil || !SameOwed(got, o) && got.Context != o.Context {
t.Errorf("round trip = %+v, %v", got, err)
}
if _, err := EncodeCheckerOwed(o); err == nil {
t.Error("the checker's bucket takes the gate's recheck status")
}
if _, _, _, err := ParseCheckerOwedKey(key); err == nil {
t.Error("a checker owed key of the recheck status accepted")
}
}
// 1445/AC5 (layout): a head that does not merge is a preview naming its conflicting files, with no merge commit and no
// modules; a preview with conflicts and a merge or modules is refused; a preview of today's shape (no conflicts)
// decodes with none, and a reader of today's shape reads one with conflicts.
func TestPreviewOfAHeadInConflict(t *testing.T) {
c := Preview{Repository: "novox/mesh-delivery", Head: "aaaa", Trunk: "bbbb", Conflicts: []string{"a.txt", "modules/m/module.json"}}
data, err := EncodePreview(c)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"conflicts":["a.txt","modules/m/module.json"]`) || strings.Contains(string(data), `"merge"`) {
t.Errorf("at %s", data)
}
got, err := DecodePreview(data)
c.V = Version
if err != nil || !SamePreview(got, c) {
t.Errorf("round trip = %+v, %v", got, err)
}
withMerge := c
withMerge.Merge = "cccc"
withModules := c
withModules.Modules = samplePreview().Modules
emptyPath := c
emptyPath.Conflicts = []string{""}
for name, p := range map[string]Preview{"a merge commit": withMerge, "modules": withModules, "an empty conflicting path": emptyPath} {
if _, err := EncodePreview(p); err == nil || !strings.Contains(err.Error(), "onflict") {
t.Errorf("a preview in conflict with %s: %v", name, err)
}
}
// today's bytes: a clean preview has no conflicts and decodes with none.
clean, _ := EncodePreview(samplePreview())
if strings.Contains(string(clean), "conflicts") {
t.Errorf("a clean preview carries conflicts: %s", clean)
}
if p, err := DecodePreview(clean); err != nil || p.Conflicts != nil {
t.Errorf("a clean preview decodes with conflicts %v, %v", p.Conflicts, err)
}
var old struct {
V int `json:"v"`
Repository string `json:"repository"`
Head string `json:"head"`
Trunk string `json:"trunk"`
Merge string `json:"merge"`
}
if err := json.Unmarshal(data, &old); err != nil || old.Head != "aaaa" || old.Merge != "" {
t.Errorf("the old reader: %+v %v", old, err)
}
}
+28 -1
View File
@@ -28,6 +28,20 @@ const (
AnswersHistory = 5
)
// The contexts of the statuses the mesh owes the forge: the gate's, the delivery's view of a head and of a group
// (design 47), and the checker's. From R4e the checker's mesh/check is the one status a repository's main requires,
// in place of mesh/merge-gate and mesh/repo-check (ADR 0503 clauses 3 and 5); a main that requires only
// mesh/repo-check (hq) keeps it, and the controller writes it for a repository the checker writes no status for.
const (
StatusContextMergeGate = "mesh/merge-gate"
StatusContextDelivery = "mesh/delivery"
StatusContextDeliveryGroup = "mesh/delivery-group"
StatusContextRepoCheck = "mesh/repo-check" // the gate's pending status of a recheck (mesh-delivery.recheck)
StatusContextCheck = "mesh/check"
)
var statusContexts = []string{StatusContextMergeGate, StatusContextDelivery, StatusContextDeliveryGroup, StatusContextRepoCheck, StatusContextCheck}
// AskKindMerge is the kind of the gate's ask for a group's merge, the first part of its key.
const AskKindMerge = "merge"
@@ -131,6 +145,16 @@ func AskingID(askKey string, asking int) string {
return AskID(askKey + "." + strconv.Itoa(asking))
}
// checkStatusContext refuses a status context the mesh does not owe the forge.
func checkStatusContext(context string) error {
for _, c := range statusContexts {
if context == c {
return nil
}
}
return fmt.Errorf("delivery: the status context %q is not one of %s", context, strings.Join(statusContexts, ", "))
}
func checkOwedAct(act string) (kind, context string, err error) {
kind, context, hasContext := strings.Cut(act, "=")
switch kind {
@@ -138,7 +162,7 @@ func checkOwedAct(act string) (kind, context string, err error) {
if !hasContext {
return "", "", fmt.Errorf("delivery: the owed act %q is a status without its context, status=<context>", act)
}
if err := checkName("status context", context); err != nil {
if err := checkStatusContext(context); err != nil {
return "", "", err
}
return kind, context, nil
@@ -345,6 +369,9 @@ func (o Owed) check() error {
if o.Context == "" {
return fmt.Errorf("delivery: an owed status has no context")
}
if err := checkStatusContext(o.Context); err != nil {
return err
}
if o.Target == "" {
return fmt.Errorf("delivery: an owed status has no target")
}
+32
View File
@@ -0,0 +1,32 @@
package delivery
import (
"strings"
"testing"
)
// From R4e a modules value needs no old form (novox/hq ADR 0453 clause 6; issue 1851): a value with only the new
// fingerprint, or only the reason it has none, is written and read; an old value with fp or none still reads; a value
// with no fingerprint of either form and no reason is still refused.
func TestAModulesValueNeedsNoOldForm(t *testing.T) {
fp2 := SourcePrefix2 + strings.Repeat("a", 64)
for _, m := range []Module{
{FP2: fp2, Repository: "a/b", Commit: "c"},
{None2: "no lock", Repository: "a/b", Commit: "c"},
} {
data, err := EncodeModule(m)
if err != nil {
t.Fatalf("%+v was refused: %v", m, err)
}
back, err := DecodeModule(data)
if err != nil || back.FP2 != m.FP2 || back.None2 != m.None2 || back.FP != "" || back.None != "" {
t.Fatalf("decoded %+v, %v", back, err)
}
}
if _, err := DecodeModule([]byte(`{"v":1,"fp":"src1:x","repository":"a/b","commit":"c"}`)); err != nil {
t.Fatalf("an old value no longer reads: %v", err)
}
if _, err := EncodeModule(Module{Repository: "a/b", Commit: "c"}); err == nil {
t.Fatal("a value with no fingerprint and no reason was accepted")
}
}
+33
View File
@@ -0,0 +1,33 @@
package delivery
import (
"strings"
"testing"
)
// The indexer's modules value carries each artifact's named paths (novox/hq ADR 0499; the lead's decision L8 of R4e):
// the checker tells from it whether a change touches lib's paths without reading a manifest. An older value has none.
func TestAModulesValueCarriesEachArtifactsNamedPaths(t *testing.T) {
m := Module{FP2: SourcePrefix2 + strings.Repeat("a", 64), Repository: "novox/mesh-sdk", Commit: "c",
ArtifactPaths: map[string][]string{"lib": {"package-lock.json", "package.json", "src", "test", "tsconfig.json"}}}
data, err := EncodeModule(m)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"paths":{"lib":[`) {
t.Fatalf("the value carries no paths: %s", data)
}
back, err := DecodeModule(data)
if err != nil || len(back.ArtifactPaths["lib"]) != 5 || back.ArtifactPaths["lib"][2] != "src" {
t.Fatalf("decoded %+v, %v", back.ArtifactPaths, err)
}
if old, err := DecodeModule([]byte(`{"v":1,"fp":"src1:x","repository":"a/b","commit":"c"}`)); err != nil || old.ArtifactPaths != nil {
t.Fatalf("an older value: %+v, %v", old, err)
}
for _, bad := range []map[string][]string{{"": {"src"}}, {"lib": {}}, {"lib": {"../x"}}, {"lib": {""}}} {
m.ArtifactPaths = bad
if _, err := EncodeModule(m); err == nil {
t.Errorf("%v was accepted", bad)
}
}
}
+47
View File
@@ -0,0 +1,47 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"path"
"sort"
"strings"
)
// NamedTreePrefix opens the tree input of a module whose every artifact names its paths, so such a value is never read
// as a git tree id and a reader of a kept Tree can tell the two apart.
const NamedTreePrefix = "named:"
// NamedTree is the tree input of the new-form fingerprint of a module whose every artifact names the paths it is built
// from (novox/hq ADR 0499 clause 2): `named:` and the hex sha256 of the lines "manifest <digest>" and, sorted,
// "path <path> <object>", joined by a newline. manifest is the module's ManifestDigest (the sha256 of its module.json's
// exact bytes, as Module.Manifest and the merge preview carry it); paths are the named paths, each clean and relative to
// the module's directory as the manifest names them (catalogue.Manifest.NamedPaths), with the git object id each has at
// the commit (a blob for a file, a tree for a directory).
//
// **One function computes it** (clause 2): the indexer, its merge preview (buildstep.ModuleTrees) and the build modules
// call this one, and no second computation exists. It refuses no path, a named path with no object (absent at the
// commit), a path that is not clean, and a manifest that is not a digest, naming them: the indexer then gives the
// module none, with that reason (clause 3).
func NamedTree(manifest string, paths map[string]string) (string, error) {
if _, err := ManifestsKey(manifest); err != nil {
return "", err
}
if len(paths) == 0 {
return "", fmt.Errorf("delivery: the module names no path, so it has no named tree")
}
lines := make([]string, 0, len(paths))
for p, object := range paths {
if p == "" || p != path.Clean(p) || strings.HasPrefix(p, "/") || p == ".." || strings.HasPrefix(p, "../") {
return "", fmt.Errorf("delivery: the named path %q is not a clean path inside the module's directory", p)
}
if object == "" {
return "", fmt.Errorf("delivery: the named path %q has no git object at the commit", p)
}
lines = append(lines, "path "+p+" "+object)
}
sort.Strings(lines)
sum := sha256.Sum256([]byte("manifest " + manifest + "\n" + strings.Join(lines, "\n")))
return NamedTreePrefix + hex.EncodeToString(sum[:]), nil
}
+79
View File
@@ -0,0 +1,79 @@
package delivery
import (
"strings"
"testing"
)
// The named paths' git objects travel beside a module's other new-form inputs (novox/hq ADR 0499 clauses 2 and 3): in
// the forge bridge's messages (Forge.Paths), in the merge preview (PreviewModule.Paths) and in what the indexer keeps
// (KeptInputs2.Paths), so a computation from kept inputs gives the merge's value. Each is read as absent by an older
// reader, and an older value, without them, still decodes.
func TestTheNamedPathsObjectsRoundTripBesideTheOtherInputs(t *testing.T) {
manifest := sha256Of(`{"module":"mesh-sdk"}`)
paths := map[string]string{"src": strings.Repeat("a", 40), "package.json": strings.Repeat("b", 40)}
tree, err := NamedTree(manifest, paths)
if err != nil {
t.Fatal(err)
}
m := Module{FP2: SourcePrefix2 + strings.Repeat("1", 64), Manifest: manifest, Repository: "novox/mesh-sdk",
Commit: strings.Repeat("c", 40), Inputs2: &KeptInputs2{Tree: tree, Paths: paths}}
m.FP = SourcePrefix + strings.Repeat("0", 64)
data, err := EncodeModule(m)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"paths":{`) {
t.Fatalf("the kept inputs carry no paths: %s", data)
}
back, err := DecodeModule(data)
if err != nil || back.Inputs2.Paths["src"] != paths["src"] || back.Inputs2.Tree != tree {
t.Fatalf("decoded %+v, %v", back.Inputs2, err)
}
// What the indexer keeps must compute to the tree it kept: a mismatch is a value no recompute could give.
bad := m
bad.Inputs2 = &KeptInputs2{Tree: tree, Paths: map[string]string{"src": strings.Repeat("d", 40)}}
if _, err := EncodeModule(bad); err == nil || !strings.Contains(err.Error(), "named tree") {
t.Fatalf("kept paths that do not compute to the kept tree were accepted: %v", err)
}
// An older value has no paths, and its directory tree stays its tree.
old := `{"v":1,"fp":"` + m.FP + `","manifest":"` + manifest + `","inputs2":{"tree":"` + strings.Repeat("e", 40) +
`"},"repository":"novox/x","commit":"` + strings.Repeat("c", 40) + `"}`
if o, err := DecodeModule([]byte(old)); err != nil || o.Inputs2.Paths != nil {
t.Fatalf("an older value: %+v, %v", o, err)
}
f := Forge{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 1, Head: strings.Repeat("h", 40),
Merge: strings.Repeat("m", 40), Root: strings.Repeat("r", 40), Trees: map[string]string{".": strings.Repeat("r", 40)},
Paths: map[string]map[string]string{".": paths}}
data, err = EncodeForge(f)
if err != nil {
t.Fatal(err)
}
if fb, err := DecodeForge(data); err != nil || fb.Paths["."]["package.json"] != paths["package.json"] {
t.Fatalf("decoded forge %+v, %v", fb.Paths, err)
}
f.Paths = map[string]map[string]string{"other": paths}
if _, err := EncodeForge(f); err == nil {
t.Fatal("named paths of a directory the message has no tree of were accepted")
}
f.Paths = map[string]map[string]string{".": {"src": ""}}
if _, err := EncodeForge(f); err == nil {
t.Fatal("a named path with no object was accepted")
}
p := Preview{Repository: "novox/mesh-sdk", Head: strings.Repeat("a", 40), Trunk: strings.Repeat("b", 40),
Merge: strings.Repeat("c", 40), Modules: map[string]PreviewModule{".": {Tree: tree, Manifest: manifest, Paths: paths}}}
data, err = EncodePreview(p)
if err != nil {
t.Fatal(err)
}
if pb, err := DecodePreview(data); err != nil || pb.Modules["."].Paths["src"] != paths["src"] {
t.Fatalf("decoded preview %+v, %v", pb.Modules, err)
}
p.Modules["."] = PreviewModule{Tree: strings.Repeat("e", 40), Manifest: manifest, Paths: paths}
if _, err := EncodePreview(p); err == nil || !strings.Contains(err.Error(), "named tree") {
t.Fatalf("a preview whose tree is not its paths' named tree was accepted: %v", err)
}
}
+68
View File
@@ -0,0 +1,68 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"strings"
"testing"
)
func sha256Of(s string) string { h := sha256.Sum256([]byte(s)); return hex.EncodeToString(h[:]) }
// A module whose every artifact names its paths takes its tree from its manifest and those paths alone (novox/hq ADR
// 0499 clause 2): a change outside them leaves the value equal, a change to a named path or to the manifest changes it,
// and the order the paths are given in changes nothing.
func TestTheNamedTreeIsOfTheManifestAndTheNamedPathsAlone(t *testing.T) {
manifest := sha256Of(`{"module":"mesh-sdk"}`)
paths := map[string]string{"src": strings.Repeat("a", 40), "package.json": strings.Repeat("b", 40)}
v, err := NamedTree(manifest, paths)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(v, NamedTreePrefix) || !isHex64(strings.TrimPrefix(v, NamedTreePrefix)) {
t.Fatalf("the value %q is not %s and 64 hex digits", v, NamedTreePrefix)
}
// A change outside the named paths is no input at all: the same inputs give the same value, in any map order.
again, _ := NamedTree(manifest, map[string]string{"package.json": strings.Repeat("b", 40), "src": strings.Repeat("a", 40)})
if again != v {
t.Fatalf("the same inputs gave %q and %q", v, again)
}
changed, _ := NamedTree(manifest, map[string]string{"src": strings.Repeat("c", 40), "package.json": strings.Repeat("b", 40)})
if changed == v {
t.Fatal("a changed named path left the value equal")
}
otherManifest, _ := NamedTree(sha256Of(`{"module":"mesh-sdk","version":"2"}`), paths)
if otherManifest == v {
t.Fatal("a changed manifest left the value equal")
}
renamed, _ := NamedTree(manifest, map[string]string{"lib": strings.Repeat("a", 40), "package.json": strings.Repeat("b", 40)})
if renamed == v {
t.Fatal("a path named differently with the same object left the value equal")
}
// The lines are fixed, so every computation of it agrees: pinned here.
want := "named:" + sha256Of("manifest "+manifest+"\npath package.json "+strings.Repeat("b", 40)+"\npath src "+strings.Repeat("a", 40))
if v != want {
t.Fatalf("NamedTree = %q, want %q", v, want)
}
}
func TestTheNamedTreeRefusesWhatItCannotPin(t *testing.T) {
manifest := sha256Of("{}")
for _, c := range []struct {
manifest string
paths map[string]string
says string
}{
{manifest, nil, "names no path"},
{manifest, map[string]string{"src": ""}, `the named path "src" has no git object`},
{"abc", map[string]string{"src": "x"}, "not a manifest digest"},
{manifest, map[string]string{"src/../x": "x"}, "is not a clean path"},
{manifest, map[string]string{"": "x"}, "is not a clean path"},
{manifest, map[string]string{"/src": "x"}, "is not a clean path"},
} {
if _, err := NamedTree(c.manifest, c.paths); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("NamedTree(%q, %v) = %v, wanted a refusal saying %q", c.manifest, c.paths, err, c.says)
}
}
}
+187
View File
@@ -0,0 +1,187 @@
package delivery
import (
"encoding/json"
"fmt"
"reflect"
"strings"
)
// The checker's other buckets (ADR 0484 clauses 2 and 3): owed, the acts it owes the forge, which the forge bridge
// performs once per revision of a key, and previews, what the fetch step answered for a head merged onto a trunk commit.
const (
// CheckerOwedBucket is the checker's bucket of owed acts, in the gate's owed shape: a status with the context
// StatusContextCheck for each head it judged. One part writes each key: the checker.
CheckerOwedBucket = CheckerModule + "_owed"
// CheckerOwedHistory is the revisions the checker's owed bucket keeps per key.
CheckerOwedHistory = 5
// PreviewsBucket is the checker's bucket of merge previews; the indexer reads it for the candidates' fingerprints.
PreviewsBucket = CheckerModule + "_previews"
// PreviewsHistory is the revisions the previews bucket keeps per key.
PreviewsHistory = 5
)
// CheckerOwedKey is the key, in CheckerOwedBucket, of the status the checker owes on a pull request at a head: the
// gate's OwedKey with the act `status=mesh/check`, the only act the checker owes.
func CheckerOwedKey(repository string, number int, head string) (string, error) {
return OwedKey(repository, number, head, OwedStatus+"="+StatusContextCheck)
}
// ParseCheckerOwedKey is the inverse of CheckerOwedKey; it refuses an owed key of any other act.
func ParseCheckerOwedKey(key string) (repository string, number int, head string, err error) {
repository, number, head, act, err := ParseOwedKey(key)
if err != nil {
return "", 0, "", err
}
if act != OwedStatus+"="+StatusContextCheck {
return "", 0, "", fmt.Errorf("delivery: %q is not a checker owed key: the checker owes only status=%s, not %q", key, StatusContextCheck, act)
}
return repository, number, head, nil
}
// EncodeCheckerOwed is o as JSON at Version; it refuses what EncodeOwed refuses and any act but a status with the
// context StatusContextCheck.
func EncodeCheckerOwed(o Owed) ([]byte, error) {
if err := checkCheckerOwed(o); err != nil {
return nil, err
}
return EncodeOwed(o)
}
func checkCheckerOwed(o Owed) error {
if o.Act != OwedStatus || o.Context != StatusContextCheck {
return fmt.Errorf("delivery: the checker owes only a status with the context %s, not the act %q with the context %q", StatusContextCheck, o.Act, o.Context)
}
return nil
}
// DecodeCheckerOwed reads an owed act of the checker's bucket.
func DecodeCheckerOwed(data []byte) (Owed, error) {
o, err := DecodeOwed(data)
if err != nil {
return Owed{}, err
}
if err := checkCheckerOwed(o); err != nil {
return Owed{}, err
}
return o, nil
}
// PreviewKey is the key, in PreviewsBucket, of a head merged onto a trunk commit: `<repository>.<head>.<trunk>`.
func PreviewKey(repository, head, trunk string) (string, error) {
repo, err := escapeRepository(repository)
if err != nil {
return "", err
}
if err := checkName("head", head); err != nil {
return "", err
}
if err := checkName("trunk commit", trunk); err != nil {
return "", err
}
return repo + "." + head + "." + trunk, nil
}
// ParsePreviewKey is the inverse of PreviewKey.
func ParsePreviewKey(key string) (repository, head, trunk string, err error) {
parts := strings.Split(key, ".")
if len(parts) != 3 {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key, <repository>.<head>.<trunk>", key)
}
repository, err = unescapeRepository(parts[0])
if err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key: %w", key, err)
}
if _, err := PreviewKey(repository, parts[1], parts[2]); err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key: %w", key, err)
}
return repository, parts[1], parts[2], nil
}
// PreviewModule is what the preview holds of one module directory: the tree sha of the directory (or its named tree), the digest of its
// manifest (the key in the manifests bucket) and the hex sha256 of its lock files by path relative to it.
type PreviewModule struct {
Tree string `json:"tree"`
Manifest string `json:"manifest"`
Locks map[string]string `json:"locks,omitempty"`
// Paths are, of a module whose every artifact names its paths, each named path's git object id at the merge; Tree
// is then their NamedTree (novox/hq ADR 0499 clause 2). Absent otherwise.
Paths map[string]string `json:"paths,omitempty"`
}
// Preview is the value of a preview key: the repository, the head and the trunk commit it was merged onto, the merge
// commit and, for each module directory of the merge, a PreviewModule. A head that does not merge is a preview too: it
// names the files in Conflicts and has no merge commit and no modules (issue 1445).
type Preview struct {
V int `json:"v"`
Repository string `json:"repository"`
Head string `json:"head"`
Trunk string `json:"trunk"`
Merge string `json:"merge,omitempty"`
Conflicts []string `json:"conflicts,omitempty"`
Modules map[string]PreviewModule `json:"modules,omitempty"`
}
func (p Preview) check() error {
if _, err := PreviewKey(p.Repository, p.Head, p.Trunk); err != nil {
return err
}
for _, f := range p.Conflicts {
if f == "" {
return fmt.Errorf("delivery: a preview names a conflicting file with an empty path")
}
}
if len(p.Conflicts) > 0 {
if p.Merge != "" || len(p.Modules) > 0 {
return fmt.Errorf("delivery: a preview of a head in conflict has no merge commit and no modules")
}
return nil
}
if p.Merge == "" {
return fmt.Errorf("delivery: a preview names no merge commit")
}
for dir, m := range p.Modules {
if dir == "" {
return fmt.Errorf("delivery: a preview names a module with an empty directory")
}
if m.Tree == "" {
return fmt.Errorf("delivery: the preview's module %q has no tree", dir)
}
if _, err := ManifestsKey(m.Manifest); err != nil {
return fmt.Errorf("delivery: the preview's module %q: %w", dir, err)
}
for path, hash := range m.Locks {
if path == "" || hash == "" {
return fmt.Errorf("delivery: the lock %q of the preview's module %q has an empty path or hash", path, dir)
}
}
if err := checkNamed(m.Tree, m.Manifest, m.Paths, fmt.Sprintf("the preview's module %q", dir)); err != nil {
return err
}
}
return nil
}
// EncodePreview is p as JSON at Version.
func EncodePreview(p Preview) ([]byte, error) {
if err := p.check(); err != nil {
return nil, err
}
p.V = Version
return json.Marshal(p)
}
// DecodePreview reads a preview.
func DecodePreview(data []byte) (Preview, error) {
var p Preview
if err := decode(data, &p); err != nil {
return Preview{}, err
}
if err := p.check(); err != nil {
return Preview{}, err
}
return p, nil
}
// SamePreview says whether two previews are the same value.
func SamePreview(a, b Preview) bool { return reflect.DeepEqual(a, b) }
+86 -11
View File
@@ -5,6 +5,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"path"
"reflect"
"sort"
"strings"
@@ -276,6 +277,9 @@ type CandidateModule struct {
FP2 string `json:"fp2,omitempty"`
None2 string `json:"none2,omitempty"`
Manifest string `json:"manifest,omitempty"`
// Paths are, of a module whose every artifact names its paths, each named path's git object id at the preview
// (novox/hq ADR 0499 clause 2), so the builder's fetch request for the candidate names them. Absent otherwise.
Paths map[string]string `json:"paths,omitempty"`
}
func (m CandidateModule) check() error {
@@ -285,6 +289,9 @@ func (m CandidateModule) check() error {
case m.FP2 == "" && m.None2 == "":
return fmt.Errorf("delivery: a module has neither fp2 nor none2")
}
if err := checkPathObjects(m.Paths); err != nil {
return err
}
return checkNewForm(m.FP2, m.Manifest)
}
@@ -356,7 +363,7 @@ func (c Candidate) check() error {
if err := m.check(); err != nil {
return fmt.Errorf("delivery: the candidate's module %q: %w", module, err)
}
if before, ok := seen[module]; ok && before != m {
if before, ok := seen[module]; ok && !reflect.DeepEqual(before, m) {
return fmt.Errorf("delivery: two pull requests of the candidate give the module %q different fingerprints", module)
}
seen[module] = m
@@ -370,12 +377,13 @@ func (c Candidate) check() error {
// Module is the value of a modules key: the module's source fingerprint, or none and why (a build that resolves
// packages when built gets none, said here), and where it was computed: Repository and Commit are the trunk merge,
// Src the source revision then (0 before the gate writes source.main). Exactly one of FP and None is set.
// Src the source revision then (0 before the gate writes source.main). At most one of FP and None is set.
//
// From R4c the indexer writes the new form beside it (novox/hq ADR 0453 clause 6): FP2, the fingerprint of the new
// form (`src2:` and 64 hex digits), or None2, why it has none; one of them whenever either is, never both; and
// Manifest, the digest of the module's manifest, the key of the manifests bucket. A value written before has none of
// the three, and a reader of today's shape ignores them.
// the three, and a reader of today's shape ignores them. From R4e a value may carry the new form alone (ADR 0453 clause
// 6; issue 1851): one of FP, None, FP2 and None2 is set, and the old form stays readable.
type Module struct {
V int `json:"v"`
FP string `json:"fp,omitempty"`
@@ -398,6 +406,11 @@ type Module struct {
// builds the fingerprint only with these. Empty when FP2 is empty or the module has no toolchain; a value written
// before has none (issue 1615).
Toolchains map[string]ToolchainUsed `json:"toolchains,omitempty"`
// ArtifactPaths are, of each artifact that names the paths it is built from, those paths as its manifest names
// them, by artifact (novox/hq ADR 0499 clause 1): the indexer fills them from each catalogue.Artifact.Paths it reads,
// so the checker tells whether a change touches an artifact's build source without reading a manifest (the lead's
// decision L8 of R4e). Absent for an artifact that names none, and in a value written before.
ArtifactPaths map[string][]string `json:"paths,omitempty"`
}
// RootPath is Module.Path of a module whose manifest is its repository's root.
@@ -411,6 +424,39 @@ type KeptInputs2 struct {
Builder string `json:"builder,omitempty"`
Contexts map[string]string `json:"contexts,omitempty"`
Locks map[string]string `json:"locks,omitempty"`
// Paths are, of a module whose every artifact names its paths, each named path's git object id at the merge, by
// path relative to the module's directory; Tree is then their NamedTree with the module's manifest digest, so a
// computation from kept inputs gives the merge's value (novox/hq ADR 0499 clause 2). Absent for every other module,
// whose Tree is its directory's tree, and in a value written before.
Paths map[string]string `json:"paths,omitempty"`
}
// checkPathObjects refuses a named path that is not clean and relative to the module's directory, or names no object.
func checkPathObjects(paths map[string]string) error {
for p, object := range paths {
if p == "" || p != path.Clean(p) || strings.HasPrefix(p, "/") || p == ".." || strings.HasPrefix(p, "../") {
return fmt.Errorf("delivery: the named path %q is not a clean path inside the module's directory", p)
}
if object == "" {
return fmt.Errorf("delivery: the named path %q names no git object", p)
}
}
return nil
}
// checkNamed refuses kept named paths that do not compute to the kept tree with the module's manifest digest.
func checkNamed(tree, manifest string, paths map[string]string, of string) error {
if len(paths) == 0 {
return nil
}
named, err := NamedTree(manifest, paths)
if err != nil {
return fmt.Errorf("delivery: %s: %w", of, err)
}
if named != tree {
return fmt.Errorf("delivery: %s keeps the tree %q, and its named paths give the named tree %q", of, tree, named)
}
return nil
}
// With is the full inputs of module m: the kept ones and the given toolchains.
@@ -423,8 +469,8 @@ func (m Module) check() error {
switch {
case m.FP != "" && m.None != "":
return fmt.Errorf("delivery: a module has both a fingerprint and a reason for none")
case m.FP == "" && m.None == "":
return fmt.Errorf("delivery: a module has neither a fingerprint nor a reason for none")
case m.FP == "" && m.None == "" && m.FP2 == "" && m.None2 == "":
return fmt.Errorf("delivery: a module has neither a fingerprint nor a reason for none, of either form")
case m.FP != "" && !strings.HasPrefix(m.FP, SourcePrefix):
return fmt.Errorf("delivery: the module's fingerprint %q does not start with %s", m.FP, SourcePrefix)
case m.FP2 != "" && m.None2 != "":
@@ -438,6 +484,21 @@ func (m Module) check() error {
if m.Inputs2 != nil && m.Manifest == "" {
return fmt.Errorf("delivery: a module keeps new-form inputs but has no manifest digest")
}
for artifact, paths := range m.ArtifactPaths {
if artifact == "" || len(paths) == 0 {
return fmt.Errorf("delivery: the module's named paths of the artifact %q are empty or name no artifact", artifact)
}
for _, p := range paths {
if p == "" || p != path.Clean(p) || strings.HasPrefix(p, "/") || p == ".." || strings.HasPrefix(p, "../") {
return fmt.Errorf("delivery: the artifact %q names the path %q, not a clean path inside the module's directory", artifact, p)
}
}
}
if m.Inputs2 != nil {
if err := checkNamed(m.Inputs2.Tree, m.Manifest, m.Inputs2.Paths, "the module's kept inputs"); err != nil {
return err
}
}
return checkNewForm(m.FP2, m.Manifest)
}
@@ -465,9 +526,13 @@ type Forge struct {
// relative to it (novox/hq ADR 0465 clause 1). Older writers send neither.
Manifests map[string]string `json:"manifests,omitempty"`
Locks map[string]map[string]string `json:"locks,omitempty"`
Act string `json:"act,omitempty"`
Words string `json:"words,omitempty"`
Open []OpenPull `json:"open,omitempty"`
// Paths are, of each module directory whose manifest names paths on every artifact, each named path's git object
// id at the same commit as its manifest, by path relative to the directory (novox/hq ADR 0499 clause 3). The
// indexer names the paths; a message without them gives such a module none. Older writers send none.
Paths map[string]map[string]string `json:"paths,omitempty"`
Act string `json:"act,omitempty"`
Words string `json:"words,omitempty"`
Open []OpenPull `json:"open,omitempty"`
}
func (f Forge) check() error {
@@ -537,8 +602,8 @@ func (f Forge) check() error {
if len(f.Dirs) > 0 || f.DirsKnown {
return fmt.Errorf("delivery: a refused message carries no module directories")
}
if len(f.Manifests) > 0 || len(f.Locks) > 0 {
return fmt.Errorf("delivery: a refused message carries no manifests or lock hashes")
if len(f.Manifests) > 0 || len(f.Locks) > 0 || len(f.Paths) > 0 {
return fmt.Errorf("delivery: a refused message carries no manifests, lock hashes or named paths")
}
if f.Act == "" {
return fmt.Errorf("delivery: a refused message needs the act it answers")
@@ -571,6 +636,16 @@ func (f Forge) checkInputs(known func(dir string) bool, of string) error {
}
}
}
for dir, paths := range f.Paths {
if !known(dir) {
return fmt.Errorf("delivery: the named paths of the directory %q are not of %s", dir, of)
}
for path, object := range paths {
if path == "" || object == "" {
return fmt.Errorf("delivery: the named path %q of the directory %q has an empty path or object", path, dir)
}
}
}
return nil
}
@@ -578,7 +653,7 @@ func (f Forge) checkInputs(known func(dir string) bool, of string) error {
// an open pull request that is not `<owner>/<name>` and a number.
func (f Forge) checkOpen() error {
if f.Repository != "" || f.Branch != "" || f.Number != 0 || f.Head != "" || f.Action != "" || len(f.Dirs) > 0 || f.DirsKnown ||
f.Merge != "" || len(f.Trees) > 0 || len(f.Manifests) > 0 || len(f.Locks) > 0 || f.Root != "" || f.Act != "" || f.Words != "" {
f.Merge != "" || len(f.Trees) > 0 || len(f.Manifests) > 0 || len(f.Locks) > 0 || len(f.Paths) > 0 || f.Root != "" || f.Act != "" || f.Words != "" {
return fmt.Errorf("delivery: an open message carries only the open pull requests it names")
}
for _, p := range f.Open {
+2
View File
@@ -1,3 +1,5 @@
//go:build integration
package forgekey
import (
+80
View File
@@ -0,0 +1,80 @@
package stdio
import (
"encoding/json"
"fmt"
"os"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// What this bundle said it keeps (novox/hq ADR 0495 clause 2; issue 1790).
var (
keepsMu sync.Mutex
kept map[string]bool // nil until Keeps: no state is refused then
announced []byte // the announcement, published once the bundle serves
)
// Keeps says on the bus, as this bundle's module, every state it keeps and which of them others may read (novox/hq
// ADR 0495 clause 2): what a reader is granted is what the owner announced (clause 3). Call it before Serve; the
// announcement is published through the runtime once the bundle serves, or at once when it already does. From then
// on a state of the module's own that it did not announce is refused at the call, naming it, so code and announcement
// cannot drift (clause 5). It refuses a state no bucket could be named by and a state given twice.
func Keeps(states ...announce.State) error {
module := os.Getenv("MESH_MODULE")
if module == "" {
module = os.Getenv(ServedAs)
}
data, err := announce.Encode(announce.States{Module: module, Node: os.Getenv("MESH_NODE"), States: states})
if err != nil {
return err
}
names := map[string]bool{}
for _, s := range states {
names[s.Name] = true
}
keepsMu.Lock()
kept, announced = names, data
keepsMu.Unlock()
currentMu.Lock()
serving := current != nil
currentMu.Unlock()
if serving {
return announce1()
}
return nil
}
// announce1 publishes the announcement Keeps made, if any.
func announce1() error {
keepsMu.Lock()
data := announced
keepsMu.Unlock()
if data == nil {
return nil
}
return Emit(announce.Event, json.RawMessage(data))
}
// refusedUnannounced is the error of a call on a state of the module's own that Keeps did not name, or nil. A read of
// another module's state (`<module>.<name>`) is the grant's business.
func refusedUnannounced(name string) error {
if strings.Contains(name, ".") {
return nil
}
keepsMu.Lock()
defer keepsMu.Unlock()
if kept != nil && !kept[name] {
return fmt.Errorf("the state %q is not among the states this module announced it keeps (stdio.Keeps)", name)
}
return nil
}
// forgetKeeps is for a test: nothing announced.
func forgetKeeps() {
keepsMu.Lock()
kept, announced = nil, nil
keepsMu.Unlock()
}
+63
View File
@@ -0,0 +1,63 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// A Go bundle says what it keeps from its code (novox/hq ADR 0495 clauses 1 and 2): Keeps, called before Serve,
// announces through the runtime once the bundle serves, as the module the runtime names; and a state of its own that it
// did not announce is refused at the call, naming it, so code and announcement cannot drift.
func TestABundleAnnouncesWhatItKeepsAndBindsNothingElse(t *testing.T) {
t.Setenv(ServedAs, "chime")
t.Setenv("MESH_MODULE", "chime")
t.Setenv("MESH_NODE", "anchor")
defer forgetKeeps()
// Not yet serving: an earlier test's channel is gone with its test.
currentMu.Lock()
current = nil
currentMu.Unlock()
if err := Keeps(announce.State{Name: "rings", Readable: true}, announce.State{Name: "own"}); err != nil {
t.Fatal(err)
}
if _, err := State("other").Get("k"); err == nil || !strings.Contains(err.Error(), `"other"`) {
t.Fatalf("a state not announced was bound: %v", err)
}
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = ServeOn("", nil, inR, outW) }()
defer inW.Close()
lines := bufio.NewScanner(outR)
got := make(chan map[string]any, 1)
go func() {
if lines.Scan() {
var m map[string]any
_ = json.Unmarshal(lines.Bytes(), &m)
got <- m
}
}()
var asked map[string]any
select {
case asked = <-got:
case <-time.After(3 * time.Second):
t.Fatal("the bundle announced nothing")
}
p, _ := asked["params"].(map[string]any)
if asked["method"] != "mesh/publish" || p["key"] != announce.Event {
t.Fatalf("the announcement was not published through the runtime: %v", asked)
}
raw, _ := json.Marshal(p["body"])
s, err := announce.Decode(raw)
if err != nil || s.Module != "chime" || s.Node != "anchor" || !s.Readable("rings") || !s.Keeps("own") || s.Readable("own") {
t.Fatalf("announced %s (%v)", raw, err)
}
if err := Keeps(announce.State{Name: "bad_name"}); err == nil {
t.Fatal("an announcement no bucket could be named by was accepted")
}
}
+165
View File
@@ -0,0 +1,165 @@
// Package membership is what the mesh issues an assignment to serve and to reach (novox/hq ADR 0160), as a
// process reads it from the bus: the Membership shape, the one subject it is published on, the stream that
// holds it, and the bus address helpers a reader needs. The controller composes it and the route proxy reads
// it; both use this one definition, so there is no second copy to drift (novox/hq issue 1633).
package membership
import (
"encoding/json"
"strings"
)
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = "ASSIGNMENTS"
// Subject is the one address a runtime derives for itself: where its own membership is published, from
// the two names its credential carries. Everything else is in the membership.
func Subject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// DirectGetSubject is the subject-addressed direct get of a membership from its stream: the one request an
// account may make of the stream.
func DirectGetSubject(node, module string) string {
return "$JS.API.DIRECT.GET." + AssignmentsStream + "." + Subject(node, module)
}
// Membership is one assignment's subjects: what this instance of a module on this machine serves,
// and what it may reach.
type Membership struct {
Node string `json:"node"`
Module string `json:"module"`
// Serves is every address a tool of this instance answers on. `{tool}` stands for the tool's
// own name. An address with a queue is shared with the module's other instances.
Serves []Served `json:"serves"`
// Seats is every verb of a seat this instance holds, at the subject the seat's callers use.
Seats []SeatServed `json:"seats,omitempty"`
// Emits is where an event of this module lands; `{event}` stands for the event's name.
Emits string `json:"emits"`
// Reaches is each tool this module may call, `<module>.<tool>`, to the subjects that reach it.
Reaches map[string][]string `json:"reaches,omitempty"`
// Tools is where this instance answers what it serves.
Tools string `json:"tools"`
// Receives is what this assignment is given for each requirement it receives, by requirement (novox/hq
// ADR 0167): kept as JSON because the catalogue owns the shape of a contribution.
Receives map[string]json.RawMessage `json:"receives,omitempty"`
// Mesh is every machine's address on the private network (novox/hq ADR 0167).
Mesh []string `json:"mesh,omitempty"`
// State is every bucket this module's code may reach, and whether it may write it (novox/hq ADR 0201).
State []StateIssued `json:"state,omitempty"`
// Logs is every log this module's code may reach (novox/hq ADR 0297).
Logs []LogIssued `json:"logs,omitempty"`
// SeatTraffic is what this module's code may submit, say, hear, take, ask, answer and read on seats
// that name their caller or their kind (novox/hq ADR 0259 §3).
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are this module's own streams, Hears the module streams it hears (novox/hq ADR 0363).
Streams []StreamIssued `json:"streams,omitempty"`
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2).
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is one address a tool is answered on.
type Served struct {
Subject string `json:"subject"`
Queue string `json:"queue,omitempty"`
}
// SeatServed is one verb of a held seat, where its callers ask.
type SeatServed struct {
Seat string `json:"seat"`
Verb string `json:"verb"`
Subject string `json:"subject"`
}
// StateIssued is one bucket an assignment may reach, by the name its module uses for it (novox/hq ADR 0201).
// Key, when set, is the one key it may reach (novox/hq ADR 0260): this machine's.
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
Key string `json:"key,omitempty"`
}
// LogIssued is one log an assignment may reach, by the name its module uses for it (novox/hq ADR 0297).
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes"`
}
// Worker is one durable consumer a holder pulls a seat's work from.
type Worker struct {
Stream string
Consumer string
Filter string
}
// SeatTraffic is one module's seat traffic beyond tools. Publish and Subscribe are subject patterns, in the
// server's wildcards.
type SeatTraffic struct {
Publish []string `json:"publish,omitempty"`
Subscribe []string `json:"subscribe,omitempty"`
Answers []string `json:"answers,omitempty"`
Workers []Worker `json:"workers,omitempty"`
Records []string `json:"records,omitempty"`
Kinds []KindHeld `json:"kinds,omitempty"`
}
// KindHeld is one kind of a kinded bench and who holds it.
type KindHeld struct {
Seat string `json:"seat"`
Kind string `json:"kind"`
Module string `json:"module"`
Node string `json:"node"`
Capabilities []string `json:"capabilities,omitempty"`
}
// StreamIssued is one of a module's own streams as its membership names it (ADR 0363 §2).
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears (ADR 0363 §8).
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// WaitIssued is one family as a membership issues it: the name a session asks for, and its subject.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// CredentialIn reads the credential out of a bus address, `[scheme://]user:password@host:port`, splitting on
// the last at-sign so a password containing one stays whole, and returns the address without it.
func CredentialIn(address string) (user, password, bare string) {
at := strings.LastIndex(address, "@")
if at < 0 {
return "", "", address
}
scheme := ""
rest := address[:at]
if i := strings.Index(rest, "://"); i >= 0 {
scheme, rest = rest[:i+3], rest[i+3:]
}
user, password, _ = strings.Cut(rest, ":")
return user, password, scheme + address[at+1:]
}
// BareAddress is a bus address with any credential stripped, for something that only needs to know
// whether a server is answering there.
func BareAddress(address string) string {
_, _, bare := CredentialIn(address)
if bare == "" {
return address
}
if strings.Contains(bare, "://") {
return bare
}
return "nats://" + bare
}
+76
View File
@@ -0,0 +1,76 @@
package membership
import (
"encoding/json"
"reflect"
"testing"
)
func TestACredentialIsReadOutOfABusAddress(t *testing.T) {
for _, c := range []struct{ in, user, password, bare string }{
{"nats://controller:secret@127.0.0.1:4222", "controller", "secret", "nats://127.0.0.1:4222"},
{"controller:secret@127.0.0.1:4222", "controller", "secret", "127.0.0.1:4222"},
{"nats://127.0.0.1:4222", "", "", "nats://127.0.0.1:4222"},
{"127.0.0.1:4222", "", "", "127.0.0.1:4222"},
// A password containing an at-sign: split on the last one.
{"nats://controller:a@b@127.0.0.1:4222", "controller", "a@b", "nats://127.0.0.1:4222"},
} {
user, password, bare := CredentialIn(c.in)
if user != c.user || password != c.password || bare != c.bare {
t.Errorf("%q read as %q/%q at %q; wanted %q/%q at %q", c.in, user, password, bare, c.user, c.password, c.bare)
}
}
}
func TestABareAddressCarriesNoCredentialAndAScheme(t *testing.T) {
for in, want := range map[string]string{
"nats://controller:secret@10.0.0.1:4222": "nats://10.0.0.1:4222",
"controller:secret@10.0.0.1:4222": "nats://10.0.0.1:4222",
"nats://10.0.0.1:4222": "nats://10.0.0.1:4222",
"10.0.0.1:4222": "nats://10.0.0.1:4222",
} {
if got := BareAddress(in); got != want {
t.Errorf("BareAddress(%q) = %q, want %q", in, got, want)
}
}
}
func TestAMembershipIsPublishedAndReadAtTheSubjectDerivedFromTheTwoNames(t *testing.T) {
if got := Subject("ace", "route-proxy"); got != "mesh.assignment.ace.route-proxy" {
t.Errorf("subject: %q", got)
}
if got := DirectGetSubject("ace", "route-proxy"); got != "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.ace.route-proxy" {
t.Errorf("direct get: %q", got)
}
if AssignmentsStream != "ASSIGNMENTS" {
t.Errorf("stream: %q", AssignmentsStream)
}
}
// The shape on the wire is the controller's: a membership as issued reads back whole, with the routes kept raw.
func TestAMembershipRoundTripsAsIssued(t *testing.T) {
in := Membership{Node: "ace", Module: "route-proxy", Emits: "e", Tools: "t",
Serves: []Served{{Subject: "s", Queue: "q"}},
Receives: map[string]json.RawMessage{"route": json.RawMessage(`[{"name":"a"}]`)},
Mesh: []string{"10.10.0.7"},
State: []StateIssued{{Name: "n", Bucket: "b", Writes: true, Key: "ace"}},
SeatTraffic: &SeatTraffic{Publish: []string{"p"}, Kinds: []KindHeld{{Seat: "channel", Kind: "tg"}}}}
raw, err := json.Marshal(in)
if err != nil {
t.Fatal(err)
}
var out Membership
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in, out) {
t.Fatalf("round trip changed it:\n%+v\n%+v", in, out)
}
var wire map[string]json.RawMessage
_ = json.Unmarshal(raw, &wire)
for _, k := range []string{"receives", "mesh", "seat-traffic"} {
if _, ok := wire[k]; !ok {
t.Errorf("field %q is not on the wire under that name", k)
}
}
}
+7 -2
View File
@@ -10,13 +10,17 @@
# 1. the unit level: `go test ./...` untagged, no bus or other process, timed and held to a bound
# (unit_bound below); over it the check fails and names the slowest tests (issue 895);
# 2. the integration level: the whole suite with `-tags integration`, each test against a bus of its
# own (internal/testbus, which issue 1171 brings with the first test that needs a bus) so nothing is
# own (testbus, which issue 1171 brings with the first test that needs a bus) so nothing is
# serialised. With no tagged test yet, the level runs the unit tests again and says their count.
#
# **A test that starts a bus or another process carries `//go:build integration`** as the first line of
# its file; a file that held both kinds is two files (`<name>_integration_test.go` beside `<name>_test.go`).
# Each level says its test count and its time.
#
# **A test that needs bubblewrap carries `//go:build integration && isolation`** (issue 1309's proof that a run step
# is shut off): this script vets it but does not run it, since the toolchain it runs in has no bubblewrap; the checker
# runs the `isolation` tag on the build nodes, which have it (R4e).
#
# No dependency is committed (no vendor directory: .gitignore; the operator, 2026-10-19; issue 1304). Every dependency
# is resolved by the version and checksum go.sum names (novox/hq ADR 0405 §5), from its public source: this binding
# requires nothing of the mesh's own forge. `go mod verify` fails by module name when a downloaded dependency does not
@@ -98,6 +102,7 @@ fi
go vet ./...
# The files tagged `integration` are vetted too, or nothing would read them.
go vet -tags integration ./...
go vet -tags integration,isolation ./...
# NOT RACE-CHECKED here: the SDK's suite is small and carries no store or other shared state between
# packages, so the race detector is not required of this check (novox/hq issue 1175); a module that needs
@@ -117,7 +122,7 @@ if [ "$status" -ne 0 ]; then
fi
judge_unit_run "$work/unit.out" "$unit_bound"
# The integration level: the whole suite with the tag, each test against its own bus (internal/testbus).
# The integration level: the whole suite with the tag, each test against its own bus (testbus).
integration_start=$(date +%s)
status=0
go test -tags integration -count=1 -v ./... >"$work/integration.out" 2>&1 || status=$?
+12 -4
View File
@@ -24,6 +24,12 @@ type Conn struct {
mu sync.Mutex
next uint64
pending map[string]map[uint64]context.CancelCauseFunc
// What Keeps announced (keeps.go): the states' names (nil before Keeps), the announcement, and whether a
// reconnect announces it again.
kept map[string]bool
announced []byte
reannounces bool
}
// errPermissionRefused marks a context cancelled because the bus refused this account's publish of the
@@ -40,7 +46,7 @@ var refusedPublish = regexp.MustCompile(`(?i)Publish to "(\S+)"`)
// other certificate), authenticated as its user and password, on the inbox prefix _INBOX.<user> and no
// other, reconnecting without limit as the tool runner does (novox/hq issue 1192: a process outlives a
// bus restart, never giving up the way the client library's own default of 60 attempts would). Ported
// from mesh-controller internal/broker/jetstream.go:106-132 (DialPinned, PinnedToFingerprint) since this
// from mesh-controller internal/broker/jetstream.go (DialPinned, PinnedToFingerprint), which now uses this one since this
// package may not import the controller.
//
// An ErrorHandler among opts is chained behind this package's own, which watches every async error for a
@@ -78,7 +84,7 @@ func (c *Conn) options(opts []nats.Option) []nats.Option {
nats.MaxReconnects(-1),
}
if c.account.Fingerprint != "" {
all = append(all, nats.Secure(pinnedToFingerprint(c.account.Fingerprint)))
all = append(all, nats.Secure(PinnedToFingerprint(c.account.Fingerprint)))
}
all = append(all, opts...)
return append(all, nats.ErrorHandler(func(nc *nats.Conn, sub *nats.Subscription, err error) {
@@ -140,9 +146,11 @@ func (c *Conn) publishRefused(err error) bool {
return len(cancels) > 0
}
// pinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other — the leaf alone, its
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other — the leaf alone, its
// name and chain deliberately not consulted, as mesh-controller's own pin does (jetstream.go:116-132).
func pinnedToFingerprint(want string) *tls.Config {
// PinnedToFingerprint is public so that a process dialling the bus on its own terms, and the controller,
// share this one pin (novox/hq issue 1633).
func PinnedToFingerprint(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
+1 -1
View File
@@ -22,7 +22,7 @@ import (
)
// selfSignedServer is a nats-server with JetStream and TLS, self-signed, for this test alone (mirroring
// go/internal/testbus.Start, with a certificate added since testbus has none): a mesh's bus presents its own
// go/testbus.Start, with a certificate added since testbus has none): a mesh's bus presents its own
// certificate in no public trust store, which is exactly what a process pins to (ADR 0436 clause 4).
func selfSignedServer(t *testing.T) (url, fingerprint string) {
t.Helper()
+1
View File
@@ -38,6 +38,7 @@ var sendGrants = []string{
"$KV.mesh-mail_sends.>",
"mesh.mod.mesh-mail.event.mail.send.failed",
"mesh.mod.mesh-mail.event.mail.send.sent",
"mesh.mod.mesh-mail.event.states",
"mesh.seat.operator-channel.call.tell.*.novox~mesh-mail~send",
"mesh.seat.operator-channel.call.tell.novox~mesh-mail~send",
"mesh.seat.operator-channel.tool.tell",
+53
View File
@@ -0,0 +1,53 @@
package process
import (
"encoding/json"
"fmt"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/membership"
)
// FollowMembership reads this account's own membership and follows it (novox/hq ADR 0160): apply is called
// with every membership the mesh issues this account's node and module, the one already issued first and
// each later one as it is published, so a change reaches a running process without a restart. A body that is
// not a membership is handed to onBad (nil to ignore it) and skipped.
//
// The subject is followed before it is read, so an issue landing between the two is applied, not missed. The
// read is the subject-addressed direct get, the one request this account may make of the stream; no
// membership issued yet, or the read failing, is not an error: the process follows the subject and is told
// when one arrives. The returned error is a refusal to follow at all.
func (c *Conn) FollowMembership(apply func(membership.Membership), onBad func(error)) error {
if c.account.Node == "" || c.account.Module == "" {
return fmt.Errorf("the account names no node or module, so it has no membership")
}
bad := func(err error) {
if onBad != nil {
onBad(err)
}
}
take := func(body []byte) {
var issued membership.Membership
if err := json.Unmarshal(body, &issued); err != nil {
bad(fmt.Errorf("a membership arrived that is not one: %w", err))
return
}
apply(issued)
}
subject := membership.Subject(c.account.Node, c.account.Module)
if _, err := c.nc.Subscribe(subject, func(m *nats.Msg) { take(m.Data) }); err != nil {
return fmt.Errorf("cannot follow %s: %w", subject, err)
}
got, err := c.nc.Request(membership.DirectGetSubject(c.account.Node, c.account.Module), nil, 5*time.Second)
switch {
case err != nil:
bad(fmt.Errorf("cannot read the membership issued on %s yet (%v); following it", subject, err))
case got.Header.Get("Status") != "" || len(got.Data) == 0:
// None issued yet: the follow above delivers it.
default:
take(got.Data)
}
return nil
}
+45
View File
@@ -0,0 +1,45 @@
//go:build integration
package process
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/membership"
)
// A process connected pinned to a test bus hears the membership published for its node and module, and
// ignores what is not one (novox/hq issue 1633, AC2).
func TestFollowMembershipHearsWhatIsIssuedToTheAccount(t *testing.T) {
url, fingerprint := selfSignedServer(t)
account := mailSenderAccount(t, url, fingerprint)
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
got := make(chan membership.Membership, 1)
bad := make(chan error, 4)
if err := conn.FollowMembership(func(m membership.Membership) { got <- m }, func(e error) { bad <- e }); err != nil {
t.Fatal(err)
}
if err := conn.nc.Publish(membership.Subject("novox", "mesh-mail"), []byte("not json")); err != nil {
t.Fatal(err)
}
if err := conn.nc.Publish(membership.Subject("novox", "mesh-mail"), []byte(`{"node":"novox","module":"mesh-mail","mesh":["10.0.0.1"]}`)); err != nil {
t.Fatal(err)
}
select {
case m := <-got:
if m.Module != "mesh-mail" || len(m.Mesh) != 1 {
t.Fatalf("membership: %+v", m)
}
case <-time.After(5 * time.Second):
t.Fatal("no membership heard")
}
if len(bad) == 0 {
t.Error("the body that is not a membership was not reported")
}
}
+71
View File
@@ -0,0 +1,71 @@
package process
import (
"encoding/json"
"fmt"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// Keeps says on the bus, as this process's module, every state it keeps and which of them others may read (novox/hq
// ADR 0495 clause 2; issue 1790), on announce.Subject; what a reader is granted is what the owner announced (clause 3).
// It publishes at once and again after every reconnect, so the controller hears it whenever it missed it. From then on
// a state of the module that it did not announce is refused at the call, naming it, so code and announcement cannot
// drift (clause 5). It refuses a state no bucket could be named by and a state given twice.
func (c *Conn) Keeps(states ...announce.State) error {
if err := c.remember(states); err != nil {
return err
}
c.mu.Lock()
first := !c.reannounces
c.reannounces = true
c.mu.Unlock()
if first {
before := c.nc.Opts.ReconnectedCB
c.nc.SetReconnectHandler(func(nc *nats.Conn) {
if before != nil {
before(nc)
}
_ = c.publishKeeps()
})
}
return c.publishKeeps()
}
// remember makes the announcement of states and binds only them from now on.
func (c *Conn) remember(states []announce.State) error {
data, err := announce.Encode(announce.States{Module: c.account.Module, Node: c.account.Node, States: states})
if err != nil {
return err
}
names := map[string]bool{}
for _, s := range states {
names[s.Name] = true
}
c.mu.Lock()
c.kept, c.announced = names, data
c.mu.Unlock()
return nil
}
func (c *Conn) publishKeeps() error {
c.mu.Lock()
data := c.announced
c.mu.Unlock()
if err := c.Emit(announce.Event, json.RawMessage(data)); err != nil {
return fmt.Errorf("announcing what %s keeps: %w", c.account.Module, err)
}
return nil
}
// refusedUnannounced is the error of a bind of a state Keeps did not name, or nil (also before Keeps).
func (c *Conn) refusedUnannounced(name string) error {
c.mu.Lock()
defer c.mu.Unlock()
if c.kept != nil && !c.kept[name] {
return fmt.Errorf("the state %q is not among the states %s announced it keeps (Conn.Keeps)", name, c.account.Module)
}
return nil
}
+45
View File
@@ -0,0 +1,45 @@
//go:build integration
package process
import (
"strings"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// 1790/S10: on a real bus, a process's Keeps lands on its module's own announcement subject, in the events stream, as
// the module's announcement; a state it did not announce is then refused at the call.
func TestAProcessAnnouncesWhatItKeepsOnTheBus(t *testing.T) {
url, account := grantedSendProcess(t)
_, admin := adminJS(t, url)
stream, err := admin.CreateStream(ctxFor(t), jetstream.StreamConfig{Name: "MAIL_STATES", Subjects: []string{announce.Subject("mesh-mail")}})
if err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if err := conn.Keeps(announce.State{Name: "accounts", Readable: true}, announce.State{Name: "drafts"}); err != nil {
t.Fatal(err)
}
msg, err := stream.GetMsg(ctxFor(t), 1)
if err != nil {
t.Fatalf("no announcement in the stream: %v", err)
}
if msg.Subject != "mesh.mod.mesh-mail.event.states" || msg.Header.Get("x-source") != "mesh-mail" {
t.Fatalf("announced on %s with %v", msg.Subject, msg.Header)
}
got, err := announce.Decode(msg.Data)
if err != nil || !got.Readable("accounts") || !got.Keeps("drafts") || got.Readable("drafts") || got.Node != "novox" {
t.Fatalf("announced %s (%v)", msg.Data, err)
}
if _, err := conn.State("outbox").Get("k"); err == nil || !strings.Contains(err.Error(), `"outbox"`) {
t.Fatalf("an unannounced state was bound: %v", err)
}
}
+28
View File
@@ -0,0 +1,28 @@
package process
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// After Keeps a process binds only the states it announced (novox/hq ADR 0495 clause 5): any other of its module's
// states is refused at the call, naming it, before the bus is asked; before Keeps nothing is refused.
func TestAProcessBindsOnlyTheStatesItAnnounced(t *testing.T) {
c := &Conn{account: Account{Module: "mesh-mail", Node: "novox"}}
if err := c.remember([]announce.State{{Name: "accounts", Readable: true}}); err != nil {
t.Fatal(err)
}
_, err := c.State("drafts").Get("k")
if err == nil || !strings.Contains(err.Error(), `"drafts"`) || !strings.Contains(err.Error(), "Keeps") {
t.Fatalf("an unannounced state was bound: %v", err)
}
if err := c.remember([]announce.State{{Name: "a_b"}}); err == nil {
t.Fatal("an announcement no bucket could be named by was accepted")
}
got, err := announce.Decode(c.announced)
if err != nil || got.Module != "mesh-mail" || got.Node != "novox" || !got.Readable("accounts") {
t.Fatalf("the announcement is %s (%v)", c.announced, err)
}
}
+23
View File
@@ -0,0 +1,23 @@
package process
import (
"crypto/sha256"
"encoding/hex"
"testing"
)
// The pin accepts exactly the leaf certificate with the named SHA-256, and no other (novox/hq issue 1633).
func TestThePinAcceptsOnlyTheNamedCertificate(t *testing.T) {
leaf := []byte("a certificate")
sum := sha256.Sum256(leaf)
cfg := PinnedToFingerprint("sha256:" + hex.EncodeToString(sum[:]))
if err := cfg.VerifyPeerCertificate([][]byte{leaf}, nil); err != nil {
t.Errorf("the pinned certificate was refused: %v", err)
}
if err := cfg.VerifyPeerCertificate([][]byte{[]byte("another")}, nil); err == nil {
t.Error("another certificate was accepted")
}
if err := cfg.VerifyPeerCertificate(nil, nil); err == nil {
t.Error("no certificate was accepted")
}
}
+153 -1
View File
@@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
@@ -27,6 +29,8 @@ type State struct {
// kv binds this state's underlying key-value store: a live connection's own (Conn.State), or, in a
// unit test, a fake that never reaches a bus.
kv func(ctx context.Context) (jetstream.KeyValue, error)
// closed says the connection is gone, so a watch ends; nil in a unit test.
closed func() bool
}
// bucket is this state's name on the bus.
@@ -38,7 +42,10 @@ func (s State) bucket() string { return s.Module + "_" + s.Name }
// not granted is refused there, promptly, as nats.ErrPermissionViolation.
func (c *Conn) State(name string) State {
bucket := c.account.Module + "_" + name
return State{Module: c.account.Module, Name: name, kv: func(ctx context.Context) (jetstream.KeyValue, error) {
return State{Module: c.account.Module, Name: name, closed: func() bool { return c.nc.IsClosed() }, kv: func(ctx context.Context) (jetstream.KeyValue, error) {
if err := c.refusedUnannounced(name); err != nil {
return nil, err
}
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
@@ -128,3 +135,148 @@ func (s State) Update(key string, value any, revision uint64) (uint64, error) {
}
return rev, nil
}
// Put writes a key whole, whatever it held, and returns the revision written.
func (s State) Put(key string, value any) (uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return 0, err
}
data, err := json.Marshal(value)
if err != nil {
return 0, err
}
rev, err := kv.Put(ctx, key, data)
if err != nil {
return 0, fmt.Errorf("%s's state %s, key %s: %w", s.Module, s.Name, key, err)
}
return rev, nil
}
// Keys is every key with a current value, sorted; none is an empty list.
func (s State) Keys() ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return nil, err
}
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return []string{}, nil
}
if err != nil {
return nil, fmt.Errorf("%s's state %s, keys: %w", s.Module, s.Name, err)
}
sort.Strings(keys)
return keys, nil
}
// watchPattern is a key narrowing as the bus names it: `*` stays one name, `**` is the rest, empty is every key.
func watchPattern(key string) string {
if key == "" {
return jetstream.AllKeys
}
tokens := strings.Split(key, ".")
for i, t := range tokens {
if t == "**" {
tokens[i] = ">"
}
}
return strings.Join(tokens, ".")
}
// Watch hands handle every current value — none that is deleted, each with Current set — and then every
// change in order, and returns once the current values have all been handled. key narrows it: a key, with
// `*` for one name and `**` for the rest; empty is every key. The changes go on for the life of the
// connection; an error from handle, or the connection closing, ends them. Before the current values are
// handled, either is returned as the error.
func (s State) Watch(key string, handle func(stdio.StateChange) error) error {
// the watch outlives this call, so its context is not the call's timeout: the connection ends it.
ctx, cancel := context.WithCancel(context.Background())
bind, stopBind := context.WithTimeout(ctx, stateTimeout)
kv, err := s.store(bind)
stopBind()
if err != nil {
cancel()
return err
}
w, err := kv.Watch(ctx, watchPattern(key))
if err != nil {
cancel()
return fmt.Errorf("%s's state %s, watching %q: %w", s.Module, s.Name, key, err)
}
end := func() { w.Stop(); cancel() }
next := func() (jetstream.KeyValueEntry, bool, error) {
tick := time.NewTicker(250 * time.Millisecond)
defer tick.Stop()
for {
select {
case e, open := <-w.Updates():
if !open {
return nil, false, fmt.Errorf("%s's state %s: the watch ended", s.Module, s.Name)
}
return e, true, nil
case <-tick.C:
if s.closed != nil && s.closed() {
return nil, false, fmt.Errorf("%s's state %s: the connection closed", s.Module, s.Name)
}
}
}
}
deliver := func(e jetstream.KeyValueEntry, current bool) error {
c := stdio.StateChange{State: s.Name, Key: e.Key(), Op: "put", Revision: e.Revision(), Current: current}
if e.Operation() == jetstream.KeyValuePut {
c.Value = json.RawMessage(e.Value())
} else {
c.Op = "delete"
}
return handle(c)
}
for {
e, _, err := next()
if err != nil {
end()
return err
}
if e == nil {
break
}
if e.Operation() != jetstream.KeyValuePut {
continue
}
if err := deliver(e, true); err != nil {
end()
return err
}
}
go func() {
defer end()
for {
e, _, err := next()
if err != nil || e == nil {
return
}
if deliver(e, false) != nil {
return
}
}
}()
return nil
}
// Delete removes a key. A key that was not there is not an error.
func (s State) Delete(key string) error {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return err
}
if err := kv.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return fmt.Errorf("%s's state %s, key %s: %w", s.Module, s.Name, key, err)
}
return nil
}
+180
View File
@@ -6,6 +6,7 @@ import (
"errors"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
@@ -94,3 +95,182 @@ func TestStateRefusesAnUngrantedBucketNamingIt(t *testing.T) {
t.Errorf("the refusal %q does not name the bucket mesh-mail_unrelated", err.Error())
}
}
// 1840/S1: given a process connection granted its bucket, when it puts keys whole and lists them, then Put
// returns a new revision each time and Keys lists them sorted (none for a bucket with none).
func TestStatePutAndKeysOnAGrantedBucket(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
s := conn.State("sends")
if keys, err := s.Keys(); err != nil || len(keys) != 0 {
t.Fatalf("Keys of an empty bucket: %v, %v", keys, err)
}
rev, err := s.Put("b", map[string]any{"v": 1})
if err != nil || rev == 0 {
t.Fatalf("Put on a granted bucket: %d, %v", rev, err)
}
rev2, err := s.Put("b", map[string]any{"v": 2})
if err != nil || rev2 <= rev {
t.Fatalf("Put over a key: %d, %v", rev2, err)
}
if _, err := s.Put("a", 1); err != nil {
t.Fatal(err)
}
keys, err := s.Keys()
if err != nil || strings.Join(keys, ",") != "a,b" {
t.Fatalf("Keys: %v, %v, want a,b", keys, err)
}
}
// 1840/S2: given a process connection granted its bucket, which already holds two keys and a deleted one,
// when it watches, then it gets the two current values marked Current, and then a later put and a delete in
// order; a narrowed watch gets only the keys it names.
func TestStateWatchOnAGrantedBucket(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
kv, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"})
if err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
ctx := ctxFor(t)
for k, v := range map[string]string{"a.x": `1`, "a.y": `2`, "b.z": `3`, "gone": `4`} {
if _, err := kv.Put(ctx, k, []byte(v)); err != nil {
t.Fatal(err)
}
}
if err := kv.Delete(ctx, "gone"); err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
s := conn.State("sends")
all := make(chan stdio.StateChange, 16)
if err := s.Watch("", func(c stdio.StateChange) error { all <- c; return nil }); err != nil {
t.Fatalf("Watch on a granted bucket: %v", err)
}
current := map[string]bool{}
for len(all) > 0 {
c := <-all
if !c.Current || c.Op != "put" {
t.Fatalf("a current value arrived as %+v", c)
}
current[c.Key] = true
}
if len(current) != 3 || current["gone"] {
t.Fatalf("current values %v, want a.x, a.y, b.z and not gone", current)
}
narrow := make(chan stdio.StateChange, 16)
if err := s.Watch("a.*", func(c stdio.StateChange) error { narrow <- c; return nil }); err != nil {
t.Fatal(err)
}
if len(narrow) != 2 {
t.Fatalf("a narrowed watch handed %d current values, want 2", len(narrow))
}
// the current values come in the order of their last revision, not of their keys: taken as a set.
narrowCurrent := map[string]bool{}
for len(narrow) > 0 {
c := <-narrow
if !c.Current || c.Op != "put" {
t.Fatalf("a narrowed watch's current value arrived as %+v", c)
}
narrowCurrent[c.Key] = true
}
if !narrowCurrent["a.x"] || !narrowCurrent["a.y"] {
t.Fatalf("a narrowed watch's current values %v, want a.x and a.y", narrowCurrent)
}
if _, err := s.Put("a.x", 5); err != nil {
t.Fatal(err)
}
if err := kv.Delete(ctx, "a.y"); err != nil {
t.Fatal(err)
}
for _, want := range []struct{ key, op string }{{"a.x", "put"}, {"a.y", "delete"}} {
select {
case c := <-all:
if c.Key != want.key || c.Op != want.op || c.Current {
t.Fatalf("got %+v, want %s %s, not current", c, want.op, want.key)
}
case <-time.After(10 * time.Second):
t.Fatalf("no %s of %s", want.op, want.key)
}
}
if _, err := s.Put("b.q", 6); err != nil {
t.Fatal(err)
}
for _, want := range []string{"a.x", "a.y"} {
select {
case c := <-narrow:
if c.Key != want || c.Current {
t.Fatalf("narrowed watch got %+v, want the change of %s", c, want)
}
case <-time.After(10 * time.Second):
t.Fatalf("narrowed watch missed %s", want)
}
}
select {
case c := <-narrow:
t.Fatalf("narrowed watch got %+v for a key it did not name", c)
case <-time.After(500 * time.Millisecond):
}
}
// 1840/AC8: given an account not granted a bucket, when it watches it, then the refusal names the bucket.
func TestStateWatchRefusesAnUngrantedBucketNamingIt(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_unrelated"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
err = conn.State("unrelated").Watch("", func(stdio.StateChange) error { return nil })
if err == nil || !errors.Is(err, nats.ErrPermissionViolation) || !strings.Contains(err.Error(), "mesh-mail_unrelated") {
t.Fatalf("Watch on an ungranted bucket: %v, want a refusal naming mesh-mail_unrelated", err)
}
}
// 1840/S1: given a process connection granted its bucket, when it deletes a key, then the key is gone and
// deleting it again is no error.
func TestStateDeleteOnAGrantedBucket(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
s := conn.State("sends")
if _, err := s.Put("k", 1); err != nil {
t.Fatal(err)
}
if err := s.Delete("k"); err != nil {
t.Fatalf("Delete on a granted bucket: %v", err)
}
if e, err := s.Get("k"); err != nil || e != nil {
t.Fatalf("Get after Delete: %+v, %v", e, err)
}
if err := s.Delete("k"); err != nil {
t.Fatalf("Delete of an absent key: %v", err)
}
}
+199
View File
@@ -3,8 +3,11 @@ package process
import (
"context"
"errors"
"sort"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
@@ -32,6 +35,9 @@ type fakeKV struct {
values map[string][]byte
revs map[string]uint64
next uint64
// initial is what a Watch hands first, before its nil marker; watcher is the last one made.
initial []jetstream.KeyValueEntry
watcher *fakeWatcher
}
func newFakeKV() *fakeKV { return &fakeKV{values: map[string][]byte{}, revs: map[string]uint64{}} }
@@ -68,6 +74,58 @@ func (f *fakeKV) Update(_ context.Context, key string, value []byte, revision ui
return f.next, nil
}
func (f *fakeKV) Put(_ context.Context, key string, value []byte) (uint64, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.next++
f.values[key], f.revs[key] = value, f.next
return f.next, nil
}
func (f *fakeKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.values, key)
delete(f.revs, key)
return nil
}
func (f *fakeKV) Keys(_ context.Context, _ ...jetstream.WatchOpt) ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
if len(f.values) == 0 {
return nil, jetstream.ErrNoKeysFound
}
keys := make([]string, 0, len(f.values))
for k := range f.values {
keys = append(keys, k)
}
sort.Sort(sort.Reverse(sort.StringSlice(keys))) // as a bucket lists them: in no order the caller may rely on
return keys, nil
}
// fakeWatcher is a jetstream.KeyWatcher fed by a test.
type fakeWatcher struct {
jetstream.KeyWatcher
updates chan jetstream.KeyValueEntry
pattern string
}
func (w *fakeWatcher) Updates() <-chan jetstream.KeyValueEntry { return w.updates }
func (w *fakeWatcher) Stop() error { return nil }
func (f *fakeKV) Watch(_ context.Context, pattern string, _ ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
f.mu.Lock()
defer f.mu.Unlock()
w := &fakeWatcher{updates: make(chan jetstream.KeyValueEntry, 16), pattern: pattern}
f.watcher = w
for _, e := range f.initial {
w.updates <- e
}
w.updates <- nil
return w, nil
}
func stateOn(kv jetstream.KeyValue) State {
return State{Module: "mesh-mail", Name: "sends", kv: func(context.Context) (jetstream.KeyValue, error) { return kv, nil }}
}
@@ -103,3 +161,144 @@ func TestStateSatisfiesAskerStore(t *testing.T) {
t.Fatal("Update did not advance the revision")
}
}
// 1840/S1: given a state, when a key is put twice, then each Put returns a new revision and the key reads
// back the last value, whether or not it had one.
func TestStatePutWritesAKeyWhole(t *testing.T) {
s := stateOn(newFakeKV())
rev, err := s.Put("k", map[string]any{"a": 1})
if err != nil || rev == 0 {
t.Fatalf("Put of a new key: %d, %v", rev, err)
}
rev2, err := s.Put("k", map[string]any{"a": 2})
if err != nil || rev2 <= rev {
t.Fatalf("Put over a key: %d, %v, want a revision after %d", rev2, err, rev)
}
e, err := s.Get("k")
if err != nil || e == nil || e.Revision != rev2 || string(e.Value) != `{"a":2}` {
t.Fatalf("Get after Put: %+v, %v", e, err)
}
}
// 1840/S1: given a state, when its keys are listed, then they come sorted, and none is an empty list, not an error.
func TestStateKeysAreSortedAndEmptyWhenNone(t *testing.T) {
s := stateOn(newFakeKV())
if keys, err := s.Keys(); err != nil || len(keys) != 0 {
t.Fatalf("Keys of an empty state: %v, %v", keys, err)
}
for _, k := range []string{"b", "c", "a"} {
if _, err := s.Put(k, 1); err != nil {
t.Fatal(err)
}
}
keys, err := s.Keys()
if err != nil || strings.Join(keys, ",") != "a,b,c" {
t.Fatalf("Keys: %v, %v, want a,b,c", keys, err)
}
}
type watchEntry struct {
fakeEntry
op jetstream.KeyValueOp
}
func (e watchEntry) Operation() jetstream.KeyValueOp { return e.op }
// 1840/S2: given a state with current values (one of them deleted), when it is watched, then the handler gets
// each current value marked Current and not the deleted one, Watch returns, and a later put and delete arrive
// in order unmarked.
func TestStateWatchHandsCurrentValuesThenChanges(t *testing.T) {
kv := newFakeKV()
kv.initial = []jetstream.KeyValueEntry{
watchEntry{fakeEntry{key: "a", value: []byte(`1`), rev: 1}, jetstream.KeyValuePut},
watchEntry{fakeEntry{key: "gone", rev: 2}, jetstream.KeyValueDelete},
watchEntry{fakeEntry{key: "b", value: []byte(`2`), rev: 3}, jetstream.KeyValuePut},
}
got := make(chan stdio.StateChange, 8)
s := stateOn(kv)
if err := s.Watch("", func(c stdio.StateChange) error { got <- c; return nil }); err != nil {
t.Fatal(err)
}
for _, want := range []stdio.StateChange{
{State: "sends", Key: "a", Op: "put", Value: []byte(`1`), Revision: 1, Current: true},
{State: "sends", Key: "b", Op: "put", Value: []byte(`2`), Revision: 3, Current: true},
} {
select {
case c := <-got:
if c.Key != want.Key || c.Op != want.Op || string(c.Value) != string(want.Value) || c.Revision != want.Revision || c.Current != want.Current || c.State != want.State {
t.Fatalf("got %+v, want %+v", c, want)
}
default:
t.Fatalf("Watch returned before handling %+v", want)
}
}
if len(got) != 0 {
t.Fatalf("a deleted key was handed: %+v", <-got)
}
kv.watcher.updates <- watchEntry{fakeEntry{key: "a", value: []byte(`3`), rev: 4}, jetstream.KeyValuePut}
kv.watcher.updates <- watchEntry{fakeEntry{key: "b", rev: 5}, jetstream.KeyValueDelete}
for _, want := range []stdio.StateChange{
{Key: "a", Op: "put", Revision: 4}, {Key: "b", Op: "delete", Revision: 5},
} {
select {
case c := <-got:
if c.Key != want.Key || c.Op != want.Op || c.Revision != want.Revision || c.Current {
t.Fatalf("got %+v, want %+v", c, want)
}
case <-time.After(5 * time.Second):
t.Fatalf("no change %+v", want)
}
}
}
// 1840/S2: a key narrowing reaches the bus as a subject pattern: `*` one name, `**` the rest, empty every key.
func TestStateWatchNarrowing(t *testing.T) {
for key, want := range map[string]string{"": ">", "a.b": "a.b", "a.*": "a.*", "a.**": "a.>", "**": ">"} {
if got := watchPattern(key); got != want {
t.Errorf("watchPattern(%q) = %q, want %q", key, got, want)
}
}
}
// 1840/S2: given a handler that refuses a current value, then Watch returns its error.
func TestStateWatchReturnsAHandlersErrorOnCurrentValues(t *testing.T) {
kv := newFakeKV()
kv.initial = []jetstream.KeyValueEntry{watchEntry{fakeEntry{key: "a", value: []byte(`1`), rev: 1}, jetstream.KeyValuePut}}
boom := errors.New("boom")
if err := stateOn(kv).Watch("", func(stdio.StateChange) error { return boom }); !errors.Is(err, boom) {
t.Fatalf("Watch: %v, want the handler's error", err)
}
}
// 1840/S2: given a connection that closes before the current values arrive, then Watch returns an error.
func TestStateWatchEndsWithAnErrorWhenTheConnectionCloses(t *testing.T) {
s := State{Module: "mesh-mail", Name: "sends", closed: func() bool { return true },
kv: func(context.Context) (jetstream.KeyValue, error) { return silentKV{}, nil }}
if err := s.Watch("", func(stdio.StateChange) error { return nil }); err == nil {
t.Fatal("Watch on a closed connection returned no error")
}
}
// silentKV watches but never delivers.
type silentKV struct{ jetstream.KeyValue }
func (silentKV) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return &fakeWatcher{updates: make(chan jetstream.KeyValueEntry)}, nil
}
// 1840/S1: given a key, when it is deleted, then it reads as absent; deleting a key that was not there is no error.
func TestStateDeleteRemovesAKeyAndAnAbsentOneIsNoError(t *testing.T) {
s := stateOn(newFakeKV())
if _, err := s.Put("k", 1); err != nil {
t.Fatal(err)
}
if err := s.Delete("k"); err != nil {
t.Fatalf("Delete of a key: %v", err)
}
if e, err := s.Get("k"); err != nil || e != nil {
t.Fatalf("Get after Delete: %+v, %v, want absent", e, err)
}
if err := s.Delete("never"); err != nil {
t.Fatalf("Delete of an absent key: %v", err)
}
}
+3
View File
@@ -50,6 +50,9 @@ func State(name string) KeptState { return KeptState{name: name} }
const stateTimeout = 30 * time.Second
func (s KeptState) ask(verb string, params map[string]any) (json.RawMessage, error) {
if err := refusedUnannounced(s.name); err != nil {
return nil, err
}
c, err := live()
if err != nil {
return nil, err
+3
View File
@@ -180,6 +180,9 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
currentMu.Lock()
current = c
currentMu.Unlock()
// What the bundle said it keeps is announced once it serves: the runtime answers the publish on this channel, so
// it is asked from beside the loop that reads the answer.
go func() { _ = announce1() }()
scanner := bufio.NewScanner(in)
scanner.Buffer(make([]byte, 1<<20), 1<<24)
for scanner.Scan() {
@@ -1,5 +1,7 @@
// Package testbus is the bus an integration test of the SDK's Go binding runs against: a server of
// its own, of the release go.mod pins, started for the one test and gone after it.
// its own, of the release go.mod pins, started for the one test and gone after it. It is exported so a
// module's own integration tests start against the same (announcetest, novox/hq ADR 0495 clause 5;
// issue 1182 names the controller's copy).
//
// **Why one per test.** A test that reaches a shared bus asserts, reads and removes objects by fixed
// name — a stream, a consumer — and two tests at once would step on each other's: one deleting what
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) 2024 Antithesis
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-214
View File
@@ -1,214 +0,0 @@
//go:build enable_antithesis_sdk
// Package assert enables defining [test properties] about your program or [workload]. It is part of the [Antithesis Go SDK], which enables Go applications to integrate with the [Antithesis platform].
//
// Code that uses this package should be instrumented with the [antithesis-go-generator] utility. This step is required for the Always, Sometime, and Reachable methods. It is not required for the Unreachable and AlwaysOrUnreachable methods, but it will improve the experience of using them.
//
// These functions are no-ops with minimal performance overhead when called outside of the Antithesis environment. However, if the environment variable ANTITHESIS_SDK_LOCAL_OUTPUT is set, these functions will log to the file pointed to by that variable using a structured JSON format defined [here]. This allows you to make use of the Antithesis assertions package in your regular testing, or even in production. In particular, very few assertions frameworks offer a convenient way to define [Sometimes assertions], but they can be quite useful even outside Antithesis.
//
// Each function in this package takes a parameter called message, which is a human readable identifier used to aggregate assertions. Antithesis generates one test property per unique message and this test property will be named "<message>" in the [triage report].
//
// This test property either passes or fails, which depends upon the evaluation of every assertion that shares its message. Different assertions in different parts of the code should have different message, but the same assertion should always have the same message even if it is moved to a different file.
//
// Each function also takes a parameter called details, which is a key-value map of optional additional information provided by the user to add context for assertion failures. The information that is logged will appear in the [triage report], under the details section of the corresponding property. Normally the values passed to details are evaluated at runtime.
//
// [Antithesis Go SDK]: https://antithesis.com/docs/using_antithesis/sdk/go/
// [Antithesis platform]: https://antithesis.com
// [test properties]: https://antithesis.com/docs/properties_assertions/properties/
// [workload]: https://antithesis.com/docs/test_templates/first_test/
// [antithesis-go-generator]: https://antithesis.com/docs/using_antithesis/sdk/go/instrumentor/
// [triage report]: https://antithesis.com/docs/reports/
// [here]: https://antithesis.com/docs/using_antithesis/sdk/fallback/
// [Sometimes assertions]: https://antithesis.com/docs/best_practices/sometimes_assertions/
//
// [details]: https://antithesis.com/docs/reports/
package assert
import (
"encoding/json"
"fmt"
)
type assertInfo struct {
Location *locationInfo `json:"location"`
Details map[string]any `json:"details"`
AssertType string `json:"assert_type"`
DisplayType string `json:"display_type"`
Message string `json:"message"`
Id string `json:"id"`
Hit bool `json:"hit"`
MustHit bool `json:"must_hit"`
Condition bool `json:"condition"`
}
// Create a custom json marshaler for assertInfo so that we can force Errors to be marshaled with their error details.
// Without this, custom errors are marshaled as an empty object because the default json marshaler doesn't include the error
// (because it's a method - not an exported struct field).
func (f assertInfo) MarshalJSON() ([]byte, error) {
type alias assertInfo // prevent infinite recursion
a := alias(f)
if a.Details != nil {
a.Details = normalizeMap(a.Details)
}
return json.Marshal(a)
}
type jsonError struct {
innerError error
}
func (e jsonError) MarshalJSON() ([]byte, error) {
// Marshal this as the debug output string instead of e.Error(). These should be equivalent, but Sprintf correctly
// handles nil values for us (which otherwise are annoying to defend against due to this - https://go.dev/doc/faq#nil_error)
return json.Marshal(fmt.Sprintf("%+v", e.innerError))
}
// Recursively replace any `error` with jsonError while doing a deep copy.
// Most of the logic is in the normalize method below. This method exists to localize the type assertions
// and provide a function that takes in/out a map instead of any.
func normalizeMap(v map[string]any) map[string]any {
return normalize(v).(map[string]any)
}
func normalize(input any) any {
// This switch will miss some cases (pointers, structs, non-any types), but should catch a very large proportion of real error interfaces
// in real details objects. We can augment this if we find other cases common enough to support.
switch inputTyped := input.(type) {
case error:
// Check if the underlying error implements json.Marshaler, so that if the error
// already knows who to marshal itself, we don't override that.
if _, ok := inputTyped.(json.Marshaler); ok {
return inputTyped
} else {
return jsonError{inputTyped}
}
case map[string]any:
out := make(map[string]any, len(inputTyped))
for k, v := range inputTyped {
out[k] = normalize(v)
}
return out
case []any:
out := make([]any, len(inputTyped))
for i := range inputTyped {
out[i] = normalize(inputTyped[i])
}
return out
default:
return input
}
}
type wrappedAssertInfo struct {
A *assertInfo `json:"antithesis_assert"`
}
// --------------------------------------------------------------------------------
// Assertions
// --------------------------------------------------------------------------------
const (
wasHit = true
mustBeHit = true
optionallyHit = false
expectingTrue = true
)
const (
universalTest = "always"
existentialTest = "sometimes"
reachabilityTest = "reachability"
)
const (
alwaysDisplay = "Always"
alwaysOrUnreachableDisplay = "AlwaysOrUnreachable"
sometimesDisplay = "Sometimes"
reachableDisplay = "Reachable"
unreachableDisplay = "Unreachable"
)
// Always asserts that condition is true every time this function is called, and that it is called at least once. The corresponding test property will be viewable in the Antithesis SDK: Always group of your triage report.
func Always(condition bool, message string, details map[string]any) {
locationInfo := newLocationInfo(offsetAPICaller)
id := makeKey(message, locationInfo)
assertImpl(condition, message, details, locationInfo, wasHit, mustBeHit, universalTest, alwaysDisplay, id)
}
// AlwaysOrUnreachable asserts that condition is true every time this function is called. The corresponding test property will pass if the assertion is never encountered (unlike Always assertion types). This test property will be viewable in the “Antithesis SDK: Always” group of your triage report.
func AlwaysOrUnreachable(condition bool, message string, details map[string]any) {
locationInfo := newLocationInfo(offsetAPICaller)
id := makeKey(message, locationInfo)
assertImpl(condition, message, details, locationInfo, wasHit, optionallyHit, universalTest, alwaysOrUnreachableDisplay, id)
}
// Sometimes asserts that condition is true at least one time that this function was called. (If the assertion is never encountered, the test property will therefore fail.) This test property will be viewable in the “Antithesis SDK: Sometimes” group.
func Sometimes(condition bool, message string, details map[string]any) {
locationInfo := newLocationInfo(offsetAPICaller)
id := makeKey(message, locationInfo)
assertImpl(condition, message, details, locationInfo, wasHit, mustBeHit, existentialTest, sometimesDisplay, id)
}
// Unreachable asserts that a line of code is never reached. The corresponding test property will fail if this function is ever called. (If it is never called the test property will therefore pass.) This test property will be viewable in the “Antithesis SDK: Reachablity assertions” group.
func Unreachable(message string, details map[string]any) {
locationInfo := newLocationInfo(offsetAPICaller)
id := makeKey(message, locationInfo)
assertImpl(false, message, details, locationInfo, wasHit, optionallyHit, reachabilityTest, unreachableDisplay, id)
}
// Reachable asserts that a line of code is reached at least once. The corresponding test property will pass if this function is ever called. (If it is never called the test property will therefore fail.) This test property will be viewable in the “Antithesis SDK: Reachablity assertions” group.
func Reachable(message string, details map[string]any) {
locationInfo := newLocationInfo(offsetAPICaller)
id := makeKey(message, locationInfo)
assertImpl(true, message, details, locationInfo, wasHit, mustBeHit, reachabilityTest, reachableDisplay, id)
}
// AssertRaw is a low-level method designed to be used by third-party frameworks. Regular users of the assert package should not call it.
func AssertRaw(cond bool, message string, details map[string]any,
classname, funcname, filename string, line int,
hit bool, mustHit bool,
assertType string, displayType string,
id string,
) {
assertImpl(cond, message, details,
&locationInfo{classname, funcname, filename, line, columnUnknown},
hit, mustHit,
assertType, displayType,
id)
}
func assertImpl(cond bool, message string, details map[string]any,
loc *locationInfo,
hit bool, mustHit bool,
assertType string, displayType string,
id string,
) {
trackerEntry := assertTracker.getTrackerEntry(id, loc.Filename, loc.Classname)
// Always grab the Filename and Classname captured when the trackerEntry was established
// This provides the consistency needed between instrumentation-time and runtime
if loc.Filename != trackerEntry.Filename {
loc.Filename = trackerEntry.Filename
}
if loc.Classname != trackerEntry.Classname {
loc.Classname = trackerEntry.Classname
}
aI := &assertInfo{
Hit: hit,
MustHit: mustHit,
AssertType: assertType,
DisplayType: displayType,
Message: message,
Condition: cond,
Id: id,
Location: loc,
Details: details,
}
trackerEntry.emit(aI)
}
func makeKey(message string, _ *locationInfo) string {
return message
}
@@ -1,16 +0,0 @@
//go:build !enable_antithesis_sdk
package assert
func Always(condition bool, message string, details map[string]any) {}
func AlwaysOrUnreachable(condition bool, message string, details map[string]any) {}
func Sometimes(condition bool, message string, details map[string]any) {}
func Unreachable(message string, details map[string]any) {}
func Reachable(message string, details map[string]any) {}
func AssertRaw(cond bool, message string, details map[string]any,
classname, funcname, filename string, line int,
hit bool, mustHit bool,
assertType string, displayType string,
id string,
) {
}

Some files were not shown because too many files have changed in this diff Show More