Compare commits

..
Author SHA1 Message Date
jschoubben e53ac973e9 mesh-delivery-gate: stop asking the controller's forget-checks (issue 1318, the gate's part)
The controller's forget-checks goes with the merge gate's checks (ADR 0503 §3)
and 1318 removes the verb in the same group, so the gate must stop calling it
first: a closed pull request is stopped and asks the controller nothing, and
the manifest no longer invokes it. delivery-check stays for hq's base until
R4g (the lead's decision for 1302).
2026-10-22 17:50:44 +02:00
jschoubben f1f18bf2bd mesh-delivery-gate: read mesh/check alone on a base that requires it (issue 1302, S1-S10)
Once main requires mesh/check, the controller publishes no verdict for it and
the gate, which moved deliveries on the controller's checked, would hold every
pull request proposed. A head's mesh/check is now its verdict, from the
forge's status event (gitea now sends the status's words) or its open pull
requests. The Approve is asked only on the checker's verdict at exactly the
heads the ask covers, so a member judged alone never passes its group; a
group on mesh/check asks no composed check and owes no mesh/merge-gate; a
member's mesh/check that changes withdraws the ask. Nothing is asked of the
controller for such a base, and recheck refuses, naming checker_pull. hq's
mesh/repo-check keeps today's path.
2026-10-22 17:50:25 +02:00
jschoubben c41c968086 gitea: judge and write only the statuses a base requires, so merges go on mesh/check (issue 1301, S5-S7)
Once main requires mesh/check, nothing writes mesh/merge-gate, and the merge
tool, which judged on it, would refuse every merge. It now judges on the
status the base requires: mesh/check's line names the trunk it judged, an
out-of-date one is refused and nothing is asked (the checker re-checks on a
trunk move, ADR 0447 §4); hq's mesh/repo-check is judged as before. The
announcer, the verdict, a trunk's move and a build's start write the
controller's two statuses only where a base still requires them, so hq keeps
mesh/repo-check and no stale mesh/merge-gate appears anywhere else.
2026-10-22 17:49:53 +02:00
jschoubben 939475cb63 gitea: only the forge bridge writes mesh/check, through a tool of its own (issue 1301, S4)
Once main requires mesh/check, a hand that set it could pass a merge past the
checker, the gap issue 293 closed for the two statuses before it.
gitea_commit_status now refuses mesh/check in every state, and the new
gitea_check_status writes it alone; the bus grants that tool only to a module
whose manifest invokes it, the forge bridge (the lead's choice A).
2026-10-22 17:49:53 +02:00
jschoubben 5e24deb0f1 gitea: require mesh/check in place of the two old statuses on every main the gate judges (issue 1301, S1-S3)
ADR 0447 §1 has main require mesh/check alone. The repositories program now
requires it in place of mesh/merge-gate and mesh/repo-check on every listed
repository and on every other one whose main requires mesh/merge-gate, so the
switch needs no hand-kept list (lead decisions L4, L11). A main requiring
mesh/repo-check alone (hq) keeps it until issue 1296 (ADR 0503 §5).
2026-10-22 17:49:53 +02:00
121 changed files with 1726 additions and 4871 deletions
+15 -9
View File
@@ -22,16 +22,18 @@ Every new head of an open pull request is announced as `pull.updated`, once, and
minutes, or when the search fails.
- **A pull request nobody checks is said — by the delivery's owner, not here.** `gitea_open_pulls` answers every
open pull request from the forge itself, with the statuses its base requires and the ones its head has. The
`mesh-delivery` module asks it every five minutes, and a head on a base requiring `mesh/merge-gate` or
`mesh/repo-check` with neither for ten minutes since it first saw it so is a delivery held past its bound
delivery gate asks it every five minutes, and a head on a base requiring `mesh/check` (or hq's
`mesh/repo-check`) with none of them for ten minutes since it first saw it so is a delivery held past its bound
(`delivery.<id>.stalled`, state `unannounced`), which the controller raises like any other (issue 347).
Each of the head's statuses comes with its state and when it was set, so a head whose check started and never
answered — a required check pending, or never set beside one that is — is said as a stalled line in the state
`unanswered` once that has lasted the delivery table's bound for a check asked and not answered, an hour, counted
from its first sight and again only when a check is asked anew (issue 438).
- **How to see it work.** A pull request opened more than a minute after its branch was pushed — this fix's own
was one, and the announcer it fixes never saw it — has `mesh/merge-gate` pending within a pass of the next sweep
at the latest, and `mesh-controller.conditions` names none for it.
was one, and the announcer it fixes never saw it — is announced within a pass of the next sweep at the latest, its
`mesh/check` is the checker's to set, and `mesh-controller.conditions` names none for it. (Since issue 1301 this
module writes `mesh/merge-gate` only where a base still requires it, and none does once the forge requires
`mesh/check`.)
- **A pull request or issue number that is not one is refused** before the forge is asked: a tool called with a
misnamed argument sent `pulls/NaN`.
@@ -154,15 +156,19 @@ listed repository:
naming the description, and a `merge-check.sh` declaring `# mesh-check-toolchain: go` that vets and runs the unit
tests, then the integration tests (`-tags integration`), of every Go module in the tree, and says so and passes on
a tree with none. The rule is then narrowed to main's rule of ADR 0398 §2 (no push, only `mesh-tools` merges),
requiring `mesh/repo-check` and `mesh/merge-gate`, and only then is the repository moved into the organisation.
requiring `mesh/check` (ADR 0447 §1, issue 1301), and only then is the repository moved into the organisation.
There main's rule is read back and put right once more (the step `rule-after-move`): the forge drops from a merge
list a user without access to the repository, and under the setup account `mesh-tools` has none;
- one found still under the setup account is finished: the steps it lacks are done, main is never written twice;
- one in the organisation is only put right: main's rule narrowed, a missing one of the two statuses added and every
status already there kept.
- one in the organisation is only put right: main's rule narrowed, `mesh/check` required in place of
`mesh/merge-gate` and `mesh/repo-check`, and every other status already there kept. A main that requires
`mesh/repo-check` without `mesh/merge-gate` is judged by its own script alone (hq, ADR 0503 §5, until issue 1296)
and keeps it; `mesh/check` is never dropped.
A step the forge refuses stops that repository, and the others go on. A repository taken out of the value is never
asked about: nothing is deleted, renamed, moved out or unprotected. The pass keeps what it did per repository in
Every other repository of the organisation whose main requires `mesh/merge-gate` has the two old statuses swapped
for `mesh/check`, and only its statuses (issue 1301): the switch reaches every repository the merge gate judged, with
no list kept by hand. A step the forge refuses stops that repository, and the others go on. Beyond that swap, a
repository taken out of the value is never changed: nothing is deleted, renamed, moved out or unprotected. The pass keeps what it did per repository in
`repositories-outcome.json`.
**How a refusal is seen.** `repositories sync` is a run-once process, declared as the module's last resource, so
@@ -0,0 +1,215 @@
package main
import (
"strings"
"testing"
)
// The tests of issue 1301 (ADR 0447 §1, §5; ADR 0503 §3, §5; lead decisions L4 and L11): main requires `mesh/check`
// alone in place of `mesh/merge-gate` and `mesh/repo-check`; a main that requires `mesh/repo-check` without
// `mesh/merge-gate` (hq) keeps it; every repository of the organisation whose main requires `mesh/merge-gate` is
// switched, listed in the value or not; no other status is dropped.
func narrowedRule(contexts ...any) map[string]any {
return map[string]any{
"enable_push": false, "enable_push_whitelist": false, "push_whitelist_usernames": []any{},
"enable_merge_whitelist": true, "merge_whitelist_usernames": []any{tools},
"enable_status_check": true, "status_check_contexts": contexts,
"block_admin_merge_override": true,
}
}
// 1301-S1: a listed repository whose main requires the two old statuses is put right to `mesh/check` in their place,
// every other status kept; a second pass changes nothing.
func TestListedRepositorySwapsTheTwoStatusesForCheck(t *testing.T) {
f := newFake()
repo := f.add("novox/mesh-delivery", map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate", "mesh/repo-check", "other/x")}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"mesh-delivery": {}})
got := strs(repo.Rules[mainRule]["status_check_contexts"])
if strings.Join(got, ",") != "other/x,mesh/check" {
t.Errorf("main requires %v, want other/x and mesh/check", got)
}
if r := result(o, "mesh-delivery"); r.Did != DidPutRight {
t.Errorf("outcome %+v, want put right", r)
}
before := len(f.writes())
run(t, s, Value{"mesh-delivery": {}})
if w := f.writes(); len(w) != before {
t.Errorf("a second pass wrote %v", w[before:])
}
}
// 1301-S1: a repository the program makes requires `mesh/check` alone from its first rule in the organisation on.
func TestNewRepositoryRequiresCheckAlone(t *testing.T) {
f := newFake()
s, _ := syncer(t, f)
run(t, s, Value{"fresh": {Description: "d"}})
rule := f.repos["novox/fresh"].Rules[mainRule]
if got := strs(rule["status_check_contexts"]); rule["enable_status_check"] != true || strings.Join(got, ",") != CheckStatus {
t.Errorf("a new repository's main requires %v, want %s alone", got, CheckStatus)
}
if readme := f.repos["novox/fresh"].Files["README.md"]; !strings.Contains(readme, CheckStatus) ||
strings.Contains(readme, "mesh/merge-gate") || strings.Contains(readme, "mesh/repo-check") {
t.Errorf("the README names the statuses of before: %q", readme)
}
}
// 1301-S1: mesh/check is never dropped, and a rule that requires it beside a retired status loses the retired one.
func TestCheckIsKeptAndARetiredStatusBesideItGoes(t *testing.T) {
f := newFake()
both := f.add("novox/both", map[string]map[string]any{mainRule: narrowedRule("mesh/check", "mesh/repo-check")}, "main")
right := f.add("novox/right", map[string]map[string]any{mainRule: narrowedRule("mesh/check")}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"both": {}, "right": {}})
if got := strs(both.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("both requires %v, want %s alone", got, CheckStatus)
}
if got := strs(right.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("right requires %v", got)
}
if result(o, "right").Did != DidUnchanged || result(o, "both").Did != DidPutRight {
t.Errorf("outcome %+v", o)
}
}
// 1301-S1: a listed repository whose main requires nothing is given mesh/check, not the old two.
func TestListedRuleRequiringNothingGetsCheck(t *testing.T) {
f := newFake()
none := f.add("novox/none", map[string]map[string]any{mainRule: {"enable_push": false}}, "main")
s, _ := syncer(t, f)
run(t, s, Value{"none": {}})
if got := strs(none.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("none requires %v, want %s", got, CheckStatus)
}
}
// 1301-S2 (ADR 0503 §5): a main that requires mesh/repo-check without mesh/merge-gate is judged by its own script
// alone (hq): it keeps mesh/repo-check and is not given mesh/check, listed or not.
func TestOwnCheckAloneIsKept(t *testing.T) {
f := newFake()
hq := f.add("novox/hq", map[string]map[string]any{mainRule: narrowedRule("mesh/repo-check")}, "main")
lab := f.add("novox/mesh-lab", map[string]map[string]any{mainRule: narrowedRule("mesh/repo-check")}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"hq": {}})
for name, repo := range map[string]*fRepo{"hq": hq, "mesh-lab": lab} {
if got := strs(repo.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != "mesh/repo-check" {
t.Errorf("%s requires %v, want mesh/repo-check alone", name, got)
}
}
if result(o, "hq").Did != DidUnchanged {
t.Errorf("hq outcome %+v, want unchanged", result(o, "hq"))
}
for _, w := range f.writes() {
t.Errorf("a repository judged by its own check alone was written: %s", w)
}
}
// 1301-S3 (L4, L11): a repository the value does not list whose main requires mesh/merge-gate has the two old
// statuses swapped for mesh/check, and only its statuses: nothing else of its rule is sent. One that requires
// neither, or mesh/repo-check alone, is not written to.
func TestUnlistedRepositoryRequiringTheGateIsSwitched(t *testing.T) {
f := newFake()
catalog := f.add("novox/mesh-catalog", map[string]map[string]any{mainRule: {
"enable_push": true, "enable_status_check": true, "status_check_contexts": []any{"mesh/merge-gate", "mesh/repo-check"},
}}, "main")
toolsGo := f.add("novox/mesh-tools-go", map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate")}, "main")
f.add("novox/plain", map[string]map[string]any{mainRule: {"enable_push": false}}, "main")
f.add("novox/hq", map[string]map[string]any{mainRule: narrowedRule("mesh/repo-check")}, "main")
f.add("novox/norule", nil, "main")
f.add("mesh-admin/elsewhere", map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate")}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{})
if got := strs(catalog.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("mesh-catalog requires %v, want %s", got, CheckStatus)
}
if catalog.Rules[mainRule]["enable_push"] != true {
t.Errorf("an unlisted repository's rule was narrowed beyond its statuses: %v", catalog.Rules[mainRule])
}
if got := strs(toolsGo.Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("mesh-tools-go requires %v, want %s", got, CheckStatus)
}
want := []string{
"PATCH /api/v1/repos/novox/mesh-catalog/branch_protections/main",
"PATCH /api/v1/repos/novox/mesh-tools-go/branch_protections/main",
}
if got := f.writes(); strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("writes:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
for _, n := range []string{"mesh-catalog", "mesh-tools-go"} {
if r := result(o, n); r.Did != DidPutRight {
t.Errorf("%s outcome %+v, want put right", n, r)
}
}
for _, n := range []string{"plain", "hq", "norule", "elsewhere"} {
if r := result(o, n); r.Name != "" {
t.Errorf("%s is in the outcome though nothing was done: %+v", n, r)
}
}
before := len(f.writes())
run(t, s, Value{})
if w := f.writes(); len(w) != before {
t.Errorf("a second pass wrote %v", w[before:])
}
}
// 1301-S3: the organisation's repositories are read page by page, so one past the first page is switched too.
func TestUnlistedRepositoriesAreReadPastTheFirstPage(t *testing.T) {
f := newFake()
f.pageSize = 2
for _, n := range []string{"a", "b", "c", "d", "e"} {
f.add("novox/"+n, map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate", "mesh/repo-check")}, "main")
}
s, _ := syncer(t, f)
run(t, s, Value{})
for _, n := range []string{"a", "b", "c", "d", "e"} {
if got := strs(f.repos["novox/"+n].Rules[mainRule]["status_check_contexts"]); strings.Join(got, ",") != CheckStatus {
t.Errorf("%s requires %v", n, got)
}
}
}
// 1301-S3: a refused swap is said with the repository and its step, and the pass goes on to the next.
func TestRefusedSwapIsSaid(t *testing.T) {
f := newFake()
f.add("novox/a", map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate")}, "main")
f.add("novox/b", map[string]map[string]any{mainRule: narrowedRule("mesh/merge-gate")}, "main")
f.refuse = func(method, path string) (int, string) {
if method == "PATCH" && strings.Contains(path, "/novox/a/") {
return 403, "no"
}
return 0, ""
}
s, _ := syncer(t, f)
o := run(t, s, Value{})
if r := result(o, "a"); r.Did != DidRefused || r.Step != StepSwap {
t.Errorf("a: %+v, want refused at %s", r, StepSwap)
}
if r := result(o, "b"); r.Did != DidPutRight {
t.Errorf("b: %+v, want put right", r)
}
}
// 1301-S3: the organisation's list refused is said, and the listed repositories are still done.
func TestOrganisationListRefusedIsSaid(t *testing.T) {
f := newFake()
f.refuse = func(method, path string) (int, string) {
if strings.HasPrefix(path, "/api/v1/orgs/") {
return 500, "down"
}
return 0, ""
}
s, _ := syncer(t, f)
o := run(t, s, Value{"x": {}})
if result(o, "x").Did != DidMade {
t.Errorf("x: %+v", result(o, "x"))
}
refusedList := false
for _, r := range o.Refused() {
refusedList = refusedList || r.Step == StepList
}
if !refusedList {
t.Errorf("the refused list is not in the outcome: %+v", o)
}
}
+29 -9
View File
@@ -11,6 +11,7 @@ import (
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"testing"
@@ -43,6 +44,8 @@ type fakeForge struct {
calls []string // "METHOD /path"
// refuse answers a call with a status and a message when it returns a status above zero.
refuse func(method, path string) (int, string)
// pageSize is how many repositories one page of the organisation's list holds; 50 when zero.
pageSize int
}
func newFake() *fakeForge { return &fakeForge{repos: map[string]*fRepo{}} }
@@ -84,6 +87,7 @@ var (
branchPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/branches/([^/]+)$`)
filesPath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/contents$`)
movePath = regexp.MustCompile(`^/api/v1/repos/([^/]+)/([^/]+)/transfer$`)
orgRepos = regexp.MustCompile(`^/api/v1/orgs/([^/]+)/repos$`)
)
func strs(v any) []string {
@@ -167,6 +171,31 @@ func (f *fakeForge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch {
case p == "/api/v1/version":
reply(200, map[string]string{"version": "1.27"})
case orgRepos.MatchString(p) && r.Method == http.MethodGet:
owner := orgRepos.FindStringSubmatch(p)[1]
var names []string
for full := range f.repos {
if o, n, _ := strings.Cut(full, "/"); o == owner {
names = append(names, n)
}
}
sort.Strings(names)
size := f.pageSize
if size == 0 {
size = 50
}
if l, _ := strconv.Atoi(r.URL.Query().Get("limit")); l > 0 && l < size {
size = l
}
page, _ := strconv.Atoi(r.URL.Query().Get("page"))
if page < 1 {
page = 1
}
out := []map[string]any{}
for i := (page - 1) * size; i < len(names) && i < page*size; i++ {
out = append(out, map[string]any{"name": names[i], "full_name": owner + "/" + names[i]})
}
reply(200, out)
case p == "/api/v1/user/repos" && r.Method == http.MethodPost:
name := fmt.Sprint(in["name"])
if f.repos[admin+"/"+name] != nil {
@@ -312,12 +341,3 @@ func result(o Outcome, name string) Result {
}
return Result{}
}
func has(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
+3 -3
View File
@@ -8,7 +8,7 @@ import "strings"
const MergeCheck = `#!/bin/sh
# mesh-check-toolchain: go
#
# This repository's own check, mesh/repo-check (novox/hq ADR 0238 §4), written with its first commit (ADR 0428 §4):
# This repository's own check (novox/hq ADR 0238 §4), run by the checker within mesh/check (ADR 0447), written with its first commit (ADR 0428 §4):
# every Go module in the tree is vetted, its unit tests run, and then its integration tests (-tags integration).
# A tree with no Go module has nothing to test, and the check says so and passes.
set -eu
@@ -37,7 +37,7 @@ func FirstCommit(name string, r Repo) map[string]string {
}
readme := "# " + name + "\n\n" + purpose + "\n\n" +
"Made by the mesh from the gitea module's config value `repositories` (novox/hq ADR 0428). Main takes no push: " +
"a change reaches it by a pull request whose `mesh/repo-check` (this repository's `merge-check.sh`) and " +
"`mesh/merge-gate` passed.\n"
"a change reaches it by a pull request whose `mesh/check` passed: the checker's verdict on its head merged onto " +
"main, this repository's `merge-check.sh` among its tests (novox/hq ADR 0447).\n"
return map[string]string{"README.md": readme, "merge-check.sh": MergeCheck}
}
+2 -1
View File
@@ -3,7 +3,8 @@
// again whenever the value changes: it reads the file the mesh renders from that value and, with the setup login,
// makes every listed repository that does not exist under that login's own account, protects its main, writes
// main's first commit, narrows main's rule and only then moves it into the organisation; one that exists only has
// its main rule put right. A repository the value does not list is never touched. It keeps what it did, per
// its main rule put right, requiring mesh/check in place of the two statuses it replaces (issue 1301). A repository
// the value does not list is touched only to swap those two for mesh/check when its main requires mesh/merge-gate. It keeps what it did, per
// repository, in the outcome file, says each change on stdout in one line and never a secret, and exits 1 when a
// step was refused, its last stderr lines naming each (outcome.go Report): the node-engine's gate on a run-once
// step (ADR 0136) reports it and runs it again on every apply until it passes.
@@ -18,7 +18,7 @@ func TestNewRepositoryRefusesTheAdministratorsOverride(t *testing.T) {
}
// 1141/AC2: a narrowed main rule that still lets an administrator override is not right: it is put right, its
// statuses kept, and a second pass changes nothing.
// statuses kept (the two old ones as mesh/check, issue 1301), and a second pass changes nothing.
func TestNarrowedRuleLettingTheAdministratorOverrideIsPutRight(t *testing.T) {
f := newFake()
repo := f.add("novox/mesh-delivery", map[string]map[string]any{mainRule: {
@@ -33,10 +33,8 @@ func TestNarrowedRuleLettingTheAdministratorOverrideIsPutRight(t *testing.T) {
if rule["block_admin_merge_override"] != true {
t.Errorf("the administrator's override is still allowed: %v", rule)
}
for _, c := range []string{"mesh/merge-gate", "mesh/repo-check", "other/x"} {
if !has(strs(rule["status_check_contexts"]), c) {
t.Errorf("a status was dropped: %v", rule["status_check_contexts"])
}
if c := strs(rule["status_check_contexts"]); strings.Join(c, ",") != "other/x,mesh/check" {
t.Errorf("statuses %v, want other/x kept and mesh/check for the two it replaces (issue 1301)", c)
}
if r := result(o, "mesh-delivery"); r.Did != DidPutRight {
t.Errorf("outcome %+v, want put right", r)
+126 -28
View File
@@ -26,9 +26,47 @@ const (
MainBranch = "main"
)
// RequiredStatuses are what main requires in every listed repository from its first commit (ADR 0428 §5): the
// repository's own check and the module graph's gate. A status already on the rule is kept beside them.
var RequiredStatuses = []string{"mesh/merge-gate", "mesh/repo-check"}
// CheckStatus is the one status main requires (ADR 0447 §1, issue 1301): the checker's verdict on a pull request's
// head merged onto its trunk's head. It replaces GateStatus and OwnCheckStatus, and is never dropped.
const CheckStatus = "mesh/check"
// GateStatus and OwnCheckStatus are the two statuses CheckStatus replaces: the controller's merge gate and the
// repository's own merge-check.sh (ADR 0238 §4, ADR 0428 §5 before ADR 0447 §1).
const (
GateStatus = "mesh/merge-gate"
OwnCheckStatus = "mesh/repo-check"
)
// wanted is what a main rule requiring have should require: have with GateStatus and OwnCheckStatus replaced by
// CheckStatus, every other status kept in its order (ADR 0447 §1). A rule requiring OwnCheckStatus with neither
// GateStatus nor CheckStatus is judged by its own script alone, as hq is until issue 1296 settles how hq is
// checked (ADR 0503 §5): the checker writes no status there (ADR 0484 §8), so it is left as it is. This reads the
// rule, not a list of names (lead decision L11 of R4e).
func wanted(have []string) []string {
if !has(have, GateStatus) && !has(have, CheckStatus) && has(have, OwnCheckStatus) {
return have
}
out := []string{}
for _, s := range have {
if s != GateStatus && s != OwnCheckStatus && s != CheckStatus {
out = append(out, s)
}
}
return append(out, CheckStatus)
}
func has(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
func sameList(a, b []string) bool {
return strings.Join(a, "\x00") == strings.Join(b, "\x00")
}
// The steps of one repository, as an outcome names a refused one (ADR 0428 §6).
const (
@@ -42,6 +80,10 @@ const (
// StepRuleAfterMove reads main's rule back once the repository is in the organisation and puts it right: the
// forge drops from a merge whitelist a user without access, and under the setup account the tools' have none.
StepRuleAfterMove = "rule-after-move"
// StepList reads the organisation's repositories, for those the value does not list (issue 1301).
StepList = "list"
// StepSwap swaps the two old statuses for CheckStatus on a repository the value does not list (issue 1301).
StepSwap = "swap"
)
// What a pass did with one repository.
@@ -157,9 +199,73 @@ func (s *Syncer) Sync(ctx context.Context, v Value) (Outcome, error) {
}
out.Repositories = append(out.Repositories, r)
}
for _, r := range s.switchUnlisted(ctx, v) {
if r.Did == DidRefused {
s.say("refused: %s/%s at the step %s: %s", s.Org, r.Name, r.Step, r.Said)
} else {
s.say("%s: %s/%s (%s in place of %s and %s)", r.Did, s.Org, r.Name, CheckStatus, GateStatus, OwnCheckStatus)
}
out.Repositories = append(out.Repositories, r)
}
return out, nil
}
// orgPage is how many repositories one read of the organisation's list asks for.
const orgPage = 50
// switchUnlisted swaps the two old statuses for CheckStatus on main's rule of every repository of the organisation
// the value does not list whose main requires GateStatus (issue 1301, lead decisions L4 and L11 of R4e): the
// repositories the merge gate judges are switched in the delivery that has the checker write mesh/check (ADR 0447
// §5), with no list kept by hand. Only the statuses of a rule covering main are sent; nothing else of a rule the
// value does not hold is changed (ADR 0428 §7 for all of it but the statuses the mesh itself requires). A main that
// does not require GateStatus gets no call. It answers only what it did: a repository switched, or refused.
func (s *Syncer) switchUnlisted(ctx context.Context, v Value) []Result {
var names []string
for page := 1; ; page++ {
var repos []struct {
Name string `json:"name"`
}
q := "?limit=" + fmt.Sprint(orgPage) + "&page=" + fmt.Sprint(page)
if err := s.Forge.GetJSON(ctx, "/api/v1/orgs/"+url.PathEscape(s.Org)+"/repos"+q, &repos); err != nil {
return []Result{refused("(the organisation's repositories)", StepList, err)}
}
for _, r := range repos {
if _, listed := v[r.Name]; !listed && r.Name != "" {
names = append(names, r.Name)
}
}
if len(repos) == 0 || page > 1000 {
break
}
}
sort.Strings(names)
var out []Result
for _, n := range names {
rules, err := s.rules(ctx, s.Org, n)
if err != nil {
out = append(out, refused(n, StepRead, err))
continue
}
var did *Result
for _, r := range rules {
if ok, _ := path.Match(ruleName(r), MainBranch); !ok || !has(required(r), GateStatus) {
continue
}
body := map[string]any{"enable_status_check": true, "status_check_contexts": wanted(required(r))}
if err := s.Forge.SendJSON(ctx, http.MethodPatch,
repoAPI(s.Org, n)+"/branch_protections/"+url.PathEscape(ruleName(r)), body, nil); err != nil {
did = &Result{Name: n, Did: DidRefused, Step: StepSwap, Said: err.Error()}
break
}
did = &Result{Name: n, Did: DidPutRight}
}
if did != nil {
out = append(out, *did)
}
}
return out
}
// waitForForge is cmd/agent-teams' (teams.go waitForForge): the forge starting with the module answers late.
func (s *Syncer) waitForForge(ctx context.Context) error {
deadline := time.Now().Add(s.WaitFor)
@@ -309,7 +415,7 @@ func strList(v any) []string {
return out
}
// statuses is what a rule requires, with RequiredStatuses added and none dropped.
// statuses is what the rules require together, as wanted makes it.
func statuses(rules ...rule) []string {
seen := map[string]bool{}
var out []string
@@ -320,16 +426,19 @@ func statuses(rules ...rule) []string {
}
}
for _, r := range rules {
if r != nil && r["enable_status_check"] == true {
for _, s := range strList(r["status_check_contexts"]) {
add(s)
}
for _, s := range required(r) {
add(s)
}
}
for _, s := range RequiredStatuses {
add(s)
return wanted(out)
}
// required is what a rule requires: its statuses when it checks them, else none.
func required(r rule) []string {
if r == nil || r["enable_status_check"] != true {
return nil
}
return out
return strList(r["status_check_contexts"])
}
// narrowed is main's rule of ADR 0398 §2 (protection.ts mainRuleBody): no push, only the tools' account merges,
@@ -344,7 +453,7 @@ func narrowed(r rule, toolsAccount string, also ...rule) map[string]any {
}
}
// isRight says whether a rule already is narrowed and requires both statuses.
// isRight says whether a rule already is narrowed and requires what wanted makes of it.
func isRight(r rule, toolsAccount string) bool {
if r["enable_push"] != false || r["enable_push_whitelist"] == true || r["enable_merge_whitelist"] != true {
return false
@@ -352,26 +461,15 @@ func isRight(r rule, toolsAccount string) bool {
if m := strList(r["merge_whitelist_usernames"]); len(m) != 1 || m[0] != toolsAccount {
return false
}
return r["enable_status_check"] == true && hasAll(strList(r["status_check_contexts"])) &&
have := strList(r["status_check_contexts"])
return r["enable_status_check"] == true && len(have) > 0 && sameList(have, wanted(have)) &&
r["block_admin_merge_override"] == true
}
func hasAll(have []string) bool {
for _, want := range RequiredStatuses {
found := false
for _, h := range have {
found = found || h == want
}
if !found {
return false
}
}
return true
}
// putRight makes main's rule in an organisation's repository the narrowed one, keeping every status it requires.
// putRight makes main's rule in an organisation's repository the narrowed one, keeping every status it requires but
// the two CheckStatus replaces (ADR 0447 §1).
// A repository with no rule named main gets one, carrying what a glob rule covering main asks (client.ts
// ensureMainMerges, issue 948). It never takes a status away and never unprotects (ADR 0428 §5, §7).
// ensureMainMerges, issue 948). It takes away no other status and never unprotects (ADR 0428 §5, §7).
func (s *Syncer) putRight(ctx context.Context, owner, name string) (bool, error) {
rules, err := s.rules(ctx, owner, name)
if err != nil {
@@ -17,8 +17,8 @@ import (
const mainRule = "main"
// 1124/AC2: a listed repository that does not exist is made under the setup account, given main's rule letting that
// login alone push, given its first commit, narrowed to no push and the tools' merge requiring mesh/repo-check (and
// mesh/merge-gate, 1124/AC5), and only then moved into the organisation. Nothing reaches the organisation before the
// login alone push, given its first commit, narrowed to no push and the tools' merge requiring mesh/check (1124/AC5
// as issue 1301 changes it), and only then moved into the organisation. Nothing reaches the organisation before the
// narrowed rule, and nothing else is pushed to main.
func TestNewRepositoryIsMadeInOrder(t *testing.T) {
f := newFake()
@@ -67,8 +67,8 @@ func TestNewRepositoryIsMadeInOrder(t *testing.T) {
t.Errorf("main's merge whitelist is not the tools' account alone: %v", rule)
}
checks := strs(rule["status_check_contexts"])
if rule["enable_status_check"] != true || !has(checks, "mesh/repo-check") || !has(checks, "mesh/merge-gate") {
t.Errorf("main does not require mesh/repo-check and mesh/merge-gate (1124/AC5): %v", rule)
if rule["enable_status_check"] != true || strings.Join(checks, ",") != CheckStatus {
t.Errorf("main does not require mesh/check alone (1124/AC5, issue 1301): %v", rule)
}
if r := result(o, "mesh-delivery"); r.Did != DidMade || r.Step != "" {
t.Errorf("outcome %+v, want made", r)
@@ -87,7 +87,7 @@ func TestNewRepositoryIsMadeInOrder(t *testing.T) {
// 1124/AC2: the forge drops from a merge whitelist a user without access to the repository, and under the setup
// account the tools' account has none; so once the repository is in the organisation its main rule is read back and
// put right: the tools' account the only one that merges, both statuses required.
// put right: the tools' account the only one that merges, mesh/check required.
func TestOrganisationRuleListsTheToolsAfterTheMove(t *testing.T) {
f := newFake()
s, _ := syncer(t, f)
@@ -104,8 +104,8 @@ func TestOrganisationRuleListsTheToolsAfterTheMove(t *testing.T) {
t.Errorf("after the move main takes a push: %v", rule)
}
checks := strs(rule["status_check_contexts"])
if rule["enable_status_check"] != true || !has(checks, "mesh/repo-check") || !has(checks, "mesh/merge-gate") {
t.Errorf("after the move main does not require both statuses: %v", rule)
if rule["enable_status_check"] != true || strings.Join(checks, ",") != CheckStatus {
t.Errorf("after the move main does not require mesh/check alone: %v", rule)
}
if r := result(o, "mesh-delivery"); r.Did != DidMade {
t.Errorf("outcome %+v, want made", r)
@@ -136,7 +136,8 @@ func TestAdminRuleLetsTheSetupLoginAlonePush(t *testing.T) {
}
// 1124/AC3: a listed repository that exists in the organisation is not made again, only its main rule put right;
// a rule still letting the setup login push is narrowed, its statuses kept (1124/AC5).
// a rule still letting the setup login push is narrowed, its statuses kept but the two mesh/check replaces (1124/AC5,
// issue 1301).
func TestExistingOrganisationRepositoryIsOnlyPutRight(t *testing.T) {
f := newFake()
hq := f.add("novox/hq", map[string]map[string]any{mainRule: {
@@ -157,14 +158,11 @@ func TestExistingOrganisationRepositoryIsOnlyPutRight(t *testing.T) {
t.Errorf("hq's rule not narrowed: %v", rule)
}
checks := strs(rule["status_check_contexts"])
for _, c := range []string{"other/x", "mesh/merge-gate", "mesh/repo-check"} {
if !has(checks, c) {
t.Errorf("hq's rule lacks %s: %v", c, checks)
}
if strings.Join(checks, ",") != "other/x,mesh/check" {
t.Errorf("hq's rule requires %v, want other/x and mesh/check", checks)
}
made := bare.Rules[mainRule]
if made == nil || made["enable_push"] != false || !has(strs(made["status_check_contexts"]), "mesh/repo-check") ||
!has(strs(made["status_check_contexts"]), "mesh/merge-gate") {
if made == nil || made["enable_push"] != false || strings.Join(strs(made["status_check_contexts"]), ",") != CheckStatus {
t.Errorf("bare's rule: %v", made)
}
if hq.Commits != 0 || bare.Commits != 0 {
@@ -214,8 +212,8 @@ func TestRepositoryUnderTheSetupAccountIsFinished(t *testing.T) {
}
}
// 1124/AC5: putting a rule right adds a missing one of mesh/repo-check and mesh/merge-gate and keeps every status
// already there; no pass drops one.
// 1124/AC5 as issue 1301 changes it: putting a rule right adds mesh/check when the rule requires mesh/merge-gate,
// and keeps every other status there; a rule requiring mesh/repo-check alone keeps it (ADR 0503 §5).
func TestStatusesAreAddedAndNeverDropped(t *testing.T) {
f := newFake()
only := f.add("novox/only", map[string]map[string]any{mainRule: {
@@ -224,15 +222,15 @@ func TestStatusesAreAddedAndNeverDropped(t *testing.T) {
}}, "main")
more := f.add("novox/more", map[string]map[string]any{mainRule: {
"enable_push": false, "enable_merge_whitelist": true, "merge_whitelist_usernames": []any{tools},
"enable_status_check": true, "status_check_contexts": []any{"mesh/merge-gate", "mesh/repo-check", "mesh/delivery"},
"enable_status_check": true, "status_check_contexts": []any{"mesh/delivery", "mesh/check"},
"block_admin_merge_override": true, // right since issue 1141
}}, "main")
s, _ := syncer(t, f)
o := run(t, s, Value{"only": {}, "more": {}})
if c := strs(only.Rules[mainRule]["status_check_contexts"]); !has(c, "mesh/merge-gate") || !has(c, "mesh/repo-check") {
if c := strs(only.Rules[mainRule]["status_check_contexts"]); strings.Join(c, ",") != "mesh/repo-check" {
t.Errorf("only: %v", c)
}
if c := strs(more.Rules[mainRule]["status_check_contexts"]); len(c) != 3 || !has(c, "mesh/delivery") {
if c := strs(more.Rules[mainRule]["status_check_contexts"]); len(c) != 2 || !has(c, "mesh/delivery") {
t.Errorf("more lost a status: %v", c)
}
for _, w := range f.writes() {
@@ -314,7 +312,8 @@ func TestRefusedStepStopsThatRepositoryOnly(t *testing.T) {
}
}
// 1124/AC7: a repository taken out of the value gets no call at all.
// 1124/AC7: a repository taken out of the value is not changed: its rules are read (issue 1301 switches one whose
// main requires mesh/merge-gate), nothing is written, and one still under the setup account is not asked about.
func TestRepositoryOutOfTheValueIsNotTouched(t *testing.T) {
f := newFake()
f.add("novox/old", map[string]map[string]any{mainRule: {"enable_push": true}}, "main")
@@ -323,7 +322,7 @@ func TestRepositoryOutOfTheValueIsNotTouched(t *testing.T) {
run(t, s, Value{})
run(t, s, Value{"new": {}})
for _, c := range f.all() {
if strings.Contains(c, "old") {
if strings.Contains(c, "older") || (strings.Contains(c, "old") && !strings.HasPrefix(c, "GET ")) {
t.Errorf("a repository out of the value was asked: %s", c)
}
}
+56 -1
View File
@@ -16,6 +16,9 @@ import { promisify } from "node:util";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus } from "./client.js";
import { checkName } from "./guard.js";
import { CHECK_STATUS } from "./pulls.js";
export { CHECK_STATUS };
/** How a command is run: docker on the forge's machine, or a test's. */
export type Runner = (file: string, args: string[]) => Promise<{ stdout: string; code: number }>;
@@ -91,11 +94,18 @@ const states = new Set(["pending", "success", "error", "failure", "warning"]);
/** The merge check's contexts (pulls.ts): a branch's protection may require them. */
const mergeCheckContexts = new Set(["mesh/merge-gate", "mesh/repo-check"]);
/** A status mesh-delivery asks for, checked: one of the forge's states, a context of the mesh's own, a
* bounded description. */
export function deliveryStatus(context: string, state: string, description: string, target?: string) {
if (!/^mesh\/[a-z-]+$/.test(context)) throw new Error(`${context} is not a status of the mesh's own`);
if (!states.has(state)) throw new Error(`${state} is not a status the forge keeps`);
// mesh/check is the checker's verdict and the one status main requires (ADR 0447 §1): no hand sets it in any
// state, as issue 293 closed for the two before it. The forge bridge writes it through gitea_check_status, a tool
// only its account may call (issue 1301).
if (context === CHECK_STATUS) {
throw new Error(`${CHECK_STATUS} is the checker's status: the forge bridge writes it through gitea_check_status, never this tool`);
}
// The merge check's statuses are its verdict's, set when the controller says it, and a branch may require
// them: never set to a verdict by hand, so nothing passes — or fails — a merge past the check (novox/hq
// issue 293). Pending is no verdict: it is what this holder says on every new head until the verdict comes,
@@ -126,7 +136,7 @@ export interface StatusWriters {
export function commitStatusTool(writers: StatusWriters): ToolDefinition {
return {
name: "gitea_commit_status",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. The merge check's statuses (mesh/merge-gate, mesh/repo-check) take pending only — a recheck putting them back until the fresh verdict sets them — on the head's whole sha, in their turn after the check's earlier writes.",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. mesh/check, the status main requires, is refused: the checker's forge bridge writes it through gitea_check_status. hq's mesh/repo-check (and mesh/merge-gate, kept for a revert) take pending only — a recheck putting them back until the fresh verdict sets them — on the head's whole sha, in their turn after the check's earlier writes.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
@@ -153,3 +163,48 @@ export function commitStatusTool(writers: StatusWriters): ToolDefinition {
},
};
}
/** What the check's status tool writes with: the forge. */
export interface CheckStatusWriter {
setCommitStatus(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void>;
}
/**
* The `gitea_check_status` tool (novox/hq issue 1301, ADR 0447 §1): the one way `mesh/check` is written. Its arguments
* are gitea_commit_status's, so the forge bridge sends an owed status's arguments unchanged to it: owner, repo, sha
* (the head's whole sha), state, description, target_url, and context, which may be left out and is mesh/check when
* given. **Who may write it is the bus's grant**: the runtime tells a tool no caller, so only a module that lists
* `gitea.gitea_check_status` in its manifest's `invokes` may call it, and the forge bridge alone does. Served beside
* gitea_commit_status, from the watcher's bundle.
*/
export function checkStatusTool(writer: CheckStatusWriter): ToolDefinition {
return {
name: "gitea_check_status",
description: "Set mesh/check, the checker's verdict and the one status main requires, on a pull request's head: pending, success, error, failure or warning, a short description, and the page it links to. The checker's forge bridge alone may call it (its manifest's invokes); no other status is written here, and no other tool writes this one.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the head's whole sha" },
context: { type: "string", description: "mesh/check, or left out" },
state: { type: "string", description: "pending | success | error | failure | warning" },
description: { type: "string", description: "one line" },
target_url: { type: "string", description: "the page it links to (the check's report)" },
},
run: async (args) => {
const owner = checkName("owner", args.owner), repo = checkName("name", args.repo), sha = String(args.sha ?? "");
if (args.context !== undefined && args.context !== null && String(args.context) !== CHECK_STATUS) {
throw new Error(`${JSON.stringify(args.context)}: this tool writes ${CHECK_STATUS} alone. Nothing was sent`);
}
if (!/^([0-9a-f]{40}|[0-9a-f]{64})$/i.test(sha)) {
throw new Error(`${JSON.stringify(args.sha)} is not a head's whole sha: ${CHECK_STATUS} names the head whole. Nothing was sent`);
}
// deliveryStatus checks the state and cuts the description; it refuses mesh/check, so it is asked for a
// context of the mesh's own and the status is named mesh/check after.
const checked = deliveryStatus("mesh/delivery", String(args.state), String(args.description ?? ""),
args.target_url ? String(args.target_url) : undefined);
const status = { ...checked, context: CHECK_STATUS };
await writer.setCommitStatus(owner, repo, sha, status);
return { set: status };
},
};
}
+2 -1
View File
@@ -212,7 +212,8 @@ export class GateWriter {
let since = h.pendingAt;
if (since === undefined) {
const gate = await this.forge.gateOf(owner, repo, sha);
if (gate && gate.state !== "pending") return;
// No gate on the forge: none was written (its base does not require it, issue 1301), and nothing is said.
if (!gate || gate.state !== "pending") return;
since = gate?.updated_at ? Date.parse(gate.updated_at) : undefined;
}
if (since !== undefined && at.getTime() < since - CLOCK_SLACK_MS) return;
+22 -11
View File
@@ -12,7 +12,9 @@
// Consumes mesh-controller.checked — a pull request's merge check, judged — and sets it as the head
// commit's statuses: `mesh/merge-gate`, the modules of the mesh's graph the change touches, and
// `mesh/repo-check`, the repository's own merge-check.sh (novox/hq ADR 0237 as amended); with a comment
// saying why when the gate is not a pass or the repository's own check failed.
// saying why when the gate is not a pass or the repository's own check failed. Of the two, only those the base
// still requires are written (issue 1301): main requires `mesh/check` — the checker's, written by its forge bridge
// through gitea_check_status — everywhere but hq, whose main requires `mesh/repo-check` alone (ADR 0503 §5).
//
// Consumes node-build-agent.started — a build machine took an ask — and says, on the head a pending check is of,
// which machine builds it since when (novox/hq issue 446). This module is the one writer of mesh/merge-gate.
@@ -38,12 +40,12 @@
import { emit, on } from "@novox/mesh-sdk/events";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { commitStatusTool } from "./delivery.js";
import { checkStatusTool, commitStatusTool } from "./delivery.js";
import { announce, keepWatermark, readWatermark } from "./announce.js";
import { mergeTool } from "./merge.js";
import { GiteaAdmin, GiteaClient, movedSince } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, commentFor, composedGate, groupGateStatus, headsToAnnounce, statusesFor, type Announced, type Checked,
repoCheckFacts, reposToLook } from "./pulls.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, commentFor, composedGate, groupGateStatus, headsToAnnounce, maintained,
requiredOf, statusesFor, writesGate, type Announced, type Checked, repoCheckFacts, reposToLook } from "./pulls.js";
import { GateWriter, clonedAs, type BuildStarted } from "./gate.js";
import { HOOK_PATH, HOOK_PORT, Pace, claimHead, ensureSystemHook, eventsTool, keptSecret, receiver, type ForgeEvent } from "./webhook.js";
import { keepSiblings, noteComposed, noteVerdict, readSiblings, siblingsFile, siblingsOf, weighSiblings } from "./siblings.js";
@@ -247,8 +249,11 @@ async function announceHead(client: GiteaClient, repo: RepoOf, pull: OpenOf): Pr
// Said, so an operator knows the mesh was asked to check it.
console.log(`[gitea] announced ${key} at ${sha.slice(0, 8)} to be checked before it merges`);
// Pending until the verdict comes, so the pull request says a check is running rather than nothing — and
// where it is: in the build queue from now, until a build machine takes it (novox/hq issue 446).
await gates.pending(repo.owner, repo.name, sha);
// where it is: in the build queue from now, until a build machine takes it (novox/hq issue 446). Only where the base
// still requires mesh/merge-gate (issue 1301): a main that requires mesh/check is the checker's, and hq's requires
// mesh/repo-check alone; a rule that cannot be read is said pending, as before.
const rule = await client.appliedProtection(repo.owner, repo.name, String(pull.base || "main")).catch(() => undefined);
if (writesGate(requiredOf(rule))) await gates.pending(repo.owner, repo.name, sha);
return true;
}
type OpenOf = Awaited<ReturnType<GiteaClient["listOpenPullsSince"]>>[number];
@@ -257,7 +262,7 @@ type OpenOf = Awaited<ReturnType<GiteaClient["listOpenPullsSince"]>>[number];
async function alreadyChecked(client: GiteaClient, repo: RepoOf, pull: OpenOf): Promise<boolean> {
try {
const checks = await client.headChecks(repo.owner, repo.name, String(pull.head_sha));
return CHECK_CONTEXT in checks || REPO_CHECK_CONTEXT in checks;
return CHECK_CONTEXT in checks || REPO_CHECK_CONTEXT in checks || CHECK_STATUS in checks;
} catch {
return false; // the forge cannot say: announced, and the bus's message id collapses it if it was
}
@@ -394,16 +399,20 @@ async function setVerdict(client: GiteaClient, event: { body: unknown }): Promis
// (novox/hq ADR 0323 §4): a check asked before the trunk moved. It is said so, pending with the reason, in place of
// its pass or failure; a trunk that cannot be read lets the verdict stand, and the merge tool compares the commits.
const trunk = base ?? c.plan?.base;
// Only the statuses the base still requires are written (issue 1301): on hq, mesh/repo-check alone.
const required = trunk ? requiredOf(await client.appliedProtection(c.owner, c.repo, trunk).catch(() => undefined)) : undefined;
const kept = maintained(required);
if (c.trunk && trunk) {
const head = await client.branchHead(c.owner, c.repo, trunk).catch(() => undefined);
if (head && !sameCommit(c.trunk, head)) {
const description = outOfDateWords(c.trunk, head);
for (const context of [...(composedGate(c) ? [] : [CHECK_CONTEXT]), ...(c["repo-check"] ? [REPO_CHECK_CONTEXT] : [])]) {
for (const context of [...(composedGate(c) ? [] : [CHECK_CONTEXT]), ...(c["repo-check"] ? [REPO_CHECK_CONTEXT] : [])]
.filter((x) => kept.includes(x))) {
await client.setCommitStatus(c.owner, c.repo, c.commit, { state: "pending", context, description,
...(target ? { target_url: target } : {}) });
}
// Its account is kept and shown all the same (ADR 0323 §7): what it judged, and that it is out of date.
const said = commentFor(c, await repoCheckFacts(client, c, trunk));
const said = commentFor(c, await repoCheckFacts(client, c, trunk), required);
if (c.number && !composedGate(c)) await client.addComment(c.owner, c.repo, c.number, lateVerdictComment(c, said, description));
console.log(`[gitea] ${c.owner}/${c.repo}#${c.number ?? "?"} at ${c.commit.slice(0, 8)}: merge check ${c.verdict}, ` +
`${description}`);
@@ -412,10 +421,10 @@ async function setVerdict(client: GiteaClient, event: { body: unknown }): Promis
}
const facts = await repoCheckFacts(client, c, base ?? c.plan?.base);
const gateNow = composedGate(c) ? (await client.headChecks(c.owner, c.repo, c.commit).catch(() => ({} as Record<string, { description?: string }>)))[CHECK_CONTEXT]?.description : undefined;
for (const status of statusesFor(c, facts, gateNow)) {
for (const status of statusesFor(c, facts, gateNow, required)) {
await client.setCommitStatus(c.owner, c.repo, c.commit, target ? { ...status, target_url: target } : status);
}
const comment = composedGate(c) ? null : commentFor(c, facts);
const comment = composedGate(c) ? null : commentFor(c, facts, required);
if (comment && c.number) await client.addComment(c.owner, c.repo, c.number, comment);
console.log(`[gitea] ${c.owner}/${c.repo}#${c.number ?? "?"} at ${c.commit.slice(0, 8)}: merge check ${c.verdict}`);
}
@@ -477,6 +486,8 @@ if (!gitea) pace.unheard("the forge is not watched: no way to a token", false);
registerModuleTools("gitea", () => [eventsTool(pace), ...(gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
}), checkStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
}), mergeTool(gitea, { gates })] : [])]);
// **A build machine took an ask** (the build seat's `started`, novox/hq ADR 0157): said on the head it checks, when
+41 -10
View File
@@ -4,13 +4,18 @@
// when a verdict is judged against an earlier trunk commit than the trunk's head, the tool asks the head's check and
// puts its statuses back to pending through that writer, so where the check runs is said and no second check is asked.
// Each bundle the runtime serves is its own process; from the tools' bundle the pending bypassed the writer's chain.
//
// **It judges on the status the base requires** (issue 1301; ADR 0447 §1, §4; ADR 0503 §5): mesh/check, the checker's,
// whose line names the trunk it judged ("pass <head12> on <trunk12>"); where a base does not require it, the controller's
// status it still requires (hq's mesh/repo-check), as before. A mesh/check of another trunk is refused and nothing is
// asked: the checker sets it out of date on the trunk's move and checks it again.
import { announce } from "./announce.js";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus, GiteaClient } from "./client.js";
import { mergeableUnder, newestByContext } from "./checks.js";
import { MERGE_METHODS, Refused, checkName } from "./guard.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT } from "./pulls.js";
import { CHECK_CONTEXT, CHECK_STATUS, REPO_CHECK_CONTEXT, maintained, requiredOf } from "./pulls.js";
import { approvedHeadsOf, groupRefusal, memberKey, namesSiblings, notApproved, readSiblings, siblingOutOfDate, siblingsFile } from "./siblings.js";
import { announcementOf, askedWords, outOfDateWords, sameCommit, tookFile, tookWords, twelve, verdictAtTrunk, waitsForComposed } from "./trunk.js";
@@ -32,7 +37,7 @@ export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition
return {
name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head " +
"and every status its base branch's protection requires succeeded there (mesh/merge-gate, mesh/repo-check), " +
"and every status its base branch's protection requires succeeded there (mesh/check; hq's mesh/repo-check), " +
"judged against the trunk's head as it is now, optionally deleting the source branch afterwards. Refused " +
"otherwise, with nothing merged: no tool merges past a check, and none merges a head that moved after it was " +
"reviewed. A verdict judged against an earlier trunk commit is refused, naming both, and the head's check against " +
@@ -108,11 +113,33 @@ export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition
// head makes — never again while one is on its way. The caller asks again at the reviewed head once it passes.
const trunkHead = await gitea.branchHead(owner, repo, pull.base);
const checks = await gitea.headChecks(owner, repo, pull.head_sha);
const at = verdictAtTrunk(checks, trunkHead);
const required = requiredOf(rule) ?? [];
if (required.includes(CHECK_STATUS)) {
// **mesh/check is the checker's** (ADR 0447 §4): it is judged here against the trunk's head, and refused when it
// is not current; nothing is written or asked — the checker checks a head again when its trunk moves.
const check = checks[CHECK_STATUS];
const at = verdictAtTrunk(checks, trunkHead, CHECK_STATUS);
if (!check || check.state === "pending") {
throw new Refused(`${owner}/${repo}#${number} is not merged: its ${CHECK_STATUS} is ` +
`${check ? `pending (${check.description ?? ""})` : "not set yet"}; the checker answers it. Ask again at ` +
`${twelve(pull.head_sha)} once it passes against ${pull.base}'s head ${twelve(trunkHead)}. Nothing was merged`);
}
if (!at.current) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its ${CHECK_STATUS} ` +
`${at.judged ? `was judged on trunk ${twelve(at.judged)}` : "names no trunk commit it was judged on"}, and ` +
`${pull.base} is now ${twelve(trunkHead)}. The checker checks it again against the trunk's head (novox/hq ` +
`ADR 0447); this tool asks nothing. Ask again at ${twelve(pull.head_sha)} once it passes. Nothing was merged`);
}
}
// Where the base still requires the controller's statuses (hq's mesh/repo-check; mesh/merge-gate only on a
// revert), the first it requires judges, as before (ADR 0323 §5).
const kept = maintained(required);
const judge = kept[0] ?? CHECK_CONTEXT;
const at = kept.length ? verdictAtTrunk(checks, trunkHead, judge) : { current: true, onItsWay: false };
if (!at.current) {
// A verdict put out of date by a sibling's move says so in its own words (ADR 0382 §5); it is asked again as a
// trunk's move is, and a failed ask puts those words back, not the trunk's.
const siblingWords = siblingOutOfDate(checks[CHECK_CONTEXT]?.description) ? checks[CHECK_CONTEXT]?.description : undefined;
const siblingWords = siblingOutOfDate(checks[judge]?.description) ? checks[judge]?.description : undefined;
const judged = siblingWords ? `was put out of date by a sibling's move (${siblingWords})`
: at.judged ? `was judged against trunk ${twelve(at.judged)}` : "names no trunk commit it was judged against";
const took = tookWords(tookFile(deps.stateDir), `${owner}/${repo}`);
@@ -120,21 +147,22 @@ export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition
// **A member that waits for its group's composed check is not announced again** (novox/hq issue 1106): its own
// check can only say "composed" once more, so an ask here brought a verdict naming no trunk and the next merge was
// refused the same way for ever. The composed check is mesh-delivery's to ask; this says whose verdict is awaited.
if (waitsForComposed(checks[CHECK_CONTEXT])) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its merge gate ${checks[CHECK_CONTEXT]?.description}; the ` +
if (waitsForComposed(checks[judge])) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its merge gate ${checks[judge]?.description}; the ` +
`verdict that names its trunk commit is its group's composed check's, and mesh-delivery asks it, this tool asks no check ` +
`(main is now ${now}). Ask again once the group's composed check has passed. Nothing was merged`);
}
if (at.onItsWay) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}. Its check is already asked and not yet answered (${checks[CHECK_CONTEXT]?.description}), so it is not ` +
`${now}. Its check is already asked and not yet answered (${checks[judge]?.description}), so it is not ` +
`asked again; ${took}. Ask again at ${head} once both statuses pass: this tool never merges by itself ` +
`(novox/hq ADR 0323). A check that never answers is asked again with mesh-delivery.recheck. Nothing was merged`);
}
// Pending first, so a second ask finds the check on its way; then the head is announced as a new head is. An
// announcement that cannot be made puts the statuses back to out of date, so the next ask asks again rather
// than wait for ever on a check nobody asked.
const contexts = [CHECK_CONTEXT, ...(checks[REPO_CHECK_CONTEXT] ? [REPO_CHECK_CONTEXT] : [])];
// The statuses the base requires of the controller's (issue 1301): hq's mesh/repo-check alone.
const contexts = kept.filter((c) => c === judge || (c === REPO_CHECK_CONTEXT && checks[REPO_CHECK_CONTEXT]));
// Through the module's one writer of the merge check (gate.ts, novox/hq issue 456): the pending opens a new
// check of the head, so where it runs is said ("building on …") and its verdict is not weighed as older.
const say = async (description: string) => {
@@ -167,9 +195,12 @@ export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition
// **A member judged beside its siblings merges only with its group passed and reviewed** (novox/hq ADR 0382 §6): the
// siblings its verdict names are the holder's record (siblings.ts); a status that says it read siblings with no record
// of them is no verdict to merge on, never read as none.
const heard = readSiblings(siblingsFile(deps.stateDir))[memberKey(owner, repo, number)];
// Under mesh/check (issue 1301) the group is the checker's one verdict on every member's head (ADR 0447 §2) and the
// Approve's heads above; a record the controller's verdicts left before the switch judges nothing there.
const heard = required.includes(CHECK_STATUS) ? undefined : readSiblings(siblingsFile(deps.stateDir))[memberKey(owner, repo, number)];
const siblings = heard && sameCommit(heard.head, pull.head_sha) ? heard.siblings : [];
if (!siblings.length && namesSiblings(statuses.find((s) => s.context === CHECK_CONTEXT)?.description)) {
if (!siblings.length && !required.includes(CHECK_STATUS) &&
namesSiblings(statuses.find((s) => s.context === CHECK_CONTEXT)?.description)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict read siblings, and which ones is not held ` +
`here, so the group cannot be weighed. Ask its check again at ${twelve(pull.head_sha)}. Nothing was merged`);
}
+1 -1
View File
@@ -172,7 +172,7 @@
},
"repositories": {
"type": "object",
"description": "The repositories of the organisation novox the mesh holds (ADR 0428): each by name, {\"<name>\": {\"private\": true, \"description\": \"...\"}}. One that does not exist is made by the module's setup login under its own account, its main protected and given a first commit (a README naming the description and a merge-check.sh), its rule narrowed to no push and the tools' merge requiring mesh/repo-check and mesh/merge-gate, and only then moved into the organisation; one that exists only has main's rule put right, its statuses kept. A repository taken out of the value is never touched.",
"description": "The repositories of the organisation novox the mesh holds (ADR 0428): each by name, {\"<name>\": {\"private\": true, \"description\": \"...\"}}. One that does not exist is made by the module's setup login under its own account, its main protected and given a first commit (a README naming the description and a merge-check.sh), its rule narrowed to no push and the tools' merge requiring mesh/check, and only then moved into the organisation; one that exists only has main's rule put right, mesh/check in place of mesh/merge-gate and mesh/repo-check and its other statuses kept (a main requiring mesh/repo-check alone keeps it). Every other repository whose main requires mesh/merge-gate has the two swapped for mesh/check; nothing else of a repository out of the value is touched.",
"trusted": true,
"default": {}
}
+4 -4
View File
@@ -1,6 +1,6 @@
// A branch's protection (novox/hq ADR 0237): what makes a pull request wait for the mesh's merge check —
// `mesh/merge-gate`, the module graph's gate, and `mesh/repo-check`, the repository's own tests — before it
// may merge. Pure, so it is tested without a forge; the client does the asking (client.ts).
// A branch's protection (novox/hq ADR 0237): what makes a pull request wait for the mesh's check — `mesh/check`,
// the checker's verdict on the head merged onto its trunk (ADR 0447 §1; hq's `mesh/repo-check` until issue 1296) —
// before it may merge. Pure, so it is tested without a forge; the client does the asking (client.ts).
import { TOOLS_ACCOUNT } from "./tools-account.js";
@@ -24,7 +24,7 @@ export interface BranchProtection {
/** What a branch's protection is set to. A field not given is left as the rule has it (or the forge's
* default on a new rule), except pushes, which a new rule refuses unless `push` says otherwise. */
export interface ProtectionWanted {
/** The statuses a pull request must have succeeded before it merges, e.g. mesh/merge-gate. Empty: none. */
/** The statuses a pull request must have succeeded before it merges, e.g. mesh/check. Empty: none. */
statusChecks: string[];
/** Whether a person may push to the branch directly. */
push?: boolean;
+45 -7
View File
@@ -19,6 +19,34 @@ export const CHECK_CONTEXT = "mesh/merge-gate";
/** The status context of the repository's own merge-check.sh, the check's second layer (novox/hq ADR 0237). */
export const REPO_CHECK_CONTEXT = "mesh/repo-check";
/** The one status main requires (novox/hq ADR 0447 §1, issue 1301): the checker's verdict, written by its forge bridge
* through gitea_check_status. Its line names the head and the trunk commit it judged: "pass <head12> on <trunk12> …". */
export const CHECK_STATUS = "mesh/check";
/** The statuses a branch's rule requires, from the rule the forge applies to it: none when it is unprotected or checks
* no status; undefined when the rule could not be read. */
export function requiredOf(rule: { enable_status_check?: boolean; status_check_contexts?: string[] } | null | undefined): string[] | undefined {
if (rule === undefined) return undefined;
if (!rule || !rule.enable_status_check) return [];
return (rule.status_check_contexts ?? []).filter(Boolean);
}
/**
* **The controller's statuses are written only where a base still requires them** (issue 1301; ADR 0503 §3, §5): of
* mesh/merge-gate and mesh/repo-check, those the base requires — hq's mesh/repo-check alone once the forge requires
* mesh/check everywhere else, so no mesh/merge-gate stands on a head that nothing will set. A base whose rule could not
* be read keeps both, as before: a status nothing requires blocks nothing, and one that is required is never left out.
*/
export function maintained(required: string[] | undefined): string[] {
const both = [CHECK_CONTEXT, REPO_CHECK_CONTEXT];
return required === undefined ? both : both.filter((c) => required.includes(c));
}
/** Whether mesh/merge-gate is written on a head of this base (maintained). */
export function writesGate(required: string[] | undefined): boolean {
return maintained(required).includes(CHECK_CONTEXT);
}
/**
* The pending words a head's merge check is announced with (novox/hq issue 446): where the check is, and since when,
* in the time the operator reads. They stood as "the mesh is mapping this head onto its module graph" for the whole
@@ -257,8 +285,14 @@ export function composedGate(c: Checked): boolean {
return c.gate?.verdict === "composed";
}
/** Every status a verdict sets: the gate's, and the repository's own check's when it was said. */
export function statusesFor(c: Checked, facts: RepoCheckFacts = {}, gateNow?: string): CommitStatus[] {
/** Every status a verdict sets: the gate's, and the repository's own check's when it was said; of them, only those the
* base requires (`required`, the base's rule: maintained). */
export function statusesFor(c: Checked, facts: RepoCheckFacts = {}, gateNow?: string, required?: string[]): CommitStatus[] {
const kept = maintained(required);
return allStatusesFor(c, facts, gateNow).filter((s) => kept.includes(s.context));
}
function allStatusesFor(c: Checked, facts: RepoCheckFacts = {}, gateNow?: string): CommitStatus[] {
if (composedGate(c)) {
// A member whose gate the group's composed check judges (ADR 0403 §1): only its repository's check is its own to
// set. The gate stays pending, saying it waits for the group, unless a composed verdict already stands on it.
@@ -292,11 +326,13 @@ function judgedFirst(c: Checked, status: CommitStatus): CommitStatus {
* change that builds something (novox/hq ADR 0238), and why, when the gate is not a pass or the
* repository's own check failed or could not run. A repository with no merge-check.sh, touching nothing,
* is said by its statuses alone, not by a comment on every push. */
export function commentFor(c: Checked, facts: RepoCheckFacts = {}): string | null {
export function commentFor(c: Checked, facts: RepoCheckFacts = {}, required?: string[]): string | null {
const gate = c.gate ?? { verdict: c.verdict, summary: c.summary };
// A base that no longer requires the gate (issue 1301) is told nothing of it: its verdict is the repository's own check.
const withGate = writesGate(required);
const repo = c["repo-check"];
const wrong = repoWrong(repo, facts);
if (gate.verdict === "pass" && !wrong && !builds(c.plan)) return null;
if ((!withGate || gate.verdict === "pass") && !wrong && !builds(c.plan)) return null;
const lines = [`**Merge check** at \`${c.commit.slice(0, 8)}\``, ""];
if (c.trunk) {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and what was read beside it.
@@ -312,9 +348,11 @@ export function commentFor(c: Checked, facts: RepoCheckFacts = {}): string | nul
} else if (c.siblings_unknown) {
lines.splice(c.trunk ? 2 : 1, 0, `Siblings could not be learned: ${c.siblings_unknown}`);
}
lines.push(`- \`${CHECK_CONTEXT}\`: **${saidAs(gate.verdict).toUpperCase()}** — ${gate.summary}` +
(gate.modules?.length ? ` (modules: ${gate.modules.join(", ")}` +
(gate.dependents?.length ? `; built after them: ${gate.dependents.join(", ")}` : "") + ")" : ""));
if (withGate) {
lines.push(`- \`${CHECK_CONTEXT}\`: **${saidAs(gate.verdict).toUpperCase()}** — ${gate.summary}` +
(gate.modules?.length ? ` (modules: ${gate.modules.join(", ")}` +
(gate.dependents?.length ? `; built after them: ${gate.dependents.join(", ")}` : "") + ")" : ""));
}
if (repo) {
const st = repoStatus(repo, facts);
lines.push(`- \`${REPO_CHECK_CONTEXT}\`: ` + (repo.verdict === "warning" && facts.defined === false
+35
View File
@@ -61,3 +61,38 @@ test("one view per pull request, found by its marker; only the mesh's statuses",
assert.equal(deliveryStatus("mesh/repo-check", "pending", "checking again: main moved on").context, "mesh/repo-check");
assert.equal(deliveryStatus("mesh/delivery", "success", "x", "javascript:alert(1)").target_url, undefined);
});
// Issue 1301-S4 (ADR 0447 §1; the lead's choice A): main requires mesh/check, so no hand may set it. The status tool
// every caller may use refuses it in every state; it is written only by gitea_check_status, which the forge bridge
// alone lists in its manifest's `invokes`, so only its account may call it.
test("mesh/check is refused by the status tool and written by the check's own tool alone", async () => {
const { commitStatusTool, checkStatusTool, deliveryStatus, CHECK_STATUS } = await import("../delivery.ts");
assert.equal(CHECK_STATUS, "mesh/check");
const sha = "0123456789abcdef0123456789abcdef01234567";
const sent: string[] = [];
const setCommitStatus = async (o: string, r: string, s: string, st: any) =>
void sent.push(`${o}/${r}@${s.slice(0, 8)} ${st.context} ${st.state} ${st.description}${st.target_url ? " " + st.target_url : ""}`);
const gates = { rechecked: async () => { throw new Error("not the gate writer's"); } };
for (const state of ["pending", "success", "failure", "error", "warning"]) {
assert.throws(() => deliveryStatus("mesh/check", state, "x"), /mesh\/check is the checker's status/);
await assert.rejects(commitStatusTool({ setCommitStatus, gates } as any).run({ owner: "novox", repo: "mesh-sdk", sha,
context: "mesh/check", state, description: "by hand" }), /gitea_check_status/);
}
assert.deepEqual(sent, [], "nothing was sent for a hand-set mesh/check");
const tool = checkStatusTool({ setCommitStatus });
assert.equal(tool.name, "gitea_check_status");
for (const k of ["owner", "repo", "sha", "state", "description", "target_url", "context"]) assert.ok(k in tool.input, k);
// The bridge's arguments as it sends them for gitea_commit_status: the context may be given, and is mesh/check.
await tool.run({ owner: "novox", repo: "mesh-sdk", sha, context: "mesh/check", state: "success", description: "pass",
target_url: "https://forge.invalid/novox/mesh-sdk/pulls/1" });
await tool.run({ owner: "novox", repo: "mesh-sdk", sha, state: "pending", description: "y".repeat(300) });
assert.equal(sent[0], "novox/mesh-sdk@01234567 mesh/check success pass https://forge.invalid/novox/mesh-sdk/pulls/1");
assert.ok(sent[1].startsWith("novox/mesh-sdk@01234567 mesh/check pending yyyy"), sent[1]);
assert.ok(sent[1].length < 200, "the description is cut as every status's is");
await assert.rejects(tool.run({ owner: "novox", repo: "mesh-sdk", sha, context: "mesh/delivery", state: "success" }), /mesh\/check alone/);
await assert.rejects(tool.run({ owner: "novox", repo: "mesh-sdk", sha: sha.slice(0, 12), state: "success" }), /whole sha/);
await assert.rejects(tool.run({ owner: "novox", repo: "mesh-sdk", sha, state: "green" }), /not a status/);
await assert.rejects(tool.run({ owner: "../x", repo: "mesh-sdk", sha, state: "success" }));
assert.equal(sent.length, 2, "nothing more was sent");
});
+10
View File
@@ -322,3 +322,13 @@ test("a repository as the mesh clones it is told from another", async () => {
assert.equal(clonedAs("http://127.0.0.1:20000/novox/mesh-hq.git", "novox", "hq"), false);
assert.equal(clonedAs("http://127.0.0.1:20000/novox/mesh-controller.git", "novox", "hq"), false);
});
// Issue 1301: a build machine's start on a head that holds no mesh/merge-gate (its base requires none, so none was written)
// says nothing: no "building on" status appears on a main that does not require the gate.
test("a start on a head with no gate on the forge writes nothing", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
const gates = new GateWriter(forge, announcedAt);
assert.equal(await gates.started("novox", "hq", SHA, startedOnShanks), false);
assert.equal(forge.kept["mesh/merge-gate"], undefined);
});
+56 -2
View File
@@ -15,7 +15,7 @@ import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { mergeTool } from "../dist/merge.js";
import { commitStatusTool } from "../dist/delivery.js";
import { checkStatusTool, commitStatusTool } from "../dist/delivery.js";
import { GateWriter } from "../dist/gate.js";
import { judgeApiCall, judgeProtectionChange, ruleCovers, routedPath } from "../dist/guard.js";
@@ -168,7 +168,7 @@ const gates = { rechecked: (o: string, r: string, sha: string, s: any) => {
} };
const tools = [...getGiteaTools(client), commitStatusTool({
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s), gates: writer,
}), mergeTool(client, { gates, stateDir, emit: async (type, body) => { state.emitted.push({ type, body }); },
}), checkStatusTool({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s) }), mergeTool(client, { gates, stateDir, emit: async (type, body) => { state.emitted.push({ type, body }); },
now: () => new Date(2026, 9, 12, 15, 1) })];
const tool = (name: string) => tools.find((t) => t.name === name)!;
const writes = () => state.sent.map((s) => `${s.method} ${s.path}`);
@@ -568,6 +568,10 @@ test("no tool's caller text becomes another path or another user: `..` and `?sud
["gitea_get_file", { ...repo, path: "README.md?sudo=jschoubben" }],
["gitea_commit_status", { ...repo, sha: "abc1234?sudo=jschoubben", context: "mesh/delivery", state: "success" }],
["gitea_commit_status", { ...repo, sha: "../../../admin/users", context: "mesh/delivery", state: "success" }],
["gitea_check_status", { ...repo, sha: "abc1234?sudo=jschoubben", state: "success" }],
["gitea_check_status", { ...repo, sha: "../../../admin/users", state: "success" }],
["gitea_check_status", { owner: "novox", repo: "..", sha: "0123456789abcdef0123456789abcdef01234567", state: "success" }],
["gitea_commit_status", { ...repo, sha: "0123456789abcdef0123456789abcdef01234567", context: "mesh/check", state: "success" }],
["gitea_commit_statuses", { ...repo, sha: ".." }],
["gitea_commit_statuses", { ...repo, sha: "main?sudo=jschoubben" }],
["gitea_branch_protection_set", { ...repo, branch: "..", status_checks: "mesh/merge-gate" }],
@@ -704,3 +708,53 @@ test("a group of two merges in one pass: the sibling's merge of the composed hea
assert.equal(done.merged, true);
assert.equal(state.emitted.length, 0, "no check was asked");
});
// **Issue 1301 (ADR 0447 §1, §4; ADR 0503 §3, §5): the merge tool judges on the status the base requires.** A main that
// requires mesh/check is judged by the checker's line ("pass <head12> on <trunk12> …"): it merges on a pass that names the
// trunk's head, and refuses one that names an older trunk, naming both, writing nothing and asking nothing — a re-check on
// a trunk's move is the checker's. No mesh/merge-gate is read or written there.
test("gitea_merge_pull_request judges a main requiring mesh/check on mesh/check alone, and asks nothing of it", async () => {
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED });
state.rules = [{ ...protectedMain(), status_check_contexts: ["mesh/check"] }];
const line = (trunk: string, s = "success", words = "pass") =>
[{ context: "mesh/check", state: s, description: `${words} ${REVIEWED.slice(0, 12)} on ${trunk.slice(0, 12)} | unit 12 run` }];
const before = state.emitted.length;
state.statuses = line(MOVED);
await assert.rejects(merge(), (err: Error) => {
assert.match(err.message, /mesh\/check/);
assert.match(err.message, new RegExp(`${MOVED.slice(0, 12)}.*${TRUNK.slice(0, 12)}`));
assert.match(err.message, /checker checks it again/i);
assert.match(err.message, /Nothing was merged/);
return true;
});
state.statuses = line(TRUNK, "pending", "checking");
await assert.rejects(merge(), /mesh\/check is pending/);
state.statuses = judgedAt(TRUNK); // the old two, current: not what this main requires
await assert.rejects(merge(), /mesh\/check/);
assert.deepEqual(writes(), [], "nothing is written: mesh/check is the checker's");
assert.equal(state.emitted.length, before, "no check is asked");
assert.equal(state.merged, false);
state.statuses = line(TRUNK);
assert.equal(((await merge()) as Record<string, unknown>).merged, true);
});
// hq's main requires mesh/repo-check alone (ADR 0503 §5): judged on its words, and an out-of-date one is asked again on
// mesh/repo-check alone; no mesh/merge-gate is written on it.
test("gitea_merge_pull_request judges a main requiring mesh/repo-check alone on it, and writes no mesh/merge-gate", async () => {
const { keepSiblings, siblingsFile } = await import("../dist/siblings.js");
keepSiblings(siblingsFile(stateDir), {}); // no sibling record left by the tests before
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED });
state.rules = [{ ...protectedMain(), status_check_contexts: ["mesh/repo-check"] }];
state.statuses = [judgedAt(MOVED)[1]];
const before = state.emitted.length;
await assert.rejects(merge(), /judged against trunk/);
assert.deepEqual(state.sent.filter((s) => s.path.includes("/statuses/")).map((s) => `${s.body.context} ${s.body.state}`),
["mesh/repo-check pending"]);
assert.equal(state.emitted.length, before + 1, "its check is asked once");
await assert.rejects(merge(), /already asked/);
state.sent = [];
state.statuses = [judgedAt(TRUNK)[1]];
assert.equal(((await merge()) as Record<string, unknown>).merged, true);
});
+24
View File
@@ -192,3 +192,27 @@ test("a member's own verdict with a composed gate layer leaves the gate to the c
// A verdict with an ordinary gate is as before.
assert.equal(composedGate({ ...c, gate: { verdict: "pass", summary: "x" } }), false);
});
// Issue 1301 (ADR 0503 §3, §5): gitea writes the controller's statuses only where the base still requires them. hq's main
// requires mesh/repo-check alone: its verdict sets mesh/repo-check and no mesh/merge-gate, and its comment names no gate.
test("a base's required statuses decide which of the controller's statuses are written", async () => {
const { maintained, writesGate, statusesFor, commentFor, CHECK_STATUS } = await import("../pulls.ts");
assert.equal(CHECK_STATUS, "mesh/check");
assert.deepEqual(maintained(undefined), ["mesh/merge-gate", "mesh/repo-check"], "a rule not read: as before");
assert.deepEqual(maintained(["mesh/repo-check"]), ["mesh/repo-check"]);
assert.deepEqual(maintained(["mesh/check", "other/x"]), []);
assert.deepEqual(maintained([]), []);
assert.equal(writesGate(undefined), true);
assert.equal(writesGate(["mesh/repo-check"]), false);
assert.equal(writesGate(["mesh/check"]), false);
assert.equal(writesGate(["mesh/merge-gate", "mesh/repo-check"]), true);
const c = { owner: "novox", repo: "hq", number: 5, commit: "a".repeat(40), id: "x", verdict: "warning", summary: "no script",
trunk: "b".repeat(12), "repo-check": { verdict: "pass", summary: "ok" } } as any;
const hq = statusesFor(c, {}, undefined, ["mesh/repo-check"]);
assert.deepEqual(hq.map((s: any) => s.context), ["mesh/repo-check"]);
assert.match(hq[0].description, /merged into trunk bbbbbbbbbbbb/);
assert.deepEqual(statusesFor(c).map((s: any) => s.context), ["mesh/merge-gate", "mesh/repo-check"], "unread: as before");
assert.deepEqual(statusesFor(c, {}, undefined, ["mesh/check"]), []);
const comment = commentFor(c, {}, ["mesh/repo-check"]);
assert.ok(comment === null || !comment.includes("mesh/merge-gate"), comment ?? "");
});
+18
View File
@@ -199,3 +199,21 @@ test("a release branch's move puts every open pull request into it out of date,
assert.equal(verdictAtTrunk({ "mesh/merge-gate": { state: "success",
description: "head x merged into trunk eeeeeeeeeeee — pass" } }, MOVED).current, false);
});
// Issue 1301: a trunk's move puts out of date only the controller's statuses the base still requires; on hq that is
// mesh/repo-check alone, judged by its own words; on a main requiring mesh/check, none (the checker's, ADR 0447 §4).
test("a trunk's move writes only the statuses the base still requires", async () => {
const { outOfDateStatuses, trunkNamed } = await import("../trunk.ts");
const old = "1".repeat(40), now = "2".repeat(40);
const repo = { state: "success", description: `head ${"a".repeat(12)} merged into trunk ${old.slice(0, 12)} — pass: y` };
const gate = { state: "success", description: `head ${"a".repeat(12)} merged into trunk ${old.slice(0, 12)} — pass: x` };
assert.deepEqual(outOfDateStatuses({ "mesh/repo-check": repo }, now, ["mesh/repo-check"]).map((s: any) => s.context), ["mesh/repo-check"]);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": gate, "mesh/repo-check": repo }, now, ["mesh/repo-check"]).map((s: any) => s.context),
["mesh/repo-check"]);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": gate, "mesh/repo-check": repo }, now, ["mesh/check"]), []);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": gate, "mesh/repo-check": repo }, now).map((s: any) => s.context),
["mesh/merge-gate", "mesh/repo-check"], "a rule not read: as before");
// The checker's line names its trunk as "<head12> on <trunk12>".
assert.equal(trunkNamed(`pass ${"a".repeat(12)} on ${old.slice(0, 12)} | unit 3 run`), old.slice(0, 12));
assert.equal(trunkNamed("building on shanks since 10:00"), undefined);
});
+11
View File
@@ -350,3 +350,14 @@ test("a pull request's edit event carries its description, and no other pull req
assert.equal(eventOf("pull_request", { action, pull_request: pull, repository }).body.body, undefined, action);
}
});
// Issue 1302: a status event carries the status's words, so the delivery gate reads mesh/check's line (the head and the trunk
// it judged) from the event itself.
test("a status event carries its context, state and words", async () => {
const { eventOf } = await import("../webhook.ts");
const e = eventOf("status", { sha: "a".repeat(40), context: "mesh/check", state: "success",
description: "pass aaaaaaaaaaaa on bbbbbbbbbbbb | unit 3 run, 0 kept", repository: { name: "x", owner: { login: "novox" } } });
assert.equal(e.name, "forge.status");
assert.equal((e.body as any).context, "mesh/check");
assert.equal((e.body as any).description, "pass aaaaaaaaaaaa on bbbbbbbbbbbb | unit 3 run, 0 kept");
});
+4 -4
View File
@@ -395,7 +395,7 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
// ---- Branch protection (novox/hq ADR 0237) ----
{
name: "gitea_branch_protection_get",
description: "A repository's branch protection: the rule for one branch (null when it has none) or every rule — whether direct pushes are refused, which commit statuses a pull request must have succeeded to merge (e.g. mesh/merge-gate), approvals, and whether an administrator may merge past them.",
description: "A repository's branch protection: the rule for one branch (null when it has none) or every rule — whether direct pushes are refused, which commit statuses a pull request must have succeeded to merge (e.g. mesh/check), approvals, and whether an administrator may merge past them.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
@@ -413,12 +413,12 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
},
{
name: "gitea_branch_protection_set",
description: "Only ever stronger (novox/hq ADR 0266): a change that drops a required status, lets a person push directly or lets an administrator merge past a status is refused, and on main so is a change that drops the merge whitelist (only mesh-tools merges there) or widens it (novox/hq ADR 0398 §2, issue 948) — that is the operator's, at the forge. Make a branch require commit statuses before a pull request merges into it — the mesh's merge check sets `mesh/merge-gate` (the module graph's gate) and `mesh/repo-check` (the repository's own tests). Edits the branch's rule (its required statuses replaced by these, everything else kept unless given) or creates one, which refuses direct pushes unless push=true. Answers the rule before and after.",
description: "Only ever stronger (novox/hq ADR 0266): a change that drops a required status, lets a person push directly or lets an administrator merge past a status is refused, and on main so is a change that drops the merge whitelist (only mesh-tools merges there) or widens it (novox/hq ADR 0398 §2, issue 948) — that is the operator's, at the forge. Make a branch require commit statuses before a pull request merges into it — the mesh's checker sets `mesh/check` (ADR 0447), and the repositories program requires it on every main. Edits the branch's rule (its required statuses replaced by these, everything else kept unless given) or creates one, which refuses direct pushes unless push=true. Answers the rule before and after.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
branch: { type: "string", description: "the branch to protect, e.g. main" },
status_checks: { type: "string", description: "comma-separated statuses required to merge, e.g. mesh/merge-gate,mesh/repo-check; empty requires none" },
status_checks: { type: "string", description: "comma-separated statuses required to merge, e.g. mesh/check; empty requires none" },
push: { type: "boolean", description: "whether a person may push to the branch directly (a new rule refuses it when not given)" },
block_admin_override: { type: "boolean", description: "stop an administrator merging past a status that has not succeeded (left as it is when not given)" },
},
@@ -478,7 +478,7 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
},
{
name: "gitea_commit_statuses",
description: "Read a commit's statuses — or a pull request's head's — as the forge keeps them: the newest of each context (mesh/merge-gate, mesh/repo-check, mesh/delivery, …) with its state, description, link, when it was set and by whom; and whether the branch it merges into would let it merge, every status that branch's protection requires being a success. Reads only.",
description: "Read a commit's statuses — or a pull request's head's — as the forge keeps them: the newest of each context (mesh/check, mesh/delivery, …) with its state, description, link, when it was set and by whom; and whether the branch it merges into would let it merge, every status that branch's protection requires being a success. Reads only.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
+24 -12
View File
@@ -12,12 +12,17 @@
// the moment it is asked, merges nothing when they differ, asks the head's check once — the same announcement a
// new head makes — and says how long the last check of that repository took. It never merges by itself after.
//
// Since issue 1301 (ADR 0447 §4) a main that requires mesh/check is the checker's: its line names the trunk commit it judged
// ("pass <head12> on <trunk12>"), the checker sets it out of date when the trunk moves, and the merge tool refuses a verdict
// of another trunk and asks nothing. The two statuses above are kept only where a base still requires them (hq's
// mesh/repo-check, pulls.ts maintained).
//
// Pure functions and two small files of the module's own state, so this is tested without a forge.
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type { CommitStatus, GiteaClient, GiteaPull, HeadCheck } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, WAITS_FOR_COMPOSED } from "./pulls.js";
import { CHECK_CONTEXT, WAITS_FOR_COMPOSED, maintained, requiredOf } from "./pulls.js";
/** A commit as a verdict names it: its first twelve characters. */
export function twelve(sha: string | undefined): string {
@@ -40,7 +45,9 @@ export function judgedWords(head: string, trunk: string): string {
* verdict put out of date, the one it had judged ("judged against trunk …"). Undefined when it names none, as a
* verdict of a build seat from before ADR 0323 does. */
export function trunkNamed(description: string | undefined): string | undefined {
const m = /(?:merged into|judged against) trunk ([0-9a-f]{7,64})\b/i.exec(description ?? "");
const m = /(?:merged into|judged against) trunk ([0-9a-f]{7,64})\b/i.exec(description ?? "")
// The checker's mesh/check line (ADR 0447 §3): "<head12> on <trunk12>".
?? /\b[0-9a-f]{12} on ([0-9a-f]{12})\b/i.exec(description ?? "");
return m ? m[1].toLowerCase() : undefined;
}
@@ -70,15 +77,15 @@ export function waitsForComposed(gate: HeadCheck | undefined): boolean {
* gate, and the repository's own check when the head has one — a pending repo-check a verdict will not set would
* stand for ever (novox/hq issue 305's rule).
*/
export function outOfDateStatuses(checks: Record<string, HeadCheck>, trunkHead: string): CommitStatus[] {
const gate = checks[CHECK_CONTEXT];
if (!gate || gate.state === "pending") return [];
const judged = trunkNamed(gate.description);
export function outOfDateStatuses(checks: Record<string, HeadCheck>, trunkHead: string, required?: string[]): CommitStatus[] {
// Only the statuses the base still requires (issue 1301): the first of them the head holds judges.
const kept = maintained(required).filter((c) => checks[c]);
const judge = kept.length ? checks[kept[0]] : undefined;
if (!judge || judge.state === "pending") return [];
const judged = trunkNamed(judge.description);
if (judged && sameCommit(judged, trunkHead)) return [];
const description = outOfDateWords(judged, trunkHead);
const out: CommitStatus[] = [{ state: "pending", context: CHECK_CONTEXT, description }];
if (checks[REPO_CHECK_CONTEXT]) out.push({ state: "pending", context: REPO_CHECK_CONTEXT, description });
return out;
return kept.map((context) => ({ state: "pending" as const, context, description }));
}
/** Where the merge tool stands with a head's verdict and the trunk's head (ADR 0323 §5). */
@@ -91,8 +98,8 @@ export interface VerdictAtTrunk {
judged?: string;
}
export function verdictAtTrunk(checks: Record<string, HeadCheck>, trunkHead: string): VerdictAtTrunk {
const gate = checks[CHECK_CONTEXT];
export function verdictAtTrunk(checks: Record<string, HeadCheck>, trunkHead: string, context = CHECK_CONTEXT): VerdictAtTrunk {
const gate = checks[context];
const judged = trunkNamed(gate?.description);
if (checkOnItsWay(gate)) return { current: false, onItsWay: true, judged };
return { current: !!gate && gate.state !== "pending" && !!judged && sameCommit(judged, trunkHead), onItsWay: false, judged };
@@ -119,6 +126,9 @@ export function lateVerdictComment(c: { commit: string; trunk?: string; verdict:
export interface TrunkForge {
listOpenPulls(owner: string, repo: string): Promise<GiteaPull[]>;
branchHead(owner: string, repo: string, branch: string): Promise<string>;
/** The rule the forge applies to a branch: which of the controller's statuses are kept there (issue 1301). Not given,
* or not read: both, as before. */
appliedProtection?(owner: string, repo: string, branch: string): Promise<{ enable_status_check?: boolean; status_check_contexts?: string[] } | null>;
}
/** What it writes with: the merge check's one writer (gate.ts), which decides in the head's own turn. */
@@ -147,11 +157,13 @@ export async function weighTrunks(forge: TrunkForge, gates: OutOfDateWriter, tru
if (before && sameCommit(before, head)) continue;
if (before) {
let put = 0;
const required = forge.appliedProtection
? requiredOf(await forge.appliedProtection(owner, name, base).catch(() => undefined)) : undefined;
for (const pull of open.filter((p) => p.base === base)) {
// The trunk is read again in the head's own turn: a verdict judged against a head newer than the pass's first
// read — the trunk moved again meanwhile, and a check answered — is not put out of date.
const written = await gates.outOfDate(owner, name, String(pull.head_sha),
async (checks) => outOfDateStatuses(checks, await forge.branchHead(owner, name, base)), pull.html_url || undefined);
async (checks) => outOfDateStatuses(checks, await forge.branchHead(owner, name, base), required), pull.html_url || undefined);
if (written.length) put++;
}
console.log(`[gitea] ${owner}/${name}'s ${base} moved from ${twelve(before)} to ${twelve(head)}: ${put} open pull ` +
+3 -1
View File
@@ -127,7 +127,9 @@ function forgeEventOf(said: string, payload: any): ForgeEvent {
return { name: kind, body: { ...base, ref: str(payload.ref), ref_type: str(payload.ref_type), sha: str(payload.sha) } };
}
if (kind === "status") {
return { name: "status", body: { ...base, sha: str(payload.sha), context: str(payload.context), state: str(payload.state) } };
// Its words too (issue 1302): mesh/check's line names the head and the trunk it judged, which the delivery gate reads.
return { name: "status", body: { ...base, sha: str(payload.sha), context: str(payload.context), state: str(payload.state),
description: str(payload.description) } };
}
const name = family === "other" ? `other.${kind.replace(/[^a-z0-9_-]/gi, "_") || "unknown"}` :
["fork"].includes(family) ? family : `${family}.${action ?? "changed"}`;
+8 -17
View File
@@ -65,16 +65,14 @@ integration-tagged test runs the real go on the fixtures.
## The integration level
`build_go_test` with `level: integration` runs `go test -tags integration,isolation` (isolation: the bubblewrap proofs,
which a build node can run) against a store and a bus started for the run alone (novox/hq ADR 0470). The store is a postgres of the installed package: `initdb` and `pg_ctl` into
`build_go_test` with `level: integration` runs `go test -tags integration` against a store and a bus started for the
run alone (novox/hq ADR 0470). The store is a postgres of the installed package: `initdb` and `pg_ctl` into
`<work>/<workspace>/run-<id>/pg`, as the module's account, listening on a unix socket there and on no TCP address, with
durability off; the tests get `MESH_TEST_POSTGRES` (the socket directory percent-encoded in the URL, because the
controller's tests cut the URL at its last slash). The bus is a nats-server from the Go library at the version this
module's `go.mod` names, run by this same binary started as `mesh-build-go bus <dir>` (`bus.go`: JetStream on, a
loopback port of its own, its address on the first line, stopped when its input closes); the tests get `MESH_TEST_NATS`.
Both run inside the step, are stopped, and the run's directory is removed whatever the tests did. The store, the bus and
the tests run together in one bubblewrap (ADR 0448, issue 1309): this binary again as `mesh-build-go level <spec>`, which
prints the level's result as one line of JSON, so the tests reach the level's own store and bus and nothing of the machine.
Both run inside the step, are stopped, and the run's directory is removed whatever the tests did.
Setup (compiling the tests, the store, the bus) is judged within 30 s and the tests' running time (the wall time of
`go test`) within 60 s, apart; a level over either says `over_bound` and names the bound, with the slowest tests for the running
@@ -89,9 +87,7 @@ is said until a source exists.
one digest of everything those tests read (ADR 0445 §5): the files, by content and under their path in the tree, of the
package and every package its test binary reaches (`go list -test -deps`) that lies in the module's own tree; the version
and checksum of every module outside it; the package's own `testdata/` files; and the level, the build tags and, at the
integration level, the postgres major and the nats-server version; and, when the fetch placed siblings beside the
workspace (`beside`), each one's repository and commit, which the answer echoes as `beside` (the checker keeps a verdict only
for the siblings it was judged with). The text's form is `DigestVersion` 3. The Go version is not in it: the toolchain enters a
integration level, the postgres major and the nats-server version. The Go version is not in it: the toolchain enters a
fingerprint once, as the version the build nodes agree on (ADR 0477 §3), and the standard library's packages are that
version's. A package with no tests at the level has no digest and is named in `without_tests`. A directory that vendors
is listed and tested from its vendor directory (`-mod=vendor`).
@@ -105,16 +101,11 @@ that vendors), `go build -ldflags "-s -w -X main.builtFor=<system>" -o .mesh-bui
module's directory, by the installed go (`GOTOOLCHAIN=local`) from the module cache the fetch filled and no network
(`GOPROXY=off`). The binary is named as the build agent names it. The output directory is packed with mesh-sdk's `Pack`,
as the build agent packs; two build nodes on the same toolchain give the same digest for the same tree (ADR 0477 §6). The step ends with the artifact, the binary, the digest, the size and
the reference of the archive: once the build ended, this process pushes it to the artifact store (novox/hq ADR 0498 §1 §2,
mesh-sdk's `buildstep.PushBlob`) under `<module>/<artifact>` by its digest, a candidate's as a trunk build's, and answers
`artifact-store://<module>/<artifact>/blobs/<digest>`; a push that fails fails the step `not stored: <artifact>: <the
store's words>`. The packed bytes never stay on the node. The store's address is `${dir:state}/registry`, placed by the
manifest, which requires `artifact-store`. The go build runs shut off in bubblewrap (ADR 0448, `isolate.go`): no
network, none of the account, the caches beneath a throwaway layer unless the step is told `publish` (a trunk build). An artifact that is neither a Go
bundle nor an archive fails the step naming it.
the path of the archive, which stays in the workspace's `artifacts` directory: publishing it is the builder's, in a later
release. An artifact that is neither a Go bundle nor an archive fails the step naming it.
The same tool packs an archive artifact of any module's tree, whatever its language (ADR 0478 §1, issue 1589): the
directory named by `from`, or a file pinned by `url` and `sha256`, packed with the same `Pack` as the build agent's `pack`
(`testdata/captured/README` says how its digest was captured). A pinned file is fetched, checked against its sha256 (a
mismatch fails the step naming both digests) and laid out as the artifact says (`file`, or unpacked from `within`; gzip, bzip2
and xz, read in Go as the build agent reads it, issue 1590); it runs no command and needs no Go.
mismatch fails the step naming both digests) and laid out as the artifact says (`file`, or unpacked from `within`; gzip and
bzip2, not yet xz); it runs no command and needs no Go.
@@ -1,20 +0,0 @@
package main
// What this module keeps on the bus, announced from its code (novox/hq ADR 0495 §2 §5; issue 1790): every state it
// keeps, each readable by others, since another module reads each today (the builder, the checker and the indexer read toolchains; the builder and the checker read steps; gitea reads keys). A state of its own that it
// did not name here is refused at the call (stdio.Keeps).
import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// Kept are the states the module keeps.
var Kept = []announce.State{
{Name: "toolchains", Readable: true},
{Name: "keys", Readable: true},
{Name: "steps", Readable: true},
}
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
func keepStates() error { return stdio.Keeps(Kept...) }
@@ -1,63 +0,0 @@
//go:build integration
package main
import (
"bufio"
"encoding/json"
"io"
"testing"
"github.com/nats-io/nats.go"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announcetest"
)
// 1790 (ADR 0495 §5): served as the runtime serves it, the module announces on the bus every state it keeps, as its
// code declares them. The runtime's part, publishing what the bundle asks to publish, is played by relay.
func TestTheModuleAnnouncesEveryStateItKeeps(t *testing.T) {
const module = "mesh-build-go"
announcetest.Check(t, module, Kept, func(url string) func() {
t.Setenv("MESH_MODULE", module)
t.Setenv("MESH_NODE", "shanks")
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
if err := keepStates(); err != nil {
t.Fatal(err)
}
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = stdio.ServeOn(module, nil, inR, outW) }()
go relay(t, module, nc, outR, inW)
return func() { _ = inW.Close(); _ = outW.Close(); nc.Close() }
})
}
// relay is the runtime's side of the channel: each mesh/publish the bundle asks is published on its module's event
// subject, mesh.mod.<module>.event.<key>, and answered.
func relay(t *testing.T, module string, nc *nats.Conn, from io.Reader, to io.Writer) {
sc := bufio.NewScanner(from)
sc.Buffer(make([]byte, 1<<20), 1<<24)
enc := json.NewEncoder(to)
for sc.Scan() {
var m struct {
ID json.RawMessage `json:"id"`
Method string `json:"method"`
Params struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
} `json:"params"`
}
if json.Unmarshal(sc.Bytes(), &m) != nil || m.Method != "mesh/publish" {
continue
}
if err := nc.Publish("mesh.mod."+module+".event."+m.Params.Key, m.Params.Body); err != nil {
t.Errorf("publishing %s: %v", m.Params.Key, err)
}
_ = nc.Flush()
_ = enc.Encode(map[string]any{"id": m.ID, "result": map[string]any{}})
}
}
@@ -24,7 +24,6 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/therootcompany/xz"
)
// maxUpstream is the largest pinned file fetched and the largest it may unpack to, as the build agent's and the
@@ -43,9 +42,9 @@ func (m *Module) httpClient() *http.Client {
// buildArchive packs one archive artifact of the module in dir: a pinned file when it names a url, else the directory
// `from` of the module's tree.
func (m *Module) buildArchive(ctx context.Context, workspace, module, name, dir string, art catalogue.Artifact) (any, error) {
func (m *Module) buildArchive(ctx context.Context, workspace, module, dir string, art catalogue.Artifact) (any, error) {
if art.URL != "" {
return m.pinnedArchive(ctx, workspace, module, name, art)
return m.pinnedArchive(ctx, workspace, module, art)
}
from := strings.Trim(path.Clean("/"+filepath.ToSlash(art.From)), "/")
if strings.TrimSpace(art.From) == "" {
@@ -77,12 +76,12 @@ func (m *Module) buildArchive(ctx context.Context, workspace, module, name, dir
if !within(realDir, realRoot) {
return nil, fmt.Errorf("the archive %s is taken from %q, which lies outside the module: it resolves to %s, and the module is %s", art.Name, art.From, realRoot, realDir)
}
return m.packInto(ctx, workspace, module, name, art.Name, root)
return m.packInto(workspace, module, art.Name, root)
}
// pinnedArchive fetches the file an artifact is pinned to, checks it against the pin, lays it out as the artifact says
// (kept whole as `file`, or unpacked from `within`) and packs it.
func (m *Module) pinnedArchive(ctx context.Context, workspace, module, name string, art catalogue.Artifact) (any, error) {
func (m *Module) pinnedArchive(ctx context.Context, workspace, module string, art catalogue.Artifact) (any, error) {
file, err := m.pinnedFile(ctx, workspace, art)
if err != nil {
return nil, fmt.Errorf("%s: %w", art.Name, err)
@@ -111,7 +110,7 @@ func (m *Module) pinnedArchive(ctx context.Context, workspace, module, name stri
} else if err := unpackUpstream(file, laid, art.Within); err != nil {
return nil, fmt.Errorf("%s: %s (%s): %w", art.Name, art.URL, art.Version, err)
}
return m.packInto(ctx, workspace, module, name, art.Name, laid)
return m.packInto(workspace, module, art.Name, laid)
}
// pinnedFile is the path of the pinned file in the workspace: kept from an earlier build when its bytes are still the
@@ -191,13 +190,10 @@ func sumOf(file string) (string, error) {
return hex.EncodeToString(hash.Sum(nil)), nil
}
// xzDictMax is the largest xz dictionary read (the build agent's): xz's own presets use at most 64 MiB (-9); a stream
// asking for more is refused rather than given the memory.
const xzDictMax = 64 << 20
// unpackUpstream unpacks a tar archive compressed with gzip, bzip2 or xz into a directory: of its entries only those
// within `within` (all, when empty), with that prefix taken off. Only files and directories, and nothing outside: the
// node-engine unpacks nothing else.
// unpackUpstream unpacks a tar archive compressed with gzip or bzip2 into a directory: of its entries only those within
// `within` (all, when empty), with that prefix taken off. Only files and directories, and nothing outside: the
// node-engine unpacks nothing else. An xz archive is refused by name; the build agent reads it, this module does not
// yet (it is a later issue, not a quiet difference in the archive).
func unpackUpstream(file, into, within string) error {
f, err := os.Open(file)
if err != nil {
@@ -218,22 +214,9 @@ func unpackUpstream(file, into, within string) error {
case bytes.HasPrefix(magic, []byte("BZh")):
stream = bzip2.NewReader(buffered)
case bytes.HasPrefix(magic, []byte{0xfd, '7', 'z', 'X', 'Z', 0x00}):
// As the build agent reads it (mesh-controller internal/builder/upstream_file.go; issue 1590): in Go
// (github.com/therootcompany/xz, a port of xz-embedded), never by a tool the node may not carry. xzDictMax is
// a real maximum: a block whose LZMA2 dictionary is larger is refused before its dictionary is allocated. The
// reader keeps a filter decoder for every block of a stream, so the file's framing is walked first and a file
// of more blocks than xzBlocksMax is refused before anything is decoded (xz_framing.go). What the stream
// unpacks to is bounded by untar (maxUpstream, maxEntries).
if err := xzWithinLimits(file); err != nil {
return err
}
zipped, err := xz.NewReader(buffered, xzDictMax)
if err != nil {
return fmt.Errorf("it is not an xz stream this build module reads (a dictionary of %d MiB at most): %v", xzDictMax>>20, err)
}
stream = zipped
return errors.New("it is an xz archive, which this build module does not read yet; the build agent does")
default:
return errors.New("it is not a tar archive compressed with gzip, bzip2 or xz; a single file is kept whole by naming it in `file`")
return errors.New("it is not a tar archive compressed with gzip or bzip2; a single file is kept whole by naming it in `file`")
}
return untar(stream, into, within)
}
@@ -1,7 +1,6 @@
package main
import (
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
@@ -41,11 +40,16 @@ const (
capturedWithGo = "go1.27.1"
)
// sameAsBuildAgent says the archive the build pushed holds the tar stream the build agent's pack wrote, and, under the Go the
// sameAsBuildAgent says the archive at res.Path holds the tar stream the build agent's pack wrote, and, under the Go the
// digests were captured with, has its digest.
func sameAsBuildAgent(t *testing.T, res BuildResult, stream, digest string) {
t.Helper()
z, err := gzip.NewReader(bytes.NewReader(archiveOf(t, res)))
f, err := os.Open(res.Path)
if err != nil {
t.Fatal(err)
}
defer f.Close()
z, err := gzip.NewReader(f)
if err != nil {
t.Fatal(err)
}
@@ -153,8 +157,8 @@ func TestAnArchiveOfADirectoryHasTheBuildAgentsDigest(t *testing.T) {
if res.Artifact != "files" || res.Size == 0 || res.Binary != "" {
t.Errorf("result = %+v", res)
}
if len(archiveOf(t, res)) != res.Size {
t.Errorf("the archive pushed is not %d bytes", res.Size)
if info, err := os.Stat(res.Path); err != nil || info.Size() != int64(res.Size) {
t.Errorf("the archive at %q: %v", res.Path, err)
}
if len(*calls) != 0 {
t.Errorf("an archive ran %d commands", len(*calls))
@@ -1,87 +0,0 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The lead's decision L5 (the checker keeps a verdict of a core repository): the fetch writes the siblings it placed
// beside the workspace, and the digest folds each one's repository and commit into every package's digest and echoes
// them, so a verdict is kept only for the siblings it was judged with.
func TestTheSiblingsCommitsAreInEveryDigestAndEchoed(t *testing.T) {
t.Parallel()
m, _ := digestFixture(t)
alone := digests(t, m, "unit")
if len(alone.Beside) != 0 {
t.Errorf("a workspace without siblings echoes %v", alone.Beside)
}
write := func(placed []Sibling) {
raw, _ := json.Marshal(placed)
if err := os.WriteFile(filepath.Join(m.Work, "ws1", besideRecord), raw, 0o600); err != nil {
t.Fatal(err)
}
}
other := strings.Repeat("b", 40)
write([]Sibling{{Repository: "novox/mesh-sdk", Commit: commitSHA, Dir: "mesh-sdk"}, {Repository: "novox/mesh-catalog", Commit: commitSHA, Dir: "mesh-catalog"}})
with := digests(t, m, "unit")
want := []string{"novox/mesh-catalog " + commitSHA, "novox/mesh-sdk " + commitSHA}
if !slices.Equal(with.Beside, want) {
t.Errorf("echoed %v, want %v", with.Beside, want)
}
write([]Sibling{{Repository: "novox/mesh-sdk", Commit: other, Dir: "mesh-sdk"}, {Repository: "novox/mesh-catalog", Commit: commitSHA, Dir: "mesh-catalog"}})
moved := digests(t, m, "unit")
for p, d := range alone.Packages {
if with.Packages[p] == d {
t.Errorf("%s: the siblings left the digest as it was without them", p)
}
if moved.Packages[p] == with.Packages[p] {
t.Errorf("%s: a sibling's new commit left the digest as it was", p)
}
}
}
// The fetch writes the siblings it placed, which the digest reads.
func TestTheFetchRecordsTheSiblingsItPlaced(t *testing.T) {
f := &fakeStep{gomods: "go.mod\n"}
m, states := fetchModule(t, f)
args := map[string]any{"repository": "novox/mesh-controller", "commit": commitSHA, "beside": []any{
map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA}}}
if _, err := fetchTool(t, m)(args); err != nil {
t.Fatal(err)
}
st := states.waitEnd(t)
var res FetchResult
json.Unmarshal(st.Result, &res)
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
raw, err := os.ReadFile(filepath.Join(m.Work, res.Workspace, besideRecord))
var placed []Sibling
if err != nil || json.Unmarshal(raw, &placed) != nil || len(placed) != 1 || placed[0].Repository != "novox/mesh-sdk" || placed[0].Commit != commitSHA {
t.Errorf("the record is %s (%v)", raw, err)
}
}
// The checker's agreement: the integration level runs and digests under the tags integration and isolation, so the
// bubblewrap proofs tagged isolation run in every check on a build node.
func TestTheIntegrationLevelDigestsUnderBothTags(t *testing.T) {
t.Parallel()
m, _ := digestFixture(t)
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
digests(t, m, LevelIntegration)
for _, s := range seen {
if !strings.Contains(s.command, " -tags integration,isolation ") {
t.Errorf("the integration digest listed %q", s.command)
}
}
if DigestVersion != "3" {
t.Errorf("the digest's form changed in kind and is still version %s", DigestVersion)
}
}
@@ -3,8 +3,8 @@ package main
// The build step (novox/hq ADR 0405 §3 §7, ADR 0453 §7; issue 1487): a Go bundle artifact of a module of a fetched
// workspace, built with today's recipe — the build agent's, mesh-controller's internal/builder/toolchain.go (the Go
// entry) and builder.go (compile, binaryName, pack) — by the installed Go and no network, packed with mesh-sdk's Pack
// so that its digest is the build agent's for the same tree, and pushed to the artifact store by this process once the
// build ended (ADR 0498 §1 §2, push.go).
// so that its digest is the build agent's for the same tree. The archive is left in the workspace; publishing it is
// the builder's, in a later release.
import (
"context"
@@ -15,7 +15,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The Go recipe of the build agent (toolchain.go, the entry of language go): static, trimmed of its own paths and of
@@ -34,9 +33,8 @@ type BuildResult struct {
Binary string `json:"binary"`
Digest string `json:"digest"`
Size int `json:"size"`
// Reference is where the archive is in the artifact store (novox/hq ADR 0498 §1):
// artifact-store://<module>/<artifact>/blobs/<digest>.
Reference string `json:"reference"`
// Path is the archive, in the workspace's directory of the work directory.
Path string `json:"path"`
}
func (m *Module) buildTool() StepTool {
@@ -51,7 +49,6 @@ func (m *Module) buildTool() StepTool {
"workspace": map[string]any{"type": "string", "description": "the workspace id build_go_fetch answered"},
"module": map[string]any{"type": "string", "description": "the module's directory in the workspace, . for the repository's root"},
"artifact": map[string]any{"type": "string", "description": "the artifact's name in the module's manifest"},
"publish": map[string]any{"type": "boolean", "description": "true for a merged trunk commit alone: the build writes the shared Go caches; without it (a candidate) they are read beneath a throwaway layer"},
},
"required": []string{"workspace", "module", "artifact"},
},
@@ -79,7 +76,6 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
workspace, _ := args["workspace"].(string)
module, _ := args["module"].(string)
name, _ := args["artifact"].(string)
publish, _ := args["publish"].(bool)
if !workspaceOK.MatchString(workspace) {
return nil, fmt.Errorf("%q is not a workspace id", workspace)
}
@@ -110,7 +106,7 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
return nil, fmt.Errorf("the module %s has no artifact %q (it has %s)", manifest.Module, name, strings.Join(names, ", "))
}
if art.Kind == "archive" {
return m.buildArchive(ctx, workspace, module, manifest.Module, dir, *art)
return m.buildArchive(ctx, workspace, module, dir, *art)
}
if art.Kind != "bundle" || art.Language != "go" {
return nil, fmt.Errorf("the artifact %s of %s is a %s of language %q: this module builds Go bundles and archives", name, manifest.Module, art.Kind, art.Language)
@@ -137,19 +133,13 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
linker += " -X " + recipeSystemStamp + "=" + system
}
cmd := []string{"-C", dir, "build", "-ldflags", linker, "-o", filepath.Join(out, binary), "./" + strings.Trim(art.From, "./")}
// Shut off (ADR 0448): the shared caches are written by a trunk build alone (publish), a candidate's go build writes
// beneath a throwaway layer.
run, err := m.stepRunner(workspace, publish)
if err != nil {
return nil, err
}
if r := run(ctx, m.buildEnv(dir), "go", cmd...); r.Status != 0 || r.Err != "" {
if r := m.Run(ctx, m.buildEnv(dir), "go", cmd...); r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("go build of %s: %s", art.Name, causeLine(r.Err+"\n"+r.Stderr+"\n"+r.Stdout))
}
if err := m.insideModule(workspace, dir, out); err != nil {
return nil, fmt.Errorf("the output directory of %s: %w", art.Name, err)
}
res, err := m.packInto(ctx, workspace, module, manifest.Module, art.Name, out)
res, err := m.packInto(workspace, module, art.Name, out)
if err != nil {
return nil, err
}
@@ -157,20 +147,22 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
return res, nil
}
// packInto packs a directory with mesh-sdk's Pack, as the build agent packs every artifact, and pushes the archive to
// the artifact store from this process, after the build step ended (novox/hq ADR 0498 §1 §2): a candidate's as a trunk
// build's, under <module>/<artifact> by its digest. The step answers the reference; a push that fails fails the step
// `not stored:`. The packed bytes never stay on the node: they are held in this process until the push answered (the
// lead's decision L9).
func (m *Module) packInto(ctx context.Context, workspace, module, name, artifact, root string) (BuildResult, error) {
// packInto packs a directory with mesh-sdk's Pack, as the build agent packs every artifact, and leaves the archive in
// the workspace's artifacts directory.
func (m *Module) packInto(workspace, module, artifact, root string) (BuildResult, error) {
archive, digest, err := buildstep.Pack(root)
if err != nil {
return BuildResult{}, fmt.Errorf("packing %s: %w", artifact, err)
}
if err := m.push(ctx, name+"/"+artifact, digest, archive); err != nil {
return BuildResult{}, fmt.Errorf("%s%s: %v", delivery.NotStored, artifact, err)
artifacts := filepath.Join(m.Work, workspace, "artifacts")
if err := os.MkdirAll(artifacts, 0o700); err != nil {
return BuildResult{}, err
}
return BuildResult{Artifact: artifact, Digest: digest, Size: len(archive), Reference: buildstep.BlobReference(name, artifact, digest)}, nil
path := filepath.Join(artifacts, strings.ReplaceAll(module, "/", "_")+"."+artifact+".tar.gz")
if err := os.WriteFile(path, archive, 0o600); err != nil {
return BuildResult{}, err
}
return BuildResult{Artifact: artifact, Digest: digest, Size: len(archive), Path: path}, nil
}
// buildEnv is the environment of the compile: the build agent's (CGO_ENABLED=0, GOFLAGS) with the installed go
@@ -41,7 +41,7 @@ func buildModule(t *testing.T, manifest, content string) (*Module, *keptStates,
if err := os.WriteFile(filepath.Join(dir, "module.json"), []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
m.Run = throughBubblewrap(t, func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
*calls = append(*calls, cmdRecord{env: env, line: name + " " + strings.Join(args, " ")})
for i, a := range args {
if a == "-o" {
@@ -51,8 +51,7 @@ func buildModule(t *testing.T, manifest, content string) (*Module, *keptStates,
}
}
return forgekey.Ran{}
}, &[]shutOff{}) // the compile runs shut off (1309)
withStore(t, m) // the archive is pushed to a store of the test's own (ADR 0498)
}
return m, states, calls
}
@@ -144,10 +143,13 @@ func TestTheArchiveHoldsOneExecutableAndTwoBuildsAreOneDigest(t *testing.T) {
m, states, _ := buildModule(t, manifestWith(goBundle), "the program")
first, _ := runBuild(t, m, states, "tools")
second, _ := runBuild(t, m, states, "tools")
if first.Digest != second.Digest || first.Reference != second.Reference {
if first.Digest != second.Digest || first.Path != second.Path {
t.Errorf("two builds: %+v and %+v", first, second)
}
raw := archiveOf(t, first)
raw, err := os.ReadFile(first.Path)
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(raw)
if first.Digest != "sha256:"+hex.EncodeToString(sum[:]) {
t.Errorf("the digest %s is not the archive's", first.Digest)
@@ -20,13 +20,7 @@ import (
)
// DigestVersion is the form of the digest's text: changed whenever what the digest hashes changes in kind.
// 2: no Go version in the text; the toolchain enters a fingerprint as the agreed version (ADR 0477 §3). 3: the siblings'
// commits (the lead's decision L5) and the integration level's tags integration,isolation (the checker's agreement).
const DigestVersion = "3"
// integrationTags are the build tags of the integration level, its run and its digest alike: integration, and isolation
// for the bubblewrap proofs (mesh-sdk's and the build modules'), which a build node can run.
var integrationTags = []string{"integration", "isolation"}
const DigestVersion = "2" // 2: no Go version in the text; the toolchain enters a fingerprint as the agreed version (ADR 0477 §3)
// goPackage is the part of `go list -json` a digest reads.
type goPackage struct {
@@ -77,12 +71,9 @@ func (p *goPackage) hasTests() bool { return len(p.TestGoFiles)+len(p.XTestGoFil
// DigestResult is what a digest step answers: one digest per package that has tests at the level, and the packages
// that have none.
type DigestResult struct {
Level string `json:"level"`
Packages map[string]string `json:"packages"`
// Beside echoes the siblings folded into every digest, "<owner/repo> <commit>" sorted: the checker keeps a verdict
// only when they are the siblings it gave the fetch.
Beside []string `json:"beside,omitempty"`
WithoutTests []string `json:"without_tests,omitempty"`
Level string `json:"level"`
Packages map[string]string `json:"packages"`
WithoutTests []string `json:"without_tests,omitempty"`
}
func (m *Module) digestTool() StepTool {
@@ -122,27 +113,20 @@ func (m *Module) digest(ctx context.Context, args map[string]any) (any, error) {
if _, err := m.checkPostgres(ctx); err != nil {
return nil, err
}
tags = integrationTags
tags = []string{"integration"}
header = append(header, "postgres "+strconv.Itoa(m.Recipe.PostgresMajor), "nats-server "+m.Recipe.NatsServer)
default:
return nil, fmt.Errorf("the level %q is neither unit nor integration", level)
}
header = append(header, "tags "+strings.Join(tags, ","))
beside, err := m.besideLines(workspace)
if err != nil {
return nil, err
}
for _, b := range beside {
header = append(header, "beside "+b)
}
dirs, err := m.goModDirs(ctx, tree)
if err != nil {
return nil, err
}
res := DigestResult{Level: level, Packages: map[string]string{}, Beside: beside}
res := DigestResult{Level: level, Packages: map[string]string{}}
for _, d := range dirs {
pkgs, err := m.listPackages(ctx, workspace, tree, d, tags)
pkgs, err := m.listPackages(ctx, tree, d, tags)
if err != nil {
return nil, err
}
@@ -155,18 +139,14 @@ func (m *Module) digest(ctx context.Context, args map[string]any) (any, error) {
}
// listPackages is `go list -test -deps -json` of the directory's packages, by import path.
func (m *Module) listPackages(ctx context.Context, workspace, tree, dir string, tags []string) (map[string]*goPackage, error) {
func (m *Module) listPackages(ctx context.Context, tree, dir string, tags []string) (map[string]*goPackage, error) {
abs := filepath.Join(tree, dir)
args := []string{"-C", abs, "list", "-test", "-deps", "-json"}
if len(tags) > 0 {
args = append(args, "-tags", strings.Join(tags, ","))
}
args = append(args, "./...")
run, err := m.stepRunner(workspace, false)
if err != nil {
return nil, err
}
r := run(ctx, m.goEnvFor(abs), "go", args...)
r := m.Run(ctx, m.goEnvFor(abs), "go", args...)
if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("go list in %s: %s", dir, causeLine(r.Err+"\n"+r.Stderr+"\n"+r.Stdout))
}
@@ -30,7 +30,7 @@ func digestFixture(t *testing.T) (*Module, string) {
m := moduleWith(newStates(), "go1.27.1")
m.Home, m.Work, m.Cache = t.TempDir(), work, t.TempDir()
real := StepRunner(m.Home)
m.Run = throughBubblewrap(t, func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
switch name {
case "git":
return forgekey.Ran{Stdout: "go.mod\n"}
@@ -38,7 +38,7 @@ func digestFixture(t *testing.T) (*Module, string) {
return forgekey.Ran{Stdout: "postgres (PostgreSQL) " + strconv.Itoa(m.Recipe.PostgresMajor) + ".1\n"}
}
return real(ctx, env, name, args...)
}, &[]shutOff{}) // go list runs shut off (1309); here the real go, unshut, reads the fixture
}
return m, tree
}
@@ -67,8 +67,6 @@ func (m *Module) fetchTool() StepTool {
"commit": map[string]any{"type": "string", "description": "the full commit, 40 hex digits"},
"ref": map[string]any{"type": "string", "description": "refs/pull/<n>/head when only a pull request's head carries the commit"},
"base": map[string]any{"type": "string", "description": "optional trunk commit, 40 hex digits: the head is merged onto it as the forge's merge would, and the answer carries the merge commit, the files in conflict (an answer, the workspace is then left at the trunk commit), the changed paths and the tree of each module directory"},
"dir": buildstep.FetchRequestProperties()["dir"],
"paths": buildstep.FetchRequestProperties()["paths"],
"beside": map[string]any{"type": "array", "description": "optional sibling repositories the tests read, each placed as a plain directory (no symlink) under the workspace's beside/<name>, which build_go_test gives the tests as MESH_CHECK_BESIDE",
"items": map[string]any{"type": "object", "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<repository> on the forge"},
@@ -83,17 +81,20 @@ func (m *Module) fetchTool() StepTool {
}
func (m *Module) fetch(ctx context.Context, args map[string]any) (any, error) {
// What the builder and the checker ask, in mesh-sdk's one form (ADR 0499 §4): with the module's directory and its
// named paths, the workspace is narrowed to them (buildstep.Narrow).
req, err := buildstep.ParseFetchRequest(args)
if err != nil {
return nil, err
repository, _ := args["repository"].(string)
commit, _ := args["commit"].(string)
ref, _ := args["ref"].(string)
base, _ := args["base"].(string)
if base != "" && !commitOK.MatchString(base) {
return nil, fmt.Errorf("%q is not a trunk commit (40 hex digits)", base)
}
repository, commit, ref, base := req.Repository, req.Commit, req.Ref, req.Base
siblings, err := siblingsArg(args["beside"])
if err != nil {
return nil, err
}
if !repositoryOK.MatchString(repository) || strings.Contains(repository, "..") {
return nil, fmt.Errorf("%q is not a repository, <owner>/<name>", repository)
}
m.CleanWorkspaces(m.now())
id := buildstep.NewID()
root := filepath.Join(m.Work, id)
@@ -114,15 +115,6 @@ func (m *Module) fetch(ctx context.Context, args map[string]any) (any, error) {
if res.Beside, err = m.placeSiblings(ctx, root, siblings); err != nil {
return nil, err
}
if (base == "" || res.Merge != "") && len(req.Paths) > 0 {
if err := buildstep.Narrow(tree, req); err != nil {
return nil, err
}
// Said beside the tree, so the steps that list the tree's go.mod files leave out those narrowed away.
if err := os.WriteFile(filepath.Join(root, narrowedMark), nil, 0o600); err != nil {
return nil, err
}
}
dirs, err := m.goModDirs(ctx, tree)
if err != nil {
return nil, err
@@ -185,11 +177,6 @@ func (m *Module) goModDirs(ctx context.Context, tree string) ([]string, error) {
dirs := []string{}
for _, l := range strings.Split(r.Stdout, "\n") {
if l = strings.TrimSpace(l); l != "" && !inTestdata(l) {
if narrowed(tree) {
if _, err := os.Stat(filepath.Join(tree, filepath.FromSlash(l))); err != nil {
continue // outside a narrowed workspace's paths (buildstep.Narrow): no module of this build
}
}
dirs = append(dirs, filepath.ToSlash(filepath.Dir(l)))
}
}
@@ -281,12 +268,3 @@ func firstLine(s string) string {
}
return ""
}
// narrowedMark is the file beside a workspace's tree saying the fetch narrowed it to a module's named paths.
const narrowedMark = "narrowed"
// narrowed says whether the fetch narrowed the workspace whose tree this is.
func narrowed(tree string) bool {
_, err := os.Stat(filepath.Join(filepath.Dir(tree), narrowedMark))
return err == nil
}
@@ -143,9 +143,8 @@ func TestForgeFilesRenderFromTheBoundValuesInTheModulesOwnHome(t *testing.T) {
// template in the files/ archive (novox/hq ADR 0349 §2).
func TestManifestRequiresForgeAccountKeepsKeysAndTakesFilesFromTemplates(t *testing.T) {
m := readForgeManifest(t)
// artifact-store: the module pushes a done build's archive itself (ADR 0498 §2).
if len(m.Requires) != 3 || m.Requires[0] != "forge-account" || m.Requires[1] != "go-toolchain" || m.Requires[2] != "artifact-store" {
t.Errorf("requires = %v, want [forge-account go-toolchain artifact-store]", m.Requires)
if len(m.Requires) != 2 || m.Requires[0] != "forge-account" || m.Requires[1] != "go-toolchain" {
t.Errorf("requires = %v, want [forge-account go-toolchain]", m.Requires)
}
keeps := false
for _, s := range m.State {
@@ -11,7 +11,6 @@ import (
"os"
"path/filepath"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
@@ -34,22 +33,11 @@ func (m *Module) integrationBounds() Bounds {
return b
}
func (m *Module) integration(ctx context.Context, workspace, tree string, packages []string, pattern string) (res *TestResult, err error) {
func (m *Module) integration(ctx context.Context, workspace, tree string, packages []string) (res *TestResult, err error) {
major, err := m.checkPostgres(ctx)
if err != nil {
return nil, err // a *buildstep.Refusal: this level only
}
if !m.Inside {
// The store, the bus and the tests run together in one shut-off process (level.go), so the tests reach the
// level's own store and bus and nothing else.
return m.integrationShutOff(ctx, workspace, packages, pattern)
}
return m.integrationHere(ctx, workspace, tree, packages, pattern, major)
}
// integrationHere is the level itself, in the shut-off process: the store and the bus started for the run, the tests
// compiled and run against them, both stopped and the run's directory removed however it ends.
func (m *Module) integrationHere(ctx context.Context, workspace, tree string, packages []string, pattern string, major int) (res *TestResult, err error) {
run := filepath.Join(m.Work, workspace, "run-"+buildstep.NewID())
if err := os.MkdirAll(run, 0o700); err != nil {
return nil, err
@@ -76,13 +64,12 @@ func (m *Module) integrationHere(ctx context.Context, workspace, tree string, pa
// Compiling counts as setup: the tests are built once, here, with the addresses a TestMain may read, and the run
// below finds them built. Whether they compile is for the run to say; its failure is not read here.
for _, d := range dirs {
m.Run(ctx, append(m.goEnvFor(filepath.Join(tree, d)), env...), "go", "-C", filepath.Join(tree, d), "test", "-tags", strings.Join(integrationTags, ","), "-run", "^$", "-count=1", "./...")
m.Run(ctx, append(m.goEnvFor(filepath.Join(tree, d)), env...), "go", "-C", filepath.Join(tree, d), "test", "-tags", "integration", "-run", "^$", "-count=1", "./...")
}
setup := m.clock().Sub(began)
return m.runLevel(ctx, tree, levelOpts{
workspace: workspace, run: pattern,
level: LevelIntegration, tags: integrationTags, packages: packages, bounds: m.integrationBounds(),
level: LevelIntegration, tags: []string{"integration"}, packages: packages, bounds: m.integrationBounds(),
env: env,
setup: setup, wall: true,
store: "postgres " + strconv.Itoa(major), bus: "nats-server " + m.Recipe.NatsServer,
@@ -107,9 +107,6 @@ func integModule(t *testing.T, f *integFake) (*Module, *keptStates) {
m := moduleWith(states, "go1.27.1")
m.Home, m.Work, m.Cache = t.TempDir(), t.TempDir(), t.TempDir()
m.Run, m.Now, m.StartBus = f.run, f.now, f.startBus
// These tests judge the level as it runs in its shut-off process (level.go, 1309-S5): the store, the bus and the
// tests side by side. That the step starts that one process shut off is isolate_test.go's.
m.Inside = true
if err := os.MkdirAll(filepath.Join(m.Work, "ws1", "tree"), 0o755); err != nil {
t.Fatal(err)
}
@@ -180,7 +177,7 @@ func TestTheStoreListensOnASocketUnderTheRunAndNowhereElse(t *testing.T) {
test = c
}
}
if !strings.Contains(test.line, "-tags integration,isolation") { // the bubblewrap proofs run too (the checker's agreement)
if !strings.Contains(test.line, "-tags integration") {
t.Errorf("the run does not ask for the tag: %s", test.line)
}
env := map[string]string{}
@@ -280,7 +277,7 @@ func TestAFailedOrPanickedRunStopsTheServersAndRemovesTheRunDirectory(t *testing
m, _ = integModule(t, g)
func() {
defer func() { recover() }()
m.integration(context.Background(), "ws1", filepath.Join(m.Work, "ws1", "tree"), nil, "")
m.integration(context.Background(), "ws1", filepath.Join(m.Work, "ws1", "tree"), nil)
}()
checkCleaned(g, m)
}
@@ -1,123 +0,0 @@
package main
// The run steps shut off (novox/hq ADR 0448; issue 1309, lead decision L2 of R4e): every command that runs a change's
// code (go test, go build, go list) runs through mesh-sdk's buildstep.Isolated, in bubblewrap: no network, none of the
// account (its home, its ssh key, its bus login, the runtime directory), the workspace read-write, the shared Go caches
// read-only under a throwaway writable layer, or read-write for a trunk build alone (publish). The fetch step (git, go
// mod download) holds the key and the network and runs none of the change's code, so it runs as before.
import (
"context"
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// BubblewrapLeast is the oldest bubblewrap a run step is shut off with: --overlay-src and --tmp-overlay, which keep the
// shared caches unchanged by a candidate, came in 0.10.
const BubblewrapLeast = "0.10"
// caches are the shared warm caches of the module: Go's module cache and its build cache.
func (m *Module) caches() []string {
return []string{filepath.Join(m.Cache, "gomod"), filepath.Join(m.Cache, "gobuild")}
}
// isolation is how a run step of the workspace is shut off: the workspace (its tree, its siblings, its run directories,
// its packed artifacts) read-write, the caches beneath a throwaway layer unless write, and an environment of a PATH and
// a locale alone (the step's HOME is its own, inside its /tmp).
func (m *Module) isolation(workspace string, write bool) buildstep.Isolation {
path := os.Getenv("PATH")
if path == "" {
path = defaultPath
}
return buildstep.Isolation{
Workspace: filepath.Join(m.Work, workspace),
Caches: m.caches(),
WriteCaches: write,
Env: []string{"PATH=" + path, "LC_ALL=C"},
}
}
// stepRunner is the runner of a run step of the workspace: buildstep.Isolated over m.Run, so every command it runs is
// shut off; write is true for a trunk build alone (the shared caches are written by trunk builds only). Inside is the
// integration level's own process, already running shut off (level.go): it runs its commands as they are.
func (m *Module) stepRunner(workspace string, write bool) (forgekey.Runner, error) {
if m.Inside {
return m.Run, nil
}
for _, c := range m.caches() {
// bubblewrap binds only a directory that is there; the fetch makes them, a step on a fresh node may come first.
if err := os.MkdirAll(c, 0o700); err != nil {
return nil, fmt.Errorf("the shared cache %s: %w", c, err)
}
}
return buildstep.Isolated(m.Run, m.isolation(workspace, write)), nil
}
// ProbeBubblewrap reads the installed bubblewrap's version (`bwrap --version`: "bubblewrap 0.13.0").
func ProbeBubblewrap(ctx context.Context) (string, error) {
ctx, cancel := context.WithTimeout(ctx, ProbeTimeout)
defer cancel()
r := forgekey.ExecRunnerClean(0)(ctx, []string{"PATH=" + defaultPath}, buildstep.Bubblewrap, "--version")
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("`%s --version` failed: %s", buildstep.Bubblewrap, firstLine(r.Err+"\n"+r.Stderr))
}
return parseBubblewrapVersion(r.Stdout)
}
var bubblewrapVersion = regexp.MustCompile(`^bubblewrap (\d+)\.(\d+)(\.\d+)?\s*$`)
func parseBubblewrapVersion(out string) (string, error) {
f := bubblewrapVersion.FindStringSubmatch(strings.TrimSpace(out))
if f == nil {
return "", fmt.Errorf("`%s --version` answered %q", buildstep.Bubblewrap, strings.TrimSpace(out))
}
return f[1] + "." + f[2] + f[3], nil
}
// bubblewrapProblem says why the installed bubblewrap cannot shut a run step off, or "" when it can: it is missing, or
// older than BubblewrapLeast.
func bubblewrapProblem(version string, err error) string {
if err != nil {
return "bubblewrap cannot be run, and every build and test step runs in it (ADR 0448): " + err.Error()
}
if !atLeast(version, BubblewrapLeast) {
return fmt.Sprintf("bubblewrap %s installed, and a build or test step needs %s or later (ADR 0448)", version, BubblewrapLeast)
}
return ""
}
// atLeast says whether the dotted version v is least or later.
func atLeast(v, least string) bool {
a, b := strings.Split(v, "."), strings.Split(least, ".")
for i := 0; i < len(b); i++ {
var x, y int
if i < len(a) {
x, _ = strconv.Atoi(a[i])
}
y, _ = strconv.Atoi(b[i])
if x != y {
return x > y
}
}
return true
}
// checkBubblewrap refuses a step when bubblewrap cannot shut it off; nil Bubblewrap probes nothing (a module that only
// guards).
func (m *Module) checkBubblewrap(ctx context.Context) error {
if m.Bubblewrap == nil {
return nil
}
v, err := m.Bubblewrap(ctx)
if why := bubblewrapProblem(v, err); why != "" {
return &buildstep.Refusal{Text: why, Want: BubblewrapLeast + " or later", Have: v}
}
return nil
}
@@ -1,295 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// shutOff is one command a step ran in bubblewrap: the bubblewrap words before "--", what it bound read-write, read-only
// and beneath a throwaway layer, and the command with its environment.
type shutOff struct {
flags []string
binds, roBinds, overlays []string
env []string
command string
chdir string
}
// bwrapArity is how many words each bubblewrap option the SDK writes takes.
var bwrapArity = map[string]int{"--unshare-all": 0, "--die-with-parent": 0, "--new-session": 0, "--clearenv": 0,
"--proc": 1, "--dev": 1, "--tmpfs": 1, "--dir": 1, "--chdir": 1, "--overlay-src": 1, "--tmp-overlay": 1,
"--ro-bind": 2, "--ro-bind-try": 2, "--bind": 2, "--symlink": 2, "--setenv": 2}
// throughBubblewrap is the fake runner of a step's tests: a bubblewrap command is read back (recorded in seen) and its
// command handed to run with the environment bubblewrap was told to set, the way it runs in the step; any other command
// (the fetch's git, postgres --version) goes to run as it is.
func throughBubblewrap(t *testing.T, run forgekey.Runner, seen *[]shutOff) forgekey.Runner {
t.Helper()
var mu sync.Mutex
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if name != buildstep.Bubblewrap {
return run(ctx, env, name, args...)
}
var s shutOff
i := 0
for ; i < len(args) && args[i] != "--"; i++ {
n, ok := bwrapArity[args[i]]
if !ok || i+n >= len(args) {
t.Errorf("bubblewrap was given %q, which the test does not know: %v", args[i], args)
return forgekey.Ran{Status: 1, Err: "unknown bubblewrap option"}
}
s.flags = append(s.flags, args[i])
switch args[i] {
case "--bind":
s.binds = append(s.binds, args[i+1])
case "--ro-bind", "--ro-bind-try":
s.roBinds = append(s.roBinds, args[i+1])
case "--overlay-src":
s.overlays = append(s.overlays, args[i+1])
case "--setenv":
s.env = append(s.env, args[i+1]+"="+args[i+2])
case "--chdir":
s.chdir = args[i+1]
}
i += n
}
if i+1 >= len(args) {
t.Errorf("bubblewrap was given no command: %v", args)
return forgekey.Ran{Status: 1, Err: "no command"}
}
s.command = args[i+1] + " " + strings.Join(args[i+2:], " ")
mu.Lock()
*seen = append(*seen, s)
mu.Unlock()
return run(ctx, s.env, args[i+1], args[i+2:]...)
}
}
// checkShutOff fails the test unless s ran with no network and none of the account: every namespace unshared, the
// environment cleared, nothing of the home or the state bound, the caches beneath a throwaway layer unless write.
func checkShutOff(t *testing.T, m *Module, s shutOff, write bool) {
t.Helper()
for _, f := range []string{"--unshare-all", "--clearenv", "--die-with-parent"} {
if !slices.Contains(s.flags, f) {
t.Errorf("%s ran without %s: %v", s.command, f, s.flags)
}
}
for _, b := range append(append([]string{}, s.binds...), s.roBinds...) {
if m.Home != "" && (b == m.Home || strings.HasPrefix(b, m.Home+"/")) {
t.Errorf("%s was given the account's home: %s", s.command, b)
}
if strings.HasPrefix(b, "/run") {
t.Errorf("%s was given the runtime directory: %s", s.command, b)
}
}
for _, e := range s.env {
if strings.HasPrefix(e, "HOME="+m.Home) && m.Home != "" {
t.Errorf("%s was told the account's home: %s", s.command, e)
}
}
caches := m.caches()
for _, c := range caches {
bound, layered := slices.Contains(s.binds, c), slices.Contains(s.overlays, c)
switch {
case write && (!bound || layered):
t.Errorf("a trunk build's %s does not write the shared cache %s (bound %v, layered %v)", s.command, c, bound, layered)
case !write && (bound || !layered):
t.Errorf("%s may write the shared cache %s (bound %v, layered %v): only a trunk build does", s.command, c, bound, layered)
}
}
if want := filepath.Join(m.Work, "ws1"); !slices.Contains(s.binds, want) {
t.Errorf("%s was not given its workspace %s read-write: %v", s.command, want, s.binds)
}
}
// 1309-S2: every go test of the unit level runs shut off in bubblewrap: no network, none of the account, the caches
// beneath a throwaway layer.
func TestTheUnitLevelRunsEveryGoTestShutOff(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "failing")
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
if _, st := runUnit(t, m, states, ws); st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
ran := 0
for _, c := range *calls {
if strings.HasPrefix(c.line, "go ") {
ran++
}
}
if ran == 0 || len(seen) != ran {
t.Fatalf("%d go commands ran, %d of them shut off: %v", ran, len(seen), *calls)
}
for _, s := range seen {
checkShutOff(t, m, s, false)
}
}
// 1309-S2: a try's tests: build_go_test's run pattern reaches go test as one -run word; a pattern that is a flag is refused.
func TestTheRunPatternReachesGoTestAndAFlagIsRefused(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "failing")
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
run := stepTool(t, m, "build_go_test")
if _, err := run(map[string]any{"workspace": ws, "level": "unit", "run": "TestA|TestB"}); err != nil {
t.Fatal(err)
}
states.waitEnd(t)
found := false
for _, c := range *calls {
found = found || strings.Contains(c.line, " -run TestA|TestB ")
}
if !found {
t.Errorf("no go test was given -run TestA|TestB: %v", *calls)
}
for _, bad := range []string{"-exec=/bin/sh", "a b", "(unclosed"} {
if runPatternOK(bad) {
t.Errorf("the run pattern %q was taken", bad)
}
}
}
// 1309-S3 and S4: a candidate's go build runs shut off with the caches beneath a throwaway layer; a trunk build
// (publish) writes them.
func TestABuildRunsShutOffAndOnlyATrunkBuildWritesTheCaches(t *testing.T) {
for _, publish := range []bool{false, true} {
m, states, _ := buildModule(t, manifestWith(goBundle), "binary")
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
run := stepTool(t, m, "build_go_build")
args := map[string]any{"workspace": "ws1", "module": "modules/demo", "artifact": "tools"}
if publish {
args["publish"] = true
}
if _, err := run(args); err != nil {
t.Fatal(err)
}
if st := states.waitEnd(t); st.State != delivery.StepPassed {
t.Fatalf("publish %v: the step ended %s: %s", publish, st.State, st.Reason)
}
if len(seen) != 1 || !strings.HasPrefix(seen[0].command, "go ") || !strings.Contains(seen[0].command, " build ") {
t.Fatalf("publish %v: shut off %v, want the one go build", publish, seen)
}
checkShutOff(t, m, seen[0], publish)
}
}
// 1309-S3: the digest's go list runs shut off.
func TestTheDigestListsPackagesShutOff(t *testing.T) {
m, _ := digestFixture(t)
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
digests(t, m, LevelUnit)
if len(seen) == 0 {
t.Fatal("the digest ran no command shut off")
}
for _, s := range seen {
if !strings.HasPrefix(s.command, "go ") || !strings.Contains(s.command, " list ") {
t.Errorf("the digest shut off %q, want go list", s.command)
}
checkShutOff(t, m, s, false)
}
}
// 1309-S5: the integration level runs its store, its bus and its tests in ONE shut-off process (this program again,
// `level <spec>`), so they reach each other and nothing else; its answer is the level's result. Nothing of the level
// runs outside but reading postgres's version.
func TestTheIntegrationLevelRunsInOneShutOffProcess(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "failing")
m.Executable = "/opt/bundle/mesh-build-go"
want := TestResult{Level: LevelIntegration, Passed: true, Tests: 3, Packages: []PackageResult{}, Store: "postgres 18"}
var outside []string
var seen []shutOff
m.Run = throughBubblewrap(t, func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
switch {
case name == "postgres" && len(args) == 1 && args[0] == "--version":
return forgekey.Ran{Stdout: "postgres (PostgreSQL) 18.6\n"}
case name == m.Executable && len(args) == 2 && args[0] == levelArg:
var spec levelSpec
if err := json.Unmarshal([]byte(args[1]), &spec); err != nil || spec.Workspace != ws || spec.Work != m.Work {
t.Errorf("the level's spec %s: %v", args[1], err)
}
raw, _ := json.Marshal(levelAnswer{Result: &want})
return forgekey.Ran{Stdout: string(raw) + "\n"}
}
outside = append(outside, name+" "+strings.Join(args, " "))
return forgekey.Ran{Status: 1, Err: "not expected here"}
}, &seen)
run := stepTool(t, m, "build_go_test")
if _, err := run(map[string]any{"workspace": ws, "level": LevelIntegration}); err != nil {
t.Fatal(err)
}
st := states.waitEnd(t)
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
var got TestResult
if err := json.Unmarshal(st.Result, &got); err != nil || got.Tests != 3 || !got.Passed || got.Store != "postgres 18" {
t.Errorf("the result %s (%v), want the shut-off process's", st.Result, err)
}
if len(outside) > 0 {
t.Errorf("ran outside the shut-off process: %v", outside)
}
if len(seen) != 1 {
t.Fatalf("shut off %d processes, want one: %v", len(seen), seen)
}
checkShutOff(t, m, seen[0], false)
if !slices.Contains(seen[0].roBinds, m.Executable) || slices.Contains(seen[0].roBinds, filepath.Dir(m.Executable)) {
t.Errorf("the program is bound %v; want the program alone, read-only", seen[0].roBinds)
}
}
// 1309-S1: a step is refused, and the module is unhealthy, while bubblewrap is missing or older than 0.10, naming it.
func TestAnOldOrMissingBubblewrapRefusesTheStepsAndIsUnhealthy(t *testing.T) {
for _, c := range []struct {
version string
err error
ok bool
}{{"0.13.0", nil, true}, {"0.10.0", nil, true}, {"0.9.0", nil, false}, {"", errors.New("bwrap: not found"), false}} {
m := moduleWith(newStates(), "go1.27.1")
m.Bubblewrap = func(context.Context) (string, error) { return c.version, c.err }
err := m.Check(context.Background())
h := m.Health(context.Background())
raw, _ := json.Marshal(h)
if c.ok {
if err != nil || strings.Contains(string(raw), "unhealthy") {
t.Errorf("bubblewrap %q: refused %v, health %s", c.version, err, raw)
}
continue
}
var ref *buildstep.Refusal
if !errors.As(err, &ref) || !strings.Contains(err.Error(), "bubblewrap") {
t.Errorf("bubblewrap %q %v: Check answered %v, want a refusal naming bubblewrap", c.version, c.err, err)
}
if !strings.Contains(string(raw), "unhealthy") || !strings.Contains(string(raw), "bubblewrap") {
t.Errorf("bubblewrap %q %v: health %s, want unhealthy naming bubblewrap", c.version, c.err, raw)
}
}
for out, want := range map[string]string{"bubblewrap 0.13.0\n": "0.13.0", "bubblewrap 0.10": "0.10"} {
if got, err := parseBubblewrapVersion(out); err != nil || got != want {
t.Errorf("%q read as %q, %v", out, got, err)
}
}
}
// stepTool is the Run of the named tool.
func stepTool(t *testing.T, m *Module, name string) func(map[string]any) (any, error) {
t.Helper()
for _, tool := range m.Tools(m.StepTools()) {
if tool.Name == name {
return tool.Run
}
}
t.Fatalf("no tool %s", name)
return nil
}
@@ -1,84 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"net"
"os"
"path/filepath"
"testing"
)
// 1309-S6 (ADR 0448's check): a test of the change, run by the unit level, reads no key and no login of the account,
// resolves no name, reaches no socket of the machine (a TCP port on its loopback, a unix socket) and leaves the shared
// caches unchanged. The probe is testdata/repos/probe.
func TestARunStepReachesNoKeyLoginNameOrSocketAndLeavesTheCachesUnchanged(t *testing.T) {
needTools(t, "the shut-off unit level", "go", "git", "bwrap")
tree := repository(t, "probe")
work, err := os.MkdirTemp("", "w")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(work) })
if err := os.MkdirAll(filepath.Join(work, "ws1"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Rename(tree, filepath.Join(work, "ws1", "tree")); err != nil {
t.Fatal(err)
}
home := t.TempDir()
key, login := filepath.Join(home, ".ssh", "mesh_ed25519"), filepath.Join(home, "broker")
for _, f := range []string{key, login} {
if err := os.MkdirAll(filepath.Dir(f), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(f, []byte("secret"), 0o600); err != nil {
t.Fatal(err)
}
}
tcp, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer tcp.Close()
sockDir := t.TempDir()
unix, err := net.Listen("unix", filepath.Join(sockDir, "s"))
if err != nil {
t.Fatal(err)
}
defer unix.Close()
m := &Module{Recipe: Current(), Home: home, Work: work, Cache: t.TempDir()}
m.Run = StepRunner(home)
for _, c := range m.caches() {
if err := os.MkdirAll(c, 0o700); err != nil {
t.Fatal(err)
}
}
res, err := m.runLevel(context.Background(), filepath.Join(work, "ws1", "tree"), levelOpts{workspace: "ws1", level: LevelUnit,
bounds: Bounds{Run: UnitBound * 100}, env: []string{"PROBE_KEY=" + key, "PROBE_LOGIN=" + login,
"PROBE_TCP=" + tcp.Addr().String(), "PROBE_SOCKET=" + filepath.Join(sockDir, "s"), "PROBE_CACHE=" + m.caches()[1]}})
if err != nil {
t.Fatal(err)
}
if !res.Passed {
t.Fatalf("the probe did not run: %+v", res)
}
raw, err := os.ReadFile(filepath.Join(work, "ws1", "tree", "seen.json"))
if err != nil {
t.Fatalf("the probe wrote nothing: %v", err)
}
var seen map[string]bool
if err := json.Unmarshal(raw, &seen); err != nil {
t.Fatal(err)
}
for what, reached := range seen {
if reached && what != "wroteCache" {
t.Errorf("the run step reached %s: %s", what, raw)
}
}
if _, err := os.Stat(filepath.Join(m.caches()[1], "written-by-a-candidate")); err == nil {
t.Errorf("a candidate's step changed the shared cache")
}
}
@@ -1,121 +0,0 @@
package main
// The integration level shut off (novox/hq ADR 0448, ADR 0470; issue 1309 S5): its store, its bus and its tests run in
// one bubblewrap of their own, so the tests reach the level's postgres (a socket in the run's directory) and its bus (a
// loopback port of the step's own network) and nothing of the machine. The process in it is this binary started again
// as `mesh-build-go level <spec>`: it runs the level as integrationHere does and prints its answer as one line of JSON.
// The tools bundle never runs a level in its own process, and a level never serves tools.
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// levelArg is the first argument that makes this binary an integration level's process.
const levelArg = "level"
// levelSpec is what the shut-off process is asked: the workspace, the packages and the -run pattern, the level's bounds.
type levelSpec struct {
Work string `json:"work"`
Cache string `json:"cache"`
Workspace string `json:"workspace"`
Packages []string `json:"packages,omitempty"`
Run string `json:"run,omitempty"`
Setup time.Duration `json:"setup"`
Bound time.Duration `json:"bound"`
}
// levelAnswer is the one line the process prints: the level's result, or why it could not run.
type levelAnswer struct {
Result *TestResult `json:"result,omitempty"`
Error string `json:"error,omitempty"`
}
// executable is this program's path, bound read-only into the step: Executable when a test set it.
func (m *Module) executable() (string, error) {
if m.Executable != "" {
return m.Executable, nil
}
exe, err := os.Executable()
if err != nil {
return "", err
}
return filepath.EvalSymlinks(exe)
}
// integrationShutOff runs the integration level in one shut-off process and reads its answer.
func (m *Module) integrationShutOff(ctx context.Context, workspace string, packages []string, pattern string) (*TestResult, error) {
exe, err := m.executable()
if err != nil {
return nil, fmt.Errorf("the integration level's program: %w", err)
}
b := m.integrationBounds()
spec, err := json.Marshal(levelSpec{Work: m.Work, Cache: m.Cache, Workspace: workspace, Packages: packages, Run: pattern, Setup: b.Setup, Bound: b.Run})
if err != nil {
return nil, err
}
for _, c := range m.caches() {
if err := os.MkdirAll(c, 0o700); err != nil {
return nil, fmt.Errorf("the shared cache %s: %w", c, err)
}
}
iso := m.isolation(workspace, false)
iso.ReadOnly = append(iso.ReadOnly, exe) // the program alone, not the directory it lies in
r := buildstep.Isolated(m.Run, iso)(ctx, nil, exe, levelArg, string(spec))
var answer levelAnswer
line := lastLine(r.Stdout)
if err := json.Unmarshal([]byte(line), &answer); err != nil || (answer.Result == nil && answer.Error == "") {
return nil, fmt.Errorf("the integration level's shut-off process answered nothing it says (status %d %s): %s", r.Status, r.Err, firstLine(r.Stderr+"\n"+r.Stdout))
}
if answer.Error != "" {
return nil, fmt.Errorf("%s", answer.Error)
}
return answer.Result, nil
}
func lastLine(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
// runLevelProcess is the shut-off process: args is the spec. It prints its answer and returns the exit status, 0 when it
// printed one.
func runLevelProcess(args []string) int {
say := func(a levelAnswer) int {
raw, _ := json.Marshal(a)
fmt.Println(string(raw))
return 0
}
if len(args) != 1 {
fmt.Fprintln(os.Stderr, "usage: mesh-build-go level <spec>")
return 2
}
var spec levelSpec
if err := json.Unmarshal([]byte(args[0]), &spec); err != nil {
return say(levelAnswer{Error: "the level's spec: " + err.Error()})
}
if !workspaceOK.MatchString(spec.Workspace) {
return say(levelAnswer{Error: fmt.Sprintf("%q is not a workspace id", spec.Workspace)})
}
m := &Module{Inside: true, Recipe: Current(), Work: spec.Work, Cache: spec.Cache, Home: os.Getenv("HOME"),
IntegrationBounds: Bounds{Setup: spec.Setup, Run: spec.Bound}, Say: func(string, ...any) {}}
m.Run = StepRunner(m.Home)
ctx := context.Background()
major, err := m.checkPostgres(ctx)
if err != nil {
return say(levelAnswer{Error: err.Error()})
}
tree := filepath.Join(spec.Work, spec.Workspace, "tree")
res, err := m.integrationHere(ctx, spec.Workspace, tree, spec.Packages, spec.Run, major)
if err != nil {
return say(levelAnswer{Error: err.Error()})
}
return say(levelAnswer{Result: res})
}
@@ -29,9 +29,6 @@ func main() {
if len(os.Args) > 1 && os.Args[1] == busArg {
os.Exit(runBus(os.Args[2:]))
}
if len(os.Args) > 1 && os.Args[1] == levelArg {
os.Exit(runLevelProcess(os.Args[2:]))
}
ctx := context.Background()
m := &Module{
Node: strings.TrimSpace(os.Getenv("MESH_NODE")),
@@ -50,19 +47,12 @@ func main() {
m.Home = u.HomeDir
}
m.Run = StepRunner(m.Home)
m.RegistryFile = m.Home + "/registry"
m.Probe = ProbeGo(m.Home, m.Cache)
m.Bubblewrap = ProbeBubblewrap
// Outside a launch by the runtime — a build, a check — there is no node and no state to write.
if m.Node != "" {
go m.Start(ctx)
}
go startKeeper(ctx)
// Every state it keeps, said on the bus before it serves (ADR 0495 §2; issue 1790).
if err := keepStates(); err != nil {
logf("announcing the states it keeps: %v", err)
os.Exit(1)
}
if err := stdio.Serve("", m.Tools(m.StepTools())); err != nil {
logf("%v", err)
os.Exit(1)
@@ -64,7 +64,7 @@ func TestManifestIsValidAndDeclaresTheAccountTheCapabilityAndThePackages(t *test
}
}
slices.Sort(packages)
if want := []string{"bubblewrap", "git", "openssh", "postgresql"}; !slices.Equal(packages, want) {
if want := []string{"git", "openssh", "postgresql"}; !slices.Equal(packages, want) {
t.Errorf("packages = %v, want %v", packages, want)
}
}
@@ -144,34 +144,7 @@ func TestGiteaReadsTheBuildModulesKeysAndTheCatalogueAcceptsThemTogether(t *test
t.Errorf("gitea reads %v, lacks %s", gitea.Reads, want)
}
}
}
// 1309-S1: every run step runs in bubblewrap, so the module declares its package (the lead's decision: each build module
// that runs a step through it declares it), beside the user-namespaces capability bubblewrap needs.
func TestTheModuleDeclaresBubblewrapAndTheCapabilityItNeeds(t *testing.T) {
m := readManifest(t)
if r := resource(t, m, "bubblewrap"); r["type"] != "package" || r["package"] != "bubblewrap" {
t.Errorf("the bubblewrap resource is %v, want the package bubblewrap", r)
}
if !slices.Contains(m.Capabilities, "user-namespaces") {
t.Errorf("capabilities %v lack user-namespaces", m.Capabilities)
}
}
// 1587-S1 (ADR 0498 §2): the module requires the artifact store and is told its address in a file of its home, which no
// run step is given (a run step has no network either, ADR 0448).
func TestTheModuleRequiresTheArtifactStoreAndIsToldItsAddress(t *testing.T) {
m := readManifest(t)
if !slices.Contains(m.Requires, "artifact-store") {
t.Errorf("requires %v lack artifact-store", m.Requires)
}
r := resource(t, m, "registry")
// The validator reads the template (files/registry) into the resource's content.
if content, _ := r["content"].(string); r["type"] != "file" || r["path"] != "${dir:state}/registry" || !strings.Contains(content, "${seat:mesh-artifact-store:reach}") {
t.Errorf("the registry resource is %v", r)
}
raw, err := os.ReadFile(filepath.Join(moduleDir, "files", "registry"))
if err != nil || strings.TrimSpace(string(raw)) != "${seat:mesh-artifact-store:reach}" {
t.Errorf("files/registry is %q (%v), want the store seat's reach", raw, err)
if problems := catalogue.StateReadsNothingDeclares(all); len(problems) > 0 {
t.Errorf("a read names a state its owner does not keep: %v", problems)
}
}
@@ -50,18 +50,6 @@ type Module struct {
IntegrationBounds Bounds
// Now is the clock the levels' times are read from; time.Now when nil.
Now func() time.Time
// Bubblewrap reads the installed bubblewrap's version (ProbeBubblewrap): a step is refused, and the module is
// unhealthy, while it is missing or older than BubblewrapLeast. Nil probes nothing.
Bubblewrap func(ctx context.Context) (string, error)
// Inside is the integration level's own process, started shut off in bubblewrap by a test step (level.go): its
// commands run as they are, never shut off a second time.
Inside bool
// RegistryFile holds the artifact store's address (host:port), placed by the manifest; Push puts a blob there
// (buildstep.PushBlob when nil; a test's fake store).
RegistryFile string
Push func(ctx context.Context, registry, repository, digest string, body []byte) error
// Executable is this program's path, which the integration level starts again shut off; os.Executable when empty.
Executable string
// StartBus starts the throwaway bus of a run in dir and answers its address and what stops it (the module's own
// program, StartBus in bus.go, when nil).
StartBus func(ctx context.Context, dir string) (url string, stop func(), err error)
@@ -81,10 +69,7 @@ func (m *Module) guard(installed string, probeErr error) buildstep.Guard {
// CheckRound.
func (m *Module) Check(ctx context.Context) error {
installed, err := m.Probe(ctx)
if err := m.guard(installed, err).Check(ctx); err != nil {
return err
}
return m.checkBubblewrap(ctx)
return m.guard(installed, err).Check(ctx)
}
// Start checks once, as the module starts, and then every CheckRound until ctx ends.
@@ -110,9 +95,6 @@ func (m *Module) Health(ctx context.Context) health.Answer {
if err := m.guard(installed, perr).Check(ctx); err != nil {
return health.Unhealthy(err.Error())
}
if err := m.checkBubblewrap(ctx); err != nil {
return health.Unhealthy(err.Error())
}
return health.Healthy(fmt.Sprintf("%s installed, the version the module builds with (recipe %s)", installed, m.Recipe.Version()))
}
@@ -1,51 +0,0 @@
//go:build integration
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0499 §4 (issue 1758): a fetch asked with the module's directory and its named paths leaves the workspace holding
// only those, the manifest and the lock files; the rest of the commit is gone before any run step; a request naming
// paths without a directory is refused before anything is fetched.
func TestAFetchWithNamedPathsNarrowsTheWorkspace(t *testing.T) {
f := newForge(t)
_, head := f.repo("novox/narrow", map[string]string{
"README.md": "the repository\n",
"lib/go.mod": "module fixture/lib\n\ngo 1.25.0\n",
"lib/module.json": `{"module":"lib","version":"1"}`,
"lib/cmd/x/x.go": "package main\n\nfunc main() {}\n",
"lib/docs/a.md": "not named\n",
"other/go.mod": "module fixture/other\n\ngo 1.25.0\n",
"other/o.go": "package other\n",
})
m, states := f.module(t)
res, st := fetched(t, m, states, map[string]any{"repository": "novox/narrow", "commit": head, "dir": "lib", "paths": []any{"cmd"}})
if st.State != delivery.StepPassed {
t.Fatalf("the fetch ended %s: %s", st.State, st.Reason)
}
tree := filepath.Join(m.Work, res.Workspace, "tree")
for _, kept := range []string{"lib/go.mod", "lib/module.json", "lib/cmd/x/x.go"} {
if _, err := os.Stat(filepath.Join(tree, kept)); err != nil {
t.Errorf("%s was narrowed away: %v", kept, err)
}
}
for _, gone := range []string{"README.md", "lib/docs/a.md", "other/o.go"} {
if _, err := os.Stat(filepath.Join(tree, gone)); err == nil {
t.Errorf("%s is still in the workspace", gone)
}
}
if strings.Join(res.Modules, ",") != "lib" {
t.Errorf("the fetch answered the modules %v, want lib alone", res.Modules)
}
_, st = fetched(t, m, states, map[string]any{"repository": "novox/narrow", "commit": head, "paths": []any{"cmd"}})
if st.State != delivery.StepFailed || !strings.Contains(st.Reason, "no module directory") {
t.Errorf("paths without a directory: %s %q", st.State, st.Reason)
}
}
@@ -41,7 +41,6 @@ func TestRealFetchAndBuildOfAMemberPinningAPullHead(t *testing.T) {
}
_, memberHead := f.repo("novox/member", member)
m, states := f.module(t)
withStore(t, m) // the build pushes its archive (ADR 0498)
res, st := fetched(t, m, states, map[string]any{"repository": "novox/member", "commit": memberHead})
if st.State != delivery.StepPassed {
@@ -50,9 +49,6 @@ func TestRealFetchAndBuildOfAMemberPinningAPullHead(t *testing.T) {
if _, err := os.Stat(filepath.Join(m.Work, res.Workspace, "pinned", "dep@"+head, "beside", "dep", "go", "dep.go")); err != nil {
t.Errorf("the pull request's repository was not placed beside the workspace: %v", err)
}
// The build runs shut off in bubblewrap (1309), which a build node has and build-agent's check container has not:
// there the build half is not run, said loudly; the fetch half above, the pin's own part (issue 1148), always is.
needTools(t, "the shut-off build of a member pinning a pull head", "bwrap")
built, err := m.build(context.Background(), map[string]any{"workspace": res.Workspace, "module": ".", "artifact": "tools"})
if err != nil {
t.Fatal(fmt.Errorf("the build: %w", err))
@@ -6,7 +6,6 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io/fs"
"os"
@@ -105,47 +104,9 @@ func (m *Module) placeSiblings(ctx context.Context, root string, siblings []sibl
placed = append(placed, Sibling{Repository: s.repository, Commit: s.commit, Dir: name})
}
sort.Slice(placed, func(i, j int) bool { return placed[i].Dir < placed[j].Dir })
if len(placed) > 0 {
// What the digest folds in (the lead's decision L5): the siblings this workspace's tests read, by commit.
raw, err := json.Marshal(placed)
if err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(root, besideRecord), raw, 0o600); err != nil {
return nil, err
}
}
return placed, nil
}
// besideRecord is the file in a workspace's root naming the siblings the fetch placed beside it.
const besideRecord = "beside.json"
// besideLines are the siblings a workspace's fetch placed, as the digest folds them in and echoes them: "<owner/repo>
// <commit>", sorted; none when it placed none.
func (m *Module) besideLines(workspace string) ([]string, error) {
raw, err := os.ReadFile(filepath.Join(m.Work, workspace, besideRecord))
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, err
}
var placed []Sibling
if err := json.Unmarshal(raw, &placed); err != nil {
return nil, fmt.Errorf("the workspace's record of its siblings: %w", err)
}
var out []string
for _, s := range placed {
if !repositoryOK.MatchString(s.Repository) || !commitOK.MatchString(s.Commit) {
return nil, fmt.Errorf("the workspace's record of its siblings names %q at %q", s.Repository, s.Commit)
}
out = append(out, s.Repository+" "+s.Commit)
}
sort.Strings(out)
return out, nil
}
func firstSymlink(dir string) string {
var found string
filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
@@ -171,7 +171,6 @@ func TestRealFetchMergesConflictsAndPlainlyFetches(t *testing.T) {
// sibling and passes with it; the sibling is a plain directory with no symlink in it; and a sibling that holds a
// symlink fails the fetch naming it.
func TestRealTestsReadTheSiblingBesideTheTree(t *testing.T) {
needTools(t, "the shut-off unit level", "go", "git", "bwrap") // the tests run in bubblewrap (1309)
f := newForge(t)
files := map[string]string{}
for _, n := range []string{"go.mod", "beside.go", "beside_test.go"} {
@@ -173,7 +173,7 @@ func TestSiblingsArePlacedBesideAndTheUnitLevelIsGivenTheirDirectory(t *testing.
}
calls := &[]cmdRecord{}
m.Run = throughBubblewrap(t, capturedRunner(t, "failing", calls), &[]shutOff{})
m.Run = capturedRunner(t, "failing", calls)
if err := os.MkdirAll(filepath.Join(m.Work, res.Workspace, "tree"), 0o755); err != nil {
t.Fatal(err)
}
@@ -1,42 +0,0 @@
package main
// The push of a done build's archive (novox/hq ADR 0498 §1 §2; issue 1587): from this module's own process, after the
// build step ended, with mesh-sdk's one client (buildstep.PushBlob), to the artifact store whose address the manifest
// binds into RegistryFile. No build step reaches the store: a run step has no network (ADR 0448).
import (
"context"
"fmt"
"net/http"
"os"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// registry is the artifact store's address, host:port, as the manifest bound it into RegistryFile.
func (m *Module) registry() (string, error) {
raw, err := os.ReadFile(m.RegistryFile)
if err != nil {
return "", fmt.Errorf("the artifact store's address cannot be read: %w", err)
}
addr := strings.TrimSpace(string(raw))
if addr == "" || strings.Contains(addr, "${") || strings.HasPrefix(addr, ":") || strings.HasSuffix(addr, ":") {
return "", fmt.Errorf("the artifact store's address in %s is %q, not host:port", m.RegistryFile, addr)
}
return addr, nil
}
// push puts body into the artifact store as a blob of repository (<module>/<artifact>) by its digest: Push when a test
// set it, else buildstep.PushBlob.
func (m *Module) push(ctx context.Context, repository, digest string, body []byte) error {
addr, err := m.registry()
if err != nil {
return err
}
if m.Push != nil {
return m.Push(ctx, addr, repository, digest, body)
}
return buildstep.PushBlob(ctx, &http.Client{Timeout: 10 * time.Minute}, addr, repository, digest, body)
}
@@ -1,120 +0,0 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// pushed are the blobs the fake artifact store took, by digest, for the tests that read an archive back.
var pushed sync.Map
// fakeStore makes m push to a store of the test's own: its address in RegistryFile, each blob kept by repository and
// digest; fail, when set, is the store's refusal.
type fakeStore struct {
mu sync.Mutex
blobs map[string][]byte // "<repository>@<digest>"
fail error
addr string
}
func withStore(t *testing.T, m *Module) *fakeStore {
t.Helper()
s := &fakeStore{blobs: map[string][]byte{}}
m.RegistryFile = filepath.Join(t.TempDir(), "registry")
if err := os.WriteFile(m.RegistryFile, []byte("store.test:5000\n"), 0o644); err != nil {
t.Fatal(err)
}
m.Push = func(ctx context.Context, registry, repository, digest string, body []byte) error {
s.mu.Lock()
defer s.mu.Unlock()
s.addr = registry
if s.fail != nil {
return s.fail
}
sum := sha256.Sum256(body)
if "sha256:"+hex.EncodeToString(sum[:]) != digest {
t.Errorf("pushed %s with bytes of another digest", digest)
}
s.blobs[repository+"@"+digest] = append([]byte(nil), body...)
pushed.Store(digest, s.blobs[repository+"@"+digest])
return nil
}
return s
}
// archiveOf is the archive a build pushed, by its digest.
func archiveOf(t *testing.T, res BuildResult) []byte {
t.Helper()
raw, ok := pushed.Load(res.Digest)
if !ok {
t.Fatalf("nothing was pushed under %s", res.Digest)
}
return raw.([]byte)
}
// 1587-S2 (ADR 0498 §1): a done build answers a reference of the stated form whose tail is the build's digest, and the
// store holds the archive there; a bundle and an archive alike.
func TestADoneBuildAnswersTheReferenceOfTheArchiveItPushed(t *testing.T) {
m, states, _ := buildModule(t, manifestWith(goBundle), "the program")
store := withStore(t, m)
res, st := runBuild(t, m, states, "tools")
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
if want := "artifact-store://demo/tools/blobs/" + res.Digest; res.Reference != want {
t.Errorf("reference %q, want %q", res.Reference, want)
}
if store.addr != "store.test:5000" || store.blobs["demo/tools@"+res.Digest] == nil {
t.Errorf("the store at %q holds %d blobs, not demo/tools@%s", store.addr, len(store.blobs), res.Digest)
}
a, states2, _ := archiveModule(t, "")
withStore(t, a)
arch, st := runBuild(t, a, states2, "files")
if st.State != delivery.StepPassed || arch.Reference != "artifact-store://demo/files/blobs/"+arch.Digest {
t.Errorf("the archive artifact: %s %q, reference %q", st.State, st.Reason, arch.Reference)
}
}
// 1587-S3 (ADR 0498 §2): a push the store refuses fails the step `not stored: <artifact>: <the store's words>`, and so
// does a store whose address the manifest did not place.
func TestARefusedPushFailsTheStepNotStored(t *testing.T) {
m, states, _ := buildModule(t, manifestWith(goBundle), "the program")
store := withStore(t, m)
store.fail = errors.New("store.test:5000 refused the blob: 500 Internal Server Error disk full")
if _, st := runBuild(t, m, states, "tools"); st.State != delivery.StepFailed || !strings.HasPrefix(st.Reason, delivery.NotStored+"tools: ") || !strings.Contains(st.Reason, "disk full") {
t.Errorf("the step ended %s: %q", st.State, st.Reason)
}
m.RegistryFile = filepath.Join(t.TempDir(), "absent")
if _, st := runBuild(t, m, states, "tools"); st.State != delivery.StepFailed || !strings.HasPrefix(st.Reason, delivery.NotStored+"tools: ") {
t.Errorf("no address: the step ended %s: %q", st.State, st.Reason)
}
}
// 1587-S4 (the lead's decision L9): the packed file is not left on the node once its push answered, a candidate's
// included: the workspace holds no archive, pushed or refused.
func TestThePackedFileLeavesTheNodeOnceItsPushAnswered(t *testing.T) {
for _, fail := range []error{nil, errors.New("refused")} {
m, states, _ := buildModule(t, manifestWith(goBundle), "the program")
withStore(t, m).fail = fail
runBuild(t, m, states, "tools")
var archives []string
filepath.WalkDir(m.Work, func(p string, d os.DirEntry, err error) error {
if err == nil && strings.HasSuffix(p, ".tar.gz") {
archives = append(archives, p)
}
return nil
})
if len(archives) != 0 {
t.Errorf("push answered %v: archives left on the node: %v", fail, archives)
}
}
}
@@ -1,112 +0,0 @@
//go:build integration
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/distribution/distribution/v3/configuration"
"github.com/distribution/distribution/v3/registry/handlers"
_ "github.com/distribution/distribution/v3/registry/storage/driver/inmemory"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// realStore starts the artifact store's own program (the distribution registry, modules/distribution) inside this
// test, with its in-memory storage, on a loopback port, and points m at it with the real push client. It is imported
// by this integration test alone: the module's program does not carry it.
func realStore(t *testing.T, m *Module) string {
t.Helper()
config := &configuration.Configuration{Storage: configuration.Storage{"inmemory": configuration.Parameters{}}}
config.HTTP.Secret = "a throwaway store"
config.Log.Level = "error"
config.Log.AccessLog.Disabled = true
srv := httptest.NewServer(handlers.NewApp(context.Background(), config))
t.Cleanup(srv.Close)
addr := strings.TrimPrefix(srv.URL, "http://")
m.RegistryFile = filepath.Join(t.TempDir(), "registry")
if err := os.WriteFile(m.RegistryFile, []byte(addr+"\n"), 0o644); err != nil {
t.Fatal(err)
}
m.Push = nil
return addr
}
// resolve reads a done artifact's reference from the store, as the controller will: artifact-store://<module>/
// <artifact>/blobs/<digest> is <store>/v2/<module>/<artifact>/blobs/<digest>.
func resolve(t *testing.T, addr string, a delivery.BuiltArtifact) []byte {
t.Helper()
path, ok := strings.CutPrefix(a.Reference, "artifact-store://")
if !ok {
t.Fatalf("%s's reference %q is not the artifact store's", a.Name, a.Reference)
}
resp, err := http.Get("http://" + addr + "/v2/" + path)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
t.Fatalf("the store answered %s for %s", resp.Status, a.Reference)
}
return body
}
// 1587-S9 (ADR 0498's check): a done build's every artifact names a reference whose tail is its digest, and that
// reference resolves in a real artifact store to bytes of that digest. The build is the real go's, shut off, of the
// fixture testdata/repos/build, with an archive beside its bundle.
func TestADoneBuildsArtifactsResolveInARealArtifactStore(t *testing.T) {
needTools(t, "the build", "go", "bwrap")
work := t.TempDir()
tree := filepath.Join(work, "ws1", "tree")
if err := os.MkdirAll(tree, 0o755); err != nil {
t.Fatal(err)
}
if out, err := exec.Command("cp", "-r", filepath.Join("testdata", "repos", "build")+"/.", tree).CombinedOutput(); err != nil {
t.Fatalf("%v %s", err, out)
}
manifest := `{"module":"fixture-build","version":"1","build":{"artifacts":[` +
`{"name":"tools","kind":"bundle","language":"go","system":"arch","from":"cmd/prog","binary":"prog-bin","loads":["prog-bin"]},` +
`{"name":"files","kind":"archive","from":"cmd"}]}}`
if err := os.WriteFile(filepath.Join(tree, "module.json"), []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
home := t.TempDir()
m := &Module{Home: home, Work: work, Cache: t.TempDir()}
m.Run = StepRunner(home)
addr := realStore(t, m)
done := delivery.BuilderBuild{V: 1, State: delivery.BuildDone, Manifest: strings.Repeat("0", 64)}
for _, a := range []struct{ name, kind string }{{"tools", "bundle"}, {"files", "archive"}} {
out, err := m.build(context.Background(), map[string]any{"workspace": "ws1", "module": ".", "artifact": a.name})
if err != nil {
t.Fatalf("%s: %v", a.name, err)
}
r := out.(BuildResult)
done.Artifacts = append(done.Artifacts, delivery.BuiltArtifact{Name: a.name, Kind: a.kind, Digest: r.Digest, Reference: r.Reference})
}
if _, err := delivery.EncodeBuilderBuild(done); err != nil {
t.Errorf("the done key the builder would write is refused: %v", err)
}
if ok, missing := done.Referenced(); !ok {
t.Errorf("the done key is not deliverable, missing %v", missing)
}
for _, a := range done.Artifacts {
if !strings.HasSuffix(a.Reference, "/blobs/"+a.Digest) {
t.Errorf("%s: the reference %q does not end in its digest %s", a.Name, a.Reference, a.Digest)
}
sum := sha256.Sum256(resolve(t, addr, a))
if got := "sha256:" + hex.EncodeToString(sum[:]); got != a.Digest {
t.Errorf("%s resolves to %s, its digest is %s", a.Name, got, a.Digest)
}
}
}
@@ -33,7 +33,6 @@ func (m *Module) testTool() StepTool {
"level": map[string]any{"type": "string", "enum": []string{LevelUnit, LevelIntegration}, "description": "the level"},
"packages": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "package patterns of each module, ./... when none"},
"run": map[string]any{"type": "string", "description": "optional go test -run pattern: only the tests it matches run (a try's tests)"},
},
"required": []string{"workspace", "level"},
},
@@ -57,10 +56,6 @@ func (m *Module) test(ctx context.Context, args map[string]any) (any, error) {
if err != nil {
return nil, err
}
pattern, _ := args["run"].(string)
if !runPatternOK(pattern) {
return nil, fmt.Errorf("%q is not a go test -run pattern", pattern)
}
if !workspaceOK.MatchString(workspace) {
return nil, fmt.Errorf("%q is not a workspace id", workspace)
}
@@ -70,9 +65,9 @@ func (m *Module) test(ctx context.Context, args map[string]any) (any, error) {
}
switch level {
case LevelUnit:
return m.runLevel(ctx, tree, levelOpts{workspace: workspace, level: LevelUnit, packages: packages, run: pattern, bounds: Bounds{Run: m.unitBound()}, env: m.besideEnv(workspace)})
return m.runLevel(ctx, tree, levelOpts{level: LevelUnit, packages: packages, bounds: Bounds{Run: m.unitBound()}, env: m.besideEnv(workspace)})
case LevelIntegration:
return m.integration(ctx, workspace, tree, packages, pattern)
return m.integration(ctx, workspace, tree, packages)
}
return nil, fmt.Errorf("the level %q is neither unit nor integration", level)
}
@@ -108,10 +103,6 @@ func packagesArg(v any) ([]string, error) {
// levelOpts are what a level asks of runLevel.
type levelOpts struct {
// workspace is the workspace's id: its commands run shut off in it (stepRunner).
workspace string
// run is go test's -run pattern, "" for every test.
run string
level string
tags []string
packages []string
@@ -138,10 +129,6 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
if err != nil {
return nil, err
}
run, err := m.stepRunner(o.workspace, false)
if err != nil {
return nil, err
}
began := m.clock()
for _, d := range dirs {
packages := byDir[d]
@@ -152,11 +139,8 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
if len(o.tags) > 0 {
args = append(args, "-tags", strings.Join(o.tags, ","))
}
if o.run != "" {
args = append(args, "-run", o.run)
}
args = append(args, packages...)
r := run(ctx, append(m.goEnvFor(filepath.Join(tree, d)), o.env...), "go", args...)
r := m.Run(ctx, append(m.goEnvFor(filepath.Join(tree, d)), o.env...), "go", args...)
rd := newReader(res, d)
noise := rd.read(r.Stdout)
rd.finish()
@@ -218,22 +202,3 @@ func modulePath(gomod string) string {
}
return ""
}
// runPatternOK says whether a -run pattern is one go test takes as a single argument: no flag, no space, no control
// character, and short ("" is every test).
func runPatternOK(p string) bool {
if len(p) > 512 || strings.HasPrefix(p, "-") {
return false
}
for _, r := range p {
if r <= ' ' || r == 0x7f {
return false
}
}
if p != "" {
if _, err := regexp.Compile(strings.Split(p, "/")[0]); err != nil {
return false
}
}
return true
}
@@ -21,9 +21,6 @@ func TestMain(m *testing.M) {
if len(os.Args) > 1 && os.Args[1] == busArg {
os.Exit(runBus(os.Args[2:]))
}
if len(os.Args) > 1 && os.Args[1] == levelArg {
os.Exit(runLevelProcess(os.Args[2:])) // and the integration level's shut-off process (level.go)
}
os.Exit(m.Run())
}
@@ -57,7 +54,7 @@ func repository(t *testing.T, name string) string {
// test_test.go read the output captured from them, and this one proves the capture is what the real go says today. It
// starts go test, so it carries the tag.
func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
needTools(t, "the unit level on fixtures", "go", "git", "bwrap")
needTools(t, "the unit level on fixtures", "go", "git")
want := map[string]struct {
passed, noTests bool
failed string
@@ -69,18 +66,11 @@ func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
"broken": {failed: "fixture/broken"},
}
for name, w := range want {
tree := repository(t, name)
home := t.TempDir()
m := &Module{Home: home, Work: t.TempDir(), Cache: t.TempDir()}
// The tree lies in a workspace of the work directory, as a fetch leaves it: the level runs shut off in it (1309).
tree := filepath.Join(m.Work, "ws1", "tree")
if err := os.MkdirAll(filepath.Dir(tree), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Rename(repository(t, name), tree); err != nil {
t.Fatal(err)
}
m.Run = StepRunner(home)
res, err := m.runLevel(context.Background(), tree, levelOpts{workspace: "ws1", level: LevelUnit, bounds: Bounds{Run: UnitBound}})
res, err := m.runLevel(context.Background(), tree, levelOpts{level: LevelUnit, bounds: Bounds{Run: UnitBound}})
if err != nil {
t.Fatalf("%s: %v", name, err)
}
@@ -106,7 +96,7 @@ func processesNamed(text string) []string {
// integration level passes, and afterwards no postgres or bus process of the run is left and its directory is gone.
// It starts postgres and a bus, so it runs in the merge check and in a try, never on the way.
func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.T) {
needTools(t, "the integration level", "go", "git", "initdb", "pg_ctl", "postgres", "bwrap")
needTools(t, "the integration level", "go", "git", "initdb", "pg_ctl", "postgres")
tree := repository(t, "store")
// A short work directory: a unix socket's path is bounded at 108 bytes.
work, err := os.MkdirTemp("/tmp", "w")
@@ -123,7 +113,7 @@ func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.
home := t.TempDir()
m := &Module{Recipe: Current(), Home: home, Work: work, Cache: t.TempDir()}
m.Run = StepRunner(home)
res, err := m.integration(context.Background(), "ws1", filepath.Join(work, "ws1", "tree"), nil, "")
res, err := m.integration(context.Background(), "ws1", filepath.Join(work, "ws1", "tree"), nil)
if err != nil {
t.Fatalf("the integration level: %v", err)
}
@@ -144,7 +134,7 @@ func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.
// 1487/AC2: given a fixture Go module, when it is built twice, both digests are equal and the archive holds one
// executable named as binaryName says. It runs the real go, so it carries the tag.
func TestBuildTwiceGivesOneDigestAndOneExecutable(t *testing.T) {
needTools(t, "the build", "go", "bwrap")
needTools(t, "the build", "go")
work := t.TempDir()
tree := filepath.Join(work, "ws1", "tree")
if err := os.MkdirAll(tree, 0o755); err != nil {
@@ -161,7 +151,6 @@ func TestBuildTwiceGivesOneDigestAndOneExecutable(t *testing.T) {
home := t.TempDir()
m := &Module{Home: home, Work: work, Cache: t.TempDir()}
m.Run = StepRunner(home)
withStore(t, m)
args := map[string]any{"workspace": "ws1", "module": ".", "artifact": "tools"}
first, err := m.build(context.Background(), args)
if err != nil {
@@ -175,7 +164,11 @@ func TestBuildTwiceGivesOneDigestAndOneExecutable(t *testing.T) {
if a.Digest != b.Digest || a.Binary != "prog-bin" {
t.Errorf("two builds of one tree: %+v and %+v", a, b)
}
zr, err := gzip.NewReader(bytes.NewReader(archiveOf(t, a)))
raw, err := os.ReadFile(a.Path)
if err != nil {
t.Fatal(err)
}
zr, err := gzip.NewReader(bytes.NewReader(raw))
if err != nil {
t.Fatal(err)
}
@@ -44,7 +44,7 @@ func workspaceFor(t *testing.T, captured string) (*Module, *keptStates, string,
states := newStates()
m := moduleWith(states, "go1.27.1")
m.Home, m.Work, m.Cache = t.TempDir(), t.TempDir(), t.TempDir()
m.Run = throughBubblewrap(t, capturedRunner(t, captured, calls), &[]shutOff{}) // the step's go runs shut off (1309)
m.Run = capturedRunner(t, captured, calls)
if err := os.MkdirAll(filepath.Join(m.Work, "ws1", "tree"), 0o755); err != nil {
t.Fatal(err)
}
@@ -185,12 +185,12 @@ func TestUnitLevelRunsGoTestJsonInTheModulesDirectoryWithTheStepsEnvironment(t *
// A go test that said nothing a test says (it could not run) ends the step failed, not the level.
func TestAGoTestThatCouldNotRunEndsTheStepFailed(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "empty")
m.Run = throughBubblewrap(t, func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if name == "go" {
return forgekey.Ran{Status: 1, Stderr: "go: cannot find main module\n"}
}
return forgekey.Ran{Stdout: "go.mod\n"}
}, &[]shutOff{})
}
_, st := runUnit(t, m, states, ws)
if st.State != delivery.StepFailed || !strings.Contains(st.Reason, "go: cannot find main module") {
t.Errorf("the step ended %s: %q", st.State, st.Reason)
@@ -1,3 +0,0 @@
module fixture/probe
go 1.25.0
@@ -1,34 +0,0 @@
package probe
import (
"encoding/json"
"net"
"os"
"testing"
"time"
)
// TestProbe tries what a change's code must not reach, named by PROBE_* (the test that runs it sets them through
// the environment of the step), and writes what it saw to seen.json beside it.
func TestProbe(t *testing.T) {
seen := map[string]bool{}
read := func(p string) bool { _, err := os.ReadFile(p); return err == nil }
seen["key"] = read(os.Getenv("PROBE_KEY"))
seen["login"] = read(os.Getenv("PROBE_LOGIN"))
_, err := net.LookupHost("git.novox.be")
seen["resolved"] = err == nil
reach := func(network, addr string) bool {
c, err := net.DialTimeout(network, addr, 2*time.Second)
if err == nil {
c.Close()
}
return err == nil
}
seen["tcp"] = reach("tcp", os.Getenv("PROBE_TCP"))
seen["socket"] = reach("unix", os.Getenv("PROBE_SOCKET"))
seen["wroteCache"] = os.WriteFile(os.Getenv("PROBE_CACHE")+"/written-by-a-candidate", []byte("x"), 0o644) == nil
raw, _ := json.Marshal(seen)
if err := os.WriteFile("seen.json", raw, 0o644); err != nil {
t.Fatal(err)
}
}
@@ -1,293 +0,0 @@
package main
// Copied from mesh-controller origin/main internal/builder/xz_framing.go (issue 1590): the build agent's walk of an xz
// file's framing before its reader decodes it. "The build seat" below is the build module now.
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"hash/crc32"
"io"
"os"
)
// xzBlocksMax and xzStreamsMax are the most blocks and the most concatenated streams an xz file may hold for
// the build seat to read it.
//
// **Why a block count is limited at all.** The xz reader (github.com/therootcompany/xz v1.0.1) keeps one
// filter decoder for every block that names a delta or BCJ filter, and frees them only when a stream ends:
// its block counters are reset per stream, not per block. A stream of many tiny blocks therefore grows the
// heap with its length: an `xz --block-size=1 --delta` file of 6 MB kept about 67 MiB live, which comes to
// about 5.6 GiB at the 512 MiB download cap (review 68 on mesh-controller pull request 264, hq issue 345).
//
// **Why 1,024.** At about 340 bytes of decoder a block, 1,024 blocks keep at most about 0.35 MiB live. A
// real release is far below it: Betterbird's archive is 1 stream with 1 block, and multi-threaded xz cuts
// blocks of three times the dictionary (24 MiB at the default preset), so even 512 MiB unpacked in blocks
// of 512 KiB stays within it. Streams are limited to 16 because a release is one stream; the decoders are
// reused from one stream to the next, so the stream limit only bounds the work of this walk.
const (
xzBlocksMax = 1024
xzStreamsMax = 16
)
// xzWithinLimits walks an xz file's framing, without decompressing anything, and refuses one holding more
// blocks or streams than the build seat reads.
//
// **The framing, not the index.** Each stream's index at its end lists its blocks, but the reader compares
// the index with the blocks only after it has decoded them all, so a file whose index lies would still be
// decoded block by block. This walk follows the same framing the reader follows: stream header, each block
// header, the block's LZMA2 chunks (each chunk header carries its own compressed size, so a chunk is skipped
// without being decoded), its padding and check, then the index and the footer. It counts the blocks the
// reader would decode, and it also refuses an index whose count of blocks is not the count walked.
func xzWithinLimits(file string) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
_, _, err = xzFraming(bufio.NewReader(f))
if errors.Is(err, errXzTooManyBlocks) || errors.Is(err, errXzTooManyStreams) {
return err
}
if err != nil {
return fmt.Errorf("it is not an xz file this build seat reads: %v", err)
}
return nil
}
var (
errXzTooManyBlocks = fmt.Errorf("it holds more than %d xz blocks, the most this build seat reads: "+
"its xz reader keeps a filter decoder for every block (hq issue 345)", xzBlocksMax)
errXzTooManyStreams = fmt.Errorf("it holds more than %d concatenated xz streams, the most this build "+
"seat reads", xzStreamsMax)
)
// xzFraming counts the streams and blocks of an xz file, checking its framing as it goes. It stops with
// errXzTooManyBlocks or errXzTooManyStreams at the first block or stream over the limit.
func xzFraming(r *bufio.Reader) (streams, blocks int, err error) {
full := func(n int) ([]byte, error) {
b := make([]byte, n)
if _, err := io.ReadFull(r, b); err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
return nil, err
}
return b, nil
}
skip := func(n int) error {
if got, err := r.Discard(n); err != nil || got != n {
return io.ErrUnexpectedEOF
}
return nil
}
for {
header, err := full(12)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: header: %v", streams+1, err)
}
if !bytes.Equal(header[:6], []byte{0xfd, '7', 'z', 'X', 'Z', 0x00}) {
return streams, blocks, fmt.Errorf("stream %d: no xz stream header", streams+1)
}
if header[6] != 0 || header[7]&0xf0 != 0 ||
crc32.ChecksumIEEE(header[6:8]) != binary.LittleEndian.Uint32(header[8:12]) {
return streams, blocks, fmt.Errorf("stream %d: a corrupt stream header", streams+1)
}
if streams++; streams > xzStreamsMax {
return streams, blocks, errXzTooManyStreams
}
checkSize := 0
if check := int(header[7]); check != 0 {
checkSize = 4 << ((check - 1) / 3)
}
inStream := 0
for {
first, err := r.ReadByte()
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: no index: %v", streams, io.ErrUnexpectedEOF)
}
if first == 0 {
break // the index indicator
}
rest, err := full(int(first)*4 + 3)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: header: %v", streams, inStream+1, err)
}
if err := xzBlockHeader(append([]byte{first}, rest...)); err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: %v", streams, inStream+1, err)
}
inStream++
if blocks++; blocks > xzBlocksMax {
return streams, blocks, errXzTooManyBlocks
}
// The block's LZMA2 chunks, up to the end marker.
size := 0
for {
control, err := r.ReadByte()
if err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: %v", streams, inStream, io.ErrUnexpectedEOF)
}
size++
if control == 0 {
break
}
switch {
case control == 1 || control == 2: // stored: a 16-bit size less one, then the bytes
b, err := full(2)
if err != nil {
return streams, blocks, err
}
n := int(binary.BigEndian.Uint16(b)) + 1
if err := skip(n); err != nil {
return streams, blocks, err
}
size += 2 + n
case control >= 0x80: // LZMA: unpacked and packed sizes, the properties byte on a reset
b, err := full(4)
if err != nil {
return streams, blocks, err
}
n := int(binary.BigEndian.Uint16(b[2:])) + 1
size += 4
if control >= 0xc0 {
if err := skip(1); err != nil {
return streams, blocks, err
}
size++
}
if err := skip(n); err != nil {
return streams, blocks, err
}
size += n
default:
return streams, blocks, fmt.Errorf("stream %d, block %d: an LZMA2 chunk of kind %#x",
streams, inStream, control)
}
}
if pad := (4 - size%4) % 4; pad > 0 {
b, err := full(pad)
if err != nil || !bytes.Equal(b, make([]byte, pad)) {
return streams, blocks, fmt.Errorf("stream %d, block %d: corrupt block padding", streams, inStream)
}
}
if err := skip(checkSize); err != nil {
return streams, blocks, err
}
}
// The index: its count of records, two numbers each, padding, CRC32.
used := 1
count, n, err := xzVLI(r)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
used += n
if count != uint64(inStream) {
return streams, blocks, fmt.Errorf("stream %d: its index lists %d blocks, but it holds %d",
streams, count, inStream)
}
for i := 0; i < 2*inStream; i++ {
_, n, err := xzVLI(r)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
used += n
}
if err := skip((4-used%4)%4 + 4); err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
footer, err := full(12)
if err != nil || footer[10] != 'Y' || footer[11] != 'Z' {
return streams, blocks, fmt.Errorf("stream %d: no stream footer", streams)
}
// Stream padding: zero bytes, a multiple of four, then the end or another stream.
zeros := 0
for {
b, err := r.ReadByte()
if err == io.EOF {
if zeros%4 != 0 {
return streams, blocks, errors.New("stream padding not a multiple of four bytes")
}
return streams, blocks, nil
}
if err != nil {
return streams, blocks, err
}
if b != 0 {
_ = r.UnreadByte()
break
}
zeros++
}
if zeros%4 != 0 {
return streams, blocks, errors.New("stream padding not a multiple of four bytes")
}
}
}
// xzBlockHeader checks one block header: its CRC32, its flags, and that its last filter is LZMA2, the only
// one whose chunks this walk can skip (and the only last filter the reader decodes).
func xzBlockHeader(h []byte) error {
size := len(h)
if crc32.ChecksumIEEE(h[:size-4]) != binary.LittleEndian.Uint32(h[size-4:]) {
return errors.New("a corrupt block header")
}
flags := h[1]
if flags&0x3c != 0 {
return errors.New("a block header with reserved flags")
}
body := bytes.NewReader(h[2 : size-4])
if flags&0x40 != 0 {
if _, _, err := xzVLI(body); err != nil {
return err
}
}
if flags&0x80 != 0 {
if _, _, err := xzVLI(body); err != nil {
return err
}
}
var last uint64
for i := 0; i <= int(flags&3); i++ {
id, _, err := xzVLI(body)
if err != nil {
return err
}
props, _, err := xzVLI(body)
if err != nil {
return err
}
if props > uint64(body.Len()) {
return errors.New("a corrupt block header")
}
if _, err := body.Seek(int64(props), io.SeekCurrent); err != nil {
return err
}
last = id
}
if last != 0x21 {
return fmt.Errorf("a block whose last filter is %#x, not LZMA2", last)
}
return nil
}
// xzVLI reads one of xz's variable-length integers (at most 9 bytes, 7 bits each), and how many bytes it took.
func xzVLI(r io.ByteReader) (uint64, int, error) {
var v uint64
for i := 0; i < 9; i++ {
b, err := r.ReadByte()
if err != nil {
return 0, i, io.ErrUnexpectedEOF
}
v |= uint64(b&0x7f) << (7 * i)
if b&0x80 == 0 {
if i > 0 && b == 0 {
return 0, i + 1, errors.New("a number not in its shortest form")
}
return v, i + 1, nil
}
}
return 0, 9, errors.New("a number longer than nine bytes")
}
@@ -1,180 +0,0 @@
package main
// Copied from mesh-controller origin/main internal/builder/xz_framing_test.go (issue 1590), with xz_framing.go: the
// build agent's walk of an xz file's framing, which this module now uses as the build agent does.
import (
"archive/tar"
"bufio"
"bytes"
"encoding/binary"
"errors"
"hash/crc32"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
// xzStream is one xz stream (no integrity check) of data cut into blocks of blockSize bytes, each block's
// LZMA2 data stored uncompressed, each block a delta filter before LZMA2 when delta is set: the shape of
// `xz --block-size=N --filters=delta,lzma2`, built without any xz tool. listed is how many blocks its index
// lists (-1: as many as it holds), so a stream whose index lies can be built too.
func xzStream(data []byte, blockSize int, delta bool, listed int) []byte {
var out bytes.Buffer
flags := []byte{0, 0}
out.Write([]byte{0xfd, '7', 'z', 'X', 'Z', 0x00})
out.Write(flags)
out.Write(binary.LittleEndian.AppendUint32(nil, crc32.ChecksumIEEE(flags)))
type record struct{ unpadded, uncompressed uint64 }
var records []record
for at := 0; at < len(data); at += blockSize {
block := data[at:min(at+blockSize, len(data))]
if delta {
// Delta-encoded with a distance of one; the decoder's history starts empty at each block.
encoded := make([]byte, len(block))
for i := range block {
encoded[i] = block[i]
if i > 0 {
encoded[i] -= block[i-1]
}
}
block = encoded
}
header := []byte{2, 0} // 12 bytes
if delta {
header[1] = 1 // two filters
header = append(header, 0x03, 0x01, 0x00)
}
header = append(header, 0x21, 0x01, 0x00) // LZMA2, a 4 KiB dictionary
for len(header) < 8 {
header = append(header, 0)
}
header = binary.LittleEndian.AppendUint32(header, crc32.ChecksumIEEE(header))
out.Write(header)
size := 0
for i, chunk := 0, block; len(chunk) > 0; i++ {
n := min(len(chunk), 1<<16)
control := byte(2)
if i == 0 {
control = 1 // the first chunk of a block resets the dictionary
}
out.WriteByte(control)
out.Write(binary.BigEndian.AppendUint16(nil, uint16(n-1)))
out.Write(chunk[:n])
size += 3 + n
chunk = chunk[n:]
}
out.WriteByte(0)
size++
out.Write(make([]byte, (4-size%4)%4))
records = append(records, record{uint64(len(header) + size), uint64(len(block))})
}
if listed >= 0 {
for len(records) < listed {
records = append(records, records[0])
}
records = records[:listed]
}
vli := func(b []byte, v uint64) []byte {
for v >= 0x80 {
b = append(b, byte(v)|0x80)
v >>= 7
}
return append(b, byte(v))
}
index := vli([]byte{0}, uint64(len(records)))
for _, r := range records {
index = vli(vli(index, r.unpadded), r.uncompressed)
}
for len(index)%4 != 0 {
index = append(index, 0)
}
index = binary.LittleEndian.AppendUint32(index, crc32.ChecksumIEEE(index))
out.Write(index)
footer := binary.LittleEndian.AppendUint32(nil, uint32(len(index)/4-1))
footer = append(footer, flags...)
out.Write(binary.LittleEndian.AppendUint32(nil, crc32.ChecksumIEEE(footer)))
out.Write(footer)
out.WriteString("YZ")
return out.Bytes()
}
// tarHolding is a tar holding betterbird/data with body.
func tarHolding(t *testing.T, body []byte) []byte {
t.Helper()
var out bytes.Buffer
w := tar.NewWriter(&out)
if err := w.WriteHeader(&tar.Header{Name: "betterbird/data", Mode: 0o644, Size: int64(len(body)),
Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
w.Write(body)
if err := w.Close(); err != nil {
t.Fatal(err)
}
return out.Bytes()
}
func writeTemp(t *testing.T, body []byte) string {
t.Helper()
file := filepath.Join(t.TempDir(), "r.tar.xz")
if err := os.WriteFile(file, body, 0o644); err != nil {
t.Fatal(err)
}
return file
}
func TestAnXzFileOfManyDeltaBlocksIsRefusedBeforeItIsDecoded(t *testing.T) {
// Review 68 on pull request 264: `xz --block-size=1 --delta` of a 200 kB tar kept about 67 MiB live in
// the reader, one delta decoder per block. Its index here lists one block, as a file crafted to pass a
// check of the index alone would; the walk counts the blocks the reader would decode.
body := make([]byte, 200_000)
for i := range body {
body[i] = byte(i * 7)
}
crafted := xzStream(tarHolding(t, body), 1, true, 1)
path := writeTemp(t, crafted)
var before, after runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&before)
err := unpackUpstream(path, t.TempDir(), "betterbird")
runtime.ReadMemStats(&after)
if err == nil || !strings.Contains(err.Error(), "more than 1024 xz blocks") {
t.Fatalf("a file of %d bytes in blocks of one byte was not refused for its blocks: %v", len(crafted), err)
}
if grew := after.TotalAlloc - before.TotalAlloc; grew > 4<<20 {
t.Fatalf("%d bytes were allocated for a refused file", grew)
}
}
func TestTheXzBlockLimitIsExact(t *testing.T) {
data := make([]byte, xzBlocksMax+1)
streams, blocks, err := xzFraming(bufio.NewReader(bytes.NewReader(xzStream(data[:xzBlocksMax], 1, true, -1))))
if err != nil || streams != 1 || blocks != xzBlocksMax {
t.Fatalf("%d blocks: %d streams, %d blocks, %v", xzBlocksMax, streams, blocks, err)
}
if _, _, err := xzFraming(bufio.NewReader(bytes.NewReader(xzStream(data, 1, true, -1)))); !errors.Is(err, errXzTooManyBlocks) {
t.Fatalf("%d blocks: %v", xzBlocksMax+1, err)
}
}
func TestAnXzFileOfTooManyStreamsIsRefused(t *testing.T) {
var file []byte
for i := 0; i <= xzStreamsMax; i++ {
file = append(file, xzStream([]byte("x"), 1, false, -1)...)
}
err := unpackUpstream(writeTemp(t, file), t.TempDir(), "")
if err == nil || !strings.Contains(err.Error(), "more than 16 concatenated xz streams") {
t.Fatalf("%v", err)
}
}
func TestAnXzIndexThatDoesNotListItsBlocksIsRefused(t *testing.T) {
file := xzStream(tarHolding(t, []byte("x")), 512, false, 1)
err := unpackUpstream(writeTemp(t, file), t.TempDir(), "betterbird")
if err == nil || !strings.Contains(err.Error(), "its index lists 1 blocks, but it holds 4") {
t.Fatalf("%v", err)
}
}
@@ -1,52 +0,0 @@
package main
import (
"context"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// releaseXzSHA256 is the sha256 of testdata/upstream-release.tar.xz: the tar stream of upstream-release.tar.gz,
// compressed with `xz -9e` (one stream, one block).
const releaseXzSHA256 = "7e59472615d7d654d6f72e8275c83e36203da9db1b91372dcd38a3ff7984f997"
// pinnedModuleServing is pinnedModule for any fixture file, served at /<its name>.
func pinnedModuleServing(t *testing.T, fixture, sha, extra string) (*Module, *keptStates) {
t.Helper()
body, err := os.ReadFile(filepath.Join("testdata", fixture))
if err != nil {
t.Fatal(err)
}
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/"+fixture {
http.NotFound(w, r)
return
}
w.Write(body)
}))
t.Cleanup(srv.Close)
m, states, _ := archiveModule(t, pinned("https://example.com/"+fixture, sha, extra))
client := srv.Client()
client.Transport.(*http.Transport).DialContext = func(ctx context.Context, network, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String())
}
m.HTTP = client
return m, states
}
// Issue 1590: a pinned release compressed with xz (as betterbird's is) is unpacked as the build agent unpacks it, and
// gives the build agent's archive: the same tar under gzip and under xz packs to one stream.
func TestAPinnedXzArchiveIsUnpackedAsTheBuildAgentDoes(t *testing.T) {
m, states := pinnedModuleServing(t, "upstream-release.tar.xz", releaseXzSHA256, `,"within":"release/app"`)
res, st := runBuild(t, m, states, "files")
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %q", st.State, st.Reason)
}
sameAsBuildAgent(t, res, buildAgentWithinStream, buildAgentWithinDigest)
}
-1
View File
@@ -1 +0,0 @@
${seat:mesh-artifact-store:reach}
+1 -63
View File
@@ -3,81 +3,19 @@ module meshbuildgo
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52
github.com/distribution/distribution/v3 v3.1.2
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
github.com/therootcompany/xz v1.0.1
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/docker-credential-helpers v0.9.5 // indirect
github.com/docker/go-events v0.0.0-20250808211157-605354379745 // indirect
github.com/docker/go-metrics v0.1.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/handlers v1.5.2 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
github.com/hashicorp/golang-lru/arc/v2 v2.0.5 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.5 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/otlptranslator v1.0.0 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/redis/go-redis/extra/rediscmd/v9 v9.0.5 // indirect
github.com/redis/go-redis/extra/redisotel/v9 v9.0.5 // indirect
github.com/redis/go-redis/v9 v9.7.3 // indirect
github.com/sirupsen/logrus v1.10.2 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/bridges/prometheus v0.67.0 // indirect
go.opentelemetry.io/contrib/exporters/autoexport v0.67.0 // indirect
go.opentelemetry.io/otel v1.45.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.21.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.43.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.64.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.21.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.42.0 // indirect
go.opentelemetry.io/otel/log v0.21.0 // indirect
go.opentelemetry.io/otel/metric v1.45.0 // indirect
go.opentelemetry.io/otel/sdk v1.45.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.21.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect
go.opentelemetry.io/otel/trace v1.45.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/net v0.59.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.11 // indirect
)
+6 -155
View File
@@ -1,67 +1,17 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52 h1:5k04XjpfYCq6j7rCM6Dy6FfrsBk5iUJlbwGf6WhUOUc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021093645-465beb4ea5e3 h1:OKxU64kHN1yQ/GOa6LwoUwVmiQ74d83WMSZJRFPVm9c=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021093645-465beb4ea5e3/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021185947-afbe371003a5 h1:1BiPKRPTYpgex8rSHWT9az0BteOafGWRDzd/Ba6GAcA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021185947-afbe371003a5/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc h1:ZJScJSnF6LgFl55/VJ9RT/lNroVWun469sXtx0nlVYw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bsm/ginkgo/v2 v2.7.0/go.mod h1:AiKlXPm7ItEHNc/2+OkrNG4E0ITzojb9/xWzvQ9XZ9w=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.26.0/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/distribution/distribution/v3 v3.1.2 h1:/Bv2YIqqSR00HiO49FT93hovy6r9IlrzwUDsJjCN5qg=
github.com/distribution/distribution/v3 v3.1.2/go.mod h1:Oz0Drs2Tx3iTXzK3DhUD4otfTfqlCxlWVC7QJeXWBP0=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/docker-credential-helpers v0.9.5 h1:EFNN8DHvaiK8zVqFA2DT6BjXE0GzfLOZ38ggPTKePkY=
github.com/docker/docker-credential-helpers v0.9.5/go.mod h1:v1S+hepowrQXITkEfw6o4+BMbGot02wiKpzWhGUZK6c=
github.com/docker/go-events v0.0.0-20250808211157-605354379745 h1:yOn6Ze6IbYI/KAw2lw/83ELYvZh6hvsygTVkD0dzMC4=
github.com/docker/go-events v0.0.0-20250808211157-605354379745/go.mod h1:Uw6UezgYA44ePAFQYUehOuCzmy5zmg/+nl2ZfMWGkpA=
github.com/docker/go-metrics v0.1.0 h1:r76KPNpstz+IvQKSWpYegSkkyzex0V3A1ZGVx6bhGlY=
github.com/docker/go-metrics v0.1.0/go.mod h1:PciI3sONtB051kXALN1JoIlpcu54E1FuPh+4DuqEzyw=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/handlers v1.5.2 h1:cLTUSsNkgcwhgRqvCNmdbRWG0A3N4F+M2nWKdScwyEE=
github.com/gorilla/handlers v1.5.2/go.mod h1:dX+xVpaxdSw+q0Qek8SSsl3dfMk3jNddUkMzo0GtH0w=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 h1:5VipnvEpbqr2gA2VbM+nYVbkIF28c5ZQfqCBQ5g2xfk=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0/go.mod h1:Hyl3n6Twe1hvtd9XUXDec4pTvgMSEixRuQKPTMH2bNs=
github.com/hashicorp/golang-lru/arc/v2 v2.0.5 h1:l2zaLDubNhW4XO3LnliVj0GXO3+/CGNJAg1dcN2Fpfw=
github.com/hashicorp/golang-lru/arc/v2 v2.0.5/go.mod h1:ny6zBSQZi2JxIeYcv7kt2sH2PXJtirBN7RDhRpxPkxU=
github.com/hashicorp/golang-lru/v2 v2.0.5 h1:wW7h1TG88eUIJ2i69gaE3uNVtEPIagzhGvHgwfx2Vm4=
github.com/hashicorp/golang-lru/v2 v2.0.5/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
@@ -72,109 +22,10 @@ github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEoIwkU+A6qos=
github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/redis/go-redis/extra/rediscmd/v9 v9.0.5 h1:EaDatTxkdHG+U3Bk4EUr+DZ7fOGwTfezUiUJMaIcaho=
github.com/redis/go-redis/extra/rediscmd/v9 v9.0.5/go.mod h1:fyalQWdtzDBECAQFBJuQe5bzQ02jGd5Qcbgb97Flm7U=
github.com/redis/go-redis/extra/redisotel/v9 v9.0.5 h1:EfpWLLCyXw8PSM2/XNJLjI3Pb27yVE+gIAfeqp8LUCc=
github.com/redis/go-redis/extra/redisotel/v9 v9.0.5/go.mod h1:WZjPDy7VNzn77AAfnAfVjZNvfJTYfPetfZk5yoSTLaQ=
github.com/redis/go-redis/v9 v9.0.5/go.mod h1:WqMKv5vnQbRuZstUwxQI195wHy+t4PuXDOjzMvcuQHk=
github.com/redis/go-redis/v9 v9.7.3 h1:YpPyAayJV+XErNsatSElgRZZVCwXX9QzkKYNvO7x0wM=
github.com/redis/go-redis/v9 v9.7.3/go.mod h1:bGUrSggJ9X9GUmZpZNEOQKaANxSGgOEBRltRTZHSvrA=
github.com/sirupsen/logrus v1.10.2 h1:G2SED73/qrAu6YwbdxOD6peLkCBI3z7L+ykJFTXJBBo=
github.com/sirupsen/logrus v1.10.2/go.mod h1:SLEg8TqYulVKKfIGHldVp2K2aYz2DKSVBq4g/H5bR7Q=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw=
github.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY=
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/bridges/prometheus v0.67.0 h1:dkBzNEAIKADEaFnuESzcXvpd09vxvDZsOjx11gjUqLk=
go.opentelemetry.io/contrib/bridges/prometheus v0.67.0/go.mod h1:Z5RIwRkZgauOIfnG5IpidvLpERjhTninpP1dTG2jTl4=
go.opentelemetry.io/contrib/exporters/autoexport v0.67.0 h1:4fnRcNpc6YFtG3zsFw9achKn3XgmxPxuMuqIL5rE8e8=
go.opentelemetry.io/contrib/exporters/autoexport v0.67.0/go.mod h1:qTvIHMFKoxW7HXg02gm6/Wofhq5p3Ib/A/NNt1EoBSQ=
go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU=
go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 h1:WseeVYf5dJZTsyPiyW5L14k5qsSibqXAMTSiFEDiWr0=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0/go.mod h1:SiLZnQS6Qk2eCpvr2CH/XMAOa64TWGXxEZJZCpD2Lmc=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.21.0 h1:fvNHGyo3CdRv/DQveXqhqBxnKTDyRaC5sMSQxilX/A0=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.21.0/go.mod h1:zyGrjRKL2B/6+Jc/m4/otPoZqV2MY9ZjC/aBraRO7zc=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.42.0 h1:MdKucPl/HbzckWWEisiNqMPhRrAOQX8r4jTuGr636gk=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.42.0/go.mod h1:RolT8tWtfHcjajEH5wFIZ4Dgh5jpPdFXYV9pTAk/qjc=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.43.0 h1:w1K+pCJoPpQifuVpsKamUdn9U0zM3xUziVOqsGksUrY=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.43.0/go.mod h1:HBy4BjzgVE8139ieRI75oXm3EcDN+6GhD88JT1Kjvxg=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 h1:88Y4s2C8oTui1LGM6bTWkw0ICGcOLCAI5l6zsD1j20k=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0/go.mod h1:Vl1/iaggsuRlrHf/hfPJPvVag77kKyvrLeD10kpMl+A=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0 h1:zWWrB1U6nqhS/k6zYB74CjRpuiitRtLLi68VcgmOEto=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0/go.mod h1:2qXPNBX1OVRC0IwOnfo1ljoid+RD0QK3443EaqVlsOU=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 h1:3iZJKlCZufyRzPzlQhUIWVmfltrXuGyfjREgGP3UUjc=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0/go.mod h1:/G+nUPfhq2e+qiXMGxMwumDrP5jtzU+mWN7/sjT2rak=
go.opentelemetry.io/otel/exporters/prometheus v0.64.0 h1:g0LRDXMX/G1SEZtK8zl8Chm4K6GBwRkjPKE36LxiTYs=
go.opentelemetry.io/otel/exporters/prometheus v0.64.0/go.mod h1:UrgcjnarfdlBDP3GjDIJWe6HTprwSazNjwsI+Ru6hro=
go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.21.0 h1:2lpf4hnrasYIsUyEXwnTZq5lsxrMm4T2Bwb06IctAZQ=
go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.21.0/go.mod h1:YWOW6h7jwApz9Pl76ie/izUsSPj0s2MdIlpqbPqaf3U=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.42.0 h1:lSZHgNHfbmQTPfuTmWVkEu8J8qXaQwuV30pjCcAUvP8=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.42.0/go.mod h1:so9ounLcuoRDu033MW/E0AD4hhUjVqswrMF5FoZlBcw=
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.42.0 h1:s/1iRkCKDfhlh1JF26knRneorus8aOwVIDhvYx9WoDw=
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.42.0/go.mod h1:UI3wi0FXg1Pofb8ZBiBLhtMzgoTm1TYkMvn71fAqDzs=
go.opentelemetry.io/otel/log v0.21.0 h1:SLsVDGmtyBrdw8/a2Z0bOIxou/+bN4z56GebH7T0LvA=
go.opentelemetry.io/otel/log v0.21.0/go.mod h1:iReetQrZL9Wyg84cCkOoCmqDHS5RCFfyxC7J+r8fn8g=
go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M=
go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s=
go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns=
go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI=
go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw=
go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA=
go.opentelemetry.io/otel/sdk/log v0.21.0 h1:QsE7XSR0ktQdKmRKGnR+f1ObGF32WG+7MER/P9KgmYc=
go.opentelemetry.io/otel/sdk/log v0.21.0/go.mod h1:m9mApjCoD2/1QuKCAptjv+BrG9WKOvQLVdNx+iBldTo=
go.opentelemetry.io/otel/sdk/log/logtest v0.21.0 h1:X+JBBgKlswCGYsmgL0CnoUUtlE//VB345c84jYAYkdQ=
go.opentelemetry.io/otel/sdk/log/logtest v0.21.0/go.mod h1:HD1575K8e6sIFBBDd5tZB3t9DlMytWXq9FuR+Y4rfjE=
go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o=
go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA=
go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag=
go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc=
go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc=
google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d/go.mod h1:K/+WGbmBY7aNW1HDw1fJnKYo10i0DkAX6pows00dLig=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d h1:IL4hdHzcUv2l/gcg98/Rj3FbtE6axwqslOW8SW0C+S0=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU=
google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+1 -15
View File
@@ -31,8 +31,7 @@
],
"requires": [
"forge-account",
"go-toolchain",
"artifact-store"
"go-toolchain"
],
"own-secrets": {
"broker": "${dir:state}/broker"
@@ -110,14 +109,6 @@
"mode": "0644",
"template": "files/gitconfig"
},
{
"id": "registry",
"type": "file",
"path": "${dir:state}/registry",
"owner": "mesh-build-go",
"mode": "0644",
"template": "files/registry"
},
{
"id": "git",
"type": "package",
@@ -128,11 +119,6 @@
"type": "package",
"package": "openssh"
},
{
"id": "bubblewrap",
"type": "package",
"package": "bubblewrap"
},
{
"id": "postgresql",
"type": "package",
@@ -1,19 +0,0 @@
package main
// What this module keeps on the bus, announced from its code (novox/hq ADR 0495 §2 §5; issue 1790): every state it
// keeps, each readable by others, since another module reads each today (the builder reads steps; gitea reads keys). A state of its own that it
// did not name here is refused at the call (stdio.Keeps).
import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// Kept are the states the module keeps.
var Kept = []announce.State{
{Name: "keys", Readable: true},
{Name: "steps", Readable: true},
}
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
func keepStates() error { return stdio.Keeps(Kept...) }
@@ -1,63 +0,0 @@
//go:build integration
package main
import (
"bufio"
"encoding/json"
"io"
"testing"
"github.com/nats-io/nats.go"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announcetest"
)
// 1790 (ADR 0495 §5): served as the runtime serves it, the module announces on the bus every state it keeps, as its
// code declares them. The runtime's part, publishing what the bundle asks to publish, is played by relay.
func TestTheModuleAnnouncesEveryStateItKeeps(t *testing.T) {
const module = "mesh-build-image"
announcetest.Check(t, module, Kept, func(url string) func() {
t.Setenv("MESH_MODULE", module)
t.Setenv("MESH_NODE", "shanks")
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
if err := keepStates(); err != nil {
t.Fatal(err)
}
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = stdio.ServeOn(module, nil, inR, outW) }()
go relay(t, module, nc, outR, inW)
return func() { _ = inW.Close(); _ = outW.Close(); nc.Close() }
})
}
// relay is the runtime's side of the channel: each mesh/publish the bundle asks is published on its module's event
// subject, mesh.mod.<module>.event.<key>, and answered.
func relay(t *testing.T, module string, nc *nats.Conn, from io.Reader, to io.Writer) {
sc := bufio.NewScanner(from)
sc.Buffer(make([]byte, 1<<20), 1<<24)
enc := json.NewEncoder(to)
for sc.Scan() {
var m struct {
ID json.RawMessage `json:"id"`
Method string `json:"method"`
Params struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
} `json:"params"`
}
if json.Unmarshal(sc.Bytes(), &m) != nil || m.Method != "mesh/publish" {
continue
}
if err := nc.Publish("mesh.mod."+module+".event."+m.Params.Key, m.Params.Body); err != nil {
t.Errorf("publishing %s: %v", m.Params.Key, err)
}
_ = nc.Flush()
_ = enc.Encode(map[string]any{"id": m.ID, "result": map[string]any{}})
}
}
@@ -47,11 +47,6 @@ func main() {
go m.Start(ctx)
}
go startKeeper(ctx)
// Every state it keeps, said on the bus before it serves (ADR 0495 §2; issue 1790).
if err := keepStates(); err != nil {
logf("announcing the states it keeps: %v", err)
os.Exit(1)
}
if err := stdio.Serve("", m.Tools(m.StepTools())); err != nil {
logf("%v", err)
os.Exit(1)
@@ -154,6 +154,9 @@ func TestGiteaReadsTheImageBuildModulesKeys(t *testing.T) {
t.Errorf("gitea reads %v, lacks %s", gitea.Reads, want)
}
}
if problems := catalogue.StateReadsNothingDeclares(all); len(problems) > 0 {
t.Errorf("a read names a state its owner does not keep: %v", problems)
}
}
// 1704b: over the real manifests, mesh-build-image where no module provides rootless-image-build is refused, the error
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildimage
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52 h1:5k04XjpfYCq6j7rCM6Dy6FfrsBk5iUJlbwGf6WhUOUc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022163107-d1af87a83a52/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc h1:ZJScJSnF6LgFl55/VJ9RT/lNroVWun469sXtx0nlVYw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021192148-78364e318bdc/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+1 -20
View File
@@ -58,15 +58,6 @@ toolchain install: a long step answers at once with its id and ends on `<node>.<
other `@novox/` packages install. A module with packages and no lock does not fail the fetch: its result lists it
under `refused` with the reason, and a build of that bundle fails naming it (issue 1459). Every command runs through `forgekey.ExecRunnerClean(0)` with PATH, HOME, the locale and its own
words, nothing of this process's environment. Workspaces older than a day are removed at each fetch.
The arguments are mesh-sdk's `buildstep.FetchRequest` (ADR 0499 §4): with `dir` and `paths` (from the builder, never
from a manifest) the workspace is narrowed to those paths, the manifest and the lock files before anything else. A
member is built against the SDK version its own lock names (the lead's decision L10 of R4e, ADR 0500 §5-§6): the
fetch places that version in the module's `node_modules/@novox/mesh-sdk`, from the group's package when the call
names one (`package: {repository, commit, ref}`: fetched, its dependencies installed from its own lock, built and
packed shut off, published nowhere) and its version and integrity are the lock's, else from the registry (integrity
checked); a version the registry lacks fails `waits for: @novox/mesh-sdk@<version>` (the builder queues the key), or,
with a group package of another version, fails naming both. A module whose lock names no SDK keeps the toolchain's.
A module whose artifact is a `package` gets its dependencies, devDependencies included, from its own lock instead.
- `build_node_test` {workspace, module, level} (issue 1491): the unit level is the type-check (the toolchain's tsc, the
package's `tsconfig.json` or the build agent's flags over its .ts files) and the `test` script; the integration level
is the `test:integration` script, the type-check being its setup. Bounds (the Go module's; ADR 0445 names none for
@@ -79,14 +70,4 @@ toolchain install: a long step answers at once with its id and ends on `<node>.<
toolchain's packages (not typescript or esbuild) copied to `stage-<artifact>/node_modules`: the build agent's depth
(`/app/modules/module`, `/app/node_modules`), so esbuild's `../../node_modules/...` path comments, and the bundle's
digest, equal its. Copies, not links, because the mesh makes no symlink. A bundle that keeps an external gets them in its own `node_modules`.
Once the build ended, this process pushes the packed archive to the artifact store (novox/hq ADR 0498 §1 §2, mesh-sdk's
`buildstep.PushBlob`, the address in `${dir:state}/registry`) under `<module>/<artifact>` by its digest, a candidate's as a
trunk build's, and answers `reference: artifact-store://<module>/<artifact>/blobs/<digest>`; a push that fails fails the
step `not stored: <artifact>: <the store's words>`. The packed bytes never stay on the node. tsc and esbuild, like the
type-check and the test script, run shut off in bubblewrap (ADR 0448): no network, none of the account, the toolchain
read-only, the npm cache beneath a throwaway layer unless the step is told `publish` (a trunk build).
A `package` artifact (ADR 0498 §5) is built by its own `build` script and packed with `npm pack --ignore-scripts`, both
shut off; its digest is the packed file's. Only with `publish` is it published, by this process with the registry's
write credential (never in a workspace or a step's mounts): a version the registry lacks is published, the same
packed file already there is done without a publish, another file under that version fails `version taken:
<name>@<version>`; the reference `<name>@<version>` is answered only then. A candidate answers its digest alone.
The packed archive is written beside the tree as `<artifact>.tar.gz`.
@@ -1,20 +0,0 @@
package main
// What this module keeps on the bus, announced from its code (novox/hq ADR 0495 §2 §5; issue 1790): every state it
// keeps, each readable by others, since another module reads each today (the builder, the checker and the indexer read toolchains; the builder and the checker read steps; gitea reads keys). A state of its own that it
// did not name here is refused at the call (stdio.Keeps).
import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
)
// Kept are the states the module keeps.
var Kept = []announce.State{
{Name: "toolchains", Readable: true},
{Name: "keys", Readable: true},
{Name: "steps", Readable: true},
}
// keepStates announces Kept as this module's (stdio.Keeps): main calls it before it serves.
func keepStates() error { return stdio.Keeps(Kept...) }
@@ -1,63 +0,0 @@
//go:build integration
package main
import (
"bufio"
"encoding/json"
"io"
"testing"
"github.com/nats-io/nats.go"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announcetest"
)
// 1790 (ADR 0495 §5): served as the runtime serves it, the module announces on the bus every state it keeps, as its
// code declares them. The runtime's part, publishing what the bundle asks to publish, is played by relay.
func TestTheModuleAnnouncesEveryStateItKeeps(t *testing.T) {
const module = "mesh-build-node"
announcetest.Check(t, module, Kept, func(url string) func() {
t.Setenv("MESH_MODULE", module)
t.Setenv("MESH_NODE", "shanks")
nc, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
if err := keepStates(); err != nil {
t.Fatal(err)
}
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = stdio.ServeOn(module, nil, inR, outW) }()
go relay(t, module, nc, outR, inW)
return func() { _ = inW.Close(); _ = outW.Close(); nc.Close() }
})
}
// relay is the runtime's side of the channel: each mesh/publish the bundle asks is published on its module's event
// subject, mesh.mod.<module>.event.<key>, and answered.
func relay(t *testing.T, module string, nc *nats.Conn, from io.Reader, to io.Writer) {
sc := bufio.NewScanner(from)
sc.Buffer(make([]byte, 1<<20), 1<<24)
enc := json.NewEncoder(to)
for sc.Scan() {
var m struct {
ID json.RawMessage `json:"id"`
Method string `json:"method"`
Params struct {
Key string `json:"key"`
Body json.RawMessage `json:"body"`
} `json:"params"`
}
if json.Unmarshal(sc.Bytes(), &m) != nil || m.Method != "mesh/publish" {
continue
}
if err := nc.Publish("mesh.mod."+module+".event."+m.Params.Key, m.Params.Body); err != nil {
t.Errorf("publishing %s: %v", m.Params.Key, err)
}
_ = nc.Flush()
_ = enc.Encode(map[string]any{"id": m.ID, "result": map[string]any{}})
}
}
@@ -18,9 +18,12 @@ import (
"strings"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// launcherSuffix is what an entrypoint's launcher is called beside it: index.js is started as index.serve.mjs
// (novox/hq ADR 0193).
const launcherSuffix = ".serve.mjs"
// Out is where a bundle's compiled files go in the module's directory (builder toolchain.go:118).
func Out(artifact string) string { return ".mesh-build/" + artifact }
@@ -29,10 +32,7 @@ const banner = "--banner:js=import { createRequire as __meshRequire } from 'node
// artifact is the part of a module.json artifact the build reads.
type artifact struct {
// module is the manifest's module name, which the artifact is stored under.
module string
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
Entrypoints []string `json:"entrypoints"`
External []string `json:"external"`
@@ -46,10 +46,9 @@ type BuildResult struct {
// Digest is "sha256:" and the hex of the archive's bytes (buildstep.Pack).
Digest string `json:"digest"`
Bytes int `json:"bytes"`
// Reference is where the packed bundle is in the artifact store (novox/hq ADR 0498 §1):
// artifact-store://<module>/<artifact>/blobs/<digest>. The packed file never stays on the node.
Reference string `json:"reference"`
Files []string `json:"files"`
// Archive is the packed file, in the workspace beside its tree.
Archive string `json:"archive"`
Files []string `json:"files"`
}
func (m *Module) buildTool() StepTool {
@@ -64,7 +63,6 @@ func (m *Module) buildTool() StepTool {
"workspace": map[string]any{"type": "string", "description": "the workspace id a fetch step answered"},
"module": map[string]any{"type": "string", "description": "the module's directory in the workspace, '.' when it is the root"},
"artifact": map[string]any{"type": "string", "description": "the name of the bundle in the module's module.json"},
"publish": map[string]any{"type": "boolean", "description": "true for a merged trunk commit alone: the build writes the shared cache; without it (a candidate) the cache is read beneath a throwaway layer"},
},
"required": []string{"workspace", "artifact"},
},
@@ -77,7 +75,6 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
workspace, _ := args["workspace"].(string)
module, _ := args["module"].(string)
name, _ := args["artifact"].(string)
publish, _ := args["publish"].(bool)
root, dir, err := m.moduleDirOf(workspace, module)
if err != nil {
return nil, err
@@ -86,9 +83,6 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
if err != nil {
return nil, err
}
if a.Kind == "package" {
return m.buildPackage(ctx, workspace, module, root, dir, a, publish)
}
if why, err := lockProblem(dir, module); err != nil || why != "" {
if err == nil {
err = fmt.Errorf("%s was refused by the fetch: %s", name, why)
@@ -118,14 +112,14 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
tsc := []string{m.tsc(), "--module", "NodeNext", "--moduleResolution", "NodeNext", "--target", "ES2022",
"--rootDir", ".", "--outDir", out}
tsc = append(tsc, sourcesFor(a.Entrypoints, out, ".ts")...)
if why := ranFailed(m.runIn(ctx, root, publish, dir, nil, "node", tsc...)); why != "" {
if why := ranFailed(m.runIn(ctx, dir, nil, "node", tsc...)); why != "" {
return nil, fmt.Errorf("tsc for %s: %s", a.Name, why)
}
launchers, err := writeLaunchers(filepath.Join(dir, out), a)
if err != nil {
return nil, err
}
if err := m.bundle(ctx, root, publish, dir, a, launchers); err != nil {
if err := m.bundle(ctx, dir, a, launchers); err != nil {
return nil, err
}
bundleDir := filepath.Join(dir, final)
@@ -138,10 +132,9 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
if err != nil {
return nil, fmt.Errorf("packing %s: %w", a.Name, err)
}
// Pushed from this process once the build ended, a candidate's as a trunk build's (ADR 0498 §1 §2): the step answers
// the reference, or fails not stored. The packed bytes are never written to the node (the lead's decision L9).
if err := m.push(ctx, a.module+"/"+a.Name, digest, archive); err != nil {
return nil, fmt.Errorf("%s%s: %v", delivery.NotStored, a.Name, err)
file := filepath.Join(root, a.Name+".tar.gz")
if err := os.WriteFile(file, archive, 0o600); err != nil {
return nil, err
}
files := []string{}
_ = filepath.WalkDir(bundleDir, func(p string, d os.DirEntry, err error) error {
@@ -152,13 +145,12 @@ func (m *Module) build(ctx context.Context, args map[string]any) (any, error) {
return nil
})
sort.Strings(files)
return BuildResult{Workspace: workspace, Module: module, Artifact: a.Name, Digest: digest, Bytes: len(archive),
Reference: buildstep.BlobReference(a.module, a.Name, digest), Files: files}, nil
return BuildResult{Workspace: workspace, Module: module, Artifact: a.Name, Digest: digest, Bytes: len(archive), Archive: file, Files: files}, nil
}
// bundle makes every entrypoint and every launcher ONE file, with esbuild (builder.go:1971-2039): an entrypoint is
// bundled in place under its own name, a launcher keeps its name with the extension .mjs.
func (m *Module) bundle(ctx context.Context, root string, publish bool, dir string, a artifact, launchers map[string]string) error {
func (m *Module) bundle(ctx context.Context, dir string, a artifact, launchers map[string]string) error {
out, final := Out(a.Name), Out(a.Name)+".bundled"
if err := os.RemoveAll(filepath.Join(dir, final)); err != nil {
return err
@@ -185,7 +177,7 @@ func (m *Module) bundle(ctx context.Context, root string, publish bool, dir stri
return nil
}
args := append(append(append([]string{}, entries...), common...), extra...)
if why := ranFailed(m.runIn(ctx, root, publish, dir, nil, m.esbuild(), args...)); why != "" {
if why := ranFailed(m.runIn(ctx, dir, nil, m.esbuild(), args...)); why != "" {
return fmt.Errorf("esbuild for %s: %s", a.Name, why)
}
return nil
@@ -212,12 +204,15 @@ func (m *Module) bundle(ctx context.Context, root string, publish bool, dir stri
}
// writeLaunchers writes, beside every entrypoint, an executable that imports the entrypoint and serves what it
// registered over MCP on stdio (builder.go:1926-1956; novox/hq ADR 0193). Which entrypoint gets which launcher is
// mesh-sdk's one rule, buildstep.Launchers, which the controller reads a bundle by (R4e). Its answer is each
// entrypoint's launcher, relative to the bundle's root.
// registered over MCP on stdio (builder.go:1926-1956; novox/hq ADR 0193). Its answer is each entrypoint's launcher,
// relative to the bundle's root.
func writeLaunchers(root string, a artifact) (map[string]string, error) {
out := map[string]string{}
for entry, launcher := range buildstep.Launchers(a.Language, a.Entrypoints) {
for _, entry := range a.Entrypoints {
if !strings.HasSuffix(entry, ".js") {
continue
}
launcher := strings.TrimSuffix(entry, ".js") + launcherSuffix
body := "#!/usr/bin/env node\n" +
"// Written by the mesh's builder (novox/hq ADR 0193): serve what " + entry + " registers,\n" +
"// over MCP on stdio, as the module the node's runtime names in MESH_SERVED_MODULE.\n" +
@@ -265,8 +260,7 @@ func readArtifact(dir, name string) (artifact, error) {
return artifact{}, fmt.Errorf("the module has no module.json: %w", err)
}
var m struct {
Module string `json:"module"`
Build struct {
Build struct {
Artifacts []artifact `json:"artifacts"`
} `json:"build"`
}
@@ -277,17 +271,12 @@ func readArtifact(dir, name string) (artifact, error) {
if a.Name != name {
continue
}
if a.Kind == "package" && a.Language == "typescript" {
a.module = m.Module
return a, nil // a package is built by its own build script and packed (package.go), from no entrypoint
}
if a.Language != "typescript" {
return artifact{}, fmt.Errorf("artifact %q is %q, and this module builds typescript", name, a.Language)
}
if len(a.Entrypoints) == 0 {
return artifact{}, fmt.Errorf("artifact %q names no entrypoints", name)
}
a.module = m.Module
return a, nil
}
return artifact{}, fmt.Errorf("module.json declares no artifact %q", name)
@@ -17,7 +17,7 @@ import (
// registry for @novox, which reads without a credential) and makes two workspaces holding one fixture bundle.
func realMachine(t *testing.T) (*Module, string) {
t.Helper()
for _, p := range []string{"node", "npm", "bwrap"} { // bwrap: tsc, esbuild and the scripts run shut off (1309)
for _, p := range []string{"node", "npm"} {
if _, err := exec.LookPath(p); err != nil {
t.Skipf("NOT RUN: no %s installed where this check runs (the mesh's Go toolchain image holds none); proven live on a build node", p)
}
@@ -39,7 +39,6 @@ func realMachine(t *testing.T) (*Module, string) {
Toolchain: &Toolchain{Paths: Paths{Home: home, Cache: cache}, Run: run, Registry: func() (Registry, error) {
return Registry{URL: "https://git.novox.be/api/packages/novox/npm/"}, nil
}}}
withStore(t, m)
if err := m.Toolchain.Ensure(context.Background(), m.Recipe.Version()); err != nil {
t.Fatal(err)
}
@@ -90,28 +89,3 @@ func TestTheFixturePassesTheUnitLevel(t *testing.T) {
t.Errorf("result = %+v", r)
}
}
// Issue 1869: a package whose test imports its own build output (as mesh-sdk's do) passes the unit level: the test step
// emits it with the real tsc, shut off, before the test script runs.
func TestAPackageWhoseTestsImportItsBuildPassesTheUnitLevel(t *testing.T) {
if _, err := exec.LookPath("bwrap"); err != nil {
t.Skip("NOT RUN: no bubblewrap installed where this check runs; every build node has it (the module declares it)")
}
m, work := realMachine(t)
for p, body := range map[string]string{
"modules/p/package.json": `{"name":"p","type":"module","scripts":{"build":"tsc","test":"node --test test/a.test.mjs","prepare":"tsc"}}`,
"modules/p/tsconfig.json": `{"compilerOptions":{"module":"NodeNext","moduleResolution":"NodeNext","target":"ES2022","outDir":"dist","rootDir":"src"},"include":["src"]}`,
"modules/p/src/index.ts": "export const answer: number = 42;\n",
"modules/p/test/a.test.mjs": "import { test } from 'node:test';\nimport assert from 'node:assert';\nimport { answer } from '../dist/index.js';\n" +
"test('the build is there', () => assert.equal(answer, 42));\n",
} {
write(t, filepath.Join(work, "w1", "tree", filepath.FromSlash(p)), body, 0o644)
}
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/p", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed || !slices.Equal(r.Ran, []string{"build", "test"}) {
t.Errorf("result = %+v", r)
}
}
@@ -6,14 +6,12 @@ import (
"compress/gzip"
"context"
"io"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
@@ -98,10 +96,12 @@ func TestTscAndEsbuildRunWithTheBuildAgentsFlags(t *testing.T) {
}
}
// readArchive reads the archive a build pushed (ADR 0498: the packed file never stays on the node).
func readArchive(t *testing.T, r BuildResult) map[string]os.FileMode {
func readArchive(t *testing.T, path string) map[string]os.FileMode {
t.Helper()
raw := archiveOf(t, r)
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
zr, err := gzip.NewReader(bytes.NewReader(raw))
if err != nil {
t.Fatal(err)
@@ -135,7 +135,7 @@ func TestTheBundleIsPackedWithItsLauncherAndPackageJson(t *testing.T) {
if want := []string{"package.json", "tools/index.js", "tools/index.serve.mjs"}; !slices.Equal(r.Files, want) {
t.Errorf("files = %v, want %v", r.Files, want)
}
modes := readArchive(t, r)
modes := readArchive(t, r.Archive)
if len(modes) != 3 || modes["tools/index.serve.mjs"]&0o111 == 0 {
t.Errorf("archive = %v, want the three files with the launcher executable", modes)
}
@@ -143,7 +143,7 @@ func TestTheBundleIsPackedWithItsLauncherAndPackageJson(t *testing.T) {
if err != nil || again.(BuildResult).Digest != r.Digest {
t.Errorf("a second build: %v, digest %v, want %s", err, again, r.Digest)
}
launcher, _ := os.ReadFile(filepath.Join(m.Work, "w1", "stage-tools", "modules", "module", ".mesh-build", "tools", "tools", "index.serve.mjs"))
launcher, _ := os.ReadFile(filepath.Join(filepath.Dir(r.Archive), "stage-tools", "modules", "module", ".mesh-build", "tools", "tools", "index.serve.mjs"))
if !strings.Contains(string(launcher), `import { serveRegisteredOverStdio } from "@novox/mesh-sdk/stdio";`) || !strings.Contains(string(launcher), `await import("./index.js");`) {
t.Errorf("launcher = %q", launcher)
}
@@ -181,7 +181,7 @@ func TestThePackagesAreCopiedBesideTheTreeAndIntoTheBundleOnlyForExternals(t *te
t.Errorf("the toolchain's own %s was copied", p)
}
}
for f := range readArchive(t, out.(BuildResult)) {
for f := range readArchive(t, out.(BuildResult).Archive) {
if strings.HasPrefix(f, "node_modules") {
t.Errorf("a bundle without externals carries %s", f)
}
@@ -254,16 +254,3 @@ func TestTheBuildRunsWithThePackagesTwoLevelsAboveTheModule(t *testing.T) {
}
}
}
// R4e: the launchers a build writes are mesh-sdk's one rule (buildstep.Launchers), the names the controller reads a bundle
// by; an entrypoint that is not JavaScript gets none.
func TestTheLaunchersAreTheSDKsRule(t *testing.T) {
a := artifact{Language: "typescript", Entrypoints: []string{"tools/index.js", "web/server.js", "assets/x.css"}}
got, err := writeLaunchers(t.TempDir(), a)
if err != nil {
t.Fatal(err)
}
if want := buildstep.Launchers(a.Language, a.Entrypoints); !maps.Equal(got, want) || len(got) != 2 {
t.Errorf("launchers %v, mesh-sdk's rule %v", got, want)
}
}
@@ -138,25 +138,18 @@ func (m *Module) test(ctx context.Context, args map[string]any) (any, error) {
return nil
}
if len(sources) > 0 {
// A package that builds (a build or prepare script, which the fetch's --ignore-scripts never runs) is emitted
// by tsc into the workspace, shut off like every run step, so its tests find what they import of its own output
// (issue 1869: mesh-sdk's tests import ../dist); any other is type-checked without emitting.
name, line := "type-check", m.typeCheck(dir, sources)
if emit := m.emitBuild(dir, scripts); emit != nil {
name, line = "build", emit
}
start := m.now()
r := m.runIn(ctx, root, false, dir, nil, "node", line...)
r := m.runIn(ctx, dir, nil, "node", m.typeCheck(dir, sources)...)
check = m.now().Sub(start)
if err := ran(name, r); err != nil {
if err := ran("type-check", r); err != nil {
return nil, err
}
res.Ran = append(res.Ran, name)
res.Ran = append(res.Ran, "type-check")
}
if hasScript && len(failed) == 0 {
start := m.now()
env := []string{"PATH=" + binDir(root) + ":" + pathOrDefault(), "npm_config_cache=" + filepath.Join(m.Toolchain.Paths.Cache, "npm")}
r := m.runIn(ctx, root, false, dir, env, "npm", scriptArgs(script)...)
r := m.runIn(ctx, dir, env, "npm", scriptArgs(script)...)
run = m.now().Sub(start)
if err := ran(script, r); err != nil {
return nil, err
@@ -227,18 +220,6 @@ func (m *Module) typeCheck(dir string, sources []string) []string {
"--target", "ES2022", "--rootDir", "."}, sources...)
}
// emitBuild is tsc's line that builds a package as its tsconfig.json says (`tsc -p .`, emitting where it says), when the
// package builds (a build or prepare script) and has a tsconfig.json; nil otherwise.
func (m *Module) emitBuild(dir string, scripts map[string]string) []string {
if scripts["build"] == "" && scripts["prepare"] == "" {
return nil
}
if _, err := os.Stat(filepath.Join(dir, "tsconfig.json")); err != nil {
return nil
}
return []string{m.tsc(), "-p", "."}
}
// packageScripts are the scripts of the package.json in dir (none without one).
func packageScripts(dir string) (map[string]string, error) {
raw, err := os.ReadFile(filepath.Join(dir, "package.json"))
@@ -4,7 +4,6 @@ import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
@@ -123,9 +122,8 @@ func TestTheStepsCommandsGetAnEnvironmentOfTheirOwn(t *testing.T) {
t.Errorf("%s was given %s", joined(c), e)
}
}
// 1309: the HOME of a run step is its own, inside its /tmp, never the account's.
if len(c.env) < 3 || !strings.HasPrefix(c.env[0], "PATH=") || !slices.Contains(c.env, "HOME=/tmp/home") || !slices.Contains(c.env, "LC_ALL=C") {
t.Errorf("%s env = %v, want PATH, the step's own HOME and the locale", c.name, c.env)
if len(c.env) < 3 || !strings.HasPrefix(c.env[0], "PATH=") || !strings.HasPrefix(c.env[1], "HOME=") {
t.Errorf("%s env = %v, want PATH, HOME and the locale", c.name, c.env)
}
}
}
@@ -1,52 +0,0 @@
package main
import (
"context"
"slices"
"strings"
"testing"
)
// The package.json of mesh-sdk's TypeScript package (feat/r4e-sdk 9cb4d59), whose tests import its own build output
// (test/retire.test.ts imports ../dist/...): built by `tsc` (build and prepare), which a fetch with --ignore-scripts
// never runs.
const sdkPackageJSON = `{"name":"@novox/mesh-sdk","version":"0.1.15","type":"module","exports":{".":"./dist/index.js"},"files":["dist"],` +
`"scripts":{"build":"tsc","test":"node --test --experimental-strip-types 'test/*.test.ts'","prepare":"tsc"}}`
// Issue 1869: a package that builds (a build or prepare script) is emitted by tsc into the workspace, inside the shut-off
// run step, at both levels, before its tests, which import what it emits; the result names it "build". A package that
// does not build is still type-checked without emitting.
func TestAPackageThatBuildsIsEmittedBeforeItsTests(t *testing.T) {
for _, level := range []string{"unit", "integration"} {
m, f, work := newMachine(t)
pkg := sdkPackageJSON
if level == "integration" {
pkg = strings.Replace(pkg, `"prepare":"tsc"`, `"prepare":"tsc","test:integration":"node --test test/it.mjs"`, 1)
}
workspaceWith(t, work, map[string]string{"package.json": pkg, "tsconfig.json": `{"compilerOptions":{"outDir":"dist"}}`,
"src/index.ts": "export const a = 1;\n", "test/retire.test.ts": "import '../dist/index.js';\n"})
out, err := m.test(context.Background(), testArgs(level))
if err != nil {
t.Fatal(err)
}
r := out.(*TestResult)
if !slices.Contains(r.Ran, "build") || slices.Contains(r.Ran, "type-check") {
t.Errorf("%s: ran %v, want the build and no separate type-check", level, r.Ran)
}
first := joined(f.calls[0])
if strings.Contains(first, "--noEmit") || !strings.HasSuffix(first, "typescript/bin/tsc -p .") {
t.Errorf("%s: the build ran %s, want tsc -p . emitting", level, first)
}
if len(f.shut) == 0 || !strings.Contains(f.shut[0].command, "tsc -p .") {
t.Errorf("%s: the build did not run shut off", level)
}
}
m, f, work := newMachine(t)
workspaceWith(t, work, map[string]string{"package.json": `{"scripts":{"test":"node --test"}}`, "tsconfig.json": `{}`, "index.ts": "export const a = 1;\n"})
if _, err := m.test(context.Background(), testArgs("unit")); err != nil {
t.Fatal(err)
}
if first := joined(f.calls[0]); !strings.Contains(first, "--noEmit") {
t.Errorf("a package that does not build ran %s, want the type-check", first)
}
}
@@ -59,18 +59,6 @@ type FetchResult struct {
Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"`
Trees map[string]buildstep.ModuleTree `json:"trees,omitempty"`
// Group is the group's package the fetch packed and served to the TypeScript members (the call's `package`):
// "<name>@<version> <integrity>".
Group string `json:"group,omitempty"`
}
// narrowedMark is the file beside a workspace's tree saying the fetch narrowed it to a module's named paths.
const narrowedMark = "narrowed"
// narrowed says whether the fetch narrowed the workspace whose tree this is.
func narrowed(tree string) bool {
_, err := os.Stat(filepath.Join(filepath.Dir(tree), narrowedMark))
return err == nil
}
// Refusal is a module a fetch did not install the packages of.
@@ -91,12 +79,7 @@ func (m *Module) fetchTool() StepTool {
"repository": map[string]any{"type": "string", "description": "<owner>/<repository> on the forge"},
"commit": map[string]any{"type": "string", "description": "the full commit, 40 hex digits"},
"ref": map[string]any{"type": "string", "description": "refs/pull/<n>/head when only a pull request's head carries the commit"},
"dir": buildstep.FetchRequestProperties()["dir"],
"paths": buildstep.FetchRequestProperties()["paths"],
"package": map[string]any{"type": "object", "description": "optional: the group's package member (ADR 0500 §5), packed here, publishing nothing, and served to each TypeScript member whose lock names its version, in place of the registry's",
"properties": map[string]any{"repository": map[string]any{"type": "string"}, "commit": map[string]any{"type": "string"}, "ref": map[string]any{"type": "string"}},
"required": []string{"repository", "commit"}},
"base": map[string]any{"type": "string", "description": "optional trunk commit, 40 hex digits: the head is merged onto it as the forge's merge would, and the answer carries the merge commit, the files in conflict (an answer, the workspace is then left at the trunk commit), the changed paths and the tree of each module directory"},
"base": map[string]any{"type": "string", "description": "optional trunk commit, 40 hex digits: the head is merged onto it as the forge's merge would, and the answer carries the merge commit, the files in conflict (an answer, the workspace is then left at the trunk commit), the changed paths and the tree of each module directory"},
},
"required": []string{"repository", "commit"},
},
@@ -106,16 +89,15 @@ func (m *Module) fetchTool() StepTool {
}
func (m *Module) fetch(ctx context.Context, args map[string]any) (any, error) {
// What the builder and the checker ask, in mesh-sdk's one form (ADR 0499 §4): with the module's directory and its
// named paths, the workspace is narrowed to them (buildstep.Narrow).
req, err := buildstep.ParseFetchRequest(args)
if err != nil {
return nil, err
repository, _ := args["repository"].(string)
commit, _ := args["commit"].(string)
ref, _ := args["ref"].(string)
base, _ := args["base"].(string)
if base != "" && !commitOK.MatchString(base) {
return nil, fmt.Errorf("%q is not a trunk commit (40 hex digits)", base)
}
repository, commit, ref, base := req.Repository, req.Commit, req.Ref, req.Base
group, err := groupPackageArg(args["package"])
if err != nil {
return nil, err
if !repositoryOK.MatchString(repository) || strings.Contains(repository, "..") {
return nil, fmt.Errorf("%q is not a repository, <owner>/<name>", repository)
}
m.CleanWorkspaces(m.now())
id := buildstep.NewID()
@@ -141,49 +123,16 @@ func (m *Module) fetch(ctx context.Context, args map[string]any) (any, error) {
return res, nil // a conflict: the tree is at the trunk, nothing of the head is installed
}
}
if len(req.Paths) > 0 {
if err := buildstep.Narrow(tree, req); err != nil {
return nil, err
}
// Said beside the tree, so the listing of the tree's modules leaves out those narrowed away.
if err := os.WriteFile(filepath.Join(root, narrowedMark), nil, 0o600); err != nil {
return nil, err
}
}
// The group's package, packed here and served to its members in place of the registry's version (ADR 0500 §5).
var served *packed
if group != nil {
p, err := m.packGroup(ctx, root, gitRun, group)
if err != nil {
return nil, err
}
served, res.Group = &p, p.Name+"@"+p.Version+" "+p.Integrity
}
dirs, packages, err := m.typeScriptModules(ctx, tree)
dirs, err := m.typeScriptModules(ctx, tree)
if err != nil {
return nil, err
}
for _, d := range dirs {
res.Modules = append(res.Modules, d)
dir := filepath.Join(tree, filepath.FromSlash(d))
if packages[d] {
// A package is built from its own lock's dependencies, its devDependencies included (ADR 0498 §5).
if err := m.installDev(ctx, root, dir, d); err != nil {
return nil, err
}
res.Installed = append(res.Installed, d)
continue
}
did, refused, err := m.installOwn(ctx, root, dir, d)
did, refused, err := m.installOwn(ctx, root, filepath.Join(tree, filepath.FromSlash(d)), d)
if err != nil {
return nil, err
}
if refused == "" {
// The SDK the module's own lock names, from the group or the registry (the lead's decision L10).
if err := m.placeSDK(ctx, dir, d, served); err != nil {
return nil, err
}
}
if refused != "" {
res.Refused = append(res.Refused, Refusal{Module: d, Reason: refused})
}
@@ -196,22 +145,19 @@ func (m *Module) fetch(ctx context.Context, args map[string]any) (any, error) {
// typeScriptModules are the directories of the tree whose module.json declares a typescript artifact, relative to
// it, in order.
func (m *Module) typeScriptModules(ctx context.Context, tree string) (dirs []string, packages map[string]bool, err error) {
func (m *Module) typeScriptModules(ctx context.Context, tree string) ([]string, error) {
r := m.Run(ctx, m.baseEnv(), "git", "-C", tree, "ls-files", "--", "module.json", "*/module.json")
if r.Status != 0 || r.Err != "" {
return nil, nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
return nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
}
packages = map[string]bool{}
var dirs []string
for _, l := range strings.Split(r.Stdout, "\n") {
if l = strings.TrimSpace(l); l == "" {
continue
}
raw, err := os.ReadFile(filepath.Join(tree, filepath.FromSlash(l)))
if os.IsNotExist(err) && narrowed(tree) {
continue // outside a narrowed workspace's paths (buildstep.Narrow): no module of this build
}
if err != nil {
return nil, nil, err
return nil, err
}
var mj struct {
Build struct {
@@ -223,15 +169,13 @@ func (m *Module) typeScriptModules(ctx context.Context, tree string) (dirs []str
}
for _, a := range mj.Build.Artifacts {
if a.Language == "typescript" {
d := filepath.ToSlash(filepath.Dir(l))
dirs = append(dirs, d)
packages[d] = a.Kind == "package"
dirs = append(dirs, filepath.ToSlash(filepath.Dir(l)))
break
}
}
}
sort.Strings(dirs)
return dirs, packages, nil
return dirs, nil
}
// ownDependencies are the package.json's dependencies beyond the SDK and beyond what is build-time: the types and
@@ -285,6 +229,13 @@ func (m *Module) installOwn(ctx context.Context, root, dir, rel string) (install
if err != nil {
return false, "", err
}
if m.Toolchain == nil || m.Toolchain.Registry == nil {
return false, "", fmt.Errorf("the module has no package registry to install %s from", rel)
}
reg, err := m.Toolchain.Registry()
if err != nil {
return false, "", err
}
// A scratch copy, so the module's package.json and lock are never rewritten (dependencies.go installSteps).
scratch := filepath.Join(root, "npm-"+strings.NewReplacer("/", "-", ".", "root").Replace(rel))
if err := os.MkdirAll(scratch, 0o700); err != nil {
@@ -304,11 +255,10 @@ func (m *Module) installOwn(ctx context.Context, root, dir, rel string) (install
if err := os.WriteFile(filepath.Join(scratch, "package-lock.json"), lock, 0o600); err != nil {
return false, "", err
}
// The registry is given through a config file of its own, never on the command line, outside the workspace, and
// removed after (npmrcIn).
npmrc, err := m.npmrcIn(root)
if err != nil {
return false, "", fmt.Errorf("installing the packages of %s: %w", rel, err)
// The registry is given through a config file of its own, never on the command line, and removed after.
npmrc := filepath.Join(scratch, ".npmrc-install")
if err := os.WriteFile(npmrc, []byte(reg.Npmrc()), 0o600); err != nil {
return false, "", err
}
env := append(m.baseEnv(), "npm_config_cache="+filepath.Join(m.Toolchain.Paths.Cache, "npm"))
r := m.Run(ctx, env, "npm", "ci", "--omit=dev", "--omit=peer", "--ignore-scripts", "--no-bin-links",
@@ -27,9 +27,7 @@ type call struct {
type fakeMachine struct {
mu sync.Mutex
calls []call
// shut are the commands that ran in bubblewrap, read back (isolate_test.go).
shut []shutOff
now time.Time
now time.Time
// cost is how long a program takes, by the base name of its path; answer is what it prints and its status;
// effect runs the program's effects on the disk.
cost map[string]time.Duration
@@ -76,10 +74,7 @@ func newMachine(t *testing.T) (*Module, *fakeMachine, string) {
Toolchain: &Toolchain{Paths: Paths{Home: home, Cache: cache}, Registry: func() (Registry, error) {
return Registry{URL: "https://git.novox.be/api/packages/novox/npm/"}, nil
}},
Work: work, Clock: f.clock, Ctx: context.Background()}
// A run step's commands run in bubblewrap (1309); the fake reads each back and runs its command.
m.Run = throughBubblewrap(t, f.run, &f.shut)
withStore(t, m) // a build pushes its bundle to a store of the test's own (ADR 0498)
Work: work, Run: f.run, Clock: f.clock, Ctx: context.Background()}
// The toolchain directory as npm ci leaves it, with no bin links.
nm := filepath.Join(m.toolchainDir(), "node_modules")
for _, p := range []string{"typescript/bin/tsc", "esbuild/bin/esbuild", "@esbuild/linux-x64/bin/esbuild",
@@ -1,110 +0,0 @@
package main
// The run steps shut off (novox/hq ADR 0448; issue 1309, lead decision L2 of R4e): the type-check, the package's test
// script, tsc and esbuild run the change's code, so each runs through mesh-sdk's buildstep.Isolated, in bubblewrap: no
// network, none of the account (its home, its ssh key, its registry secret, its bus login, the runtime directory), the
// workspace read-write, the toolchain read-only, the shared npm cache read-only beneath a throwaway writable layer, or
// read-write for a trunk build alone (publish). The fetch step (git, npm ci of the module's own packages) holds the key
// and the network and runs none of the change's code, so it runs as before.
import (
"context"
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// BubblewrapLeast is the oldest bubblewrap a run step is shut off with: --overlay-src and --tmp-overlay, which keep the
// shared cache unchanged by a candidate, came in 0.10.
const BubblewrapLeast = "0.10"
// npmCache is the shared npm cache a package's test script reads.
func (m *Module) npmCache() string { return filepath.Join(m.Toolchain.Paths.Cache, "npm") }
// isolation is how a run step in the workspace root is shut off.
func (m *Module) isolation(root string, write bool) buildstep.Isolation {
return buildstep.Isolation{
Workspace: root,
ReadOnly: []string{m.toolchainDir()},
Caches: []string{m.npmCache()},
WriteCaches: write,
Env: []string{"PATH=" + pathOrDefault(), "LC_ALL=C"},
}
}
// stepRunner is the runner of a run step in the workspace root: buildstep.Isolated over m.Run, every command shut off;
// write is a trunk build's alone.
func (m *Module) stepRunner(root string, write bool) (forgekey.Runner, error) {
// bubblewrap binds only a directory that is there.
if err := os.MkdirAll(m.npmCache(), 0o700); err != nil {
return nil, fmt.Errorf("the shared cache %s: %w", m.npmCache(), err)
}
return buildstep.Isolated(m.Run, m.isolation(root, write)), nil
}
// ProbeBubblewrap reads the installed bubblewrap's version (`bwrap --version`: "bubblewrap 0.13.0").
func ProbeBubblewrap(ctx context.Context) (string, error) {
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
r := forgekey.ExecRunnerClean(0)(ctx, []string{"PATH=" + defaultPath}, buildstep.Bubblewrap, "--version")
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("`%s --version` failed: %s", buildstep.Bubblewrap, firstLine(r.Err+"\n"+r.Stderr))
}
return parseBubblewrapVersion(r.Stdout)
}
var bubblewrapVersion = regexp.MustCompile(`^bubblewrap (\d+)\.(\d+)(\.\d+)?\s*$`)
func parseBubblewrapVersion(out string) (string, error) {
f := bubblewrapVersion.FindStringSubmatch(strings.TrimSpace(out))
if f == nil {
return "", fmt.Errorf("`%s --version` answered %q", buildstep.Bubblewrap, strings.TrimSpace(out))
}
return f[1] + "." + f[2] + f[3], nil
}
// bubblewrapProblem says why the installed bubblewrap cannot shut a run step off, or "".
func bubblewrapProblem(version string, err error) string {
if err != nil {
return "bubblewrap cannot be run, and every build and test step runs in it (ADR 0448): " + err.Error()
}
if !atLeast(version, BubblewrapLeast) {
return fmt.Sprintf("bubblewrap %s installed, and a build or test step needs %s or later (ADR 0448)", version, BubblewrapLeast)
}
return ""
}
// atLeast says whether the dotted version v is least or later.
func atLeast(v, least string) bool {
a, b := strings.Split(v, "."), strings.Split(least, ".")
for i := range b {
var x int
if i < len(a) {
x, _ = strconv.Atoi(a[i])
}
y, _ := strconv.Atoi(b[i])
if x != y {
return x > y
}
}
return true
}
// checkBubblewrap refuses a step when bubblewrap cannot shut it off; nil Bubblewrap probes nothing.
func (m *Module) checkBubblewrap(ctx context.Context) error {
if m.Bubblewrap == nil {
return nil
}
v, err := m.Bubblewrap(ctx)
if why := bubblewrapProblem(v, err); why != "" {
return &buildstep.Refusal{Text: why, Want: BubblewrapLeast + " or later", Have: v}
}
return nil
}
@@ -1,178 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// shutOff is one command a step ran in bubblewrap: what it bound read-write, read-only and beneath a throwaway layer,
// its bubblewrap flags, and the command.
type shutOff struct {
flags []string
binds, roBinds, overlays []string
env []string
command string
}
// bwrapArity is how many words each bubblewrap option the SDK writes takes.
var bwrapArity = map[string]int{"--unshare-all": 0, "--die-with-parent": 0, "--new-session": 0, "--clearenv": 0,
"--proc": 1, "--dev": 1, "--tmpfs": 1, "--dir": 1, "--chdir": 1, "--overlay-src": 1, "--tmp-overlay": 1,
"--ro-bind": 2, "--ro-bind-try": 2, "--bind": 2, "--symlink": 2, "--setenv": 2}
// throughBubblewrap reads a bubblewrap command back (recorded in seen) and hands its command to run with the environment
// bubblewrap sets (a name set twice keeps its last value, as bubblewrap's --setenv does); any other command goes to run
// as it is.
func throughBubblewrap(t *testing.T, run forgekey.Runner, seen *[]shutOff) forgekey.Runner {
t.Helper()
var mu sync.Mutex
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if name != buildstep.Bubblewrap {
return run(ctx, env, name, args...)
}
var s shutOff
var names []string
values := map[string]string{}
i := 0
for ; i < len(args) && args[i] != "--"; i++ {
n, ok := bwrapArity[args[i]]
if !ok || i+n >= len(args) {
t.Errorf("bubblewrap was given %q, which the test does not know: %v", args[i], args)
return forgekey.Ran{Status: 1, Err: "unknown bubblewrap option"}
}
s.flags = append(s.flags, args[i])
switch args[i] {
case "--bind":
s.binds = append(s.binds, args[i+1])
case "--ro-bind", "--ro-bind-try":
s.roBinds = append(s.roBinds, args[i+1])
case "--overlay-src":
s.overlays = append(s.overlays, args[i+1])
case "--setenv":
if _, set := values[args[i+1]]; !set {
names = append(names, args[i+1])
}
values[args[i+1]] = args[i+2]
}
i += n
}
if i+1 >= len(args) {
t.Errorf("bubblewrap was given no command: %v", args)
return forgekey.Ran{Status: 1, Err: "no command"}
}
for _, n := range names {
s.env = append(s.env, n+"="+values[n])
}
s.command = args[i+1] + " " + strings.Join(args[i+2:], " ")
mu.Lock()
*seen = append(*seen, s)
mu.Unlock()
return run(ctx, s.env, args[i+1], args[i+2:]...)
}
}
// checkShutOff fails the test unless s ran with no network and none of the account, the toolchain read-only, and the
// shared cache beneath a throwaway layer unless write.
func checkShutOff(t *testing.T, m *Module, root string, s shutOff, write bool) {
t.Helper()
for _, f := range []string{"--unshare-all", "--clearenv", "--die-with-parent"} {
if !slices.Contains(s.flags, f) {
t.Errorf("%s ran without %s", s.command, f)
}
}
home := m.Toolchain.Paths.Home
for _, b := range append(append([]string{}, s.binds...), s.roBinds...) {
if b == home || strings.HasPrefix(b, home+"/") || strings.HasPrefix(b, "/run") {
t.Errorf("%s was given %s: the account's home or the runtime directory", s.command, b)
}
}
for _, e := range s.env {
if e == "HOME="+home {
t.Errorf("%s was told the account's home", s.command)
}
}
if !slices.Contains(s.roBinds, m.toolchainDir()) || slices.Contains(s.binds, m.toolchainDir()) {
t.Errorf("%s was not given the toolchain read-only: %v %v", s.command, s.roBinds, s.binds)
}
c := m.npmCache()
bound, layered := slices.Contains(s.binds, c), slices.Contains(s.overlays, c)
if write != bound || write == layered {
t.Errorf("%s: the shared cache bound %v, layered %v; a trunk build (%v) alone writes it", s.command, bound, layered, write)
}
if !slices.Contains(s.binds, root) {
t.Errorf("%s was not given its workspace %s read-write: %v", s.command, root, s.binds)
}
}
// 1309-S7: the type-check and the test script run shut off, the cache beneath a throwaway layer.
func TestTheTestStepRunsItsCommandsShutOff(t *testing.T) {
m, f, work := newMachine(t)
workspaceWith(t, work, map[string]string{"index.ts": `export const a = 1;`, "package.json": `{"scripts":{"test":"x"}}`})
if _, err := m.test(context.Background(), testArgs("unit")); err != nil {
t.Fatal(err)
}
if len(f.shut) != 2 || len(f.calls) != 2 {
t.Fatalf("shut off %d of %d commands, want the type-check and the script: %v", len(f.shut), len(f.calls), f.calls)
}
for _, s := range f.shut {
checkShutOff(t, m, work+"/w1", s, false)
}
}
// 1309-S8: tsc and esbuild of a candidate's build run shut off with the cache beneath a throwaway layer; a trunk build
// (publish) writes it.
func TestTheBuildStepRunsShutOffAndOnlyATrunkBuildWritesTheCache(t *testing.T) {
for _, publish := range []bool{false, true} {
m, f, work := builtMachine(t, "")
ws := "w1"
args := buildArgs()
if publish {
args["publish"] = true
}
if _, err := m.build(context.Background(), args); err != nil {
t.Fatalf("publish %v: %v", publish, err)
}
if len(f.shut) == 0 || len(f.shut) != len(f.calls) {
t.Fatalf("publish %v: shut off %d of %d commands", publish, len(f.shut), len(f.calls))
}
for _, s := range f.shut {
checkShutOff(t, m, work+"/"+ws, s, publish)
}
}
}
// 1309-S1: a step is refused, and the module is unhealthy, while bubblewrap is missing or older than 0.10.
func TestAnOldOrMissingBubblewrapRefusesTheStepsAndIsUnhealthy(t *testing.T) {
for _, c := range []struct {
version string
err error
ok bool
}{{"0.13.0", nil, true}, {"0.10", nil, true}, {"0.9.0", nil, false}, {"", errors.New("bwrap: not found"), false}} {
m, _, _ := newMachine(t)
m.Probe = func(context.Context) (string, error) { return "v22.20.0", nil }
m.Toolchains = func(string, any) (uint64, error) { return 1, nil }
m.Bubblewrap = func(context.Context) (string, error) { return c.version, c.err }
err := m.Check(context.Background())
raw, _ := json.Marshal(m.Health(context.Background()))
if c.ok {
if err != nil || strings.Contains(string(raw), "unhealthy") {
t.Errorf("bubblewrap %q: refused %v, health %s", c.version, err, raw)
}
continue
}
var ref *buildstep.Refusal
if !errors.As(err, &ref) || !strings.Contains(err.Error(), "bubblewrap") {
t.Errorf("bubblewrap %q %v: Check answered %v", c.version, c.err, err)
}
if !strings.Contains(string(raw), "unhealthy") || !strings.Contains(string(raw), "bubblewrap") {
t.Errorf("bubblewrap %q %v: health %s", c.version, c.err, raw)
}
}
}
@@ -1,91 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"testing"
)
// probeScript is a package's test script that tries what a change's code must not reach and writes what it saw.
const probeScript = `import fs from "node:fs";
import dns from "node:dns/promises";
import net from "node:net";
const [key, login, port, sock, cache] = process.argv.slice(2);
const seen = {};
const read = (p) => { try { fs.readFileSync(p); return true; } catch { return false; } };
seen.key = read(key);
seen.login = read(login);
try { await dns.lookup("git.novox.be"); seen.resolved = true; } catch { seen.resolved = false; }
const reach = (opts) => new Promise((ok) => { const c = net.connect(opts); c.on("connect", () => { c.destroy(); ok(true); }); c.on("error", () => ok(false)); });
seen.tcp = await reach({ host: "127.0.0.1", port: Number(port) });
seen.socket = await reach({ path: sock });
try { fs.writeFileSync(cache + "/written-by-a-candidate", "x"); seen.wroteCache = true; } catch { seen.wroteCache = false; }
fs.writeFileSync("seen.json", JSON.stringify(seen));
`
// 1309-S9 (ADR 0448's check): a package's test script, run by the test step, reads no key and no login of the account,
// resolves no name, reaches no socket of the machine (a TCP port on its loopback, a unix socket), and leaves the
// shared cache unchanged.
func TestARunStepReachesNoKeyLoginNameOrSocketAndLeavesTheCacheUnchanged(t *testing.T) {
if _, err := exec.LookPath("bwrap"); err != nil {
t.Skip("NOT RUN: no bubblewrap installed where this check runs; every build node has it (the module declares it)")
}
m, work := realMachine(t)
home := m.Toolchain.Paths.Home
key, login := filepath.Join(home, ".ssh", "mesh_ed25519"), filepath.Join(home, "broker")
write(t, key, "a key", 0o600)
write(t, login, "a login", 0o600)
tcp, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer tcp.Close()
sockDir, err := os.MkdirTemp("", "sock")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(sockDir)
sock := filepath.Join(sockDir, "s")
unix, err := net.Listen("unix", sock)
if err != nil {
t.Fatal(err)
}
defer unix.Close()
if err := os.MkdirAll(m.npmCache(), 0o700); err != nil {
t.Fatal(err)
}
port := strconv.Itoa(tcp.Addr().(*net.TCPAddr).Port)
dir := filepath.Join(work, "w1", "tree", "modules", "m")
write(t, filepath.Join(dir, "probe.mjs"), probeScript, 0o644)
write(t, filepath.Join(dir, "package.json"), `{"name":"m","type":"module","scripts":{"test":"node probe.mjs `+key+` `+login+` `+port+` `+sock+` `+m.npmCache()+`"}}`, 0o644)
out, err := m.test(context.Background(), map[string]any{"workspace": "w1", "module": "modules/m", "level": "unit"})
if err != nil {
t.Fatal(err)
}
if r := out.(*TestResult); !r.Passed {
t.Fatalf("the probe did not run: %+v", r)
}
raw, err := os.ReadFile(filepath.Join(dir, "seen.json"))
if err != nil {
t.Fatalf("the probe wrote nothing: %v", err)
}
var seen map[string]bool
if err := json.Unmarshal(raw, &seen); err != nil {
t.Fatal(err)
}
for what, reached := range seen {
if reached && what != "wroteCache" {
t.Errorf("the run step reached %s: %s", what, raw)
}
}
if _, err := os.Stat(filepath.Join(m.npmCache(), "written-by-a-candidate")); err == nil {
t.Errorf("a candidate's step changed the shared cache")
}
}
@@ -51,19 +51,12 @@ func main() {
Run: forgekey.ExecRunnerClean(0),
Ctx: ctx,
Say: logf,
Bubblewrap: ProbeBubblewrap,
RegistryFile: filepath.Join(home, "registry"),
}
// Outside a launch by the runtime — a build, a check — there is no node and no state to write.
if m.Node != "" {
go m.Start(ctx)
}
go startKeeper(ctx)
// Every state it keeps, said on the bus before it serves (ADR 0495 §2; issue 1790).
if err := keepStates(); err != nil {
logf("announcing the states it keeps: %v", err)
os.Exit(1)
}
if err := stdio.Serve("", m.Tools(m.StepTools())); err != nil {
logf("%v", err)
os.Exit(1)
@@ -66,7 +66,7 @@ func TestManifestIsValidAndDeclaresTheAccountTheCapabilityAndThePackages(t *test
}
}
slices.Sort(packages)
if want := []string{"bubblewrap", "git", "openssh"}; !slices.Equal(packages, want) {
if want := []string{"git", "openssh"}; !slices.Equal(packages, want) {
t.Errorf("packages = %v, want %v", packages, want)
}
}
@@ -194,47 +194,3 @@ func TestTheToolchainFilesAreAnArchiveUnpackedIntoTheHome(t *testing.T) {
t.Errorf("the module carries no toolchain/package-lock.json: %v", err)
}
}
// 1309-S1: every run step runs in bubblewrap, so the module declares its package (the lead's decision: each build module
// that runs a step through it declares it), beside the user-namespaces capability bubblewrap needs.
func TestTheModuleDeclaresBubblewrapAndTheCapabilityItNeeds(t *testing.T) {
m := readManifest(t)
if r := resource(t, m, "bubblewrap"); r["type"] != "package" || r["package"] != "bubblewrap" {
t.Errorf("the bubblewrap resource is %v, want the package bubblewrap", r)
}
if !slices.Contains(m.Capabilities, "user-namespaces") {
t.Errorf("capabilities %v lack user-namespaces", m.Capabilities)
}
}
// 1587-S5 (ADR 0498 §2): the module requires the artifact store and is told its address in a file of its home, which no
// run step is given (a run step has no network either, ADR 0448).
func TestTheModuleRequiresTheArtifactStoreAndIsToldItsAddress(t *testing.T) {
m := readManifest(t)
if !slices.Contains(m.Requires, "artifact-store") {
t.Errorf("requires %v lack artifact-store", m.Requires)
}
// The validator reads the template (files/registry) into the resource's content.
r := resource(t, m, "registry")
if content, _ := r["content"].(string); r["type"] != "file" || r["path"] != "${dir:state}/registry" || !strings.Contains(content, "${seat:mesh-artifact-store:reach}") {
t.Errorf("the registry resource is %v", r)
}
}
// 1587-S17 (ADR 0498 §5): the module publishes a trunk package itself, so its contribution to the package registry asks
// for write access; the credential stays with the module's own process (a run step has neither the home nor a network).
func TestTheModuleContributesWriteAccessToThePackageRegistry(t *testing.T) {
raw, err := os.ReadFile(filepath.Join(moduleDir, "module.json"))
if err != nil {
t.Fatal(err)
}
var mj struct {
Contributes map[string]map[string]any `json:"contributes"`
}
if err := json.Unmarshal(raw, &mj); err != nil {
t.Fatal(err)
}
if got := mj.Contributes["npm-package-registry"]["access"]; got != "write" {
t.Errorf("npm-package-registry access = %v, want write", got)
}
}
@@ -3,7 +3,6 @@ package main
import (
"context"
"fmt"
"net/http"
"os"
"sync"
"time"
@@ -49,15 +48,6 @@ type Module struct {
// Ctx is the process's context, which a step runs under (the tool call's is cut at 30 s, the step's is not).
Ctx context.Context
Say func(format string, a ...any)
// HTTP reads and publishes to the package registry; a client of its own when nil. A test serves its own.
HTTP *http.Client
// RegistryFile holds the artifact store's address (host:port), placed by the manifest; Push puts a blob there
// (buildstep.PushBlob when nil; a test's fake store).
RegistryFile string
Push func(ctx context.Context, registry, repository, digest string, body []byte) error
// Bubblewrap reads the installed bubblewrap's version (ProbeBubblewrap): a step is refused, and the module is
// unhealthy, while it is missing or older than BubblewrapLeast. Nil probes nothing.
Bubblewrap func(ctx context.Context) (string, error)
mu sync.Mutex // guards Recipe
install sync.Mutex // one install at a time
@@ -97,10 +87,7 @@ func (m *Module) check(ctx context.Context) (string, error) {
m.mu.Unlock()
}
installed, err := m.Probe(ctx)
if err := m.guard(m.recipe(), installed, err).Check(ctx); err != nil {
return installed, err
}
return installed, m.checkBubblewrap(ctx)
return installed, m.guard(m.recipe(), installed, err).Check(ctx)
}
// Prepare is Check and then the toolchain directory of the recipe in force: what every step runs behind.
@@ -1,527 +0,0 @@
package main
// A TypeScript package (novox/hq ADR 0498 §5, ADR 0500 §4-§6; issue 1587, the lead's decision L10): built from its own
// lock's devDependencies (the toolchain's SDK is never used to build the SDK itself), packed with npm pack, and, for a
// group, served to its TypeScript members' fetch in place of the registry's version. A member is built against the SDK
// version its own lock names: the fetch places that version in the member's node_modules, from the group's packed
// package when the group carries that version, else from the registry; a version neither holds waits for its
// publication (`waits for:`). The install of a package's devDependencies is the fetch's (it holds the network and runs
// no script); its build and pack run shut off like every run step.
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"crypto/sha512"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path"
"path/filepath"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// maxPackage is the largest packed package read: far above the SDK's.
const maxPackage = 64 << 20
// packed is a package built and packed: its name and version from its package.json, the packed file's bytes, its npm
// integrity (sha512, as a lock names it) and its digest (sha256, as a done key names it).
type packed struct {
Name, Version, Integrity, Digest string
tgz []byte
}
// integrityOf is a packed file's integrity as npm writes it in a lock: sha512- and the base64 of its sha512.
func integrityOf(b []byte) string {
sum := sha512.Sum512(b)
return "sha512-" + base64.StdEncoding.EncodeToString(sum[:])
}
func digestOf(b []byte) string {
sum := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(sum[:])
}
// packageJSON is the part of a package.json the package steps read.
type packageJSON struct {
Name string `json:"name"`
Version string `json:"version"`
Scripts map[string]string `json:"scripts"`
}
func readPackageJSON(dir string) (packageJSON, error) {
var p packageJSON
raw, err := os.ReadFile(filepath.Join(dir, "package.json"))
if err != nil {
return p, fmt.Errorf("the package has no package.json: %w", err)
}
if err := json.Unmarshal(raw, &p); err != nil {
return p, fmt.Errorf("the package.json is not JSON: %w", err)
}
if p.Name == "" || p.Version == "" {
return p, fmt.Errorf("the package.json names no name or no version")
}
return p, nil
}
// installDev installs a package's own dependencies, its devDependencies included, from its lock (the fetch: network,
// no script, no bin link), and writes a shim for each program they carry into <root>/bin-<name>, files and not links,
// so its build script finds them by name.
func (m *Module) installDev(ctx context.Context, root, dir, rel string) error {
if _, err := os.Stat(filepath.Join(dir, "package-lock.json")); err != nil {
return fmt.Errorf("the package in %s has no package-lock.json: commit one", rel)
}
npmrc, err := m.npmrcIn(root)
if err != nil {
return err
}
defer os.Remove(npmrc)
env := append(m.baseEnv(), "npm_config_cache="+m.npmCache())
r := m.Run(ctx, env, "npm", "ci", "--ignore-scripts", "--no-bin-links", "--no-audit", "--no-fund", "--prefix", dir, "--userconfig", npmrc)
if why := ranFailed(r); why != "" {
return fmt.Errorf("installing the dependencies of the package in %s: %s", rel, why)
}
return writeShims(filepath.Join(dir, "node_modules"), shimDir(root, rel))
}
// npmrcIn writes the npm configuration for the mesh's registry, with its credential, for one command of this process;
// the caller removes it. It lies in this process's temporary directory, never in a workspace: a run step is given the
// workspace and a /tmp of its own, so no step's mounts hold the credential (ADR 0498 §5, ADR 0500 §4).
func (m *Module) npmrcIn(root string) (string, error) {
if m.Toolchain == nil || m.Toolchain.Registry == nil {
return "", fmt.Errorf("the module has no package registry")
}
reg, err := m.Toolchain.Registry()
if err != nil {
return "", err
}
f, err := os.CreateTemp("", ".npmrc-")
if err != nil {
return "", err
}
defer f.Close()
if _, err := f.WriteString(reg.Npmrc()); err != nil {
os.Remove(f.Name())
return "", err
}
return f.Name(), nil
}
// shimDir is where the shims of the package in rel are written.
func shimDir(root, rel string) string {
return filepath.Join(root, "bin-"+strings.NewReplacer("/", "-", ".", "root").Replace(rel))
}
// writeShims writes, for every program a top-level package of nm names in its package.json `bin`, a shell script that
// runs it by its real path.
func writeShims(nm, into string) error {
if err := os.MkdirAll(into, 0o755); err != nil {
return err
}
var dirs []string
entries, _ := os.ReadDir(nm)
for _, e := range entries {
if !e.IsDir() || strings.HasPrefix(e.Name(), ".") {
continue
}
if strings.HasPrefix(e.Name(), "@") {
scoped, _ := os.ReadDir(filepath.Join(nm, e.Name()))
for _, s := range scoped {
if s.IsDir() {
dirs = append(dirs, filepath.Join(nm, e.Name(), s.Name()))
}
}
continue
}
dirs = append(dirs, filepath.Join(nm, e.Name()))
}
sort.Strings(dirs)
for _, d := range dirs {
raw, err := os.ReadFile(filepath.Join(d, "package.json"))
if err != nil {
continue
}
var p struct {
Name string `json:"name"`
Bin json.RawMessage `json:"bin"`
}
if json.Unmarshal(raw, &p) != nil || len(p.Bin) == 0 {
continue
}
bins := map[string]string{}
var one string
if json.Unmarshal(p.Bin, &one) == nil {
bins[path.Base(p.Name)] = one
} else if json.Unmarshal(p.Bin, &bins) != nil {
continue
}
for name, target := range bins {
if name == "" || strings.ContainsAny(name, "/\\") {
continue
}
body := "#!/bin/sh\nexec node '" + filepath.Join(d, filepath.FromSlash(target)) + "' \"$@\"\n"
p := filepath.Join(into, name)
if err := os.WriteFile(p, []byte(body), 0o755); err != nil {
return err
}
if err := os.Chmod(p, 0o755); err != nil {
return err
}
}
}
return nil
}
// buildAndPack builds the package in dir with its own build script, if it has one, and packs it with npm pack, both
// shut off (write: a trunk build's, which alone writes the shared cache). The packed file is read and removed: it never
// stays on the node.
func (m *Module) buildAndPack(ctx context.Context, root, dir, rel string, write bool) (packed, error) {
p, err := readPackageJSON(dir)
if err != nil {
return packed{}, err
}
env := []string{"PATH=" + shimDir(root, rel) + ":" + pathOrDefault(), "npm_config_cache=" + m.npmCache()}
if p.Scripts["build"] != "" {
if why := ranFailed(m.runIn(ctx, root, write, dir, env, "npm", "run", "build")); why != "" {
return packed{}, fmt.Errorf("the build of the package %s: %s", p.Name, why)
}
}
out, err := os.MkdirTemp(root, "packed-")
if err != nil {
return packed{}, err
}
defer os.RemoveAll(out)
if why := ranFailed(m.runIn(ctx, root, write, dir, env, "npm", "pack", "--ignore-scripts", "--pack-destination", out)); why != "" {
return packed{}, fmt.Errorf("packing the package %s: %s", p.Name, why)
}
files, _ := filepath.Glob(filepath.Join(out, "*.tgz"))
if len(files) != 1 {
return packed{}, fmt.Errorf("npm pack of %s left %d packed files, want one", p.Name, len(files))
}
tgz, err := os.ReadFile(files[0])
if err != nil {
return packed{}, err
}
return packed{Name: p.Name, Version: p.Version, Integrity: integrityOf(tgz), Digest: digestOf(tgz), tgz: tgz}, nil
}
// groupPackage is the fetch's `package` argument: the group's package member (ADR 0500 §5), which the fetch packs and
// serves to its TypeScript members in place of the registry's version.
type groupPackage struct{ Repository, Commit, Ref string }
func groupPackageArg(v any) (*groupPackage, error) {
if v == nil {
return nil, nil
}
o, ok := v.(map[string]any)
if !ok {
return nil, fmt.Errorf("package is {repository, commit, ref}")
}
g := &groupPackage{}
g.Repository, _ = o["repository"].(string)
g.Commit, _ = o["commit"].(string)
g.Ref, _ = o["ref"].(string)
if !repositoryOK.MatchString(g.Repository) || strings.Contains(g.Repository, "..") {
return nil, fmt.Errorf("package: %q is not a repository, <owner>/<name>", g.Repository)
}
if !commitOK.MatchString(g.Commit) {
return nil, fmt.Errorf("package: %q is not a full commit (40 hex digits)", g.Commit)
}
if g.Ref != "" && !strings.HasPrefix(g.Ref, "refs/pull/") {
return nil, fmt.Errorf("package: %q is not a pull request's head", g.Ref)
}
return g, nil
}
// packGroup fetches the group's package member into the workspace, installs its dependencies, builds and packs it,
// publishing nothing (ADR 0500 §4).
func (m *Module) packGroup(ctx context.Context, root string, run forgekey.Runner, g *groupPackage) (packed, error) {
dir := filepath.Join(root, "package", "tree")
if err := os.MkdirAll(dir, 0o700); err != nil {
return packed{}, err
}
if err := fetchCommit(ctx, run, dir, m.remote(g.Repository), g.Commit, g.Ref, m); err != nil {
return packed{}, fmt.Errorf("the group's package %s: %w", g.Repository, err)
}
if err := m.installDev(ctx, root, dir, "package"); err != nil {
return packed{}, err
}
return m.buildAndPack(ctx, root, dir, "package", false)
}
// sdkLocked is what a module's lock names of the SDK: its version, its integrity and where it was resolved from; ok is
// false when the lock names none (the toolchain's SDK is then the module's, as before).
func sdkLocked(dir string) (version, integrity string, ok bool, err error) {
raw, err := os.ReadFile(filepath.Join(dir, "package-lock.json"))
if os.IsNotExist(err) {
return "", "", false, nil
}
if err != nil {
return "", "", false, err
}
var lock struct {
Packages map[string]struct {
Version string `json:"version"`
Integrity string `json:"integrity"`
} `json:"packages"`
}
if err := json.Unmarshal(raw, &lock); err != nil {
return "", "", false, fmt.Errorf("the package-lock.json is not JSON: %w", err)
}
p, ok := lock.Packages["node_modules/"+sdkPackage]
if !ok || p.Version == "" {
return "", "", false, nil
}
return p.Version, p.Integrity, true, nil
}
// placeSDK places in the module's node_modules the SDK version its lock names (the lead's decision L10): the group's
// packed package when it is that version (its integrity must be the lock's), else the registry's; a version neither
// holds is a wait (`waits for:`) when no group package was given, and a failure naming both when one was.
func (m *Module) placeSDK(ctx context.Context, dir, rel string, group *packed) error {
version, integrity, ok, err := sdkLocked(dir)
if err != nil || !ok {
return err
}
want := sdkPackage + "@" + version
var tgz []byte
switch {
case group != nil && group.Name == sdkPackage && group.Version == version:
if integrity != "" && integrity != group.Integrity {
return fmt.Errorf("the lock of %s names %s with the integrity %s, and the group's package packs to %s", rel, want, integrity, group.Integrity)
}
tgz = group.tgz
default:
got, held, err := m.registryPackage(ctx, sdkPackage, version)
if err != nil {
return err
}
if !held {
if group != nil {
return fmt.Errorf("the lock of %s names %s, which neither the registry nor the group (%s@%s) holds", rel, want, group.Name, group.Version)
}
return fmt.Errorf("%s%s", delivery.WaitsFor, want)
}
if integrity != "" && integrityOf(got) != integrity {
return fmt.Errorf("the registry's %s is %s, and the lock of %s names %s", want, integrityOf(got), rel, integrity)
}
tgz = got
}
into := filepath.Join(dir, "node_modules", filepath.FromSlash(sdkPackage))
if err := os.RemoveAll(into); err != nil {
return err
}
return unpackPackage(tgz, into)
}
// httpClient is the client of the registry's reads and publishes: HTTP when a test set it.
func (m *Module) httpClient() *http.Client {
if m.HTTP != nil {
return m.HTTP
}
return &http.Client{Timeout: 5 * time.Minute}
}
// packument is the part of the registry's document of a package the steps read: each version's packed file.
type packument struct {
Versions map[string]struct {
Dist struct {
Tarball string `json:"tarball"`
Integrity string `json:"integrity"`
} `json:"dist"`
} `json:"versions"`
}
// registryVersion reads what the registry holds of name@version: its packed file's address and integrity, and whether
// it holds it at all.
func (m *Module) registryVersion(ctx context.Context, name, version string) (tarball, integrity string, held bool, err error) {
reg, err := m.Toolchain.Registry()
if err != nil {
return "", "", false, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reg.URL+url.PathEscape(name), nil)
if err != nil {
return "", "", false, err
}
reg.authorize(req)
resp, err := m.httpClient().Do(req)
if err != nil {
return "", "", false, fmt.Errorf("asking the registry for %s: %w", name, err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNotFound {
return "", "", false, nil
}
if resp.StatusCode != http.StatusOK {
said, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
return "", "", false, fmt.Errorf("the registry answered %s for %s: %s", resp.Status, name, strings.TrimSpace(string(said)))
}
var p packument
if err := json.NewDecoder(io.LimitReader(resp.Body, 64<<20)).Decode(&p); err != nil {
return "", "", false, fmt.Errorf("the registry's document of %s: %w", name, err)
}
v, ok := p.Versions[version]
if !ok {
return "", "", false, nil
}
return v.Dist.Tarball, v.Dist.Integrity, true, nil
}
// registryPackage downloads name@version's packed file from the registry; held is false when the registry has none.
func (m *Module) registryPackage(ctx context.Context, name, version string) ([]byte, bool, error) {
tarball, _, held, err := m.registryVersion(ctx, name, version)
if err != nil || !held {
return nil, held, err
}
reg, err := m.Toolchain.Registry()
if err != nil {
return nil, false, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, tarball, nil)
if err != nil {
return nil, false, err
}
reg.authorize(req)
resp, err := m.httpClient().Do(req)
if err != nil {
return nil, false, fmt.Errorf("downloading %s@%s: %w", name, version, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, false, fmt.Errorf("the registry answered %s for %s@%s's packed file", resp.Status, name, version)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, maxPackage+1))
if err != nil {
return nil, false, err
}
if len(body) > maxPackage {
return nil, false, fmt.Errorf("%s@%s is larger than %d bytes", name, version, maxPackage)
}
return body, true, nil
}
// unpackPackage unpacks a packed npm package (a gzipped tar whose entries lie under package/) into dir: files and
// directories only, nothing outside.
func unpackPackage(tgz []byte, dir string) error {
zr, err := gzip.NewReader(bytes.NewReader(tgz))
if err != nil {
return fmt.Errorf("the packed package is not gzip: %w", err)
}
tr := tar.NewReader(zr)
for {
h, err := tr.Next()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
name := path.Clean(strings.TrimPrefix(h.Name, "./"))
rel, ok := strings.CutPrefix(name, "package/")
if !ok || rel == "" || path.IsAbs(rel) || rel == ".." || strings.HasPrefix(rel, "../") {
if name == "package" {
continue
}
return fmt.Errorf("the packed package holds %s, outside package/", h.Name)
}
target := filepath.Join(dir, filepath.FromSlash(rel))
switch h.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(target, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
return err
}
body, err := io.ReadAll(io.LimitReader(tr, maxPackage))
if err != nil {
return err
}
if err := os.WriteFile(target, body, os.FileMode(h.Mode).Perm()|0o600); err != nil {
return err
}
default:
return fmt.Errorf("the packed package holds %s, neither a file nor a directory", h.Name)
}
}
}
// fetchCommit fetches one commit into dir as the fetch step fetches the module's own (buildstep.FetchCommit).
func fetchCommit(ctx context.Context, run forgekey.Runner, dir, remote, commit, ref string, m *Module) error {
return buildstep.FetchCommit(ctx, run, dir, remote, commit, ref, m.FetchOptions...)
}
// buildPackage builds and packs a package artifact (ADR 0498 §5): its digest is the packed file's. Only a trunk build
// (publish) publishes it, from this module's own process with the registry's write credential, which no run step is
// given: a version the registry lacks is published; the same packed file already there is done without a publish;
// another packed file under that version fails `version taken:`. The reference is <name>@<version>, answered only when
// the registry holds this build's file; a candidate answers its digest alone.
func (m *Module) buildPackage(ctx context.Context, workspace, module, root, dir string, a artifact, publish bool) (any, error) {
rel := module
if rel == "" {
rel = "."
}
p, err := m.buildAndPack(ctx, root, dir, rel, publish)
if err != nil {
return nil, err
}
res := BuildResult{Workspace: workspace, Module: module, Artifact: a.Name, Digest: p.Digest, Bytes: len(p.tgz)}
if publish {
if res.Reference, err = m.publish(ctx, root, p); err != nil {
return nil, err
}
}
return res, nil
}
// publish puts a packed package into the registry unless it holds that version, and answers <name>@<version>.
func (m *Module) publish(ctx context.Context, root string, p packed) (string, error) {
ref := p.Name + "@" + p.Version
_, integrity, held, err := m.registryVersion(ctx, p.Name, p.Version)
if err != nil {
return "", err
}
if held {
if integrity == p.Integrity {
return ref, nil // this very file, published before: done without a publish
}
return "", fmt.Errorf("%s%s: the registry holds it as %s, and this build packs %s", delivery.VersionTaken, ref, integrity, p.Integrity)
}
file, err := os.CreateTemp(root, "publish-*.tgz")
if err != nil {
return "", err
}
defer os.Remove(file.Name()) // the packed file leaves the node once its publish answered
if _, err := file.Write(p.tgz); err != nil {
file.Close()
return "", err
}
if err := file.Close(); err != nil {
return "", err
}
npmrc, err := m.npmrcIn(root)
if err != nil {
return "", err
}
defer os.Remove(npmrc)
env := append(m.baseEnv(), "npm_config_cache="+m.npmCache())
r := m.Run(ctx, env, "npm", "publish", file.Name(), "--ignore-scripts", "--userconfig", npmrc)
if why := ranFailed(r); why != "" {
return "", fmt.Errorf("publishing %s: %s", ref, why)
}
return ref, nil
}
@@ -1,125 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// sdkFixture is a package named as the SDK at version 9.9.9 with no dependency, whose build script writes dist/: what
// the group's package member is to its TypeScript members.
var sdkFixture = map[string]string{
"module.json": `{"module":"mesh-sdk","version":"1","build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript"}]}}`,
"package.json": `{"name":"@novox/mesh-sdk","version":"9.9.9","type":"module","files":["dist"],"scripts":{"build":"node build.mjs"}}`,
"package-lock.json": `{"name":"@novox/mesh-sdk","version":"9.9.9","lockfileVersion":3,"requires":true,"packages":{"":{"name":"@novox/mesh-sdk","version":"9.9.9"}}}`,
"build.mjs": "import fs from 'node:fs';\nfs.mkdirSync('dist', {recursive: true});\nfs.writeFileSync('dist/index.js', 'export const v = \"9.9.9\";\\n');\n",
}
// forgeRepo makes a bare repository o/<name> under root/forge holding files at one commit, and answers the commit.
func forgeRepo(t *testing.T, root, name string, files map[string]string) string {
t.Helper()
bare, work := filepath.Join(root, "forge", "o", name+".git"), filepath.Join(root, "work-"+name)
os.MkdirAll(bare, 0o755)
os.MkdirAll(work, 0o755)
git(t, bare, "init", "-q", "--bare")
git(t, work, "init", "-q", "-b", "main")
for n, c := range files {
write(t, filepath.Join(work, filepath.FromSlash(n)), c, 0o644)
}
git(t, work, "add", ".")
git(t, work, "commit", "-q", "-m", "c")
git(t, work, "push", "-q", bare, "HEAD:refs/heads/main")
return git(t, work, "rev-parse", "HEAD")
}
// ADR 0500 §5 with the lead's decision L10, on real git, npm and bubblewrap: the fetch packs the group's package member
// (its build script and npm pack shut off, nothing published), and a member whose lock names that version with that
// integrity gets it in its node_modules; a member whose lock names it with another integrity fails naming both.
func TestTheFetchPacksTheGroupsPackageAndServesItToAMember(t *testing.T) {
for _, tool := range []string{"node", "npm", "git", "bwrap"} {
if _, err := exec.LookPath(tool); err != nil {
t.Skipf("NOT RUN: no %s where this check runs; every build node has it", tool)
}
}
root := t.TempDir()
sdkCommit := forgeRepo(t, root, "sdk", sdkFixture)
m, _, _ := newMachine(t)
m.Run = forgekey.ExecRunner
m.RemoteBase = "file://" + filepath.Join(root, "forge") + "/"
m.FetchOptions = []buildstep.Option{buildstep.AllowFileRemote()}
fakeRegistry(t, m, nil) // the registry holds no 9.9.9: only the group can serve it
// What the group's member packs to, as the member's lock would name it: packed once here the way the fetch packs it.
probe, err := m.fetch(context.Background(), map[string]any{"repository": "o/sdk", "commit": sdkCommit,
"package": map[string]any{"repository": "o/sdk", "commit": sdkCommit}})
if err != nil {
t.Fatal(err)
}
group := strings.Fields(probe.(FetchResult).Group)
if len(group) != 2 || group[0] != "@novox/mesh-sdk@9.9.9" || !strings.HasPrefix(group[1], "sha512-") {
t.Fatalf("the fetch answered the group %q", probe.(FetchResult).Group)
}
member := func(integrity string) string {
lock, _ := json.Marshal(map[string]any{"name": "m", "lockfileVersion": 3, "requires": true, "packages": map[string]any{
"": map[string]any{"name": "m", "dependencies": map[string]any{"@novox/mesh-sdk": "9.9.9"}},
"node_modules/@novox/mesh-sdk": map[string]any{"version": "9.9.9", "integrity": integrity}}})
files := typeScriptModule(`{"name":"m","dependencies":{"@novox/mesh-sdk":"9.9.9"}}`)
files["modules/m/package-lock.json"] = string(lock)
return forgeRepo(t, root, "m-"+integrity[len(integrity)-6:len(integrity)-2], files)
}
commit := member(group[1])
name := "m-" + group[1][len(group[1])-6:len(group[1])-2]
out, err := m.fetch(context.Background(), map[string]any{"repository": "o/" + name, "commit": commit,
"package": map[string]any{"repository": "o/sdk", "commit": sdkCommit}})
if err != nil {
t.Fatalf("a member locking the group's version: %v", err)
}
placed := filepath.Join(m.Work, out.(FetchResult).Workspace, "tree", "modules", "m", "node_modules", "@novox", "mesh-sdk")
if raw, err := os.ReadFile(filepath.Join(placed, "dist", "index.js")); err != nil || !strings.Contains(string(raw), "9.9.9") {
t.Errorf("the member's SDK is not the group's built package: %q %v", raw, err)
}
other := member("sha512-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==")
_, err = m.fetch(context.Background(), map[string]any{"repository": "o/m-AAAA", "commit": other,
"package": map[string]any{"repository": "o/sdk", "commit": sdkCommit}})
if err == nil || !strings.Contains(err.Error(), group[1]) {
t.Errorf("a member locking another integrity answered %v", err)
}
}
// ADR 0499 §4 (issue 1758): a fetch asked with the module's directory and its named paths leaves only those, the
// manifest and the lock files; another module of the commit is not listed or installed.
func TestAFetchWithNamedPathsNarrowsTheWorkspace(t *testing.T) {
root := t.TempDir()
files := typeScriptModule(`{"name":"m"}`)
files["modules/m/tools/index.ts"] = "export {}\n"
files["modules/m/docs/a.md"] = "not named\n"
files["modules/other/module.json"] = files["modules/m/module.json"]
files["README.md"] = "the repository\n"
commit := forgeRepo(t, root, "narrow", files)
m, _, _ := newMachine(t)
m.Run = forgekey.ExecRunner
m.RemoteBase = "file://" + filepath.Join(root, "forge") + "/"
m.FetchOptions = []buildstep.Option{buildstep.AllowFileRemote()}
out, err := m.fetch(context.Background(), map[string]any{"repository": "o/narrow", "commit": commit, "dir": "modules/m", "paths": []any{"tools"}})
if err != nil {
t.Fatal(err)
}
r := out.(FetchResult)
tree := filepath.Join(m.Work, r.Workspace, "tree")
for _, gone := range []string{"README.md", "modules/m/docs/a.md", "modules/other/module.json"} {
if _, err := os.Stat(filepath.Join(tree, gone)); err == nil {
t.Errorf("%s is still in the workspace", gone)
}
}
if _, err := os.Stat(filepath.Join(tree, "modules/m/tools/index.ts")); err != nil || strings.Join(r.Modules, ",") != "modules/m" {
t.Errorf("the named path or the module went: %v, modules %v", err, r.Modules)
}
}
@@ -1,160 +0,0 @@
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// tgzOf is a packed npm package holding package/package.json of name@version and one more file.
func tgzOf(t *testing.T, name, version string) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for _, f := range [][2]string{{"package/package.json", `{"name":"` + name + `","version":"` + version + `"}`}, {"package/dist/index.js", "export const v = '" + version + "';\n"}} {
n, body := f[0], f[1]
tw.WriteHeader(&tar.Header{Name: n, Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg})
tw.Write([]byte(body))
}
tw.Close()
zw.Close()
return buf.Bytes()
}
// fakeRegistry serves the packument and the packed files of the versions it holds; asked counts its answers.
func fakeRegistry(t *testing.T, m *Module, held map[string][]byte) *atomic.Int32 {
t.Helper()
var asked atomic.Int32
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
asked.Add(1)
switch {
case r.URL.Path == "/npm/@novox%2fmesh-sdk" || r.URL.Path == "/npm/@novox/mesh-sdk":
if len(held) == 0 {
http.NotFound(w, r)
return
}
versions := map[string]any{}
for v, tgz := range held {
versions[v] = map[string]any{"dist": map[string]any{"tarball": srv.URL + "/files/" + v + ".tgz", "integrity": integrityOf(tgz)}}
}
json.NewEncoder(w).Encode(map[string]any{"versions": versions})
case strings.HasPrefix(r.URL.Path, "/files/"):
v := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/files/"), ".tgz")
if tgz, ok := held[v]; ok {
w.Write(tgz)
return
}
http.NotFound(w, r)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
m.Toolchain.Registry = func() (Registry, error) { return Registry{URL: srv.URL + "/npm/"}, nil }
m.HTTP = srv.Client()
return &asked
}
// memberLocking is a member's directory whose lock names @novox/mesh-sdk@version with integrity (none when version is "").
func memberLocking(t *testing.T, version, integrity string) string {
t.Helper()
dir := t.TempDir()
lock := map[string]any{"lockfileVersion": 3, "packages": map[string]any{"": map[string]any{"name": "m"}}}
if version != "" {
lock["packages"].(map[string]any)["node_modules/@novox/mesh-sdk"] = map[string]any{"version": version, "integrity": integrity}
}
raw, _ := json.Marshal(lock)
write(t, filepath.Join(dir, "package-lock.json"), string(raw), 0o644)
return dir
}
func placedVersion(dir string) string {
raw, err := os.ReadFile(filepath.Join(dir, "node_modules", "@novox", "mesh-sdk", "package.json"))
if err != nil {
return ""
}
var p packageJSON
json.Unmarshal(raw, &p)
return p.Version
}
// The lead's decision L10 (ADR 0500 §6): a member is built against the SDK version its own lock names, downloaded from
// the registry and checked against the lock's integrity; a version the registry does not hold is a wait, `waits for:`.
func TestAMemberGetsTheSDKVersionItsLockNamesOrWaitsForIt(t *testing.T) {
m, _, _ := newMachine(t)
held := tgzOf(t, sdkPackage, "0.1.15")
fakeRegistry(t, m, map[string][]byte{"0.1.15": held})
dir := memberLocking(t, "0.1.15", integrityOf(held))
if err := m.placeSDK(context.Background(), dir, "modules/m", nil); err != nil || placedVersion(dir) != "0.1.15" {
t.Errorf("the registry's 0.1.15: %v, placed %q", err, placedVersion(dir))
}
dir = memberLocking(t, "0.1.16", "sha512-x")
err := m.placeSDK(context.Background(), dir, "modules/m", nil)
if err == nil || err.Error() != delivery.WaitsFor+"@novox/mesh-sdk@0.1.16" {
t.Errorf("a version the registry lacks answered %v, want %s@novox/mesh-sdk@0.1.16", err, delivery.WaitsFor)
}
dir = memberLocking(t, "0.1.15", "sha512-another")
if err := m.placeSDK(context.Background(), dir, "modules/m", nil); err == nil || !strings.Contains(err.Error(), "sha512-another") {
t.Errorf("a registry file of another integrity than the lock's answered %v", err)
}
}
// ADR 0500 §5 (S3, S4): the group's packed package is served in place of the registry's when the member's lock names its
// version with its integrity; another integrity, or a version neither the group nor the registry holds, fails naming both.
// A member whose lock names no SDK keeps the toolchain's and asks the registry nothing.
func TestTheGroupsPackageIsServedToAMemberLockingItsVersion(t *testing.T) {
m, _, _ := newMachine(t)
asked := fakeRegistry(t, m, map[string][]byte{"0.1.14": tgzOf(t, sdkPackage, "0.1.14")})
tgz := tgzOf(t, sdkPackage, "0.1.15")
group := &packed{Name: sdkPackage, Version: "0.1.15", Integrity: integrityOf(tgz), Digest: digestOf(tgz), tgz: tgz}
dir := memberLocking(t, "0.1.15", group.Integrity)
if err := m.placeSDK(context.Background(), dir, "modules/m", group); err != nil || placedVersion(dir) != "0.1.15" || asked.Load() != 0 {
t.Errorf("the group's own version: %v, placed %q, the registry asked %d times", err, placedVersion(dir), asked.Load())
}
dir = memberLocking(t, "0.1.15", "sha512-other")
if err := m.placeSDK(context.Background(), dir, "modules/m", group); err == nil || !strings.Contains(err.Error(), "sha512-other") || !strings.Contains(err.Error(), group.Integrity) {
t.Errorf("another integrity answered %v, want both named", err)
}
dir = memberLocking(t, "0.1.14", integrityOf(tgzOf(t, sdkPackage, "0.1.14")))
if err := m.placeSDK(context.Background(), dir, "modules/m", group); err != nil || placedVersion(dir) != "0.1.14" {
t.Errorf("a member locking the registry's 0.1.14 in a group: %v, placed %q", err, placedVersion(dir))
}
dir = memberLocking(t, "0.1.13", "sha512-x")
if err := m.placeSDK(context.Background(), dir, "modules/m", group); err == nil || !strings.Contains(err.Error(), "0.1.13") || !strings.Contains(err.Error(), "0.1.15") || strings.HasPrefix(err.Error(), delivery.WaitsFor) {
t.Errorf("a version neither holds answered %v, want both named and no wait", err)
}
before := asked.Load()
dir = memberLocking(t, "", "")
if err := m.placeSDK(context.Background(), dir, "modules/m", group); err != nil || placedVersion(dir) != "" || asked.Load() != before {
t.Errorf("a lock naming no SDK: %v, placed %q, the registry asked", err, placedVersion(dir))
}
}
// The fetch's `package` argument is a repository and a full commit, with a pull request's head at most; anything else is
// refused before anything is fetched.
func TestTheGroupPackageArgumentIsRefusedWhenItIsNotOne(t *testing.T) {
for _, bad := range []any{"novox/mesh-sdk", map[string]any{"repository": "novox/mesh-sdk", "commit": "main"},
map[string]any{"repository": "../x", "commit": strings.Repeat("a", 40)},
map[string]any{"repository": "novox/mesh-sdk", "commit": strings.Repeat("a", 40), "ref": "refs/heads/main"}} {
if _, err := groupPackageArg(bad); err == nil {
t.Errorf("%v was taken", bad)
}
}
g, err := groupPackageArg(map[string]any{"repository": "novox/mesh-sdk", "commit": strings.Repeat("a", 40), "ref": "refs/pull/9/head"})
if err != nil || g.Ref != "refs/pull/9/head" {
t.Errorf("a pull request's head: %+v %v", g, err)
}
}
@@ -1,121 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/base64"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os/exec"
"path/filepath"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// npmRegistry is a registry of the test's own speaking npm's publish protocol: a PUT of the package's document with the
// packed file attached, then GET of the document and the file; it asks for the account's credential.
type npmRegistry struct {
mu sync.Mutex
versions map[string]map[string]any
files map[string][]byte
publishes int
}
func (r *npmRegistry) serve(t *testing.T) *httptest.Server {
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, q *http.Request) {
r.mu.Lock()
defer r.mu.Unlock()
if u, p, ok := q.BasicAuth(); !ok || u != "builder" || p != "the write credential" {
w.WriteHeader(http.StatusUnauthorized)
return
}
doc := strings.TrimPrefix(q.URL.EscapedPath(), "/npm/")
switch {
case q.Method == http.MethodPut && (doc == "@novox%2fmesh-sdk" || doc == "@novox%2Fmesh-sdk"):
var body struct {
Versions map[string]map[string]any `json:"versions"`
Attachments map[string]struct {
Data string `json:"data"`
} `json:"_attachments"`
}
raw, _ := io.ReadAll(q.Body)
if err := json.Unmarshal(raw, &body); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
for v, meta := range body.Versions {
for name, a := range body.Attachments {
data, _ := base64.StdEncoding.DecodeString(a.Data)
r.files[name] = data
meta["dist"] = map[string]any{"tarball": srv.URL + "/files/" + name, "integrity": integrityOf(data)}
}
r.versions[v] = meta
}
r.publishes++
w.WriteHeader(http.StatusCreated)
w.Write([]byte(`{"ok":true}`))
case q.Method == http.MethodGet && (doc == "@novox%2fmesh-sdk" || doc == "@novox%2Fmesh-sdk"):
if len(r.versions) == 0 {
http.NotFound(w, q)
return
}
json.NewEncoder(w).Encode(map[string]any{"name": "@novox/mesh-sdk", "versions": r.versions})
case strings.HasPrefix(q.URL.Path, "/files/"):
w.Write(r.files[strings.TrimPrefix(q.URL.Path, "/files/")])
default:
http.NotFound(w, q)
}
}))
t.Cleanup(srv.Close)
return srv
}
// 1587-S20 (ADR 0498 §5): the real npm publishes a trunk build's package to a registry speaking npm's protocol, with
// the module's credential, once: a second trunk build of the same package finds its file there and publishes nothing;
// a candidate publishes nothing at all.
func TestARealTrunkPackageIsPublishedOnce(t *testing.T) {
for _, tool := range []string{"node", "npm", "bwrap"} {
if _, err := exec.LookPath(tool); err != nil {
t.Skipf("NOT RUN: no %s where this check runs; every build node has it", tool)
}
}
m, _, work := newMachine(t)
m.Run = forgekey.ExecRunner
reg := &npmRegistry{versions: map[string]map[string]any{}, files: map[string][]byte{}}
srv := reg.serve(t)
m.Toolchain.Registry = func() (Registry, error) {
return Registry{URL: srv.URL + "/npm/", Username: "builder", Password: "the write credential"}, nil
}
m.HTTP = srv.Client()
files := map[string]string{}
for n, c := range sdkFixture {
files[n] = c
}
workspaceWith(t, work, files)
dir := filepath.Join(work, "w1", "tree")
if err := m.installDev(context.Background(), filepath.Join(work, "w1"), dir, "."); err != nil {
t.Fatal(err)
}
args := map[string]any{"workspace": "w1", "module": ".", "artifact": "lib"}
if out, err := m.build(context.Background(), args); err != nil || out.(BuildResult).Reference != "" || reg.publishes != 0 {
t.Fatalf("a candidate: %+v %v, %d publishes", out, err, reg.publishes)
}
args["publish"] = true
out, err := m.build(context.Background(), args)
if err != nil || out.(BuildResult).Reference != "@novox/mesh-sdk@9.9.9" || reg.publishes != 1 {
t.Fatalf("a trunk build: %+v %v, %d publishes", out, err, reg.publishes)
}
if got := digestOf(reg.files["@novox/mesh-sdk-9.9.9.tgz"]); got != out.(BuildResult).Digest {
t.Errorf("the registry holds %s, the build packed %s (files %d)", got, out.(BuildResult).Digest, len(reg.files))
}
if again, err := m.build(context.Background(), args); err != nil || again.(BuildResult).Reference != "@novox/mesh-sdk@9.9.9" || reg.publishes != 1 {
t.Errorf("the same package again: %+v %v, %d publishes", again, err, reg.publishes)
}
}
@@ -1,116 +0,0 @@
package main
import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const packageManifest = `{"module":"mesh-sdk","version":"1","build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript","paths":["src","package.json"]}]}}`
// packageMachine is a machine whose workspace w1 holds the package @novox/mesh-sdk@0.1.15 at its root; npm pack writes
// the packed file tgz into its --pack-destination, as npm does.
func packageMachine(t *testing.T, tgz []byte) (*Module, *fakeMachine, string) {
t.Helper()
m, f, work := newMachine(t)
workspaceWith(t, work, map[string]string{"module.json": packageManifest, "src/index.ts": "export const v = 1;\n",
"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.15","scripts":{"build":"tsc"}}`, "package-lock.json": `{"lockfileVersion":3}`})
f.effect = func(c call) {
if c.name != "npm" || len(c.args) == 0 || c.args[0] != "pack" {
return
}
for i, a := range c.args {
if a == "--pack-destination" {
write(t, filepath.Join(c.args[i+1], "novox-mesh-sdk-0.1.15.tgz"), string(tgz), 0o644)
}
}
}
return m, f, work
}
func publishes(f *fakeMachine) []call {
var out []call
for _, c := range f.calls {
if c.name == "npm" && len(c.args) > 0 && c.args[0] == "publish" {
out = append(out, c)
}
}
return out
}
// 1587-S15..S19 (ADR 0498 §5): a package is built by its own build script and packed, shut off; a candidate answers the
// packed file's digest, publishes nothing and asks the registry nothing; its reference is empty.
func TestACandidatesPackageIsBuiltAndPackedAndNeverPublished(t *testing.T) {
tgz := tgzOf(t, sdkPackage, "0.1.15")
m, f, _ := packageMachine(t, tgz)
asked := fakeRegistry(t, m, nil)
out, err := m.build(context.Background(), map[string]any{"workspace": "w1", "module": ".", "artifact": "lib"})
if err != nil {
t.Fatal(err)
}
r := out.(BuildResult)
if r.Digest != digestOf(tgz) || r.Reference != "" || r.Bytes != len(tgz) {
t.Errorf("result %+v, want the packed file's digest and no reference", r)
}
if len(publishes(f)) != 0 || asked.Load() != 0 {
t.Errorf("a candidate published %d times and asked the registry %d times", len(publishes(f)), asked.Load())
}
var ran []string
for _, s := range f.shut {
ran = append(ran, s.command)
}
if len(f.shut) != 2 || !strings.Contains(ran[0], "npm run build") || !strings.Contains(ran[1], "npm pack --ignore-scripts") {
t.Errorf("shut off %v, want the build and the pack", ran)
}
}
// ADR 0498 §5: a trunk build (publish) publishes a version the registry lacks and answers <name>@<version>; the same
// packed file already there is done without a publish; another file under that version fails `version taken:`. The
// packed file is gone from the node afterwards.
func TestATrunkPackageIsPublishedOnceAndAVersionTakenIsRefused(t *testing.T) {
tgz := tgzOf(t, sdkPackage, "0.1.15")
args := map[string]any{"workspace": "w1", "module": ".", "artifact": "lib", "publish": true}
m, f, work := packageMachine(t, tgz)
fakeRegistry(t, m, nil)
out, err := m.build(context.Background(), args)
if err != nil || out.(BuildResult).Reference != "@novox/mesh-sdk@0.1.15" || len(publishes(f)) != 1 {
t.Fatalf("an absent version: %+v %v, %d publishes", out, err, len(publishes(f)))
}
if p := publishes(f)[0]; !slices.Contains(p.args, "--ignore-scripts") || !strings.HasSuffix(p.args[1], ".tgz") {
t.Errorf("published with %v", p.args)
}
// ADR 0500 §4: the registry's credential is never in a workspace, which a run step is given.
for i, a := range publishes(f)[0].args {
if a == "--userconfig" && strings.HasPrefix(publishes(f)[0].args[i+1], work) {
t.Errorf("the credential was written into the workspace: %s", publishes(f)[0].args[i+1])
}
}
var left []string
filepath.WalkDir(work, func(p string, d os.DirEntry, err error) error {
if err == nil && strings.HasSuffix(p, ".tgz") || strings.Contains(p, ".npmrc") {
left = append(left, p)
}
return nil
})
if len(left) != 0 {
t.Errorf("left on the node: %v", left)
}
m, f, _ = packageMachine(t, tgz)
fakeRegistry(t, m, map[string][]byte{"0.1.15": tgz})
if out, err := m.build(context.Background(), args); err != nil || out.(BuildResult).Reference != "@novox/mesh-sdk@0.1.15" || len(publishes(f)) != 0 {
t.Errorf("the same file published before: %+v %v, %d publishes", out, err, len(publishes(f)))
}
m, f, _ = packageMachine(t, tgz)
fakeRegistry(t, m, map[string][]byte{"0.1.15": tgzOf(t, sdkPackage, "another")})
_, err = m.build(context.Background(), args)
if err == nil || !strings.HasPrefix(err.Error(), delivery.VersionTaken+"@novox/mesh-sdk@0.1.15") || len(publishes(f)) != 0 {
t.Errorf("another file under the version: %v, %d publishes", err, len(publishes(f)))
}
}
@@ -1,42 +0,0 @@
package main
// The push of a done build's archive (novox/hq ADR 0498 §1 §2; issue 1587): from this module's own process, after the
// build step ended, with mesh-sdk's one client (buildstep.PushBlob), to the artifact store whose address the manifest
// binds into RegistryFile. No build step reaches the store: a run step has no network (ADR 0448).
import (
"context"
"fmt"
"net/http"
"os"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
)
// registry is the artifact store's address, host:port, as the manifest bound it into RegistryFile.
func (m *Module) registry() (string, error) {
raw, err := os.ReadFile(m.RegistryFile)
if err != nil {
return "", fmt.Errorf("the artifact store's address cannot be read: %w", err)
}
addr := strings.TrimSpace(string(raw))
if addr == "" || strings.Contains(addr, "${") || strings.HasPrefix(addr, ":") || strings.HasSuffix(addr, ":") {
return "", fmt.Errorf("the artifact store's address in %s is %q, not host:port", m.RegistryFile, addr)
}
return addr, nil
}
// push puts body into the artifact store as a blob of repository (<module>/<artifact>) by its digest: Push when a test
// set it, else buildstep.PushBlob.
func (m *Module) push(ctx context.Context, repository, digest string, body []byte) error {
addr, err := m.registry()
if err != nil {
return err
}
if m.Push != nil {
return m.Push(ctx, addr, repository, digest, body)
}
return buildstep.PushBlob(ctx, &http.Client{Timeout: 10 * time.Minute}, addr, repository, digest, body)
}
@@ -1,99 +0,0 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// pushed are the blobs the fake artifact store took, by digest, for the tests that read an archive back.
var pushed sync.Map
// fakeStore is a store of the test's own: its address in RegistryFile, each blob kept by repository and digest; fail,
// when set, is the store's refusal.
type fakeStore struct {
mu sync.Mutex
blobs map[string][]byte // "<repository>@<digest>"
fail error
}
func withStore(t *testing.T, m *Module) *fakeStore {
t.Helper()
s := &fakeStore{blobs: map[string][]byte{}}
m.RegistryFile = filepath.Join(t.TempDir(), "registry")
if err := os.WriteFile(m.RegistryFile, []byte("store.test:5000\n"), 0o644); err != nil {
t.Fatal(err)
}
m.Push = func(ctx context.Context, registry, repository, digest string, body []byte) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.fail != nil {
return s.fail
}
if sum := sha256.Sum256(body); "sha256:"+hex.EncodeToString(sum[:]) != digest || registry != "store.test:5000" {
t.Errorf("pushed %s to %s with bytes of another digest", digest, registry)
}
s.blobs[repository+"@"+digest] = append([]byte(nil), body...)
pushed.Store(digest, s.blobs[repository+"@"+digest])
return nil
}
return s
}
// archiveOf is the archive a build pushed, by its digest.
func archiveOf(t *testing.T, r BuildResult) []byte {
t.Helper()
raw, ok := pushed.Load(r.Digest)
if !ok {
t.Fatalf("nothing was pushed under %s", r.Digest)
}
return raw.([]byte)
}
// 1587-S6 (ADR 0498 §1): a done bundle answers the reference of the archive it pushed, under the manifest's module.
func TestADoneBundleAnswersTheReferenceOfTheArchiveItPushed(t *testing.T) {
m, _, _ := builtMachine(t, ``)
store := withStore(t, m)
out, err := m.build(context.Background(), buildArgs())
if err != nil {
t.Fatal(err)
}
r := out.(BuildResult)
if want := "artifact-store://m/tools/blobs/" + r.Digest; r.Reference != want {
t.Errorf("reference %q, want %q", r.Reference, want)
}
if store.blobs["m/tools@"+r.Digest] == nil {
t.Errorf("the store holds %d blobs, not m/tools@%s", len(store.blobs), r.Digest)
}
}
// 1587-S7 and S8 (ADR 0498 §2, the lead's decision L9): a push the store refuses fails the step `not stored:
// <artifact>: <the store's words>`, and whatever the push answered, no packed file is left on the node.
func TestARefusedPushFailsNotStoredAndNoPackedFileStays(t *testing.T) {
for _, fail := range []error{nil, errors.New("store.test:5000 refused the blob: 507 Insufficient Storage")} {
m, _, work := builtMachine(t, ``)
withStore(t, m).fail = fail
_, err := m.build(context.Background(), buildArgs())
if fail != nil && (err == nil || !strings.HasPrefix(err.Error(), delivery.NotStored+"tools: ") || !strings.Contains(err.Error(), "507")) {
t.Errorf("a refused push answered %v", err)
}
var archives []string
filepath.WalkDir(work, func(p string, d os.DirEntry, err error) error {
if err == nil && strings.HasSuffix(p, ".tar.gz") {
archives = append(archives, p)
}
return nil
})
if len(archives) != 0 {
t.Errorf("push answered %v: packed files left on the node: %v", fail, archives)
}
}
}
@@ -14,7 +14,7 @@ import (
// RecipeSteps is the recipe's own version: changed whenever the steps of fetch, test or build change in a way that
// can change what they produce (ADR 0405 §7). It is one of the two values the recipe version hashes.
const RecipeSteps = "2" // 2: a member is built against the SDK its own lock names (R4e, the lead's decision L10)
const RecipeSteps = "1"
// Recipe is every value that makes up the recipe version.
type Recipe struct {
@@ -12,7 +12,6 @@ import (
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"os"
"strings"
)
@@ -80,10 +79,3 @@ func (r Registry) Npmrc() string {
}
return sb.String()
}
// authorize gives a request to the registry the account's credential, when there is one.
func (r Registry) authorize(req *http.Request) {
if r.Username != "" && r.Password != "" {
req.SetBasicAuth(r.Username, r.Password)
}
}
@@ -1,77 +0,0 @@
//go:build integration
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/distribution/distribution/v3/configuration"
"github.com/distribution/distribution/v3/registry/handlers"
_ "github.com/distribution/distribution/v3/registry/storage/driver/inmemory"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// realStore starts the artifact store's own program (the distribution registry, modules/distribution) inside this
// test, with its in-memory storage, on a loopback port, and points m at it with the real push client. It is imported
// by this integration test alone: the module's program does not carry it.
func realStore(t *testing.T, m *Module) string {
t.Helper()
config := &configuration.Configuration{Storage: configuration.Storage{"inmemory": configuration.Parameters{}}}
config.HTTP.Secret = "a throwaway store"
config.Log.Level = "error"
config.Log.AccessLog.Disabled = true
srv := httptest.NewServer(handlers.NewApp(context.Background(), config))
t.Cleanup(srv.Close)
addr := strings.TrimPrefix(srv.URL, "http://")
m.RegistryFile = filepath.Join(t.TempDir(), "registry")
if err := os.WriteFile(m.RegistryFile, []byte(addr+"\n"), 0o644); err != nil {
t.Fatal(err)
}
m.Push = nil
return addr
}
// 1587-S10 (ADR 0498's check): a done bundle names a reference whose tail is its digest, and that reference resolves in
// a real artifact store to bytes of that digest. The build is the real tsc's and esbuild's, shut off.
func TestADoneBundleResolvesInARealArtifactStore(t *testing.T) {
if _, err := exec.LookPath("bwrap"); err != nil {
t.Skip("NOT RUN: no bubblewrap installed where this check runs; every build node has it (the module declares it)")
}
m, _ := realMachine(t)
addr := realStore(t, m)
out, err := m.build(context.Background(), map[string]any{"workspace": "w1", "module": "modules/m", "artifact": "tools"})
if err != nil {
t.Fatal(err)
}
r := out.(BuildResult)
done := delivery.BuilderBuild{V: 1, State: delivery.BuildDone, Manifest: strings.Repeat("0", 64),
Artifacts: []delivery.BuiltArtifact{{Name: "tools", Kind: "bundle", Digest: r.Digest, Reference: r.Reference}}}
if _, err := delivery.EncodeBuilderBuild(done); err != nil {
t.Errorf("the done key the builder would write is refused: %v", err)
}
path, ok := strings.CutPrefix(r.Reference, "artifact-store://")
if !ok || !strings.HasSuffix(r.Reference, "/blobs/"+r.Digest) {
t.Fatalf("the reference %q", r.Reference)
}
resp, err := http.Get("http://" + addr + "/v2/" + path)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
sum := sha256.Sum256(body)
if resp.StatusCode != http.StatusOK || "sha256:"+hex.EncodeToString(sum[:]) != r.Digest {
t.Errorf("the store answered %s with bytes of sha256:%x for %s", resp.Status, sum, r.Reference)
}
}
@@ -168,16 +168,10 @@ func copyFile(from, to string) error {
return out.Close()
}
// runIn runs a command of a run step shut off (stepRunner) with dir as its working directory (`env -C`), so it is the
// plain line the build agent runs from the module's root, with the same relative paths. The command's environment is
// a PATH, a locale and extra: never the account's HOME (the step's is its own) nor this process's environment. root is
// the workspace; write is a trunk build's (publish), which alone writes the shared cache.
func (m *Module) runIn(ctx context.Context, root string, write bool, dir string, extra []string, name string, args ...string) forgekey.Ran {
run, err := m.stepRunner(root, write)
if err != nil {
return forgekey.Ran{Status: 1, Err: err.Error()}
}
return run(ctx, extra, "env", append([]string{"-C", dir, name}, args...)...)
// runIn runs a command with dir as its working directory (`env -C`), so it is the plain line the build agent runs from
// the module's root, with the same relative paths. The command's environment is baseEnv and extra.
func (m *Module) runIn(ctx context.Context, dir string, extra []string, name string, args ...string) forgekey.Ran {
return m.Run(ctx, append(m.baseEnv(), extra...), "env", append([]string{"-C", dir, name}, args...)...)
}
// ranFailed says why a command did not pass, or "" when it did.

Some files were not shown because too many files have changed in this diff Show More