Delete the builder, the build asks, the build queue and the merge gate: the builder's keys are the only way a module is built (issues 1851, 1299, 1318; C2b-C5, C10)
The controller asks no build of anybody (ADR 0501 §5, ADR 0525 clause 6, ADR 0503 §3). Every module of a walk is taken from the builder's done key; a key not done yet waits, named in the plan, and never fails it. - C2b/C2c: internal/builder and cmd/mesh-controller's asks go (askABuild, onlyTheBuildersOwn, buildOneAsked, takeIn from an outcome, buildAgentsOwn, the builder's say and its fallback, the daemon hearing an outcome). The server follows the retired seats' built subjects only to let go of them, because the installer's carried user list names them. - C2d: replay, queue, cancel, clear, kill, pause, resume, delivery-check, check-here and merge-gate are refused by name. - C3: the check queues (CHECKS_READY, CHECKS_OTHER) and their workers, the checks-running and plans-building buckets and their grants, the build queue and its hold (plans_building.go, queue.go, link/holds.go, builds_nats.go, checks_nats.go). - C4: merge_gate*.go, the pull request's repo-check path, the Checked event and its layers; a pull request asks the controller nothing but to tell mesh-config what its head declares. - C5: the builds bucket is written only by key_takein. - C10 (help and refusals): the terminal's help lists only what is served. Integration tests of the plan's tiers, sends, stops and retries now run on a stand-in builder with a done key for every module (walk_keys_scaffold_integration_test.go); the tests of the asks, the queue, the merge gate and the replays are deleted.
This commit is contained in:
@@ -1,91 +0,0 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Listing and letting go of tagged documents: the check builds the scratch namespace keeps (novox/hq ADR 0350
|
||||
// §5). A check build is found by its tag, and the build seat's holder lets each go when its pull request closes
|
||||
// or it is seven days old, so the store keeps a week of check builds and no more.
|
||||
|
||||
// Tags is every tag repository holds, in the store's order. None for a repository the store does not hold.
|
||||
func (s Store) Tags(ctx context.Context, repository string) ([]string, error) {
|
||||
if s.Address == "" {
|
||||
return nil, fmt.Errorf("this mesh has no artifact store on its network to list %s in", repository)
|
||||
}
|
||||
var tags []string
|
||||
next := "http://" + s.Address + "/v2/" + repository + "/tags/list?n=1000"
|
||||
for next != "" {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, next, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
response, err := s.client().Do(request)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot reach the artifact store at %s: %w", s.Address, err)
|
||||
}
|
||||
switch response.StatusCode {
|
||||
case http.StatusOK:
|
||||
case http.StatusNotFound:
|
||||
response.Body.Close()
|
||||
return tags, nil
|
||||
default:
|
||||
response.Body.Close()
|
||||
return nil, fmt.Errorf("the artifact store answered %s for the tags of %s", response.Status, repository)
|
||||
}
|
||||
var page struct {
|
||||
Tags []string `json:"tags"`
|
||||
}
|
||||
err = json.NewDecoder(io.LimitReader(response.Body, 8<<20)).Decode(&page)
|
||||
link := response.Header.Get("Link")
|
||||
response.Body.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the tags of %s could not be read: %w", repository, err)
|
||||
}
|
||||
tags = append(tags, page.Tags...)
|
||||
next = nextPage(link, s.Address)
|
||||
}
|
||||
return tags, nil
|
||||
}
|
||||
|
||||
// nextPage is the registry's `Link: </v2/…>; rel="next"`, made whole; empty when there is no next page.
|
||||
func nextPage(link, address string) string {
|
||||
if !strings.Contains(link, `rel="next"`) {
|
||||
return ""
|
||||
}
|
||||
start, end := strings.Index(link, "<"), strings.Index(link, ">")
|
||||
if start < 0 || end <= start+1 {
|
||||
return ""
|
||||
}
|
||||
target := link[start+1 : end]
|
||||
if strings.HasPrefix(target, "/") {
|
||||
return "http://" + address + target
|
||||
}
|
||||
return target
|
||||
}
|
||||
|
||||
// DeleteTagged lets go of the document tag names in repository: its manifest, by digest, so the next
|
||||
// collection takes its layer. Gone when the tag names nothing.
|
||||
func (s Store) DeleteTagged(ctx context.Context, repository, tag string) error {
|
||||
if s.Address == "" {
|
||||
return fmt.Errorf("this mesh has no artifact store on its network to let go of %s:%s in", repository, tag)
|
||||
}
|
||||
digest, err := s.manifestDigest(ctx, repository, tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if digest == "" {
|
||||
return Gone
|
||||
}
|
||||
err = s.remove(ctx, s.url(repository, "manifests", digest), repository+":"+tag)
|
||||
if errors.Is(err, Gone) {
|
||||
return Gone
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -8,7 +8,7 @@
|
||||
// - In a merge check, the build seat clones each core repository beside the one checked (the catalogue
|
||||
// at its main, the node-engine at the commit the mesh runs) and says where in MESH_CHECK_BESIDE. A
|
||||
// test judges against those clones, so agreement with the other repository is checked where
|
||||
// `mesh/repo-check` runs; a repository missing there fails the test, never skips it. Which clones a
|
||||
// `mesh/check` runs; a repository missing there fails the test, never skips it. Which clones a
|
||||
// check gets is chosen from the inventory: mesh-catalog by the source of the `nats` module, mesh-host
|
||||
// by the source of `mesh-host`, and mesh-sdk as the controller's sibling at the commit the controller's
|
||||
// go.mod pins. In a mesh where either module has no source repository nothing is
|
||||
|
||||
@@ -17,9 +17,9 @@ import (
|
||||
const StatesAnnouncedBucket = announce.KeptBucket
|
||||
|
||||
// controllerReadable are the controller's states others may read: the record mirrors (the packages each node declares
|
||||
// among them, which the pacman module reads, ADR 0507 §5), the chain's buckets, the checks running and the plans building.
|
||||
// among them, which the pacman module reads, ADR 0507 §5) and the chain's buckets.
|
||||
var controllerReadable = []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined, ChainBuilds, ChainRollouts,
|
||||
ChecksRunning, PlansBuilding, MirrorDeclaredPackages}
|
||||
MirrorDeclaredPackages}
|
||||
|
||||
// ControllerAnnouncement is what the controller announces it keeps (ADR 0495 clause 2): every bucket of its own by its local
|
||||
// name, the ones others may read marked readable. It announces itself, as any module does from its code.
|
||||
|
||||
@@ -38,7 +38,7 @@ func TestTheControllerAnnouncesWhatItKeeps(t *testing.T) {
|
||||
readable = append(readable, k.Name)
|
||||
}
|
||||
}
|
||||
want := []string{ChainBuilds, ChecksRunning, MirrorDeclaredConfig, PlansBuilding, MirrorPredefined, ChainRollouts,
|
||||
want := []string{ChainBuilds, MirrorDeclaredConfig, MirrorPredefined, ChainRollouts,
|
||||
MirrorAssignments, MirrorDeclaredPackages}
|
||||
slices.Sort(want)
|
||||
if !slices.Equal(readable, want) {
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
package broker
|
||||
|
||||
import "strings"
|
||||
|
||||
// The pull requests' checks have work queues of their own (novox/hq ADR 0406 §3, issue 974).
|
||||
//
|
||||
// A delivery build goes on the build seat's own queue, `mesh.seat.node-build-agent.accept.build`. A check of a pull
|
||||
// request marked ready goes on a second work-queue stream and every other check on a third. A builder takes from the
|
||||
// first, then the second, then the third; the others are taken in the order asked. **Each check stream has one subject
|
||||
// per pull request and keeps only the newest request** (a limit of one message a subject, the older discarded), so a
|
||||
// newer request replaces one still waiting and a pull request has at most one waiting check: the stream's rule, not a
|
||||
// pass of the controller that moves asks about. Nothing pauses the worker or deletes and republishes a request.
|
||||
|
||||
const (
|
||||
// ChecksReadyStream holds the checks of pull requests marked ready; ChecksOtherStream every other check.
|
||||
ChecksReadyStream = "CHECKS_READY"
|
||||
ChecksOtherStream = "CHECKS_OTHER"
|
||||
// ChecksReadyWorker and ChecksOtherWorker are the durable consumers the build seat's holders share on them, as
|
||||
// they share the seat's worker on its own queue (novox/hq ADR 0190).
|
||||
ChecksReadyWorker = "CHECKS_READY_worker"
|
||||
ChecksOtherWorker = "CHECKS_OTHER_worker"
|
||||
// TheBuildSeat is the role whose checks these streams queue. The retired build role's checks stay on its own queue.
|
||||
TheBuildSeat = "node-build-agent"
|
||||
)
|
||||
|
||||
// PullsBucket is mesh-delivery's key per pull request: its head, the trunk commit its current check is asked against and
|
||||
// its verdict (ADR 0406 §4). A running check watches its key.
|
||||
var PullsBucket = BucketName("mesh-delivery", "pulls")
|
||||
|
||||
// CheckSubjectPrefix is where a seat's checks of one kind are asked: `mesh.seat.<seat>.check.ready.` or `.other.`. The
|
||||
// `check` token keeps the subjects out of the seat's own queue, which takes `accept.>`.
|
||||
func CheckSubjectPrefix(seat string, ready bool) string {
|
||||
kind := "other"
|
||||
if ready {
|
||||
kind = "ready"
|
||||
}
|
||||
return "mesh.seat." + seat + ".check." + kind + "."
|
||||
}
|
||||
|
||||
// CheckStreams are the two check queues, in the order a builder takes from them.
|
||||
func CheckStreams() []Stream {
|
||||
return []Stream{
|
||||
{
|
||||
Name: ChecksReadyStream,
|
||||
Subjects: []string{CheckSubjectPrefix(TheBuildSeat, true) + ">"},
|
||||
Retention: RetentionWorkQueue,
|
||||
MaxAge: 7 * 24 * 60 * 60,
|
||||
MaxMsgsPerSubject: 1,
|
||||
Direct: true,
|
||||
Why: "the checks of pull requests marked ready, taken after a delivery build and before any other check " +
|
||||
"(novox/hq ADR 0406 §3); one subject a pull request, the newest request kept, so a newer one replaces a " +
|
||||
"request still waiting",
|
||||
},
|
||||
{
|
||||
Name: ChecksOtherStream,
|
||||
Subjects: []string{CheckSubjectPrefix(TheBuildSeat, false) + ">"},
|
||||
Retention: RetentionWorkQueue,
|
||||
MaxAge: 7 * 24 * 60 * 60,
|
||||
MaxMsgsPerSubject: 1,
|
||||
Direct: true,
|
||||
Why: "every other check, taken last and in the order asked (novox/hq ADR 0406 §3); one subject a pull " +
|
||||
"request, the newest request kept; one left behind for a head already judged is dropped when taken",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// CheckWorkers are the consumers the build seat's holders pull the check queues through: as the seat's own worker is,
|
||||
// one for every holder, one ask at a time, acknowledged when the check is done.
|
||||
func CheckWorkers() []Consumer {
|
||||
worker := func(name, stream string, ready bool) Consumer {
|
||||
return Consumer{
|
||||
Name: name,
|
||||
Stream: stream,
|
||||
Filters: []string{CheckSubjectPrefix(TheBuildSeat, ready) + ">"},
|
||||
AckWaitSeconds: 60,
|
||||
MaxDeliver: 5,
|
||||
Why: "every holder of " + TheBuildSeat + " pulls one check at a time from this worker and acknowledges after " +
|
||||
"the check is done, so a crash mid-check redelivers rather than loses (novox/hq ADR 0406 §3)",
|
||||
}
|
||||
}
|
||||
return []Consumer{
|
||||
worker(ChecksReadyWorker, ChecksReadyStream, true),
|
||||
worker(ChecksOtherWorker, ChecksOtherStream, false),
|
||||
}
|
||||
}
|
||||
|
||||
// holderCheckGrants are what a holder of the build seat is granted to take checks: ask about the two workers, pull from
|
||||
// them and acknowledge, and nothing of the subjects themselves.
|
||||
func holderCheckGrants() []string {
|
||||
var pub []string
|
||||
for _, w := range CheckWorkers() {
|
||||
pub = append(pub,
|
||||
"$JS.API.CONSUMER.INFO."+w.Stream+"."+w.Name,
|
||||
"$JS.API.CONSUMER.MSG.NEXT."+w.Stream+"."+w.Name,
|
||||
"$JS.ACK."+w.Stream+"."+w.Name+".>")
|
||||
}
|
||||
// And the bucket of the checks they are running: each writes the key of its own check, reads another's, and the bucket
|
||||
// is a controller's own bucket the build seat's holders write by design (ADR 0406 §3).
|
||||
pub = append(pub,
|
||||
"$KV."+CheckRunningBucket+".>",
|
||||
"$JS.API.STREAM.INFO.KV_"+CheckRunningBucket,
|
||||
"$JS.API.DIRECT.GET.KV_"+CheckRunningBucket+".>")
|
||||
// And, to hold checks while a delivery builds (ADR 0483, issues 1687 and 1688): the bucket of the plans building, **read and
|
||||
// never written** (the controller's alone, so no `$KV.` subject of it is granted), and the oldest waiting ask of each check
|
||||
// queue, read by a direct get that takes nothing from the queue and so counts against no delivery of the ask.
|
||||
pub = append(pub, readGrants(PlansBuildingBucket, "")...)
|
||||
for _, w := range CheckWorkers() {
|
||||
pub = append(pub, "$JS.API.DIRECT.GET."+w.Stream, "$JS.API.DIRECT.GET."+w.Stream+".>")
|
||||
}
|
||||
return pub
|
||||
}
|
||||
|
||||
// IsCheckStream says a stream is one of the check queues.
|
||||
func IsCheckStream(stream string) bool {
|
||||
return stream == ChecksReadyStream || stream == ChecksOtherStream
|
||||
}
|
||||
|
||||
// isCheckAsk says a message of a check queue is an ask the controller itself makes.
|
||||
func isCheckAsk(stream, subject string) bool {
|
||||
switch stream {
|
||||
case ChecksReadyStream:
|
||||
return strings.HasPrefix(subject, CheckSubjectPrefix(TheBuildSeat, true))
|
||||
case ChecksOtherStream:
|
||||
return strings.HasPrefix(subject, CheckSubjectPrefix(TheBuildSeat, false))
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
package broker
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 1155/AC1: a module whose manifest reads mesh-controller.checks-running may watch and get every key of the bucket and
|
||||
// write none.
|
||||
func TestAModuleReadingChecksRunningReadsAndWritesNone(t *testing.T) { // 1155/AC1
|
||||
if !slices.Contains(controllerReadableStates(), "checks-running") || BucketName(ControllerSeat, "checks-running") != CheckRunningBucket {
|
||||
t.Fatalf("a module cannot name %s in `reads`: %v", CheckRunningBucket, controllerReadableStates())
|
||||
}
|
||||
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "mesh-delivery-watchman", PasswordHash: "x",
|
||||
Reads: []string{"mesh-controller.checks-running"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stream := "KV_" + CheckRunningBucket
|
||||
has(t, perms.Publish, "$JS.API.STREAM.INFO."+stream)
|
||||
has(t, perms.Publish, "$JS.API.DIRECT.GET."+stream+".>")
|
||||
has(t, perms.Publish, "$JS.API.CONSUMER.CREATE."+stream+".>")
|
||||
for _, p := range perms.Publish {
|
||||
if SubjectsOverlap(p, "$KV."+CheckRunningBucket+".any.key") {
|
||||
t.Errorf("the reading module may write %s through %q", CheckRunningBucket, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 1155/AC3: the writers table names checks-running's writer as the build seat's holders, and the rule that a mirror is
|
||||
// written by the controller alone names it as the exception, with its reason.
|
||||
func TestChecksRunningIsAMirrorWrittenByTheBuildSeatsHolders(t *testing.T) { // 1155/AC3
|
||||
why := MirrorsWrittenByAnother["checks-running"]
|
||||
if why == "" {
|
||||
t.Fatal("the mirrors written by another do not name checks-running with a reason")
|
||||
}
|
||||
subject := "$KV." + CheckRunningBucket + ".novox.a.7"
|
||||
var rows int
|
||||
for _, row := range WritersTable {
|
||||
for _, s := range row.Subjects {
|
||||
if s != "$KV."+CheckRunningBucket+".>" {
|
||||
continue
|
||||
}
|
||||
rows++
|
||||
holder := Principal{Kind: KindModule, Module: "builder", Node: "one", Holds: []Seat{{Name: TheBuildSeat}}}
|
||||
other := Principal{Kind: KindModule, Module: "mesh-delivery-watchman", Node: "one"}
|
||||
if row.Writes == nil || !row.Writes(holder, subject) || row.Writes(other, subject) {
|
||||
t.Errorf("%q is not written by the build seat's holders alone", row.State)
|
||||
}
|
||||
}
|
||||
}
|
||||
if rows != 1 {
|
||||
t.Fatalf("%d rows of the writers table write %s, want one", rows, CheckRunningBucket)
|
||||
}
|
||||
for m := range MirrorsWrittenByAnother {
|
||||
if m != "checks-running" {
|
||||
t.Errorf("the mirror %q is written by another and is not checks-running", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
package broker
|
||||
|
||||
import "testing"
|
||||
|
||||
// The build agent's credential is the module principal `<node>.build-agent` (Username, KindModule): its manifest's
|
||||
// `reads: ["mesh-delivery.pulls"]` reaches that principal as a read and a watch of mesh-delivery's bucket of pull requests'
|
||||
// keys — the direct get of a key, the consumer a watch makes, the stream's info — and as no write and no direct
|
||||
// subscription of the bucket's subjects (novox/hq ADR 0406 §4, issue 974).
|
||||
func TestTheBuildAgentReadsAndWatchesThePullRequestsKeys(t *testing.T) {
|
||||
seat := Seat{Name: TheBuildSeat, Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
|
||||
p := Principal{Kind: KindModule, Node: "one", Module: "build-agent", Holds: []Seat{seat},
|
||||
Reads: []string{"mesh-delivery.pulls"}, PasswordHash: "x"}
|
||||
if got := p.Username(); got != "one.build-agent" {
|
||||
t.Fatalf("the build agent's credential is %q", got)
|
||||
}
|
||||
perms, err := PermissionsFor(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, s := range []string{
|
||||
"$JS.API.STREAM.INFO.KV_mesh-delivery_pulls",
|
||||
"$JS.API.DIRECT.GET.KV_mesh-delivery_pulls.>",
|
||||
"$JS.API.CONSUMER.CREATE.KV_mesh-delivery_pulls.>",
|
||||
"$JS.API.CONSUMER.DELETE.KV_mesh-delivery_pulls.>",
|
||||
// And the two check workers it takes from.
|
||||
"$JS.API.CONSUMER.MSG.NEXT.CHECKS_READY.CHECKS_READY_worker",
|
||||
"$JS.API.CONSUMER.MSG.NEXT.CHECKS_OTHER.CHECKS_OTHER_worker",
|
||||
"$JS.ACK.CHECKS_OTHER.CHECKS_OTHER_worker.>",
|
||||
// And the bucket of the checks running, which it writes and reads (ADR 0406 §3).
|
||||
"$KV." + CheckRunningBucket + ".>",
|
||||
"$JS.API.DIRECT.GET.KV_" + CheckRunningBucket + ".>",
|
||||
} {
|
||||
has(t, perms.Publish, s)
|
||||
}
|
||||
hasNot(t, perms.Publish, "$KV."+PullsBucket+".>")
|
||||
hasNot(t, perms.Publish, "mesh.seat.node-build-agent.check.>")
|
||||
for _, s := range perms.Subscribe {
|
||||
if s == "$KV."+PullsBucket+".>" {
|
||||
t.Fatalf("the build agent subscribes the bucket's subjects directly: %v", perms.Subscribe)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Issue 1687 (ADR 0483 clause 2): a holder of the build seat reads the bucket of the plans building and writes none of it, and
|
||||
// reads the oldest waiting ask of each check queue by a direct get, which takes nothing from the queue.
|
||||
func TestTheBuildSeatsHoldersReadThePlansBuildingAndWriteNone(t *testing.T) {
|
||||
seat := Seat{Name: TheBuildSeat, Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
|
||||
p := Principal{Kind: KindModule, Node: "one", Module: "build-agent", Holds: []Seat{seat}, PasswordHash: "x"}
|
||||
perms, err := PermissionsFor(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stream := "KV_" + PlansBuildingBucket
|
||||
for _, s := range []string{
|
||||
"$JS.API.STREAM.INFO." + stream,
|
||||
"$JS.API.DIRECT.GET." + stream + ".>",
|
||||
"$JS.API.CONSUMER.CREATE." + stream + ".>",
|
||||
"$JS.API.DIRECT.GET.CHECKS_READY",
|
||||
"$JS.API.DIRECT.GET.CHECKS_OTHER",
|
||||
} {
|
||||
has(t, perms.Publish, s)
|
||||
}
|
||||
for _, s := range perms.Publish {
|
||||
if SubjectsOverlap(s, "$KV."+PlansBuildingBucket+".plan-1") {
|
||||
t.Errorf("a holder of the build seat may write %s through %q", PlansBuildingBucket, s)
|
||||
}
|
||||
}
|
||||
// And the controller alone writes it, as the writers table says.
|
||||
for _, row := range WritersTable {
|
||||
for _, s := range row.Subjects {
|
||||
if s == "$KV."+PlansBuildingBucket+".>" && row.Writes(p, "$KV."+PlansBuildingBucket+".plan-1") {
|
||||
t.Errorf("%q lets the build seat's holder write the plans building", row.State)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,39 +50,12 @@ var (
|
||||
// controller's watcher hearing the expiry ends the wait and names the machine silent. Written by the lease
|
||||
// holder alone.
|
||||
RolloutSilentBucket = BucketName(ControllerSeat, "rollout-silent")
|
||||
// CheckRunningBucket holds the pull request checks the builders are running now (novox/hq ADR 0406 §3, issue 974): per
|
||||
// pull request a key `<owner>.<repo>.<number>` (or `group.<name>`) holding the head and trunk commit being checked, the
|
||||
// builder and its build. A builder puts it when it starts a check and again on each heartbeat, and deletes it when the
|
||||
// check ends; a key not renewed within the bucket's age is a builder that died, and is gone. A builder that takes a
|
||||
// waiting request for a head and trunk commit a key names drops it. **Written by the build seat's holders, several by
|
||||
// design** — each the key of the check it runs.
|
||||
CheckRunningBucket = BucketName(ControllerSeat, ChecksRunning)
|
||||
// DeliveryMergesBucket holds the merges mesh-delivery announced it made, each with the ask whose Approve it merged
|
||||
// on (novox/hq ADR 0420 §7, issue 1025): per merge a key `<owner>/<repo>/<number>/<head sha>` holding the
|
||||
// announcement, kept DeliveryMergesKeptFor. What lets the build of a moved module register it from its new
|
||||
// repository: the controller reads the ask's warrant from the router and accepts only the operator's Approve of
|
||||
// those heads. Written by the controller alone, from the announcement, when it holds the lease.
|
||||
DeliveryMergesBucket = BucketName(ControllerSeat, "delivery-merges")
|
||||
// PlansBuildingBucket holds one key per delivery plan in its build phase (novox/hq ADR 0483 clause 2, issue 1687): the key
|
||||
// is `<plan id>` (`plan-<digits>`), its value the plan, the time its build phase began and the time of the last refresh.
|
||||
// The controller (lease holder) puts it when the plan's first build is asked, puts it again about every thirty seconds
|
||||
// while the plan builds, and deletes it when the phase ends (every build of the last tier built, or the plan fails, stops
|
||||
// or waits on the operator); the bucket's age, PlansBuildingKeptFor, is how long a key outlives a controller that
|
||||
// stopped. **Read by the build seat's holders and the checker, written by the controller alone**: a holder takes no new
|
||||
// check while any key is there, but for one that waited past ADR 0483's bound. A reader lists the keys and reads none of
|
||||
// their values to decide that (`link.BuildingPlans`).
|
||||
PlansBuildingBucket = BucketName(ControllerSeat, PlansBuilding)
|
||||
)
|
||||
|
||||
// PlansBuilding is the local name of the bucket of the plans building, as a reader names it in `reads`
|
||||
// (`mesh-controller.plans-building`).
|
||||
const PlansBuilding = "plans-building"
|
||||
|
||||
// PlansBuildingKeptFor is the age of a plan-building key: the controller puts it again at PlansBuildingRefresh, so a
|
||||
// controller that stopped leaves checks held at most this long (ADR 0483 clause 2).
|
||||
const (
|
||||
PlansBuildingKeptFor = 2 * time.Minute
|
||||
PlansBuildingRefresh = 30 * time.Second
|
||||
)
|
||||
|
||||
// DeliveryMergesKeptFor is how long a merge mesh-delivery announced is kept: a week, as the events are.
|
||||
@@ -158,15 +131,7 @@ func ChainBuckets() []string { return []string{BuildsBucket, RolloutsBucket, Ass
|
||||
// MirrorsWrittenByAnother are the controller's mirrors whose writer is not the controller, each with its reason: the rule
|
||||
// that a mirror is written by the controller alone (TestTheMirrorsListAndTheWritersTableAgree) has these exceptions and no
|
||||
// other.
|
||||
var MirrorsWrittenByAnother = map[string]string{
|
||||
ChecksRunning: "the checks running are said by the builders themselves: every machine holding the build seat puts the " +
|
||||
"key of the check it runs, at its start and at each heartbeat, and deletes it when the check ends, so the " +
|
||||
"controller asserts the bucket and writes none of its keys (ADR 0406 §3); it is readable through `reads` so " +
|
||||
"the watchman sees a check that stopped (ADR 0433 §6)",
|
||||
}
|
||||
|
||||
// ChecksRunning is the local name of the bucket of the checks running, as a reader names it in `reads`.
|
||||
const ChecksRunning = "checks-running"
|
||||
var MirrorsWrittenByAnother = map[string]string{}
|
||||
|
||||
// The mirrors' caps (design 54 §5): one value per key, no age, 64 KiB a value, 64 MiB a bucket.
|
||||
const (
|
||||
@@ -189,10 +154,6 @@ const RolloutSilentMarkerTTL = time.Minute
|
||||
// takes nothing live.
|
||||
const RolloutSilentKeptFor = time.Hour
|
||||
|
||||
// CheckRunningKeptFor is the age of a running check's key: a few heartbeats (ten seconds each) of a builder, so a builder that
|
||||
// died is forgotten within two minutes and the check it ran is taken again then.
|
||||
const CheckRunningKeptFor = 2 * time.Minute
|
||||
|
||||
// AskedKeptFor is how long an ask is kept after it was made: a month, as the router keeps its own.
|
||||
const AskedKeptFor = 30 * 24 * time.Hour
|
||||
|
||||
@@ -222,16 +183,16 @@ func IsControllerBucket(bucket string) bool {
|
||||
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket ||
|
||||
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket || bucket == PredefinedBucket ||
|
||||
bucket == DeclaredPackagesBucket ||
|
||||
bucket == MergeWindowBucket || bucket == RolloutSilentBucket || bucket == CheckRunningBucket ||
|
||||
bucket == MergeWindowBucket || bucket == RolloutSilentBucket ||
|
||||
bucket == DeliveryMergesBucket || bucket == BuildsBucket || bucket == RolloutsBucket ||
|
||||
bucket == PlansBuildingBucket || bucket == delivery.HaveBucket || bucket == StatesAnnouncedBucket
|
||||
bucket == delivery.HaveBucket || bucket == StatesAnnouncedBucket
|
||||
}
|
||||
|
||||
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
|
||||
func ControllerBuckets() []string {
|
||||
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
|
||||
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket, DeclaredPackagesBucket, MergeWindowBucket, RolloutSilentBucket, CheckRunningBucket,
|
||||
DeliveryMergesBucket, PlansBuildingBucket, BuildsBucket, RolloutsBucket, delivery.HaveBucket, StatesAnnouncedBucket}
|
||||
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket, DeclaredPackagesBucket, MergeWindowBucket, RolloutSilentBucket,
|
||||
DeliveryMergesBucket, BuildsBucket, RolloutsBucket, delivery.HaveBucket, StatesAnnouncedBucket}
|
||||
}
|
||||
|
||||
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
|
||||
@@ -359,37 +320,6 @@ func (j *JetStream) EnsureControllerBuckets() error {
|
||||
}); err != nil {
|
||||
return fmt.Errorf("asserting bucket %s: %w", RolloutSilentBucket, err)
|
||||
}
|
||||
// **The checks running: a key that ages** (ADR 0406 §3, issue 974). One message a key, each renewal a new one, so the
|
||||
// age of the bucket is how long a key outlives the builder that stopped renewing it.
|
||||
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
|
||||
Bucket: CheckRunningBucket,
|
||||
Description: "the pull request checks the builders are running now (novox/hq ADR 0406): `<pull request>` holds the " +
|
||||
"head and trunk commit, the builder and its build; put by the builder when it starts and at each heartbeat, " +
|
||||
"deleted when the check ends, gone within two minutes of a builder that died; written by the build seat's holders",
|
||||
History: 1,
|
||||
TTL: CheckRunningKeptFor,
|
||||
MaxValueSize: 4 << 10,
|
||||
MaxBytes: 8 << 20,
|
||||
Storage: jetstream.FileStorage,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("asserting bucket %s: %w", CheckRunningBucket, err)
|
||||
}
|
||||
// **The plans building: keys that age** (ADR 0483 clause 2, issue 1687). One message a key, each refresh a new one, so the
|
||||
// age of the bucket is how long a key outlives a controller that stopped refreshing it.
|
||||
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
|
||||
Bucket: PlansBuildingBucket,
|
||||
Description: "the delivery plans in their build phase (novox/hq ADR 0483): `<plan id>` holds the plan and when its " +
|
||||
"build phase began; put when the plan's first build is asked and about every thirty seconds, deleted when the " +
|
||||
"phase ends, gone within two minutes of a controller that stopped; read by the build seat's holders, which " +
|
||||
"take no new check while a key is there; written by the controller alone",
|
||||
History: 1,
|
||||
TTL: PlansBuildingKeptFor,
|
||||
MaxValueSize: 4 << 10,
|
||||
MaxBytes: 8 << 20,
|
||||
Storage: jetstream.FileStorage,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("asserting bucket %s: %w", PlansBuildingBucket, err)
|
||||
}
|
||||
// **The states each module announced** (novox/hq ADR 0495 clauses 2 and 3, issue 1790): one key a module, its last word.
|
||||
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
|
||||
Bucket: StatesAnnouncedBucket,
|
||||
|
||||
@@ -237,9 +237,6 @@ var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"}
|
||||
// original — and no other can (novox/hq issue 334, ADR 0264's consequences: no grant over the seats'
|
||||
// queues).
|
||||
func TheControllersAsk(stream, subject string) bool {
|
||||
if isCheckAsk(stream, subject) {
|
||||
return true
|
||||
}
|
||||
for _, seat := range seatsTheControllerAsks {
|
||||
if stream == seatStreamName(seat) && strings.HasPrefix(subject, "mesh.seat."+seat+".accept.") {
|
||||
return true
|
||||
@@ -424,22 +421,6 @@ var DegradableGrants = []DegradableGrant{
|
||||
Reason: "refused: the controller closes a batch's window by a timer of its own for that batch, as the " +
|
||||
"window setting says (ADR 0407 decision 1, issue 973)",
|
||||
},
|
||||
{
|
||||
Subject: "mesh.seat." + TheBuildSeat + ".check.>",
|
||||
Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at its " +
|
||||
"first push, and a check asked before that is asked again by the next announcement of its head (ADR 0406 §3, " +
|
||||
"issue 974)",
|
||||
},
|
||||
{
|
||||
Subject: "$KV." + CheckRunningBucket + ".>",
|
||||
Reason: "refused: the controller asserts this bucket through the API and writes none of its keys — the build seat's " +
|
||||
"holders do (ADR 0406 §3, issue 974)",
|
||||
},
|
||||
{
|
||||
Subject: "$KV." + PlansBuildingBucket + ".>",
|
||||
Reason: "refused: the controller then says no plan builds, and no check is held while a delivery builds — a check run " +
|
||||
"wrongly costs a delivery a place, a check held wrongly costs a pull request its turn (ADR 0483 clause 2, issue 1687)",
|
||||
},
|
||||
{
|
||||
Subject: "$KV." + DeclaredPackagesBucket + ".>",
|
||||
Reason: "carried only at genesis: the controller composes its own permissions and pushes them to the bus at its " +
|
||||
@@ -661,11 +642,6 @@ func PermissionsFor(p Principal) (Permissions, error) {
|
||||
// the role, and whichever machine holding it is idle takes it.
|
||||
for _, seat := range seatsTheControllerAsks {
|
||||
pub = append(pub, "mesh.seat."+seat+".accept.>")
|
||||
// **And its checks, on the two queues of their own** (novox/hq ADR 0406 §3, issue 974): one subject a pull
|
||||
// request on each, under `check`; the retired role's checks stay on its accepts.
|
||||
if seat == TheBuildSeat {
|
||||
pub = append(pub, "mesh.seat."+seat+".check.>")
|
||||
}
|
||||
// **And its holders' verbs, on every machine** (novox/hq ADR 0219): what a holder is
|
||||
// building, kill it, pause it, resume it. The queue is the controller's to show and to
|
||||
// change, and what one machine is doing with an ask it took only that machine can say.
|
||||
@@ -1013,10 +989,6 @@ func PermissionsFor(p Principal) (Permissions, error) {
|
||||
"$JS.API.CONSUMER.INFO."+stream+"."+worker,
|
||||
"$JS.API.CONSUMER.MSG.NEXT."+stream+"."+worker,
|
||||
"$JS.ACK."+stream+"."+worker+".>")
|
||||
// And the checks' two workers, for the build role they are queued for (novox/hq ADR 0406 §3).
|
||||
if s.Name == TheBuildSeat {
|
||||
pub = append(pub, holderCheckGrants()...)
|
||||
}
|
||||
// **And whether an ask it took was cancelled** (novox/hq ADR 0219): one key of the
|
||||
// seat's cancelled set, read directly by its id, so a holder that fetched an ask in the
|
||||
// moment the controller cancelled it ends it instead of building it. Read, never written:
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package broker
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R4g (issue 1299, ADR 0525 clause 6): the controller declares no work-queue stream for builds or checks and no
|
||||
// plans-building or checks-running bucket. The streams and buckets a bus already holds are left until a cleanup; none
|
||||
// is asserted.
|
||||
func TestTheControllerDeclaresNoBuildQueueAndNoBuildHold(t *testing.T) {
|
||||
for _, s := range MeshStreams() {
|
||||
if strings.HasPrefix(s.Name, "CHECKS_") || strings.HasPrefix(s.Name, "SEAT_") {
|
||||
t.Errorf("the mesh's own streams hold %s, a queue of builds or checks", s.Name)
|
||||
}
|
||||
if s.Retention == RetentionWorkQueue && s.Name != "CONTROL" && s.Name != DeadLetterNoticesStream {
|
||||
t.Errorf("%s is a work queue the controller declares", s.Name)
|
||||
}
|
||||
}
|
||||
for _, c := range MeshConsumers() {
|
||||
if strings.HasSuffix(c.Name, "_worker") {
|
||||
t.Errorf("%s is a worker of a build or check queue", c.Name)
|
||||
}
|
||||
}
|
||||
for _, name := range ControllerBuckets() {
|
||||
if strings.Contains(name, "plans-building") || strings.Contains(name, "checks-running") {
|
||||
t.Errorf("the controller's buckets hold %s", name)
|
||||
}
|
||||
}
|
||||
if slices.Contains(ControllerBuckets(), "") {
|
||||
t.Error("an unnamed bucket")
|
||||
}
|
||||
}
|
||||
@@ -1,134 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package broker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
// **A holder of the build seat reads the plans building and the server refuses its write** (novox/hq ADR 0483 clause 2, issue
|
||||
// 1687): on a real server whose users are composed as the controller composes them, the controller makes the bucket and puts a
|
||||
// plan's key; the build agent's credential lists the keys and gets one, and its put of a key is refused with a permissions
|
||||
// violation. It also reads the first waiting ask of a check queue by a direct get, which takes nothing from the queue.
|
||||
func TestNatsTheBuildSeatsHoldersReadThePlansBuildingAndTheServerRefusesTheirWrite(t *testing.T) { // 1687
|
||||
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seat := Seat{Name: TheBuildSeat, Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
|
||||
users, err := Users(Records{Nodes: []string{"alpha"}, Assigned: map[string][]Declared{
|
||||
"alpha": {{Module: RuntimeModule}, {Module: "build-agent", Holds: []Seat{seat}}},
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := range users {
|
||||
users[i].PasswordHash = string(hash)
|
||||
}
|
||||
accounts, err := ComposeAccounts(users)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conf := filepath.Join(t.TempDir(), "bus.conf")
|
||||
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
|
||||
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opts, err := server.ProcessConfigFile(conf)
|
||||
if err != nil {
|
||||
t.Fatalf("the composed accounts do not parse: %v", err)
|
||||
}
|
||||
opts.NoLog, opts.NoSigs = true, true
|
||||
s, err := server.NewServer(opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go s.Start()
|
||||
if !s.ReadyForConnections(10 * time.Second) {
|
||||
t.Fatal("the bus did not come up")
|
||||
}
|
||||
t.Cleanup(s.Shutdown)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
connect := func(user string) (*nats.Conn, chan error) {
|
||||
t.Helper()
|
||||
refused := make(chan error, 8)
|
||||
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
|
||||
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused <- err }))
|
||||
if err != nil {
|
||||
t.Fatalf("%s cannot connect: %v", user, err)
|
||||
}
|
||||
t.Cleanup(nc.Close)
|
||||
return nc, refused
|
||||
}
|
||||
controller, _ := connect(ControllerName)
|
||||
cj := OnConn(controller)
|
||||
if err := AssertMeshStreams(cj); err != nil {
|
||||
t.Fatalf("the controller cannot make the streams: %v", err)
|
||||
}
|
||||
if err := cj.EnsureControllerBuckets(); err != nil {
|
||||
t.Fatalf("the controller cannot make its buckets: %v", err)
|
||||
}
|
||||
cjs, _ := jetstream.New(controller)
|
||||
ckv, err := cjs.KeyValue(ctx, PlansBuildingBucket)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ckv.Put(ctx, "plan-1", []byte(`{"plan":"plan-1"}`)); err != nil {
|
||||
t.Fatalf("the controller cannot put a plan's key: %v", err)
|
||||
}
|
||||
|
||||
holder, refused := connect("alpha.build-agent")
|
||||
hjs, err := jetstream.New(holder)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
kv, err := hjs.KeyValue(ctx, PlansBuildingBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("a holder of the build seat cannot bind %s: %v", PlansBuildingBucket, err)
|
||||
}
|
||||
keys, err := kv.Keys(ctx)
|
||||
if err != nil || len(keys) != 1 || keys[0] != "plan-1" {
|
||||
t.Fatalf("the holder lists %v (%v), want plan-1", keys, err)
|
||||
}
|
||||
if e, err := kv.Get(ctx, "plan-1"); err != nil || e == nil {
|
||||
t.Fatalf("the holder cannot get the key: %v", err)
|
||||
}
|
||||
// Its write is refused by the server.
|
||||
if err := holder.Publish("$KV."+PlansBuildingBucket+".plan-2", []byte(`{}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = holder.Flush()
|
||||
select {
|
||||
case err := <-refused:
|
||||
if !errors.Is(err, nats.ErrPermissionViolation) {
|
||||
t.Errorf("the server said %v, want a permissions violation", err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Error("the server did not refuse the holder's put of a plan's key")
|
||||
}
|
||||
if _, err := ckv.Get(ctx, "plan-2"); !errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
t.Errorf("the refused put left a key: %v", err)
|
||||
}
|
||||
|
||||
// And the direct get of a check queue's first waiting ask, which is granted.
|
||||
reply, err := holder.RequestWithContext(ctx, "$JS.API.DIRECT.GET."+ChecksOtherStream, []byte(`{"seq":1,"next_by_subj":"`+
|
||||
CheckSubjectPrefix(TheBuildSeat, false)+`>"}`))
|
||||
if err != nil {
|
||||
t.Fatalf("the holder's direct get of %s was not answered: %v", ChecksOtherStream, err)
|
||||
}
|
||||
if reply.Header.Get("Status") != "404" {
|
||||
t.Errorf("an empty queue answered %q, want 404", reply.Header.Get("Status"))
|
||||
}
|
||||
}
|
||||
@@ -1,30 +0,0 @@
|
||||
package broker
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Issue 1687 (ADR 0483 clause 2): a module whose manifest reads mesh-controller.plans-building may name it in `reads`, and
|
||||
// reads the bucket and writes none; the controller writes it, so it is no mirror written by another.
|
||||
func TestAModuleReadingPlansBuildingReadsAndWritesNone(t *testing.T) {
|
||||
if !slices.Contains(controllerReadableStates(), "plans-building") || BucketName(ControllerSeat, PlansBuilding) != PlansBuildingBucket {
|
||||
t.Fatalf("a module cannot name %s in `reads`: %v", PlansBuildingBucket, controllerReadableStates())
|
||||
}
|
||||
if _, other := MirrorsWrittenByAnother[PlansBuilding]; other {
|
||||
t.Error("the controller writes the plans building itself")
|
||||
}
|
||||
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "mesh-delivery-checker", PasswordHash: "x",
|
||||
Reads: []string{"mesh-controller.plans-building"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stream := "KV_" + PlansBuildingBucket
|
||||
has(t, perms.Publish, "$JS.API.STREAM.INFO."+stream)
|
||||
has(t, perms.Publish, "$JS.API.DIRECT.GET."+stream+".>")
|
||||
for _, p := range perms.Publish {
|
||||
if SubjectsOverlap(p, "$KV."+PlansBuildingBucket+".plan-1") {
|
||||
t.Errorf("the reading module may write %s through %q", PlansBuildingBucket, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,7 +9,12 @@ import (
|
||||
|
||||
// The catalogue's list of the mesh's own stream names, which a module's stream may not take, is the broker's
|
||||
// own list (novox/hq ADR 0363 §1): read from MeshStreams, so a stream added there cannot be forgotten there.
|
||||
//
|
||||
// **The names of streams the mesh no longer asserts stay reserved** (R4g, ADR 0525 clause 6): the pull requests' two check
|
||||
// queues are left on the bus until a cleanup removes them, and no module may take their names meanwhile. So the catalogue
|
||||
// reserves every name of the broker's own, and may reserve these retired ones beside them.
|
||||
func TestTheCataloguesReservedStreamNamesAreTheMeshsOwn(t *testing.T) {
|
||||
retired := []string{"CHECKS_OTHER", "CHECKS_READY"}
|
||||
var own []string
|
||||
for _, s := range MeshStreams() {
|
||||
own = append(own, s.Name)
|
||||
@@ -17,7 +22,24 @@ func TestTheCataloguesReservedStreamNamesAreTheMeshsOwn(t *testing.T) {
|
||||
reserved := append([]string(nil), catalogue.MeshStreamNames...)
|
||||
slices.Sort(own)
|
||||
slices.Sort(reserved)
|
||||
if !slices.Equal(own, reserved) {
|
||||
t.Fatalf("the broker's own streams are %v, the catalogue reserves %v", own, reserved)
|
||||
var extra []string
|
||||
for _, name := range reserved {
|
||||
if !slices.Contains(own, name) {
|
||||
extra = append(extra, name)
|
||||
}
|
||||
}
|
||||
var missing []string
|
||||
for _, name := range own {
|
||||
if !slices.Contains(reserved, name) {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
}
|
||||
for _, name := range extra {
|
||||
if !slices.Contains(retired, name) {
|
||||
t.Errorf("the catalogue reserves %s, which the broker neither asserts nor names retired", name)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
t.Fatalf("the broker's own streams %v are not reserved by the catalogue (it reserves %v)", missing, reserved)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -154,7 +154,7 @@ func OriginalOfAgain(subject string) (string, bool) {
|
||||
// every tool invocation in the mesh, and a tool call must never be persisted (design 25 §3 keeps
|
||||
// tools on core NATS, where a lost call is a timeout the caller already handles).
|
||||
func MeshStreams() []Stream {
|
||||
return append([]Stream{
|
||||
return []Stream{
|
||||
{
|
||||
Name: "CONTROL",
|
||||
// A build's outcome is no longer here: it is the build-machine seat's own event, so one
|
||||
@@ -228,7 +228,7 @@ func MeshStreams() []Stream {
|
||||
"failed with the reason (novox/hq ADR 0349 §4): what a walk's end and a gate wait for (ADR 0362, " +
|
||||
"ADR 0349 §15), read by subject; a week, as the events are",
|
||||
},
|
||||
}, CheckStreams()...)
|
||||
}
|
||||
}
|
||||
|
||||
// ProvAppliedStream keeps what each machine's provisioner client last said of each module's files, newest per
|
||||
@@ -495,7 +495,7 @@ const EventsStream = "EVENTS"
|
||||
// held through a store restart — the exact message the stream exists to protect — some minutes
|
||||
// before the controller had finished deciding about it.
|
||||
func MeshConsumers() []Consumer {
|
||||
return append(ControllerConsumers(), CheckWorkers()...)
|
||||
return ControllerConsumers()
|
||||
}
|
||||
|
||||
// ControllerConsumers are the consumers the controller's own event loop takes from: MeshConsumers without the build
|
||||
|
||||
@@ -134,9 +134,6 @@ func TestEachStreamCarriesTheRetentionItsShapeNeeds(t *testing.T) {
|
||||
// What each machine's provisioner client last said of each module's files (design 57 P6): the newest
|
||||
// word per module, what a walk's end reads.
|
||||
"PROV_APPLIED": RetentionLastPerSubject,
|
||||
// The checks of pull requests, in two work queues of their own (novox/hq ADR 0406 §3, issue 974).
|
||||
"CHECKS_READY": RetentionWorkQueue,
|
||||
"CHECKS_OTHER": RetentionWorkQueue,
|
||||
}
|
||||
got := map[string]Retention{}
|
||||
for _, s := range MeshStreams() {
|
||||
|
||||
+1
-1
@@ -29,7 +29,7 @@ accounts {
|
||||
]
|
||||
users = [
|
||||
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
|
||||
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_declared-packages.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_plans-building.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$KV.mesh-controller_states-announced.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.mod.mesh-controller.event.states", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
|
||||
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_declared-packages.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$KV.mesh-controller_states-announced.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.mod.mesh-controller.event.states", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
|
||||
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
|
||||
allow_responses: { max: 1, ttl: "1m" }
|
||||
} }
|
||||
|
||||
@@ -43,14 +43,6 @@ func ownMachine(p Principal, pattern string) bool {
|
||||
return p.Kind == KindNode && len(tokens) > 2 && tokens[2] == p.Node
|
||||
}
|
||||
|
||||
// holdsTheBuildSeat is a principal holding the build seat, or the controller, which asserts the bucket the holders write.
|
||||
func holdsTheBuildSeat(p Principal, _ string) bool {
|
||||
if p.Kind == KindController {
|
||||
return true
|
||||
}
|
||||
return holdsSeatOf(p, "mesh.seat."+TheBuildSeat+".x")
|
||||
}
|
||||
|
||||
// holdsSeatOf is a principal holding the seat a `mesh.seat.<seat>.…` pattern names: its module's own
|
||||
// principal, or the node tools carrying a module that holds it (ADR 0175, the runtime is its modules).
|
||||
func holdsSeatOf(p Principal, pattern string) bool {
|
||||
@@ -184,19 +176,6 @@ var WritersTable = []WriterRow{
|
||||
Subjects: []string{"mesh.seat.node-build-agent.event.built", "mesh.seat.mesh-build-machine.event.built"},
|
||||
Writes: holdsSeatOf,
|
||||
Shared: "every machine holding the build seat answers the asks it took; each outcome names its ask"},
|
||||
{State: "a pull request's check asked", Writer: "controller",
|
||||
KeptIn: "the work queues " + ChecksReadyStream + " and " + ChecksOtherStream + ", one subject a pull request, the newest kept",
|
||||
Others: "the build seat's holders take it",
|
||||
Subjects: []string{CheckSubjectPrefix(TheBuildSeat, true) + ">", CheckSubjectPrefix(TheBuildSeat, false) + ">"},
|
||||
Writes: isController},
|
||||
{State: "a pull request's check running", Writer: "the build seat's holders — several by design, each the key of the check it runs",
|
||||
KeptIn: "key-value " + CheckRunningBucket + ", aged out when not renewed", Others: "the build seat's holders read it to drop a duplicate request",
|
||||
Subjects: kvOf(CheckRunningBucket), Writes: holdsTheBuildSeat,
|
||||
Shared: "every machine holding the build seat puts the key of the check it runs and deletes it when the check ends"},
|
||||
{State: "the plans-building mirror", Writer: "controller (lease holder), at the first build's ask and at each refresh",
|
||||
KeptIn: "key-value " + PlansBuildingBucket + ", aged out when not refreshed",
|
||||
Others: "the build seat's holders read it and take no new check while a key is there (ADR 0483)",
|
||||
Subjects: kvOf(PlansBuildingBucket), Writes: isController},
|
||||
// What a module keeps and who may read it (novox/hq ADR 0495 clauses 2 and 3, issue 1790): said by the module, kept by
|
||||
// the controller.
|
||||
{State: "a module's states announced", Writer: "the module itself, from its code, under its own name",
|
||||
|
||||
@@ -27,10 +27,6 @@ var designRows = []string{
|
||||
"the healers' acts and their brake",
|
||||
"stream definitions and bus permissions",
|
||||
"builds and their outcomes",
|
||||
"a pull request's check asked",
|
||||
"a pull request's check running",
|
||||
// The plans in their build phase, a bucket the build seat's holders read (ADR 0483 clause 2, issue 1687).
|
||||
"the plans-building mirror",
|
||||
// What each module keeps and who may read it, said by the module and kept by the controller (ADR 0495, issue 1790).
|
||||
"a module's states announced",
|
||||
"the states announced",
|
||||
|
||||
@@ -1,149 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A link as a parent of an archive's `from` must not lead the build out of the module: the repository
|
||||
// is somebody else's, and a build that followed it would pack, register and serve any directory the
|
||||
// build account can read (issue 1609).
|
||||
func TestAnArchiveWhoseFromGoesThroughALinkOutOfTheModuleIsRefused(t *testing.T) {
|
||||
outside := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(outside, "x"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(outside, "x", "secret"), []byte("s"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tree := t.TempDir()
|
||||
if err := os.Symlink(outside, filepath.Join(tree, "link")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := &recorded{}
|
||||
a := catalogue.Artifact{Name: "all", Kind: catalogue.ArtifactArchive, From: "link/x"}
|
||||
_, err := one(context.Background(), nil, r, "m", tree, t.TempDir(), "c", "", a, nil, nil, "", Npmrc{}, nil,
|
||||
nil, func(string, string, ...any) {})
|
||||
if err == nil {
|
||||
t.Fatal("an archive was packed through a link out of the module")
|
||||
}
|
||||
realOutside, _ := filepath.EvalSymlinks(filepath.Join(outside, "x"))
|
||||
realTree, _ := filepath.EvalSymlinks(tree)
|
||||
for _, want := range []string{realOutside, realTree, "link/x"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("the refusal %q does not name %q", err, want)
|
||||
}
|
||||
}
|
||||
if len(r.archives) != 0 {
|
||||
t.Errorf("a refused archive left %d published", len(r.archives))
|
||||
}
|
||||
}
|
||||
|
||||
// A link that stays inside the module is still followed, and its archive keeps its digest.
|
||||
func TestAnArchiveWhoseFromGoesThroughALinkInsideTheModuleIsPacked(t *testing.T) {
|
||||
tree := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(tree, "real", "x"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(tree, "real", "x", "f"), []byte("f"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink("real", filepath.Join(tree, "link")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := &recorded{}
|
||||
a := catalogue.Artifact{Name: "all", Kind: catalogue.ArtifactArchive, From: "link/x"}
|
||||
if _, err := one(context.Background(), nil, r, "m", tree, t.TempDir(), "c", "", a, nil, nil, "", Npmrc{}, nil,
|
||||
nil, func(string, string, ...any) {}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// A module directory that is itself a link out of the fetched tree is refused, naming both places.
|
||||
func TestAModuleDirectoryThatIsALinkOutOfTheRepositoryIsRefused(t *testing.T) {
|
||||
outside := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(outside, ManifestName), []byte("{}"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tree := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(tree, "modules"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(outside, filepath.Join(tree, "modules", "evil")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := inside(tree, "modules/evil")
|
||||
if err == nil {
|
||||
t.Fatal("a module directory behind a link out of the repository was accepted")
|
||||
}
|
||||
realOutside, _ := filepath.EvalSymlinks(outside)
|
||||
realTree, _ := filepath.EvalSymlinks(tree)
|
||||
for _, want := range []string{realOutside, realTree, "modules/evil"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("the refusal %q does not name %q", err, want)
|
||||
}
|
||||
}
|
||||
// A link that stays inside the repository, and a module directory not there, are left as they were.
|
||||
if err := os.MkdirAll(filepath.Join(tree, "real"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink("real", filepath.Join(tree, "modules", "fine")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := inside(tree, "modules/fine"); err != nil {
|
||||
t.Errorf("a link inside the repository was refused: %v", err)
|
||||
}
|
||||
if _, err := inside(tree, "modules/none"); err != nil {
|
||||
t.Errorf("an absent module directory was refused here: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A repository that commits `.mesh-build` as a link must not make a bundle's build write into, or pack,
|
||||
// a directory outside the module (issue 1609): the compile, the bundling and the pack each refuse it.
|
||||
func TestABundleWhoseOutputDirectoryIsALinkOutOfTheModuleIsRefused(t *testing.T) {
|
||||
outside := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(outside, "secret"), []byte("s"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tree := t.TempDir()
|
||||
if err := os.Symlink(outside, filepath.Join(tree, ".mesh-build")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a := catalogue.Artifact{Name: "b", Kind: catalogue.ArtifactBundle, Language: "go", From: "cmd/b"}
|
||||
ran := false
|
||||
run := func(context.Context, string, string, ...string) (string, error) { ran = true; return "", nil }
|
||||
realOutside, _ := filepath.EvalSymlinks(outside)
|
||||
realTree, _ := filepath.EvalSymlinks(tree)
|
||||
|
||||
check := func(what string, err error) {
|
||||
t.Helper()
|
||||
if err == nil {
|
||||
t.Fatalf("%s went through a link out of the module", what)
|
||||
}
|
||||
for _, want := range []string{realOutside, realTree} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%s: the refusal %q does not name %q", what, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
_, err = compile(context.Background(), run, tree, chain, "base", a)
|
||||
check("compile", err)
|
||||
_, err = bundled(context.Background(), run, tree, chain, "base", a, nil)
|
||||
check("bundled", err)
|
||||
check("the pack", outsideTheModule(tree, filepath.Join(tree, Out(a.Name))))
|
||||
if ran {
|
||||
t.Error("a command ran for a refused bundle")
|
||||
}
|
||||
if entries, _ := os.ReadDir(outside); len(entries) != 1 {
|
||||
t.Errorf("a refused bundle wrote into the directory outside: %d entries", len(entries))
|
||||
}
|
||||
}
|
||||
@@ -1,44 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// The name a machine runs a binary by is not always the name of the package that built it. The host's
|
||||
// command is cmd/mesh-host and every machine runs it as nox-mesh-host — the path it is installed at,
|
||||
// the name in its unit, and the name its launcher looks for inside a delivered version.
|
||||
//
|
||||
// A bundle carrying the package's name was delivered to a machine correctly, reported "created … 1
|
||||
// file(s)", and was invisible to the launcher (novox/hq 04-ISSUES/142). Found by reading the delivered
|
||||
// directory rather than by trusting the line that said it worked.
|
||||
|
||||
func TestACompiledArtifactNamesTheBinaryAMachineWillRun(t *testing.T) {
|
||||
got := binaryName(catalogue.Artifact{
|
||||
Name: "host-arch", From: "cmd/mesh-host", Binary: "nox-mesh-host",
|
||||
})
|
||||
if got != "nox-mesh-host" {
|
||||
t.Fatalf("the binary is named %q, and the launcher looks for nox-mesh-host", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSayingNothingKeepsWhatTheCompilerWouldHaveChosen(t *testing.T) {
|
||||
// go build names its output after the package, so an artifact that says nothing gets the same
|
||||
// thing it got before this existed.
|
||||
if got := binaryName(catalogue.Artifact{Name: "host-arch", From: "cmd/mesh-host"}); got != "mesh-host" {
|
||||
t.Fatalf("an artifact naming no binary produced %q", got)
|
||||
}
|
||||
if got := binaryName(catalogue.Artifact{Name: "host-arch", From: "./cmd/agent/"}); got != "agent" {
|
||||
t.Fatalf("a from with slashes produced %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABundleBuiltFromTheModuleRootFallsBackToItsArtifactName(t *testing.T) {
|
||||
// A single-command repository names no package, and `go build -o <dir>` would then write a file
|
||||
// named after the module directory — which is not something the manifest states. The artifact's
|
||||
// own name is what the manifest does state.
|
||||
if got := binaryName(catalogue.Artifact{Name: "tool"}); got != "tool" {
|
||||
t.Fatalf("a bundle built from the root produced %q", got)
|
||||
}
|
||||
}
|
||||
@@ -1,227 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/novox/mesh-controller/internal/gosource"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
)
|
||||
|
||||
// What a build says it was made from, as files (novox/hq ADR 0267), and what a build compiling a Go
|
||||
// program is handed.
|
||||
|
||||
// onTrunk answers the trunk's questions as a clone of a commit on main would, or off it.
|
||||
type onTrunk struct {
|
||||
*recorded
|
||||
off bool
|
||||
// behind is a commit on the trunk that is not its head: an older commit built by hand.
|
||||
behind bool
|
||||
}
|
||||
|
||||
func (o onTrunk) run(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "git" && len(args) > 0 && args[0] == "symbolic-ref" {
|
||||
return "origin/main\n", nil
|
||||
}
|
||||
if name == "git" && len(args) > 1 && args[0] == "rev-parse" && args[1] == "origin/main" && o.behind {
|
||||
return "feedfacefeedfacefeedfacefeedfacefeedface\n", nil
|
||||
}
|
||||
if name == "git" && len(args) > 0 && args[0] == "merge-base" && o.off {
|
||||
return "", os.ErrNotExist
|
||||
}
|
||||
return compiling{o.recorded}.run(ctx, dir, name, args...)
|
||||
}
|
||||
|
||||
// aSharedRepository is a repository holding two programs that share a package, as the controller's does.
|
||||
func aSharedRepository(readme, shared string) map[string]string {
|
||||
return map[string]string{
|
||||
"go.mod": "module example.com/ctl\n\ngo 1.22\n",
|
||||
"go.sum": "",
|
||||
"README.md": readme,
|
||||
"cmd/ctl/main.go": "package main\n\nimport _ \"example.com/ctl/internal/shared\"\n\nfunc main() {}\n",
|
||||
"proxy/main.go": "package main\n\nimport _ \"example.com/ctl/internal/shared\"\n\nfunc main() {}\n",
|
||||
"internal/shared/s.go": "package shared\n\nconst S = " + shared + "\n",
|
||||
"internal/only/o.go": "package only\n",
|
||||
}
|
||||
}
|
||||
|
||||
const aProxy = `{"module":"route-proxy","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile","compiles":"proxy",
|
||||
"context":{"repository":"https://forge.invalid/ctl.git","ref":"main"}},
|
||||
{"name":"trust","kind":"upstream","from":"alpine@sha256:` + "3333333333333333333333333333333333333333333333333333333333333333" + `"}]}}`
|
||||
|
||||
func buildTheProxy(t *testing.T, context_ map[string]string, off bool, behind ...bool) (Result, *recorded, string) {
|
||||
t.Helper()
|
||||
r := &recorded{
|
||||
contents: map[string]string{"modules/route-proxy/" + ManifestName: aProxy, "modules/route-proxy/Dockerfile": "FROM scratch\nCOPY . .\n", "modules/route-proxy/README.md": "x"},
|
||||
secondary: map[string]map[string]string{"https://forge.invalid/ctl.git": context_},
|
||||
}
|
||||
workspace := t.TempDir()
|
||||
got, err := Build(context.Background(), onTrunk{r, off, len(behind) > 0 && behind[0]}.run, r,
|
||||
"https://forge.invalid/catalogue.git", "modules/route-proxy", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return got, r, workspace
|
||||
}
|
||||
|
||||
func TestAnImageCompilingGoIsHandedItsBuildSourceAndSaysIt(t *testing.T) {
|
||||
got, r, workspace := buildTheProxy(t, aSharedRepository("one", "1"), false)
|
||||
|
||||
// Built in the narrowed tree, which holds the program's closure and nothing else.
|
||||
at := ""
|
||||
for i, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build ") {
|
||||
at = r.dirs[i]
|
||||
}
|
||||
}
|
||||
if filepath.Base(at) != "narrow-server" {
|
||||
t.Fatalf("docker build ran in %q, not the narrowed build source", at)
|
||||
}
|
||||
for file, want := range map[string]bool{"proxy/main.go": true, "internal/shared/s.go": true, "go.mod": true,
|
||||
"cmd/ctl/main.go": false, "internal/only/o.go": false, "README.md": false} {
|
||||
held := slices.Contains(r.builtFrom, file)
|
||||
if held != want {
|
||||
t.Errorf("%s handed to the recipe: %v, wanted %v", file, held, want)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(workspace, "narrow-server")); err == nil {
|
||||
t.Error("the narrowed build source was left in the workspace")
|
||||
}
|
||||
|
||||
// Said per repository: its own, the manifest and the recipe; the context's, the closure.
|
||||
if len(got.Sources) != 2 {
|
||||
t.Fatalf("sources %+v", got.Sources)
|
||||
}
|
||||
own, ctx := got.Sources[0], got.Sources[1]
|
||||
if own.Repository != "" || !slices.Equal(own.Paths, []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}) {
|
||||
t.Errorf("its own build source: %+v", own)
|
||||
}
|
||||
if ctx.Repository != "https://forge.invalid/ctl.git" || ctx.Ref != "main" {
|
||||
t.Errorf("the context's build source names %q at %q", ctx.Repository, ctx.Ref)
|
||||
}
|
||||
for file, want := range map[string]bool{"proxy/main.go": true, "internal/shared/s.go": true, "go.mod": true,
|
||||
"cmd/ctl/main.go": false, "README.md": false} {
|
||||
if gosource.SourceHolds(ctx.Paths, file) != want {
|
||||
t.Errorf("the context's build source holds %s: %v, wanted %v (%v)", file, !want, want, ctx.Paths)
|
||||
}
|
||||
}
|
||||
|
||||
// **The fingerprint is over the build source** (rule 5): a change outside it is one build, inside it another.
|
||||
readme, _, _ := buildTheProxy(t, aSharedRepository("two", "1"), false)
|
||||
if readme.Source != got.Source {
|
||||
t.Errorf("a README of the context changed the fingerprint: %s %s", got.Source, readme.Source)
|
||||
}
|
||||
shared, _, _ := buildTheProxy(t, aSharedRepository("one", "2"), false)
|
||||
if shared.Source == got.Source {
|
||||
t.Error("a change to the program's closure kept its fingerprint")
|
||||
}
|
||||
}
|
||||
|
||||
// A build off the trunk, or of a trunk commit that is not its head, says no build source: the planner maps
|
||||
// a merge onto the trunk head's, and an older commit's closure lacks what was imported since.
|
||||
func TestABuildOffTheTrunksHeadSaysNoBuildSource(t *testing.T) {
|
||||
got, _, _ := buildTheProxy(t, aSharedRepository("one", "1"), true)
|
||||
if len(got.Sources) != 0 {
|
||||
t.Fatalf("a build off the trunk said %+v", got.Sources)
|
||||
}
|
||||
got, _, _ = buildTheProxy(t, aSharedRepository("one", "1"), false, true)
|
||||
if len(got.Sources) != 0 {
|
||||
t.Fatalf("a build of an older trunk commit said %+v", got.Sources)
|
||||
}
|
||||
}
|
||||
|
||||
// An archive of the module's whole directory holds every file of it.
|
||||
func TestAnArchiveOfTheWholeDirectoryHoldsIt(t *testing.T) {
|
||||
manifest := `{"module":"look","version":"1","build":{"artifacts":[{"name":"all","kind":"archive","from":"."}]},
|
||||
"resources":[{"id":"files","type":"archive","path":"/opt/look","artifact":"all"}]}`
|
||||
r := &recorded{contents: map[string]string{"modules/look/" + ManifestName: manifest, "modules/look/a/b.css": "x"}}
|
||||
got, err := Build(context.Background(), onTrunk{recorded: r}.run, r,
|
||||
"https://forge.invalid/catalogue.git", "modules/look", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Sources) != 1 || !gosource.SourceHolds(got.Sources[0].Paths, "modules/look/a/b.css") ||
|
||||
gosource.SourceHolds(got.Sources[0].Paths, "modules/other/x") {
|
||||
t.Fatalf("sources %+v", got.Sources)
|
||||
}
|
||||
}
|
||||
|
||||
// A recipe reading past its build source fails, naming what it could not find — in the real docker build;
|
||||
// here, the file is simply not in the tree it is handed, which is what makes that so.
|
||||
func TestAnImageCompilingANonexistentPackageFails(t *testing.T) {
|
||||
r := &recorded{
|
||||
contents: map[string]string{ManifestName: strings.Replace(aProxy, `"compiles":"proxy"`, `"compiles":"nowhere"`, 1),
|
||||
"Dockerfile": "FROM scratch\n"},
|
||||
secondary: map[string]map[string]string{"https://forge.invalid/ctl.git": aSharedRepository("one", "1")},
|
||||
}
|
||||
_, err := Build(context.Background(), onTrunk{recorded: r}.run, r,
|
||||
"https://forge.invalid/catalogue.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "nowhere") {
|
||||
t.Fatalf("a package that is not there built: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A Go bundle of a module built from its repository's root says its import closure, and the manifest;
|
||||
// an image that compiles nothing it was told of leaves the module's source whole, and says none.
|
||||
func TestAGoBundleSaysItsClosureAndAnUntoldImageNothing(t *testing.T) {
|
||||
files := aSharedRepository("one", "1")
|
||||
manifest := `{"module":"ctl","version":"1","build":{"artifacts":[
|
||||
{"name":"controller","kind":"bundle","language":"go","system":"arch","from":"cmd/ctl","binary":"ctl"}]},
|
||||
"resources":[{"id":"controller","type":"process","name":"ctl","artifact":"controller","run":["./ctl"]}]}`
|
||||
r := &recorded{contents: map[string]string{ManifestName: manifest}}
|
||||
for k, v := range files {
|
||||
r.contents[k] = v
|
||||
}
|
||||
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
|
||||
got, err := Build(context.Background(), onTrunk{recorded: r}.run, r,
|
||||
"https://forge.invalid/ctl.git", "", "", t.TempDir(), held, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Sources) != 1 || got.Sources[0].Repository != "" {
|
||||
t.Fatalf("sources %+v", got.Sources)
|
||||
}
|
||||
for file, want := range map[string]bool{"cmd/ctl/main.go": true, "internal/shared/s.go": true, ManifestName: true,
|
||||
"go.sum": true, "proxy/main.go": false, "README.md": false, "internal/only/o.go": false} {
|
||||
if gosource.SourceHolds(got.Sources[0].Paths, file) != want {
|
||||
t.Errorf("%s: held %v, wanted %v (%v)", file, !want, want, got.Sources[0].Paths)
|
||||
}
|
||||
}
|
||||
|
||||
untold := `{"module":"ctl","version":"1","build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile"}]}}`
|
||||
r = &recorded{contents: map[string]string{ManifestName: untold, "Dockerfile": "FROM scratch\n"}}
|
||||
got, err = Build(context.Background(), onTrunk{recorded: r}.run, r,
|
||||
"https://forge.invalid/ctl.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Sources) != 0 {
|
||||
t.Fatalf("an image compiling nothing it was told of said a build source: %+v", got.Sources)
|
||||
}
|
||||
}
|
||||
|
||||
// 1215/AC1 (compiled here, run by the merge check; the unit level has 1215/AC1 over the same inputs): a build's
|
||||
// result carries every input of its fingerprint, and the SDK function over them gives it again.
|
||||
func TestABuildSaysTheInputsItsFingerprintWasComputedFrom(t *testing.T) {
|
||||
got, _, _ := buildTheProxy(t, aSharedRepository("one", "1"), false)
|
||||
if got.SourceInputs == nil {
|
||||
t.Fatal("the result carries no source inputs")
|
||||
}
|
||||
in := *got.SourceInputs
|
||||
if in.Module != "route-proxy" || in.Tree == "" || in.Contexts["server"] == "" {
|
||||
t.Errorf("the inputs miss the module, its tree or the server's context: %+v", in)
|
||||
}
|
||||
fp, none := delivery.SourceFingerprint(in)
|
||||
if fp == "" || none != "" || fp != got.Source {
|
||||
t.Errorf("the inputs give %q (%q); the build recorded %q", fp, none, got.Source)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,362 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
func TestABuildProducesAManifestThePinsAreIn(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
|
||||
})
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Commit != "c0ffeec0ffeec0ffeec0ffeec0ffeec0ffeec0ff" {
|
||||
t.Fatalf("the commit was not recorded: %q", got.Commit)
|
||||
}
|
||||
if got.Manifest.Resources[0]["image"] == nil {
|
||||
t.Fatalf("the image was not pinned: %v", got.Manifest.Resources[0])
|
||||
}
|
||||
digest, _ := got.Manifest.Resources[1]["digest"].(string)
|
||||
if !strings.HasPrefix(digest, "sha256:") {
|
||||
t.Fatalf("the archive was not pinned: %v", got.Manifest.Resources[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestTwoBuildsOfOneCommitProduceOneDigest(t *testing.T) {
|
||||
// Or nothing downstream can tell "this changed" from "this was built again", and every
|
||||
// rebuild looks like a change to every machine holding it.
|
||||
var digests []string
|
||||
for i := 0; i < 2; i++ {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/a.conf": "one", "files/b.conf": "two",
|
||||
})
|
||||
// A year apart, so a packer carrying timestamps cannot accidentally agree.
|
||||
r.stamped = time.Date(2020+i, time.March, 3, 4, 5, 6, 0, time.UTC)
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, b := range got.Built {
|
||||
if b.Kind == catalogue.ArtifactArchive {
|
||||
digests = append(digests, b.Digest)
|
||||
}
|
||||
}
|
||||
}
|
||||
if digests[0] != digests[1] {
|
||||
t.Fatalf("two builds of one commit produced %s and %s", digests[0], digests[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestNothingIsPublishedUntilEverythingIsBuilt(t *testing.T) {
|
||||
// Half a module in the store under a digest the mesh never records is reachable,
|
||||
// unreferenced, and indistinguishable from something in use.
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch"})
|
||||
// `files` is missing, so packing the archive fails — after the image would have been pushed.
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a build with a missing input succeeded")
|
||||
}
|
||||
if len(r.archives) != 0 {
|
||||
t.Fatalf("an archive was published by a failed build: %v", r.archives)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryWithNoManifestSaysSo(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
r := &recorded{contents: map[string]string{"README.md": "nothing to see"}}
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a repository with nothing saying what it is was built")
|
||||
}
|
||||
if !strings.Contains(err.Error(), ManifestName) {
|
||||
t.Fatalf("the failure does not name what is missing: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModuleThatBuildsNothingStillProducesAManifest(t *testing.T) {
|
||||
// Most of what a person installs is configuration.
|
||||
r, workspace := aRepository(t, `{"module":"shell","version":"1","resources":[
|
||||
{"id":"rc","type":"file","path":"/etc/zsh/zshrc","content":"setopt"}]}`, nil)
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/shell.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Built) != 0 {
|
||||
t.Fatalf("something was built: %v", got.Built)
|
||||
}
|
||||
if got.Manifest.Module != "shell" || len(got.Manifest.Resources) != 1 {
|
||||
t.Fatalf("got %+v", got.Manifest)
|
||||
}
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker") {
|
||||
t.Fatalf("docker was run for a module that builds nothing: %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheTreeIsFreshEveryTime(t *testing.T) {
|
||||
// A build that reuses a working tree can succeed because of something a previous build left
|
||||
// behind, and that is a build nobody can reproduce.
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/a": "b",
|
||||
})
|
||||
leftover := filepath.Join(workspace, "source", "files", "from-last-time")
|
||||
if err := os.MkdirAll(filepath.Dir(leftover), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(leftover, []byte("stale"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(leftover); err == nil {
|
||||
t.Fatal("a previous build's file survived into this one")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatCannotPushFails(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/a": "b",
|
||||
})
|
||||
r.failPush = true
|
||||
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err == nil {
|
||||
t.Fatal("a build that could publish nothing reported success")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpstreamImageIsMirroredRatherThanBuilt(t *testing.T) {
|
||||
// A module usually runs software it did not write. Naming the upstream reference directly
|
||||
// would need every machine to reach a public registry, and would pin to a tag somebody else
|
||||
// can move.
|
||||
const mirrors = `{"module":"postgres","version":"1",
|
||||
"build":{"artifacts":[{"name":"store","kind":"upstream","from":"postgres:17-alpine"}]},
|
||||
"resources":[{"id":"db","type":"container","name":"mesh-postgres","artifact":"store"}]}`
|
||||
|
||||
r, workspace := aRepository(t, mirrors, nil)
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/postgres.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Pulled, not built.
|
||||
var pulled, built bool
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker pull postgres:17-alpine") {
|
||||
pulled = true
|
||||
}
|
||||
if strings.HasPrefix(line, "docker build") {
|
||||
built = true
|
||||
}
|
||||
}
|
||||
if !pulled {
|
||||
t.Fatalf("the upstream image was not fetched: %v", r.ran)
|
||||
}
|
||||
if built {
|
||||
t.Fatalf("something was built for an image that is mirrored: %v", r.ran)
|
||||
}
|
||||
// And the resource names what this registry serves, pinned by the digest it assigned.
|
||||
image, _ := got.Manifest.Resources[0]["image"].(string)
|
||||
if !strings.Contains(image, "@sha256:") {
|
||||
t.Fatalf("the mirrored image is not pinned by digest: %q", image)
|
||||
}
|
||||
if strings.Contains(image, "17-alpine") {
|
||||
t.Fatalf("the resource still names the upstream tag: %q", image)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModuleIsBuiltFromItsPathWithinTheRepository(t *testing.T) {
|
||||
r := &recorded{contents: map[string]string{
|
||||
"README.md": "this repository holds several modules",
|
||||
"modules/shell/" + ManifestName: withBoth,
|
||||
"modules/shell/Dockerfile": "FROM scratch",
|
||||
"modules/shell/files/theme.conf": "dark",
|
||||
// A second module beside it, so what is built is chosen by the path rather than by
|
||||
// happening to be the only manifest in the clone.
|
||||
"modules/other/" + ManifestName: `{"module":"other","version":"1"}`,
|
||||
}}
|
||||
got, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/catalogue.git", "modules/shell", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Manifest.Module != "meshboard" {
|
||||
t.Fatalf("built %q, which is not the module at the path asked for", got.Manifest.Module)
|
||||
}
|
||||
if got.Manifest.Resources[0]["image"] == nil {
|
||||
t.Fatalf("the image was not pinned: %v", got.Manifest.Resources[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPathThatLeavesTheRepositoryIsRefused(t *testing.T) {
|
||||
for _, escaping := range []string{"../../etc", "/etc"} {
|
||||
r := &recorded{contents: map[string]string{ManifestName: withBoth}}
|
||||
_, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/x.git", escaping, "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatalf("%q was accepted as a module's path", escaping)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "leaves the repository") &&
|
||||
!strings.Contains(err.Error(), "absolute path") {
|
||||
t.Fatalf("the refusal of %q does not say why: %v", escaping, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArtifactWithItsOwnContextIsBuiltFromThere(t *testing.T) {
|
||||
const withContext = `{"module":"route-proxy","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile",
|
||||
"context":{"repository":"https://forge.invalid/source.git","ref":"main"}}]}}`
|
||||
r := &recorded{
|
||||
contents: map[string]string{
|
||||
ManifestName: withContext,
|
||||
// The recipe lives with the packaging, not the source — read from here regardless of
|
||||
// where the build context comes from. FROM scratch declares no base, so what is under
|
||||
// test — where the context comes from — is not entangled with ADR 0097's own checks.
|
||||
"Dockerfile": "FROM scratch\nCOPY go.mod ./\n",
|
||||
},
|
||||
secondary: map[string]map[string]string{
|
||||
// go.mod exists only in the second repository. A build context taken from the wrong
|
||||
// place would never find it, which a real docker build would refuse on — the fake
|
||||
// does not read files, so what is checked below is that the build was even pointed
|
||||
// at the right place, not that COPY would have succeeded.
|
||||
"https://forge.invalid/source.git": {"go.mod": "module route-proxy\n"},
|
||||
},
|
||||
}
|
||||
_, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/catalogue.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var clonedSource bool
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "git clone") && strings.Contains(line, "https://forge.invalid/source.git") {
|
||||
clonedSource = true
|
||||
}
|
||||
}
|
||||
if !clonedSource {
|
||||
t.Fatalf("the artifact's own context was never cloned: %v", r.ran)
|
||||
}
|
||||
|
||||
buildIndex := -1
|
||||
for i, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build ") {
|
||||
buildIndex = i
|
||||
}
|
||||
}
|
||||
if buildIndex == -1 {
|
||||
t.Fatal("no docker build was run")
|
||||
}
|
||||
build := r.ran[buildIndex]
|
||||
buildDir := r.dirs[buildIndex]
|
||||
|
||||
if !strings.Contains(buildDir, "context-server") {
|
||||
t.Errorf("docker build ran from %q, not the artifact's own cloned context", buildDir)
|
||||
}
|
||||
recipe := strings.SplitN(strings.SplitN(build, "-f ", 2)[1], " ", 2)[0]
|
||||
if !filepath.IsAbs(recipe) {
|
||||
t.Errorf("the recipe %q is not an absolute path, so it is read relative to whatever "+
|
||||
"directory the build context moved to rather than where it actually is", recipe)
|
||||
}
|
||||
if !strings.HasSuffix(recipe, string(filepath.Separator)+"Dockerfile") {
|
||||
t.Errorf("the recipe is not the module's own Dockerfile: %q", recipe)
|
||||
}
|
||||
if !strings.HasSuffix(build, " .") {
|
||||
t.Errorf("the build was not given a context: %s", build)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildOffersTheForgesCredentialThroughGitsOwnStore(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
|
||||
})
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "",
|
||||
workspace, nil, Npmrc{},
|
||||
GitCredential{URL: "http://mesh_novox_builder:sw0rdfi5h@forge.invalid:20000"}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
clone := r.ran[0]
|
||||
stored := storeNamedIn(t, clone)
|
||||
for _, line := range r.ran {
|
||||
if strings.Contains(line, "sw0rdfi5h") {
|
||||
t.Fatalf("the secret is in a command line, readable by anything that can list processes: %s", line)
|
||||
}
|
||||
}
|
||||
// Outside the workspace a check's container mounts, and gone once the build is (novox/hq issue 462); what
|
||||
// it held while git read it is checked with the check's clones (TestACheckContainerSeesNoForgeCredential).
|
||||
if withinDir(workspace, stored) {
|
||||
t.Fatalf("the credential store %s is inside the workspace %s", stored, workspace)
|
||||
}
|
||||
if _, err := os.Stat(stored); !os.IsNotExist(err) {
|
||||
t.Fatalf("the credential store outlives the build: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(workspace, "git-credentials")); !os.IsNotExist(err) {
|
||||
t.Fatal("a credential file is left in the workspace")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildWithNoCredentialClonesExactlyAsBefore(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
|
||||
})
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "",
|
||||
workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(r.ran[0], "git clone --quiet ") {
|
||||
t.Fatalf("a credential-less clone grew flags: %s", r.ran[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(workspace, "git-credentials")); !os.IsNotExist(err) {
|
||||
t.Fatal("a credential file was written with no credential to put in it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAContextCloneCarriesTheSameCredentialStore(t *testing.T) {
|
||||
const withContext = `{"module":"route-proxy","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile",
|
||||
"context":{"repository":"https://forge.invalid/source.git","ref":"main"}}]}}`
|
||||
r := &recorded{
|
||||
contents: map[string]string{
|
||||
ManifestName: withContext,
|
||||
"Dockerfile": "FROM scratch\nCOPY go.mod ./\n",
|
||||
},
|
||||
secondary: map[string]map[string]string{
|
||||
"https://forge.invalid/source.git": {"go.mod": "module route-proxy\n"},
|
||||
},
|
||||
}
|
||||
workspace := t.TempDir()
|
||||
_, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/catalogue.git", "", "", workspace, nil, Npmrc{},
|
||||
GitCredential{URL: "https://builder:s3cret@forge.invalid"}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stored := storeNamedIn(t, r.ran[0])
|
||||
var contextClone string
|
||||
for _, line := range r.ran {
|
||||
if strings.Contains(line, "clone") && strings.Contains(line, "source.git") {
|
||||
contextClone = line
|
||||
}
|
||||
}
|
||||
if contextClone == "" {
|
||||
t.Fatalf("the context was never cloned: %v", r.ran)
|
||||
}
|
||||
if !strings.Contains(contextClone, "credential.helper=store --file="+stored) {
|
||||
t.Fatalf("the context clone does not name the credential store: %s", contextClone)
|
||||
}
|
||||
}
|
||||
@@ -1,199 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A repository becoming artifacts the mesh can pin.
|
||||
//
|
||||
// git and docker are injected rather than run, because what is under test is the ORDER and the
|
||||
// refusals — that nothing is published until everything is built, that a build reads only its own
|
||||
// tree, that two builds of one commit produce one digest. Running docker here would test docker.
|
||||
|
||||
type recorded struct {
|
||||
ran []string
|
||||
// dirs is the directory each entry in ran was run from, same index — so a test can ask not
|
||||
// only what ran but where.
|
||||
dirs []string
|
||||
images map[string]string
|
||||
archives map[string]string
|
||||
failPush bool
|
||||
// contents is what a clone of this repository lands, so the fake clone can restore the tree
|
||||
// Build deliberately removes first.
|
||||
contents map[string]string
|
||||
// secondary is what a clone of a repository OTHER than the one under test lands, keyed by
|
||||
// that repository's URL — an artifact's own build context, cloned apart from the module.
|
||||
secondary map[string]map[string]string
|
||||
// stamped is the modification time the clone gives every file. Set differently between two
|
||||
// builds of one commit, because otherwise both land in the same second and a packer that
|
||||
// carried timestamps would still produce one digest — which is a test that passes for a
|
||||
// reason that has nothing to do with what it claims.
|
||||
stamped time.Time
|
||||
// tree is what git names as the tree of the module's directory; empty answers as for a commit.
|
||||
tree string
|
||||
// builtFrom is the files (slash-separated, relative) of the folder the last `docker build` ran in, read as it ran:
|
||||
// the build removes that folder when it ends (novox/hq issue 1834).
|
||||
builtFrom []string
|
||||
}
|
||||
|
||||
func (r *recorded) run(_ context.Context, dir, name string, args ...string) (string, error) {
|
||||
line := name + " " + strings.Join(args, " ")
|
||||
r.ran = append(r.ran, line)
|
||||
r.dirs = append(r.dirs, dir)
|
||||
// A clone may carry `-c` configuration in front of the verb — the credential store — so the
|
||||
// verb is found rather than assumed first.
|
||||
isClone := false
|
||||
for _, a := range args {
|
||||
if a == "clone" {
|
||||
isClone = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if name == "docker" && len(args) > 0 && args[0] == "build" {
|
||||
r.builtFrom = nil
|
||||
_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
|
||||
if err == nil && !d.IsDir() {
|
||||
rel, _ := filepath.Rel(dir, p)
|
||||
r.builtFrom = append(r.builtFrom, filepath.ToSlash(rel))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
switch {
|
||||
case name == "git" && isClone:
|
||||
repository := args[len(args)-2]
|
||||
tree := args[len(args)-1]
|
||||
if err := os.MkdirAll(tree, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
lands := r.contents
|
||||
if by, is := r.secondary[repository]; is {
|
||||
lands = by
|
||||
}
|
||||
for path, body := range lands {
|
||||
full := filepath.Join(tree, path)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !r.stamped.IsZero() {
|
||||
if err := os.Chtimes(full, r.stamped, r.stamped); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", nil
|
||||
case name == "git" && len(args) > 1 && args[0] == "rev-parse" && strings.HasPrefix(args[1], "HEAD:") && r.tree != "":
|
||||
return r.tree + "\n", nil
|
||||
case name == "git" && len(args) > 0 && args[0] == "rev-parse":
|
||||
return "c0ffeec0ffeec0ffeec0ffeec0ffeec0ffeec0ff\n", nil
|
||||
case name == "git" && len(args) > 1 && args[0] == "rev-list" && args[1] == "--count":
|
||||
return "42\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (r *recorded) PublishImage(_ context.Context, localTag, repository string) (string, error) {
|
||||
if r.failPush {
|
||||
return "", os.ErrPermission
|
||||
}
|
||||
if r.images == nil {
|
||||
r.images = map[string]string{}
|
||||
}
|
||||
r.images[repository] = localTag
|
||||
return "registry.invalid/" + repository + "@sha256:" + strings.Repeat("a", 64), nil
|
||||
}
|
||||
|
||||
func (r *recorded) PublishArchive(_ context.Context, repository string, body []byte, digest string) (string, error) {
|
||||
if r.failPush {
|
||||
return "", os.ErrPermission
|
||||
}
|
||||
if r.archives == nil {
|
||||
r.archives = map[string]string{}
|
||||
}
|
||||
r.archives[repository] = digest
|
||||
_ = body
|
||||
return "https://store.invalid/" + repository, nil
|
||||
}
|
||||
|
||||
// aRepository is a workspace whose clone lands a manifest and some files.
|
||||
func aRepository(t *testing.T, manifest string, files map[string]string) (*recorded, string) {
|
||||
t.Helper()
|
||||
contents := map[string]string{ManifestName: manifest}
|
||||
for name, body := range files {
|
||||
contents[name] = body
|
||||
}
|
||||
return &recorded{contents: contents}, t.TempDir()
|
||||
}
|
||||
|
||||
const withBoth = `{"module":"meshboard","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile"},
|
||||
{"name":"look","kind":"archive","from":"files"}]},
|
||||
"resources":[
|
||||
{"id":"svc","type":"container","name":"meshboard","artifact":"server"},
|
||||
{"id":"theme","type":"archive","path":"/opt/meshboard","artifact":"look"}]}`
|
||||
|
||||
func TestAnUpstreamImageWithNoTagIsRefused(t *testing.T) {
|
||||
// What gets mirrored would be whatever `latest` means today, and a module pinned to that is
|
||||
// not pinned.
|
||||
_, err := catalogue.ParseManifest([]byte(`{"module":"a","version":"1","build":{"artifacts":[
|
||||
{"name":"x","kind":"upstream","from":"postgres"}]}}`))
|
||||
if err == nil {
|
||||
t.Fatal("an untagged upstream reference was accepted")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no tag or digest") {
|
||||
t.Fatalf("unhelpful refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpstreamReferenceIsNotAPathInTheRepository(t *testing.T) {
|
||||
// The rule that a build reads only its own repository must not refuse every reference with a
|
||||
// registry host in it.
|
||||
if _, err := catalogue.ParseManifest([]byte(`{"module":"a","version":"1","build":{"artifacts":[
|
||||
{"name":"x","kind":"upstream","from":"registry.example/library/postgres:17"}]}}`)); err != nil {
|
||||
t.Fatalf("a perfectly ordinary upstream reference was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// **A module is a repository and a path within it** (novox/hq ADR 0069). The catalogue holds its
|
||||
// modules one to a directory and the system this replaces has always built one that way, so a
|
||||
// builder that could only read a repository's root could build none of what exists.
|
||||
|
||||
// A build reads only its own tree. A path climbing out of the clone would otherwise let a build
|
||||
// read — and an archive artifact publish — whatever the build machine happens to hold, which is
|
||||
// the one thing a machine that builds other people's repositories must not do.
|
||||
|
||||
// **Packaging and source are allowed to live apart** — a module that ships only the recipe for
|
||||
// source that lives in a second repository (the reference route-proxy, packaged in the catalogue
|
||||
// but built from mesh-controller's own repository) names where that source actually is, rather
|
||||
// than vendoring a second copy the two could drift from.
|
||||
|
||||
// The forge credential is offered through git's own credential store — a file, never argv — and
|
||||
// git decides when it applies. What is checked: the clone names the store, the secret never
|
||||
// appears in a command line, and the file holds exactly the URL at 0600.
|
||||
|
||||
// storeNamedIn is the credential store a git command line offers.
|
||||
func storeNamedIn(t *testing.T, line string) string {
|
||||
t.Helper()
|
||||
_, after, ok := strings.Cut(line, "credential.helper=store --file=")
|
||||
if !ok {
|
||||
t.Fatalf("the clone does not name the credential store: %s", line)
|
||||
}
|
||||
return strings.Fields(after)[0]
|
||||
}
|
||||
|
||||
// Without a credential, a clone is exactly the invocation it always was, and no credential file
|
||||
// appears — the builder a mesh of public repositories runs is unchanged.
|
||||
|
||||
// An artifact's own context is cloned with the same offer: a private module whose context is a
|
||||
// second private repository on the same forge builds, and the secret still never reaches argv.
|
||||
@@ -1,166 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestABundleIsCompiledAndPackedWithNoDockerfile(t *testing.T) {
|
||||
r, workspace := aRepository(t, aBundle, map[string]string{"index.ts": "console.log(1)"})
|
||||
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
|
||||
|
||||
got, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a module with a language and no Dockerfile did not build: %v", err)
|
||||
}
|
||||
|
||||
// Compiled in the toolchain the mesh chose, not in one the module named.
|
||||
var compiled string
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker run") {
|
||||
compiled = line
|
||||
}
|
||||
}
|
||||
if compiled == "" {
|
||||
t.Fatalf("nothing was compiled:\n%s", strings.Join(r.ran, "\n"))
|
||||
}
|
||||
if !strings.Contains(compiled, "mesh-tools/build@sha256:") {
|
||||
t.Fatalf("the compile did not run in the mesh's own toolchain: %s", compiled)
|
||||
}
|
||||
if strings.Contains(strings.Join(r.ran, "\n"), "docker build") {
|
||||
t.Fatalf("a bundle invoked a Dockerfile build, which is the thing it exists to avoid:\n%s",
|
||||
strings.Join(r.ran, "\n"))
|
||||
}
|
||||
|
||||
// And pinned by a digest of what came out, like any other artifact.
|
||||
digest, _ := got.Manifest.Resources[0]["digest"].(string)
|
||||
if !strings.HasPrefix(digest, "sha256:") {
|
||||
t.Fatalf("the bundle was not pinned: %v", got.Manifest.Resources[0])
|
||||
}
|
||||
|
||||
// **One file per entrypoint and launcher, in the toolchain's bundler** (novox/hq ADR 0193). A
|
||||
// second run in the same toolchain image bundles each into the artifact's bundled output, the SDK
|
||||
// inlined, refusing by name in an image that predates the bundler; and the toolchain's
|
||||
// node_modules is no longer copied into a bundle that keeps nothing external.
|
||||
var bundling []string
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "esbuild") {
|
||||
bundling = append(bundling, line)
|
||||
}
|
||||
}
|
||||
if len(bundling) != 2 {
|
||||
t.Fatalf("want one bundling run for the entrypoints and one for the launchers:\n%s", strings.Join(r.ran, "\n"))
|
||||
}
|
||||
for _, want := range []string{"mesh-tools/build@sha256:", "predates one-file bundles", "--bundle", "--format=esm",
|
||||
"--platform=node", "--outdir=" + Out("code") + ".bundled", Out("code") + "/index.js"} {
|
||||
if !strings.Contains(bundling[0], want) {
|
||||
t.Errorf("the entrypoints' bundling lacks %q: %s", want, bundling[0])
|
||||
}
|
||||
}
|
||||
if !strings.Contains(bundling[1], Out("code")+"/index.serve.mjs") || !strings.Contains(bundling[1], "--out-extension:.js=.mjs") {
|
||||
t.Errorf("the launcher is not bundled under its own name: %s", bundling[1])
|
||||
}
|
||||
if strings.Contains(strings.Join(r.ran, "\n"), "/app/runtime") {
|
||||
t.Errorf("the toolchain's node_modules was copied into a bundle that keeps nothing external:\n%s", strings.Join(r.ran, "\n"))
|
||||
}
|
||||
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "esbuild") {
|
||||
t.Fatal("the bundler ran before the compile wrote its output")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABundleKeepingAPackageExternalCarriesTheToolchainsModules(t *testing.T) {
|
||||
manifest := strings.Replace(aBundle, `"entrypoints":["index.js"]`, `"entrypoints":["index.js"],"external":["sharp"]`, 1)
|
||||
r, workspace := aRepository(t, manifest, map[string]string{"index.ts": "console.log(1)"})
|
||||
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
|
||||
if _, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all := strings.Join(r.ran, "\n")
|
||||
if !strings.Contains(all, "--external:sharp") || !strings.Contains(all, "/app/runtime") {
|
||||
t.Errorf("an external package was not kept as an import with the toolchain's modules beside it:\n%s", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABundleWhoseToolchainIsNotHeldIsRefusedFirst(t *testing.T) {
|
||||
r, workspace := aRepository(t, aBundle, map[string]string{"index.ts": "console.log(1)"})
|
||||
|
||||
_, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a bundle was built with no toolchain to compile it in")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "mesh-tools") {
|
||||
t.Fatalf("the refusal does not name what has to be built first: %v", err)
|
||||
}
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker run") {
|
||||
t.Fatalf("a compile was attempted before the refusal: %s", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABundleInAnUnknownLanguageIsRefused(t *testing.T) {
|
||||
manifest := strings.Replace(aBundle, `"language":"typescript"`, `"language":"cobol"`, 1)
|
||||
r, workspace := aRepository(t, manifest, map[string]string{"index.ts": "x"})
|
||||
|
||||
_, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace,
|
||||
map[string]string{"mesh-tools/build": "registry.invalid/x@sha256:" + strings.Repeat("c", 64)}, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a language nothing can compile was accepted")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "typescript") {
|
||||
t.Fatalf("the refusal does not say what would have worked: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTwoBundlesInOneModuleArePackedSeparately(t *testing.T) {
|
||||
const two = `{"module":"greeter","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"daemon","kind":"bundle","language":"typescript","entrypoints":["index.js"]},
|
||||
{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["index.js"]}]},
|
||||
"resources":[
|
||||
{"id":"a","type":"archive","path":"/opt/greeter/daemon","artifact":"daemon"},
|
||||
{"id":"b","type":"archive","path":"/opt/greeter/tools","artifact":"tools"}]}`
|
||||
|
||||
r, workspace := aRepository(t, two, map[string]string{"index.ts": "console.log(1)"})
|
||||
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
|
||||
|
||||
got, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a module with two bundles did not build: %v", err)
|
||||
}
|
||||
|
||||
// Compiled into two different places. Only the compile lines: the copy of each bundle's
|
||||
// dependencies names the same directory again, deliberately.
|
||||
var outputs []string
|
||||
for _, line := range r.ran {
|
||||
if !strings.Contains(line, "--outDir") {
|
||||
continue
|
||||
}
|
||||
for _, part := range strings.Fields(line) {
|
||||
if strings.HasPrefix(part, ".mesh-build/") {
|
||||
outputs = append(outputs, part)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(outputs) != 2 || outputs[0] == outputs[1] {
|
||||
t.Fatalf("two bundles did not get their own output directories: %v", outputs)
|
||||
}
|
||||
|
||||
// And published as two artifacts, each with its own digest.
|
||||
if len(r.archives) != 2 {
|
||||
t.Fatalf("expected two archives published, got %v", r.archives)
|
||||
}
|
||||
first, _ := got.Manifest.Resources[0]["digest"].(string)
|
||||
second, _ := got.Manifest.Resources[1]["digest"].(string)
|
||||
if first == "" || second == "" {
|
||||
t.Fatalf("a bundle was not pinned: %v", got.Manifest.Resources)
|
||||
}
|
||||
}
|
||||
@@ -1,80 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const aBundle = `{"module":"greeter","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"code","kind":"bundle","language":"typescript","entrypoints":["index.js"]}]},
|
||||
"resources":[
|
||||
{"id":"files","type":"archive","path":"/opt/greeter","artifact":"code"}]}`
|
||||
|
||||
// compiling is a runner that behaves like a toolchain: when asked to compile, it leaves output
|
||||
// where the toolchain says output lands. Without this the pack step has nothing to pack, and the
|
||||
// test would be asserting on a failure rather than on a build.
|
||||
type compiling struct{ *recorded }
|
||||
|
||||
func (c compiling) run(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
out, err := c.recorded.run(ctx, dir, name, args...)
|
||||
if name != "docker" || len(args) == 0 || args[0] != "run" {
|
||||
return out, err
|
||||
}
|
||||
// **Writes where it was TOLD to**, rather than to a fixed directory. A fake that always wrote
|
||||
// to one place would pass whether or not the builder gave each artifact its own — which is the
|
||||
// thing being tested.
|
||||
where := ""
|
||||
for i, a := range args {
|
||||
if a == "--outDir" && i+1 < len(args) {
|
||||
where = args[i+1]
|
||||
}
|
||||
}
|
||||
if where == "" {
|
||||
return out, err
|
||||
}
|
||||
made := filepath.Join(dir, where)
|
||||
if err := os.MkdirAll(made, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(made, "index.js"), []byte("console.log(1)"), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
|
||||
// **A module says what it is written in, and needs no Dockerfile.** This is the whole point of the
|
||||
// bundle recipe: the same module previously needed a hand-written recipe repeating an incantation
|
||||
// that is easy to get wrong in ways that fail somewhere else.
|
||||
|
||||
// A bundle naming packages it keeps external is bundled with them as imports, and carries the
|
||||
// toolchain's node_modules for them — the one case it still does.
|
||||
|
||||
// A language whose bundle carries its own dependencies copies nothing in: a Go binary is static.
|
||||
func TestOnlyALanguageWithARuntimeDirectoryCopiesDependenciesIn(t *testing.T) {
|
||||
ts, _ := ToolchainFor("typescript")
|
||||
if ts.Dependencies != "/app/runtime" {
|
||||
t.Fatalf("typescript bundles run with %q", ts.Dependencies)
|
||||
}
|
||||
for _, language := range []string{"go", "python"} {
|
||||
chain, _ := ToolchainFor(language)
|
||||
if chain.Dependencies != "" {
|
||||
t.Fatalf("%s copies %q into every bundle, and its bundles carry their own", language, chain.Dependencies)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// **Refused before anything is built, naming what to build first.** A base the mesh has not built
|
||||
// is not a compile that fails on its first line — it is a question somebody can answer, and saying
|
||||
// it early is the difference between a fixable message and one about a missing image.
|
||||
|
||||
// A language the mesh does not build is refused the same way, and names what it can build.
|
||||
|
||||
// **One module, two bundles, and neither packs the other.**
|
||||
//
|
||||
// The case that matters for real modules: a module is one piece of software and may still carry a
|
||||
// daemon in one language and tools in another (ADR 0040). An earlier version of this compiled
|
||||
// every bundle into the toolchain's single output directory, so two of them would overwrite each
|
||||
// other and then be packed together — one artifact containing both, twice.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,134 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// **The base branch's check judges a pull request** (novox/hq issue 310).
|
||||
//
|
||||
// A repository's own check is its merge-check.sh, the parts it declares (`# mesh-check-also:`) and the
|
||||
// toolchain it names. Were the check the one in the pull request's head, a change could delete or weaken
|
||||
// the script and so skip its own tests — and a branch cut before the repository had a script escaped
|
||||
// its tests without deleting anything. So when the base branch holds a merge-check.sh, **the base
|
||||
// branch's** script, its declared parts and its toolchain run against the change's tree, and the verdict
|
||||
// says so whenever the change lacks the script or alters any of it.
|
||||
//
|
||||
// A change that legitimately alters the check is judged by the check it replaces: its own version judges
|
||||
// the pull requests after it merges. A base branch with no script keeps what was so before — the head's,
|
||||
// when it brings one, and a warning when it brings none.
|
||||
|
||||
// Judging is the check that judges a change, and what of it the base branch puts in the change's tree.
|
||||
type Judging struct {
|
||||
// Script is the merge-check.sh that judges, nil when neither the base branch nor the change holds one.
|
||||
Script []byte
|
||||
// Said is what the verdict says of it, empty when the change's own script runs as the base's would.
|
||||
Said string
|
||||
// files are the base branch's scripts, by path in the tree, put there before the check runs.
|
||||
files map[string][]byte
|
||||
}
|
||||
|
||||
// Put writes the base branch's scripts into the change's tree, in place of the change's own.
|
||||
func (j Judging) Put(tree string) error {
|
||||
for path, body := range j.files {
|
||||
at := filepath.Join(tree, path)
|
||||
if err := os.MkdirAll(filepath.Dir(at), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(at, body, 0o755); err != nil {
|
||||
return fmt.Errorf("the base branch's %s cannot be put in the change's tree: %w", path, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// TheCheckThatJudges is the check that judges the change cloned at tree, merging into base: the base
|
||||
// branch's when it holds one, the change's own otherwise. An error is that the base branch could not be
|
||||
// read — never a reason to fall back to the change's script.
|
||||
func TheCheckThatJudges(ctx context.Context, tree, base string) (Judging, error) {
|
||||
head, err := os.ReadFile(filepath.Join(tree, CheckScript))
|
||||
if err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return Judging{}, err
|
||||
}
|
||||
if err != nil {
|
||||
head = nil
|
||||
}
|
||||
if base == "" {
|
||||
return Judging{Script: head}, nil
|
||||
}
|
||||
ref := "origin/" + base
|
||||
if _, err := gitIn(ctx, tree, "rev-parse", "--verify", "--quiet", ref+"^{commit}"); err != nil {
|
||||
return Judging{}, fmt.Errorf("the base branch %s cannot be read, and the check it holds judges the change: %w",
|
||||
base, err)
|
||||
}
|
||||
script, held, err := onBranch(ctx, tree, ref, CheckScript)
|
||||
if err != nil {
|
||||
return Judging{}, err
|
||||
}
|
||||
if !held {
|
||||
return Judging{Script: head}, nil
|
||||
}
|
||||
|
||||
j := Judging{Script: script, files: map[string][]byte{CheckScript: script}}
|
||||
var altered []string
|
||||
if head != nil && !bytes.Equal(head, script) {
|
||||
altered = append(altered, CheckScript)
|
||||
}
|
||||
for _, part := range ScriptParts(script)[1:] {
|
||||
if !filepath.IsLocal(part.Script) {
|
||||
continue // refused by name when the check runs
|
||||
}
|
||||
body, held, err := onBranch(ctx, tree, ref, part.Script)
|
||||
if err != nil {
|
||||
return Judging{}, err
|
||||
}
|
||||
if !held {
|
||||
continue // the base branch's own check lacks it: the change's, if it brings it
|
||||
}
|
||||
j.files[filepath.Clean(part.Script)] = body
|
||||
if own, err := os.ReadFile(filepath.Join(tree, part.Script)); err != nil || !bytes.Equal(own, body) {
|
||||
altered = append(altered, part.Script)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case head == nil:
|
||||
j.Said = fmt.Sprintf("the change holds no %s: %s's judged it", CheckScript, base)
|
||||
if len(altered) > 0 {
|
||||
j.Said += fmt.Sprintf(", and the change alters its part(s) %s", strings.Join(altered, ", "))
|
||||
}
|
||||
case len(altered) > 0:
|
||||
j.Said = fmt.Sprintf("THE CHANGE ALTERS ITS OWN CHECK (%s): %s's version judged it; the change's judges "+
|
||||
"the pull requests after it merges", strings.Join(altered, ", "), base)
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// onBranch is a file as a ref has it, and whether the ref holds it at all.
|
||||
func onBranch(ctx context.Context, tree, ref, path string) ([]byte, bool, error) {
|
||||
at := ref + ":" + filepath.ToSlash(filepath.Clean(path))
|
||||
// ls-tree answers nothing for a path the ref does not hold, and fails only when it cannot read.
|
||||
listed, err := gitIn(ctx, tree, "ls-tree", ref, "--", filepath.ToSlash(filepath.Clean(path)))
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("%s cannot be read: %w", ref, err)
|
||||
}
|
||||
if fields := strings.Fields(string(listed)); len(fields) < 2 || fields[1] != "blob" {
|
||||
return nil, false, nil // the ref holds no such file
|
||||
}
|
||||
body, err := gitIn(ctx, tree, "cat-file", "blob", at)
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("%s cannot be read: %w", at, err)
|
||||
}
|
||||
return body, true, nil
|
||||
}
|
||||
|
||||
func gitIn(ctx context.Context, dir string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, "git", args...)
|
||||
cmd.Dir = dir
|
||||
return cmd.Output()
|
||||
}
|
||||
@@ -1,212 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// novox/hq issue 310: the build seat ran the merge-check.sh found in the pull request's head, so a change
|
||||
// could delete or gut it and skip its own tests, and a branch cut before the script escaped them. The base
|
||||
// branch's script — its parts and its toolchain with it — judges the change's tree.
|
||||
|
||||
// aPullRequest is a repository whose main holds base, and a branch "change" off it that holds change on
|
||||
// top (a nil body deletes the file): cloned as the check clones it, at the change's head. It answers the
|
||||
// clone's tree.
|
||||
func aPullRequest(t *testing.T, base, change map[string]*string) string {
|
||||
t.Helper()
|
||||
origin := t.TempDir()
|
||||
git := func(dir string, args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
write := func(files map[string]*string) {
|
||||
for name, body := range files {
|
||||
at := filepath.Join(origin, name)
|
||||
if body == nil {
|
||||
if err := os.Remove(at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(at), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(at, []byte(*body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
git(origin, "add", "-A")
|
||||
git(origin, "commit", "--quiet", "--allow-empty", "-m", "x")
|
||||
}
|
||||
git(origin, "init", "--quiet", "-b", "main")
|
||||
write(base)
|
||||
git(origin, "checkout", "--quiet", "-b", "change")
|
||||
write(change)
|
||||
head := git(origin, "rev-parse", "HEAD")
|
||||
git(origin, "checkout", "--quiet", "main")
|
||||
|
||||
root := t.TempDir()
|
||||
git(root, "clone", "--quiet", origin, "checked")
|
||||
tree := filepath.Join(root, "checked")
|
||||
git(tree, "checkout", "--quiet", head)
|
||||
return tree
|
||||
}
|
||||
|
||||
func s(body string) *string { return &body }
|
||||
|
||||
// judged runs the check that judges the change as the build seat would — each part a plain sh here.
|
||||
func judged(t *testing.T, tree, base string) (Judging, *Layer) {
|
||||
t.Helper()
|
||||
j, err := TheCheckThatJudges(t.Context(), tree, base)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if j.Script == nil {
|
||||
return j, nil
|
||||
}
|
||||
if err := j.Put(tree); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out tail
|
||||
layer := ownCheck(t.Context(), CheckSpec{Toolchain: "go-image", Toolchains: map[string]string{"go": "go-image",
|
||||
"typescript": "ts-image"}}, ScriptParts(j.Script), tree, &out, func() bool { return false },
|
||||
func(image, script string) *exec.Cmd {
|
||||
cmd := exec.CommandContext(t.Context(), "sh", script)
|
||||
cmd.Dir = tree
|
||||
return cmd
|
||||
}, func(string, string, ...any) {})
|
||||
return j, layer
|
||||
}
|
||||
|
||||
// main's check: the suite fails unless the tree holds what main requires.
|
||||
const mainsCheck = "#!/bin/sh\nset -eu\ntest -f covered || { echo 'FAIL: the suite does not pass'; exit 1; }\n"
|
||||
|
||||
func TestAHeadWithoutTheScriptIsJudgedByMains(t *testing.T) {
|
||||
// Deleted by the change — or never there, on a branch cut before main had it: the head lacks it.
|
||||
tree := aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)}, map[string]*string{CheckScript: nil})
|
||||
j, layer := judged(t, tree, "main")
|
||||
if layer == nil || layer.Verdict != "fail" || !strings.Contains(layer.Summary, "does not pass") {
|
||||
t.Fatalf("a head without the script answered %+v — main's script was not what judged it", layer)
|
||||
}
|
||||
if !strings.Contains(j.Said, "the change holds no "+CheckScript) || !strings.Contains(j.Said, "main's judged it") {
|
||||
t.Errorf("the verdict does not say main's judged it: %q", j.Said)
|
||||
}
|
||||
|
||||
// And a head that holds what main's script asks passes by it.
|
||||
tree = aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)},
|
||||
map[string]*string{CheckScript: nil, "covered": s("x")})
|
||||
if _, layer = judged(t, tree, "main"); layer == nil || layer.Verdict != "pass" {
|
||||
t.Fatalf("a head main's script passes answered %+v", layer)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeadThatGutsTheScriptIsStillJudgedByMains(t *testing.T) {
|
||||
tree := aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)},
|
||||
map[string]*string{CheckScript: s("#!/bin/sh\nexit 0\n")})
|
||||
j, layer := judged(t, tree, "main")
|
||||
if layer == nil || layer.Verdict != "fail" {
|
||||
t.Fatalf("a gutted script answered %+v — the change's own copy judged it", layer)
|
||||
}
|
||||
if !strings.Contains(j.Said, "ALTERS ITS OWN CHECK") || !strings.Contains(j.Said, CheckScript) {
|
||||
t.Errorf("the verdict does not say the change alters its check: %q", j.Said)
|
||||
}
|
||||
|
||||
// Gutting the declared part instead, or declaring none: main's lines and main's part still judge.
|
||||
two := "#!/bin/sh\n# mesh-check-toolchain: typescript\n# mesh-check-also: go replays/merge-check.sh\nexit 0\n"
|
||||
tree = aPullRequest(t, map[string]*string{CheckScript: s(two), "replays/merge-check.sh": s(mainsCheck)},
|
||||
map[string]*string{CheckScript: s("#!/bin/sh\n# mesh-check-toolchain: typescript\nexit 0\n"),
|
||||
"replays/merge-check.sh": s("exit 0\n")})
|
||||
j, layer = judged(t, tree, "main")
|
||||
if layer == nil || layer.Verdict != "fail" || !strings.Contains(layer.Summary, "replays/merge-check.sh failed") {
|
||||
t.Fatalf("a gutted part answered %+v", layer)
|
||||
}
|
||||
if parts := ScriptParts(j.Script); len(parts) != 2 || parts[0].Toolchain != "typescript" {
|
||||
t.Errorf("main's toolchain and parts did not judge: %+v", parts)
|
||||
}
|
||||
if !strings.Contains(j.Said, CheckScript+", replays/merge-check.sh") {
|
||||
t.Errorf("the verdict does not name both altered scripts: %q", j.Said)
|
||||
}
|
||||
}
|
||||
|
||||
// A change that legitimately alters the check is judged by main's, says so, and passes when main's passes:
|
||||
// its own version judges the pull requests after it.
|
||||
func TestAChangeToTheCheckIsJudgedByTheOneItReplaces(t *testing.T) {
|
||||
tree := aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck), "covered": s("x")},
|
||||
map[string]*string{CheckScript: s(mainsCheck + "test -f also-covered\n")})
|
||||
j, layer := judged(t, tree, "main")
|
||||
if layer == nil || layer.Verdict != "pass" || !strings.Contains(j.Said, "ALTERS ITS OWN CHECK") {
|
||||
t.Fatalf("a change to the check answered %+v, said %q", layer, j.Said)
|
||||
}
|
||||
// The same script as main's: nothing to say.
|
||||
tree = aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck), "covered": s("x")},
|
||||
map[string]*string{"other.go": s("package x\n")})
|
||||
if j, layer = judged(t, tree, "main"); layer == nil || layer.Verdict != "pass" || j.Said != "" {
|
||||
t.Fatalf("an unchanged check answered %+v, said %q", layer, j.Said)
|
||||
}
|
||||
}
|
||||
|
||||
// Where main holds no script, what was so stays: the head's, or none.
|
||||
func TestWhereMainHoldsNoScriptTheHeadsRuns(t *testing.T) {
|
||||
tree := aPullRequest(t, map[string]*string{"README.md": s("x")},
|
||||
map[string]*string{CheckScript: s("echo 'the first check'; exit 0\n")})
|
||||
j, layer := judged(t, tree, "main")
|
||||
if layer == nil || layer.Verdict != "pass" || j.Said != "" {
|
||||
t.Fatalf("the head's first script answered %+v, said %q", layer, j.Said)
|
||||
}
|
||||
tree = aPullRequest(t, map[string]*string{"README.md": s("x")}, map[string]*string{"b.md": s("y")})
|
||||
if j, _ = judged(t, tree, "main"); j.Script != nil {
|
||||
t.Fatalf("neither holds a script, and one judged: %q", j.Script)
|
||||
}
|
||||
// No base asked (a check by hand without one): the head's, as before.
|
||||
tree = aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)}, map[string]*string{CheckScript: s("exit 0\n")})
|
||||
if j, layer = judged(t, tree, ""); layer == nil || layer.Verdict != "pass" {
|
||||
t.Fatalf("with no base the head's answered %+v", layer)
|
||||
}
|
||||
}
|
||||
|
||||
// A base branch that cannot be read is an error — never a fall back to the change's own script.
|
||||
func TestABaseThatCannotBeReadIsAnError(t *testing.T) {
|
||||
tree := aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)}, map[string]*string{CheckScript: s("exit 0\n")})
|
||||
if _, err := TheCheckThatJudges(t.Context(), tree, "no-such-branch"); err == nil ||
|
||||
!strings.Contains(err.Error(), "no-such-branch") {
|
||||
t.Fatalf("an unreadable base said %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The whole check, against a real container runtime: a head without the script fails by main's.
|
||||
func TestACheckOfAHeadWithoutTheScriptRunsMains(t *testing.T) {
|
||||
registry := checkEnvironment(t)
|
||||
tree := aPullRequest(t, map[string]*string{CheckScript: s(mainsCheck)}, map[string]*string{CheckScript: nil})
|
||||
head, err := gitIn(t.Context(), tree, "rev-parse", "HEAD")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
origin, err := gitIn(t.Context(), tree, "remote", "get-url", "origin")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-base-%d", time.Now().UnixNano()),
|
||||
Repository: strings.TrimSpace(string(origin)), Ref: strings.TrimSpace(string(head)), Owner: "novox",
|
||||
Repo: "hq", Base: "main", Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "fail" || !strings.Contains(v.Repo.Summary, "main's judged it") {
|
||||
t.Fatalf("a head without the script answered %+v\n%s", v.Repo, v.Report)
|
||||
}
|
||||
}
|
||||
@@ -1,355 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// The builds a merge check makes for its merge (novox/hq ADR 0350 §1): every artifact of every module the delivery
|
||||
// plan moves, built from the merged tree the check judged into the scratch namespace, kept under its reuse key, so
|
||||
// the walk of the merge commit takes it instead of building again. Built after the check has judged, and only when
|
||||
// it did not fail: a change that fails is not merged, and nothing it built would be taken.
|
||||
|
||||
// CheckBuildSpec is one module a check builds for its merge, in the walk's order (Tier).
|
||||
type CheckBuildSpec struct {
|
||||
Module string
|
||||
// Repository, Path and Ref are where it is built from; Merged says from the merge the check made instead — the
|
||||
// module is built from the repository checked, at the branch merged into.
|
||||
Repository string
|
||||
Path string
|
||||
Ref string
|
||||
Merged bool
|
||||
Tier int
|
||||
// Skip says why it cannot be built as the walk would build it: not built, and nothing after it.
|
||||
Skip string
|
||||
}
|
||||
|
||||
// CheckBuilding is what a check builds with: where it publishes and keeps its builds, and what a build is handed.
|
||||
type CheckBuilding struct {
|
||||
Publish Publisher
|
||||
Scratch Scratch
|
||||
Held map[string]string
|
||||
// Running is each artifact of the build the mesh has registered for its module, keyed "<module>/<artifact>"
|
||||
// (novox/hq issue 721): what a build for the merge is compared with — never Held's newest build, which may be one
|
||||
// its gate failed and the mesh put back.
|
||||
Running map[string]string
|
||||
Npmrc Npmrc
|
||||
Seats map[string]string
|
||||
// Named are the builds the build seat's verdicts of earlier checks of this pull request at this head name
|
||||
// (novox/hq issue 703): a build kept under a module's key is kept again, instead of built, only when one of them
|
||||
// names its key and every digest, and the build is this pull request's at this head. The record alone is never
|
||||
// enough.
|
||||
Named []NamedBuild
|
||||
}
|
||||
|
||||
// CheckBuilt is one module a check built for its merge, as its verdict lists it (§1): its key and its artifacts, or
|
||||
// why it was not built.
|
||||
type CheckBuilt struct {
|
||||
Module string
|
||||
Key string
|
||||
Made []catalogue.Built
|
||||
// NotBuilt says why it was not: no key, a build that failed, or a module after one of those.
|
||||
NotBuilt string
|
||||
Took time.Duration
|
||||
// Kept is the check whose build this check kept instead of building, its key unchanged (novox/hq issue 703):
|
||||
// the verdict names that build, by key and digest, as that check's. Empty when this check built it.
|
||||
Kept string
|
||||
// Same says every artifact came out the same as the build the mesh holds of the module (novox/hq issue 721), and
|
||||
// SameAs names, per artifact, the held reference it was compared with: the walk leaves the module where it is
|
||||
// while the mesh still holds that build. Empty when it differs, or could not be compared.
|
||||
Same string
|
||||
SameAs map[string]string
|
||||
}
|
||||
|
||||
// skipsTheBuilds is why a check builds nothing for its merge; empty when it builds.
|
||||
func skipsTheBuilds(v CheckVerdict) string {
|
||||
for _, l := range []*Layer{v.Gate, v.Repo} {
|
||||
if l != nil && (l.Verdict == "fail" || l.Verdict == "error") {
|
||||
return "the check did not pass, so nothing is built for its merge"
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// buildForTheMerge builds each module the plan moves into the scratch namespace (makeForTheMerge), and keeps what it
|
||||
// built under its keys (keepForTheMerge): the order of a check that builds after it has judged.
|
||||
func buildForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with CheckBuilding, workspace, tree, merge string,
|
||||
judged CheckVerdict, forge GitCredential, log Log, say func(step, format string, args ...any)) []CheckBuilt {
|
||||
out, kept := makeForTheMerge(ctx, run, spec, with, workspace, tree, merge, judged, forge, log, say)
|
||||
return keepForTheMerge(ctx, spec, with, out, kept, say)
|
||||
}
|
||||
|
||||
// makeForTheMerge builds each module the plan moves, tier by tier, into the scratch namespace — **the modules of a
|
||||
// tier at the same time** (novox/hq issue 716): none of them stands on another, and a tier waits only for the ones
|
||||
// before it, whose builds it is handed. A module standing after one not built is not built either: what it stands on
|
||||
// in the walk is not what it would stand on here. tree is the check's copy of the merge and merge the merge commit it
|
||||
// made there; workspace is where each build's working copy is made, one directory per module, which the host's
|
||||
// container engine sees at the same path (novox/hq issue 713). Answers what each module came to, in the plan's order,
|
||||
// and the builds to keep — nothing is kept here.
|
||||
func makeForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with CheckBuilding, workspace, tree, merge string,
|
||||
judged CheckVerdict, forge GitCredential, log Log, say func(step, format string, args ...any)) ([]CheckBuilt, []CheckBuild) {
|
||||
if len(spec.Builds) == 0 || with.Publish == nil || with.Scratch == nil {
|
||||
return nil, nil
|
||||
}
|
||||
// What an earlier check left, ended before it could remove it, is not built on.
|
||||
place := filepath.Join(workspace, "for-the-merge")
|
||||
_ = os.RemoveAll(place)
|
||||
defer os.RemoveAll(place)
|
||||
// One log, said to by builds running at once: a line at a time.
|
||||
var saying sync.Mutex
|
||||
if log != nil {
|
||||
inner := log
|
||||
log = func(step, message string) {
|
||||
saying.Lock()
|
||||
defer saying.Unlock()
|
||||
inner(step, message)
|
||||
}
|
||||
}
|
||||
inner := say
|
||||
say = func(step, format string, args ...any) {
|
||||
saying.Lock()
|
||||
defer saying.Unlock()
|
||||
inner(step, format, args...)
|
||||
}
|
||||
ref := merge
|
||||
if ref == "" {
|
||||
ref = spec.Ref
|
||||
}
|
||||
held := map[string]string{}
|
||||
for k, v := range with.Held {
|
||||
held[k] = v
|
||||
}
|
||||
out := make([]CheckBuilt, len(spec.Builds))
|
||||
results := make([]Result, len(spec.Builds))
|
||||
var kept []CheckBuild
|
||||
stopAt, stopWhy := -1, ""
|
||||
for start := 0; start < len(spec.Builds); {
|
||||
tier := spec.Builds[start].Tier
|
||||
end := start
|
||||
for end < len(spec.Builds) && spec.Builds[end].Tier == tier {
|
||||
end++
|
||||
}
|
||||
// What this tier's builds are handed: the builds of the tiers before it, read alone while they run.
|
||||
handed := map[string]string{}
|
||||
for k, v := range held {
|
||||
handed[k] = v
|
||||
}
|
||||
var building sync.WaitGroup
|
||||
for i := start; i < end; i++ {
|
||||
b := spec.Builds[i]
|
||||
if stopAt >= 0 && b.Tier > stopAt {
|
||||
out[i] = CheckBuilt{Module: b.Module, NotBuilt: stopWhy}
|
||||
continue
|
||||
}
|
||||
if b.Skip != "" {
|
||||
say("build", "%s not built for the merge: %s", b.Module, b.Skip)
|
||||
out[i] = CheckBuilt{Module: b.Module, NotBuilt: b.Skip}
|
||||
continue
|
||||
}
|
||||
repository, at := b.Repository, b.Ref
|
||||
if b.Merged {
|
||||
repository, at = tree, ref
|
||||
}
|
||||
name := b.Module
|
||||
if !safeName.MatchString(name) {
|
||||
name = fmt.Sprintf("module-%d", i)
|
||||
}
|
||||
building.Add(1)
|
||||
go func(i int, b CheckBuildSpec, repository, at, dir string) {
|
||||
defer building.Done()
|
||||
began := time.Now()
|
||||
say("build", "for the merge: %s (tier %d)", b.Module, b.Tier)
|
||||
var mirrored []string
|
||||
result, err := build(ctx, run, scratchPublisher{to: with.Publish}, repository, b.Path, at, dir, handed,
|
||||
with.Npmrc, forge, log, &mirrored, &Reuse{keyFirst: true, Scratch: with.Scratch, Named: with.Named,
|
||||
pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, head: judged.Head},
|
||||
with.Seats)
|
||||
built := CheckBuilt{Module: b.Module, Key: result.ReuseKey, Took: time.Since(began)}
|
||||
switch {
|
||||
case errors.Is(err, errNoKey):
|
||||
built.NotBuilt = err.Error()
|
||||
case err != nil:
|
||||
built.NotBuilt = "its build failed: " + err.Error()
|
||||
case result.ReuseKey == "":
|
||||
built.NotBuilt = "it has no reuse key: " + result.NoKey
|
||||
}
|
||||
if built.NotBuilt == "" {
|
||||
// **Compared with what runs** (novox/hq issue 721): a failed comparison says nothing, and the
|
||||
// module moves as before.
|
||||
same, against, err := sameAsHeld(ctx, run, dir, b.Module, result.Built, with.Running)
|
||||
switch {
|
||||
case err != nil:
|
||||
say("build", "%s could not be compared with the build the mesh holds, so it moves: %v", b.Module, err)
|
||||
case same != "":
|
||||
built.Same, built.SameAs = same, against
|
||||
say("build", "%s %s", b.Module, same)
|
||||
}
|
||||
}
|
||||
out[i], results[i] = built, result
|
||||
}(i, b, repository, at, filepath.Join(place, name))
|
||||
}
|
||||
building.Wait()
|
||||
for i := start; i < end; i++ {
|
||||
b, built, result := spec.Builds[i], &out[i], results[i]
|
||||
if built.NotBuilt != "" {
|
||||
if built.Key != "" || built.Took > 0 {
|
||||
say("build", "%s not built for the merge: %s", b.Module, built.NotBuilt)
|
||||
}
|
||||
if stopAt < 0 || b.Tier < stopAt {
|
||||
stopAt, stopWhy = b.Tier, "it stands after "+b.Module+", which was not built for the merge"
|
||||
}
|
||||
continue
|
||||
}
|
||||
built.Made = result.Built
|
||||
for _, m := range result.Built {
|
||||
held[b.Module+"/"+m.Name] = m.Reference
|
||||
}
|
||||
if result.Reused != nil {
|
||||
// **Kept, not built** (novox/hq issue 703): the earlier check's build, named as that check's, so the
|
||||
// walk takes it as the verdict names it; this check's document names it too, so it stays while this
|
||||
// check does (ADR 0350 §5).
|
||||
built.Kept = result.Reused.From.Check
|
||||
say("build", "%s kept for the merge under %s: %s", b.Module, twelve(result.ReuseKey[3:]),
|
||||
result.Reused.From.Said())
|
||||
kept = append(kept, result.Reused.From)
|
||||
continue
|
||||
}
|
||||
say("build", "%s built for the merge under %s (%s)", b.Module, twelve(result.ReuseKey[3:]),
|
||||
built.Took.Round(time.Second))
|
||||
if len(result.Built) > 0 {
|
||||
kept = append(kept, CheckBuild{Key: result.ReuseKey, Module: b.Module, Parts: result.ReuseParts,
|
||||
Made: result.Built, Pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number},
|
||||
Head: judged.Head, Trunk: judged.Trunk, Check: spec.ID, At: time.Now().UTC()})
|
||||
}
|
||||
}
|
||||
start = end
|
||||
}
|
||||
return out, kept
|
||||
}
|
||||
|
||||
// keepForTheMerge keeps a check's builds under their keys, its own document naming them, so the walk of its merge
|
||||
// finds them; what cannot be kept is let go of, said, and each module says it — never the check's verdict.
|
||||
func keepForTheMerge(ctx context.Context, spec CheckSpec, with CheckBuilding, out []CheckBuilt, kept []CheckBuild,
|
||||
say func(step, format string, args ...any)) []CheckBuilt {
|
||||
if len(kept) == 0 {
|
||||
return out
|
||||
}
|
||||
if err := with.Scratch.Keep(ctx, PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, spec.ID,
|
||||
time.Now().UTC(), kept); err != nil {
|
||||
// Built and not findable: the walk builds, and says no check built it. Said, never the check's verdict.
|
||||
say("build", "the builds for the merge could not be kept, and the walk will build: %v", err)
|
||||
if err := with.Scratch.Discard(ctx, kept); err != nil {
|
||||
say("build", "and what they made could not all be let go of: %v", err)
|
||||
}
|
||||
for i := range out {
|
||||
if out[i].NotBuilt == "" {
|
||||
out[i].NotBuilt = fmt.Sprintf("built, and could not be kept: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// discardForTheMerge lets go of builds a check made beside its tests when the check did not pass (novox/hq issue 716):
|
||||
// a change that fails is not merged, and nothing it built is kept. A build another check names stays (Discard).
|
||||
func discardForTheMerge(ctx context.Context, with CheckBuilding, kept []CheckBuild, why string,
|
||||
say func(step, format string, args ...any)) {
|
||||
say("check", "%s: the %d build(s) made beside it are let go of", why, len(kept))
|
||||
if len(kept) == 0 {
|
||||
return
|
||||
}
|
||||
if err := with.Scratch.Discard(ctx, kept); err != nil {
|
||||
say("build", "what they made could not all be let go of, and goes with the week's sweep: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// besideTheTests are a check's builds for its merge running while its gate and its own check run (novox/hq issue
|
||||
// 716): started once every clone is made, waited for when the check has judged, kept when it passed and let go of
|
||||
// when it did not.
|
||||
type besideTheTests struct {
|
||||
spec CheckSpec
|
||||
with CheckBuilding
|
||||
cancel context.CancelFunc
|
||||
done chan struct{}
|
||||
place string
|
||||
out []CheckBuilt
|
||||
kept []CheckBuild
|
||||
began time.Time
|
||||
settled bool
|
||||
say func(step, format string, args ...any)
|
||||
}
|
||||
|
||||
// buildsBesideTheTests is where a check's builds run beside its tests: a directory made for this check under the judge
|
||||
// cache's homes/ (ADR 0358 §6), on the host path the build agent declares — so the host's container engine mounts what
|
||||
// the agent sees (issue 713) — and never mounted by a check's container, so nothing a pull request's script runs can
|
||||
// change a build while it is made (issue 703). Empty, and the builds wait for the check, without a judge cache.
|
||||
func buildsBesideTheTests(cache, workspace, id string) (string, error) {
|
||||
if cache == "" {
|
||||
return "", nil
|
||||
}
|
||||
if withinDir(workspace, cache) {
|
||||
return "", fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers mount: "+
|
||||
"nothing is built for the merge where a pull request could reach it", cache, workspace)
|
||||
}
|
||||
if !safeName.MatchString(id) {
|
||||
return "", fmt.Errorf("the check's id %q cannot name its place in the judge cache", id)
|
||||
}
|
||||
place := filepath.Join(cache, "homes", id+"-builds")
|
||||
if err := os.RemoveAll(place); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.MkdirAll(place, 0o700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return place, nil
|
||||
}
|
||||
|
||||
// startBesideTheTests starts the builds for the merge in place, from merged, the check's copy of the merge.
|
||||
func startBesideTheTests(ctx context.Context, run Runner, spec CheckSpec, place, merged string, judged CheckVerdict,
|
||||
forge GitCredential, log Log, say func(step, format string, args ...any)) *besideTheTests {
|
||||
building, cancel := context.WithCancel(ctx)
|
||||
b := &besideTheTests{spec: spec, with: *spec.BuildWith, cancel: cancel, done: make(chan struct{}), place: place,
|
||||
began: time.Now(), say: say}
|
||||
say("check", "building what the merge moves (%d module(s)) beside its tests", len(spec.Builds))
|
||||
go func() {
|
||||
defer close(b.done)
|
||||
b.out, b.kept = makeForTheMerge(building, run, spec, b.with, place, merged, judged.Merge, judged, forge, log, say)
|
||||
}()
|
||||
return b
|
||||
}
|
||||
|
||||
// settle waits for the builds and keeps them when the check passed; a check that did not pass lets go of them, and
|
||||
// its verdict names none.
|
||||
func (b *besideTheTests) settle(ctx context.Context, verdict CheckVerdict) []CheckBuilt {
|
||||
<-b.done
|
||||
b.settled = true
|
||||
defer os.RemoveAll(b.place)
|
||||
if why := skipsTheBuilds(verdict); why != "" {
|
||||
discardForTheMerge(ctx, b.with, b.kept, why, b.say)
|
||||
return nil
|
||||
}
|
||||
built := keepForTheMerge(ctx, b.spec, b.with, b.out, b.kept, b.say)
|
||||
b.say("check", "built for the merge: %d of %d module(s), beside the tests (%s)", builtCount(built), len(built),
|
||||
time.Since(b.began).Round(time.Second))
|
||||
return built
|
||||
}
|
||||
|
||||
// abandon ends builds a check that ended without a verdict leaves, and lets go of what they made.
|
||||
func (b *besideTheTests) abandon() {
|
||||
if b.settled {
|
||||
return
|
||||
}
|
||||
b.cancel()
|
||||
<-b.done
|
||||
defer os.RemoveAll(b.place)
|
||||
letGo, done := context.WithTimeout(context.Background(), time.Minute)
|
||||
defer done()
|
||||
discardForTheMerge(letGo, b.with, b.kept, "the check ended without a verdict", b.say)
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// manyPaths is a list shaped like the real one of mesh-delivery feat/delivery-parts 9e9c2a7 (novox/hq issue
|
||||
// 1247): 2494 vendored files of nats-server across four new modules, about 208 KB comma-joined.
|
||||
func manyPaths() []string {
|
||||
modules := []string{"mesh-delivery-forge-bridge", "mesh-delivery-indexer", "mesh-delivery-teller", "mesh-delivery-view"}
|
||||
var paths []string
|
||||
for i := 0; i < 2494; i++ {
|
||||
paths = append(paths, fmt.Sprintf("modules/%s/vendor/github.com/nats-io/nats-server/v2/server/file_%04d.go",
|
||||
modules[i%len(modules)], i))
|
||||
}
|
||||
return paths
|
||||
}
|
||||
|
||||
func envValue(env []string, name string) string {
|
||||
for _, e := range env {
|
||||
if v, ok := strings.CutPrefix(e, name+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return "\x00unset"
|
||||
}
|
||||
|
||||
func TestALongListOfChangedPathsGoesInAFileAndTheStringHoldsTheModules(t *testing.T) { // 1247/AC1
|
||||
paths := manyPaths()
|
||||
if n := len(strings.Join(paths, ",")); n < 200<<10 {
|
||||
t.Fatalf("the fixture is only %d bytes: not the real list's shape", n)
|
||||
}
|
||||
root := t.TempDir()
|
||||
var said []string
|
||||
env, err := changedEnvironment(root, paths, func(l string) { said = append(said, l) })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, e := range env {
|
||||
if len(e) > 64<<10 {
|
||||
t.Errorf("an environment string of %d bytes is handed to docker: %.60s", len(e), e)
|
||||
}
|
||||
}
|
||||
raw, err := os.ReadFile(envValue(env, EnvChangedFile))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := strings.Split(strings.TrimSuffix(string(raw), "\n"), "\n"); strings.Join(got, "\n") != strings.Join(paths, "\n") {
|
||||
t.Errorf("the file holds %d lines, not the %d paths in the order given", len(got), len(paths))
|
||||
}
|
||||
want := "modules/mesh-delivery-forge-bridge/,modules/mesh-delivery-indexer/,modules/mesh-delivery-teller/,modules/mesh-delivery-view/"
|
||||
if got := envValue(env, EnvChanged); got != want {
|
||||
t.Errorf("%s is %q, want each module's directory once: %q", EnvChanged, got, want)
|
||||
}
|
||||
if len(said) != 1 || !strings.Contains(said[0], EnvChangedFile) {
|
||||
t.Errorf("the check's output says %q, want one line naming %s", said, EnvChangedFile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAShortListOfChangedPathsStaysAsItWasAndIsInTheFileToo(t *testing.T) { // 1247/AC2
|
||||
var said []string
|
||||
env, err := changedEnvironment(t.TempDir(), []string{"a.go", "b.go"}, func(l string) { said = append(said, l) })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := envValue(env, EnvChanged); got != "a.go,b.go" {
|
||||
t.Errorf("%s is %q", EnvChanged, got)
|
||||
}
|
||||
raw, err := os.ReadFile(envValue(env, EnvChangedFile))
|
||||
if err != nil || string(raw) != "a.go\nb.go\n" {
|
||||
t.Errorf("the file holds %q, %v", raw, err)
|
||||
}
|
||||
if len(said) != 0 {
|
||||
t.Errorf("a short list said %q", said)
|
||||
}
|
||||
}
|
||||
|
||||
// theGatesCommand is the command gateLayer hands the toolchain for the merge gate, given the environment of the check.
|
||||
func theGatesCommand(t *testing.T, env []string) string {
|
||||
t.Helper()
|
||||
var args []string
|
||||
gateLayer(t.Context(), CheckSpec{}, t.TempDir(), t.TempDir(), "gate", filepath.Join(t.TempDir(), "verdict.json"), env,
|
||||
func(dir string, env []string, command ...string) []string { return command },
|
||||
func(a []string) error { args = a; return errors.New("stop here") }, &tail{}, "bus", "name", false, "", nil,
|
||||
func(string, string, ...any) {})
|
||||
return strings.Join(args, " ")
|
||||
}
|
||||
|
||||
func TestTheGateIsGivenAShortListAsBeforeSoAnOlderJudgeStillRuns(t *testing.T) { // 1247/AC3
|
||||
env, err := changedEnvironment(t.TempDir(), []string{"a.go", "b.go"}, func(string) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
command := theGatesCommand(t, env)
|
||||
if !strings.Contains(command, `--changed "$MESH_CHECK_CHANGED"`) || strings.Contains(command, "--changed-file") {
|
||||
t.Errorf("a short list is not given to the gate as --changed alone: %s", command)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGateReadsACutListFromTheFile(t *testing.T) { // 1247/AC3
|
||||
env, err := changedEnvironment(t.TempDir(), manyPaths(), func(string) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
command := theGatesCommand(t, env)
|
||||
if !strings.Contains(command, `--changed-file "$MESH_CHECK_CHANGED_FILE"`) || strings.Contains(command, `--changed "`) {
|
||||
t.Errorf("a cut list is not given to the gate through the file alone: %s", command)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACommandThatCannotStartFailsWithItsError(t *testing.T) { // 1247/AC4
|
||||
var out tail
|
||||
layer := ownCheck(t.Context(), CheckSpec{Toolchain: "go-image"}, []ScriptPart{{Toolchain: "go", Script: CheckScript}}, t.TempDir(),
|
||||
&out, func() bool { return false }, func(image, script string) *exec.Cmd {
|
||||
return exec.CommandContext(t.Context(), "/nonexistent/docker", script)
|
||||
}, func(string, string, ...any) {})
|
||||
if layer == nil || layer.Verdict != "fail" {
|
||||
t.Fatalf("a command that cannot start answered %+v", layer)
|
||||
}
|
||||
if !strings.Contains(layer.Summary, CheckScript) || !strings.Contains(layer.Summary, "no such file or directory") ||
|
||||
strings.HasSuffix(strings.TrimSpace(layer.Summary), "failed:") {
|
||||
t.Errorf("the summary does not say why: %q", layer.Summary)
|
||||
}
|
||||
}
|
||||
@@ -1,180 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestACheckContainerSeesNoForgeCredential(t *testing.T) {
|
||||
repo, head := aCheckedRepository(t, map[string]string{CheckScript: "echo checked\n"})
|
||||
besideRepo, besideHead := aCheckedRepository(t, map[string]string{"README": "beside"})
|
||||
// A clone source that carries userinfo, as a forge's clone URL may: git records it as given in the
|
||||
// clone's .git/config, which the container reads.
|
||||
besideURL := "file://beside-user:" + besideSecret + "@" + besideRepo
|
||||
workspace := t.TempDir()
|
||||
|
||||
var stores []string
|
||||
reached := false
|
||||
var leaks []string
|
||||
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
switch name {
|
||||
case "git":
|
||||
for _, a := range args {
|
||||
if f, ok := strings.CutPrefix(a, "credential.helper=store --file="); ok {
|
||||
stores = append(stores, f)
|
||||
raw, err := os.ReadFile(f)
|
||||
if err != nil || strings.TrimSpace(string(raw)) != forgeURL {
|
||||
t.Errorf("git is offered a store that does not hold the credential as given: %q, %v", raw, err)
|
||||
}
|
||||
if info, err := os.Stat(f); err == nil && info.Mode().Perm() != 0o600 {
|
||||
t.Errorf("the credential store is readable beyond its owner: %v", info.Mode())
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(args) >= 2 && args[len(args)-3] == "--quiet" && hasString(args, "clone") {
|
||||
source := args[len(args)-2]
|
||||
if u, err := url.Parse(source); err == nil && u.User != nil {
|
||||
// Git cannot reach a file:// URL with userinfo; clone it without, then record it as git
|
||||
// would have: as given.
|
||||
clone := append(append([]string{}, args[:len(args)-2]...), bareURL(t, source), args[len(args)-1])
|
||||
if out, err := Command(ctx, dir, "git", clone...); err != nil {
|
||||
return out, err
|
||||
}
|
||||
return Command(ctx, filepath.Join(dir, args[len(args)-1]), "git", "remote", "set-url", "origin", source)
|
||||
}
|
||||
}
|
||||
return Command(ctx, dir, name, args...)
|
||||
case "docker":
|
||||
if !reached {
|
||||
reached = true
|
||||
// The first container: everything the workspace holds is what the toolchain container sees.
|
||||
filepath.WalkDir(workspace, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
raw, _ := os.ReadFile(path)
|
||||
s := string(raw)
|
||||
if strings.Contains(s, forgeSecret) || strings.Contains(s, besideSecret) ||
|
||||
strings.Contains(s, npmSecret) || d.Name() == "git-credentials" || d.Name() == ".npmrc" ||
|
||||
strings.Contains(s, "credential.helper") {
|
||||
leaks = append(leaks, path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
for _, f := range stores {
|
||||
if _, err := os.Stat(f); !errors.Is(err, os.ErrNotExist) {
|
||||
leaks = append(leaks, f+" (still there when the first container runs)")
|
||||
}
|
||||
if rel, err := filepath.Rel(workspace, f); err == nil && !strings.HasPrefix(rel, "..") {
|
||||
leaks = append(leaks, f+" (inside the workspace the container mounts)")
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(args) > 0 && args[0] == "ps" {
|
||||
return "", nil
|
||||
}
|
||||
return "", errors.New("no container runtime in this test")
|
||||
}
|
||||
return "", errors.New("unexpected command " + name)
|
||||
}
|
||||
// A credential an older builder left in the workspace is removed too: the forge's, and the .npmrc a
|
||||
// build wrote into the tree it cloned.
|
||||
if err := os.WriteFile(filepath.Join(workspace, "git-credentials"), []byte(forgeURL+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, dir := range []string{"source/x", "context-server"} {
|
||||
if err := os.MkdirAll(filepath.Join(workspace, dir), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(workspace, dir, ".npmrc"),
|
||||
[]byte("//forge.invalid/api/packages/novox/npm/:_authToken="+npmSecret+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
_, err := Check(t.Context(), run, CheckSpec{ID: "check-462", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-controller", Number: 1, Toolchain: "golang", Beside: map[string]Beside{
|
||||
"mesh-catalog": {Repository: besideURL, Ref: besideHead}}}, workspace, aFactsRegistry(t),
|
||||
GitCredential{URL: forgeURL}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("the check ran past its first container in a test with none")
|
||||
}
|
||||
if !reached {
|
||||
t.Fatalf("the check never reached its first container: %v", err)
|
||||
}
|
||||
if len(stores) == 0 {
|
||||
t.Fatal("no clone was offered the forge credential")
|
||||
}
|
||||
if len(leaks) > 0 {
|
||||
t.Fatalf("the check's container sees the credential:\n%s", strings.Join(leaks, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckLinePublishedToTheLogIsRedacted(t *testing.T) {
|
||||
var said []string
|
||||
say := func(step, format string, args ...any) {
|
||||
if step == "output" && len(args) > 0 {
|
||||
said = append(said, args[0].(string))
|
||||
}
|
||||
}
|
||||
var out tail
|
||||
layer := ownCheck(t.Context(), CheckSpec{Toolchain: "golang"}, []ScriptPart{{Toolchain: "go", Script: CheckScript}},
|
||||
t.TempDir(), &out, func() bool { return false }, func(string, string) *exec.Cmd {
|
||||
return exec.CommandContext(t.Context(), "sh", "-c", "echo cloning http://mesh_builder:t0ps3cret-forge@forge.invalid/novox/x.git; "+
|
||||
"echo token ghp_abcdefghijklmnopqrstuvwxyz0123456789")
|
||||
}, say)
|
||||
if layer == nil || layer.Verdict != "pass" {
|
||||
t.Fatalf("the check answered %+v\n%s", layer, out.String())
|
||||
}
|
||||
joined := strings.Join(said, "\n")
|
||||
if strings.Contains(joined, "t0ps3cret-forge") || strings.Contains(joined, "ghp_abcdef") {
|
||||
t.Fatalf("a secret the check printed is published to the build's log:\n%s", joined)
|
||||
}
|
||||
if !strings.Contains(joined, "http://mesh_builder:[redacted: a password in a URI]@forge.invalid/novox/x.git") {
|
||||
t.Fatalf("the line is not said with what was there named:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildsCloneRecordsNoUserinfo(t *testing.T) {
|
||||
repo, _ := aCheckedRepository(t, map[string]string{ManifestName: `{"module":"plain","version":"1"}`})
|
||||
source := "file://build-user:" + besideSecret + "@" + repo
|
||||
workspace := t.TempDir()
|
||||
tree := filepath.Join(workspace, "source")
|
||||
var configs []string
|
||||
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "git" && hasString(args, "clone") && args[len(args)-2] == source {
|
||||
clone := append(append([]string{}, args[:len(args)-2]...), bareURL(t, source), args[len(args)-1])
|
||||
if out, err := Command(ctx, dir, "git", clone...); err != nil {
|
||||
return out, err
|
||||
}
|
||||
return Command(ctx, args[len(args)-1], "git", "remote", "set-url", "origin", source)
|
||||
}
|
||||
if name == "git" && len(args) > 0 && args[0] == "rev-parse" {
|
||||
raw, _ := os.ReadFile(filepath.Join(tree, ".git", "config"))
|
||||
configs = append(configs, string(raw))
|
||||
}
|
||||
return Command(ctx, dir, name, args...)
|
||||
}
|
||||
if _, err := Build(t.Context(), run, &recorded{}, source, "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(configs) == 0 {
|
||||
t.Fatal("the build never read its clone")
|
||||
}
|
||||
for _, c := range configs {
|
||||
if strings.Contains(c, besideSecret) || strings.Contains(c, "build-user") {
|
||||
t.Fatalf("the build's clone records the credential it was cloned with:\n%s", c)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(tree); !os.IsNotExist(err) {
|
||||
t.Fatalf("the build's tree outlives the build: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,82 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/facts"
|
||||
)
|
||||
|
||||
// **A check's container never sees the forge credential** (novox/hq issue 462): the toolchain container
|
||||
// mounts the workspace as HOME, so a credential kept there — or a clone's .git/config carrying one — is
|
||||
// readable by any pull request's merge-check.sh, and printed, kept on the bus for days.
|
||||
|
||||
const (
|
||||
forgeSecret = "sw0rdfi5h-forge"
|
||||
forgeURL = "http://mesh_novox_builder:" + forgeSecret + "@forge.invalid:20000"
|
||||
besideSecret = "b3side-t0ken"
|
||||
npmSecret = "npm-s3cret-t0ken"
|
||||
)
|
||||
|
||||
// aFactsRegistry is an artifact store holding the facts snapshot, and nothing else.
|
||||
func aFactsRegistry(t *testing.T) string {
|
||||
t.Helper()
|
||||
body, err := json.Marshal(facts.Facts{Format: facts.Format, Taken: time.Now().UTC(),
|
||||
Versions: facts.Versions{Bus: "2.11.17", Store: "17.11"}, Machines: []facts.Machine{{Name: "abcdef", Length: 6}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
manifest, _ := json.Marshal(map[string]any{"schemaVersion": 2, "layers": []map[string]any{
|
||||
{"mediaType": facts.MediaType, "digest": digest, "size": len(body)}}})
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.Contains(r.URL.Path, "/manifests/"):
|
||||
w.Write(manifest)
|
||||
case strings.HasSuffix(r.URL.Path, "/blobs/"+digest):
|
||||
w.Write(body)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return strings.TrimPrefix(srv.URL, "http://")
|
||||
}
|
||||
|
||||
// bareURL is a URL with its userinfo left out.
|
||||
func bareURL(t *testing.T, raw string) string {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u.User = nil
|
||||
return u.String()
|
||||
}
|
||||
|
||||
// Every line a repository's own check prints is published to the build's log redacted.
|
||||
|
||||
func TestTheRedactorHidesTheForgeCredentialAndShapes(t *testing.T) {
|
||||
r := redactorFor(GitCredential{URL: forgeURL})
|
||||
for in, want := range map[string]string{
|
||||
"the secret alone: " + forgeSecret: "the secret alone: [redacted: the forge credential]",
|
||||
"go test ./... ok": "go test ./... ok",
|
||||
"--password hunter22 and done": "--password [redacted: the word after --password] and done",
|
||||
"commit 3b6b54a0c1d2e3f4a5b6c7d8e9f0": "commit 3b6b54a0c1d2e3f4a5b6c7d8e9f0",
|
||||
} {
|
||||
if got := r.redact(in); got != want {
|
||||
t.Errorf("%q redacted as %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A build's clone of a URL carrying userinfo records it without, so no build leaves a credential in
|
||||
// workspace/source/.git/config while it runs either (novox/hq issue 462); the tree itself is gone when the
|
||||
// build ends.
|
||||
@@ -1,33 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 1826/AC2: the held values are written whole to held.json, and the gate is given the flag only when there are some.
|
||||
func TestHeldValuesAreWrittenForTheGateAndNoneWriteNothing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if file, err := writeHeld(root, nil); err != nil || file != "" {
|
||||
t.Fatalf("none wrote %q %v", file, err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, "held.json")); err == nil {
|
||||
t.Fatal("none wrote a file")
|
||||
}
|
||||
file, err := writeHeld(root, []HeldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
|
||||
Value: []byte("2000"), Approved: "the operator"}})
|
||||
if err != nil || file != filepath.Join(root, "held.json") {
|
||||
t.Fatalf("wrote %q %v", file, err)
|
||||
}
|
||||
raw, _ := os.ReadFile(file)
|
||||
for _, want := range []string{`"node":"ace"`, `"module":"n8n"`, `"key":"media-gid"`, `"value":2000`} {
|
||||
if !strings.Contains(string(raw), want) {
|
||||
t.Fatalf("%s lacks %s", raw, want)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(heldFlag, `--held "$MESH_CHECK_HELD"`) || !strings.HasPrefix(heldFlag, "${MESH_CHECK_HELD:+") {
|
||||
t.Fatalf("the flag is %s", heldFlag)
|
||||
}
|
||||
}
|
||||
@@ -1,429 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheThrowawayStoreFlushesNothing(t *testing.T) {
|
||||
said := strings.Join(StoreSettings, " ")
|
||||
for _, want := range []string{"-c fsync=off", "-c synchronous_commit=off", "-c full_page_writes=off"} {
|
||||
if !strings.Contains(said, want) {
|
||||
t.Errorf("the store is raised with %q, not %q", said, want)
|
||||
}
|
||||
}
|
||||
if os.Getenv("MESH_TEST_DOCKER") != "1" {
|
||||
t.Skip("MESH_TEST_DOCKER=1: the settings are proven on a raised store")
|
||||
}
|
||||
id := fmt.Sprintf("check-store-%d", time.Now().UnixNano())
|
||||
labelledRun := Labelled(Command, id)
|
||||
t.Cleanup(func() { _, _ = RemoveContainersOf(context.Background(), Command, id) })
|
||||
if _, err := throwaway(t.Context(), labelledRun, Command, id+"-store", StoreImage("17.11"), 5432,
|
||||
[]string{"-e", "POSTGRES_PASSWORD=check"}, StoreSettings); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := waitFor(t.Context(), Command, id+"-store", []string{"pg_isready", "-U", "postgres"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for setting, want := range map[string]string{"fsync": "off", "synchronous_commit": "off", "full_page_writes": "off"} {
|
||||
var got string
|
||||
for try := 0; try < 10; try++ {
|
||||
out, err := exec.Command("docker", "exec", id+"-store", "psql", "-U", "postgres", "-tAc", "SHOW "+setting).CombinedOutput()
|
||||
if got = strings.TrimSpace(string(out)); err == nil {
|
||||
break
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("%s is %q on the raised store", setting, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func aCheckedRepository(t *testing.T, files map[string]string) (string, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
git := func(args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git("init", "--quiet", "-b", "main")
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
git("add", "-A")
|
||||
git("commit", "--quiet", "-m", "x")
|
||||
return dir, git("rev-parse", "HEAD")
|
||||
}
|
||||
|
||||
func labelled(id string) []string {
|
||||
out, _ := exec.Command("docker", "ps", "-aq", "--filter", "label="+BuildLabel+"="+id).Output()
|
||||
return strings.Fields(string(out))
|
||||
}
|
||||
|
||||
func TestACheckRunsTheRepositorysOwnScriptBesideTheMeshsVersionsAndLeavesNothing(t *testing.T) {
|
||||
registry := checkEnvironment(t)
|
||||
// The script proves what it was given: the facts, a store that answers, a bus that answers — and no
|
||||
// container runtime socket.
|
||||
script := `set -e
|
||||
test -s "$MESH_FACTS"
|
||||
grep -q '"bus": "2.11.17"' "$MESH_FACTS" || grep -q '"bus":"2.11.17"' "$MESH_FACTS"
|
||||
case "$MESH_TEST_POSTGRES" in postgres://*127.0.0.1:*) ;; *) echo "no store: $MESH_TEST_POSTGRES"; exit 1;; esac
|
||||
case "$MESH_TEST_NATS" in nats://127.0.0.1:*) ;; *) echo "no bus: $MESH_TEST_NATS"; exit 1;; esac
|
||||
test "$MESH_CHECK_REPOSITORY" = "novox/hq"
|
||||
test "$MESH_CHECK_CHANGED" = "a.go,b.go"
|
||||
test ! -S /var/run/docker.sock || { echo "the check holds the container runtime's socket"; exit 1; }
|
||||
echo checked
|
||||
`
|
||||
repo, head := aCheckedRepository(t, map[string]string{CheckScript: script})
|
||||
id := fmt.Sprintf("check-test-%d", time.Now().UnixNano())
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "hq", Number: 7, Paths: []string{"a.go", "b.go"}, Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Nothing of the graph touched: the gate a pass that says so, the repository's own check run.
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || v.Gate.Summary != noModule {
|
||||
t.Errorf("the gate answered %+v", v.Gate)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" || !strings.Contains(v.Report, "checked") {
|
||||
t.Fatalf("the repository's check answered %+v\n%s", v.Repo, v.Report)
|
||||
}
|
||||
if left := labelled(id); len(left) > 0 {
|
||||
t.Errorf("the check left %d container(s) behind", len(left))
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckOfAChangeTouchingMoreFilesThanOneArgumentHoldsRunsWhole(t *testing.T) { // 1247/AC5
|
||||
registry := checkEnvironment(t)
|
||||
// 3000 paths of about 70 bytes: over 128 KiB comma-joined, which no environment string can carry.
|
||||
var paths []string
|
||||
for i := 0; i < 3000; i++ {
|
||||
paths = append(paths, fmt.Sprintf("modules/m%d/vendor/github.com/nats-io/nats-server/v2/server/file_%05d.go", i%4, i))
|
||||
}
|
||||
script := `set -e
|
||||
test -s "$MESH_CHECK_CHANGED_FILE"
|
||||
test "$(wc -l < "$MESH_CHECK_CHANGED_FILE")" = 3000
|
||||
head -n 1 "$MESH_CHECK_CHANGED_FILE" | grep -q '^modules/m0/vendor/.*file_00000.go$'
|
||||
test "${#MESH_CHECK_CHANGED}" -lt 65536
|
||||
case "$MESH_CHECK_CHANGED" in modules/m0/,*) ;; *) echo "changed: $MESH_CHECK_CHANGED"; exit 1;; esac
|
||||
echo checked
|
||||
`
|
||||
repo, head := aCheckedRepository(t, map[string]string{CheckScript: script})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-many-%d", time.Now().UnixNano()), Repository: repo,
|
||||
Ref: head, Owner: "novox", Repo: "hq", Number: 8, Paths: paths, Toolchain: goToolchain}, t.TempDir(), registry,
|
||||
GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" || !strings.Contains(v.Report, "checked") {
|
||||
t.Fatalf("the repository's check answered %+v\n%s", v.Repo, v.Report)
|
||||
}
|
||||
if !strings.Contains(v.Report, EnvChangedFile) {
|
||||
t.Errorf("the report does not say the list went into a file:\n%s", v.Report)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailingCheckFailsAndOneThatCannotRunIsNeverAPass(t *testing.T) {
|
||||
registry := checkEnvironment(t)
|
||||
repo, head := aCheckedRepository(t, map[string]string{CheckScript: "echo 'resource \"x.service\": refused'; exit 3\n"})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-fail-%d", time.Now().UnixNano()),
|
||||
Repository: repo, Ref: head, Owner: "novox", Repo: "hq", Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "fail" || !strings.Contains(v.Repo.Summary, "refused") {
|
||||
t.Fatalf("a failing script answered %+v", v.Repo)
|
||||
}
|
||||
|
||||
// A script in a toolchain the mesh does not hold: an error, never a pass.
|
||||
repo, head = aCheckedRepository(t, map[string]string{CheckScript: "# mesh-check-toolchain: cobol\nexit 0\n"})
|
||||
v, err = Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-cobol-%d", time.Now().UnixNano()),
|
||||
Repository: repo, Ref: head, Owner: "novox", Repo: "hq", Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil || v.Repo == nil || v.Repo.Verdict != "error" {
|
||||
t.Fatalf("a script in a toolchain nobody holds answered %+v, %v", v.Repo, err)
|
||||
}
|
||||
|
||||
// Past its bound: an error, not a pass.
|
||||
was := CheckTimeout
|
||||
CheckTimeout = 25 * time.Second
|
||||
t.Cleanup(func() { CheckTimeout = was })
|
||||
repo, head = aCheckedRepository(t, map[string]string{CheckScript: "sleep 120\n"})
|
||||
v, err = Check(context.Background(), Command, CheckSpec{ID: fmt.Sprintf("check-slow-%d", time.Now().UnixNano()),
|
||||
Repository: repo, Ref: head, Owner: "novox", Repo: "hq", Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err == nil && (v.Repo == nil || v.Repo.Verdict != "error") {
|
||||
t.Fatalf("a check past its bound answered %+v", v.Repo)
|
||||
}
|
||||
|
||||
// No facts: it cannot run, and says so.
|
||||
repo, head = aCheckedRepository(t, map[string]string{CheckScript: "exit 0\n"})
|
||||
_, err = Check(t.Context(), Command, CheckSpec{ID: "check-nofacts", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "hq", Toolchain: goToolchain}, t.TempDir(), "127.0.0.1:1", GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "facts snapshot") {
|
||||
t.Fatalf("a check with no facts said %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckWithNothingToRunSaysSo(t *testing.T) {
|
||||
repo, head := aCheckedRepository(t, map[string]string{"README.md": "x"})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-nothing", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "hq"}, t.TempDir(), "", GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || v.Repo == nil || v.Repo.Verdict != "warning" ||
|
||||
!strings.Contains(v.Repo.Summary, CheckScript) {
|
||||
t.Fatalf("a check with nothing to run said %+v / %+v", v.Gate, v.Repo)
|
||||
}
|
||||
// A gated change never takes that path: with no store to read the facts from, it cannot run.
|
||||
_, err = Check(t.Context(), Command, CheckSpec{ID: "check-gated", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-catalog", Modules: []string{"gitea"}}, t.TempDir(), "", GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a change touching a module ran nothing and was not refused")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGateRunsWhenTheGraphIsTouchedBesideTheRepositorysOwnCheck(t *testing.T) {
|
||||
registry := checkEnvironment(t)
|
||||
judgeRepo, judgeHead := aCheckedRepository(t, aJudge)
|
||||
beside := map[string]Beside{"mesh-controller": {Repository: judgeRepo, Ref: judgeHead}}
|
||||
check := func(files map[string]string, judge string) CheckVerdict {
|
||||
t.Helper()
|
||||
repo, head := aCheckedRepository(t, files)
|
||||
id := fmt.Sprintf("check-gate-%d", time.Now().UnixNano())
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-catalog", Number: 9, Paths: []string{"modules/gitea/index.ts"}, Beside: beside,
|
||||
Modules: []string{"gitea"}, Manifests: []string{"modules/gitea/module.json"}, Judge: judge,
|
||||
Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if left := labelled(id); len(left) > 0 {
|
||||
t.Errorf("the check left %d container(s) behind", len(left))
|
||||
}
|
||||
return v
|
||||
}
|
||||
v := check(map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`, CheckScript: "echo own; exit 0\n"}, "")
|
||||
if v.Gate == nil || v.Gate.Verdict != "warning" || v.Gate.Summary != "a merge rebuilds 14 module(s)" ||
|
||||
strings.Join(v.Gate.Modules, ",") != "gitea" || v.Verdict != "warning" {
|
||||
t.Fatalf("the gate answered %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" {
|
||||
t.Fatalf("its own check answered %+v", v.Repo)
|
||||
}
|
||||
|
||||
v = check(map[string]string{"modules/gitea/module.json": `{"module":"gitea","broken":true}`}, "")
|
||||
if v.Gate.Verdict != "fail" || !strings.Contains(v.Gate.Summary, "module check") || v.Repo.Verdict != "warning" {
|
||||
t.Fatalf("a manifest the judge refuses answered %+v / %+v\n%s", v.Gate, v.Repo, v.Report)
|
||||
}
|
||||
|
||||
// A fault the base branch already had is said and fails nothing; one the change brings fails.
|
||||
repo, _ := aCheckedRepository(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea","broken":true}`})
|
||||
git := func(args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = repo
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git("checkout", "--quiet", "-b", "change")
|
||||
if err := os.WriteFile(filepath.Join(repo, "modules/gitea/index.ts"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git("add", "-A")
|
||||
git("commit", "--quiet", "-m", "y")
|
||||
id := fmt.Sprintf("check-base-%d", time.Now().UnixNano())
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: repo, Ref: git("rev-parse", "HEAD"), Owner: "novox",
|
||||
Repo: "mesh-catalog", Base: "main", Paths: []string{"modules/gitea/index.ts"}, Beside: beside,
|
||||
Modules: []string{"gitea"}, Manifests: []string{"modules/gitea/module.json"}, Toolchain: goToolchain},
|
||||
t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate.Verdict != "warning" || !strings.Contains(v.Report, "already") {
|
||||
t.Fatalf("a fault the base already had failed the change: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
|
||||
// A delivery group (novox/hq ADR 0239): the other heads cloned beside it at their heads and handed to the
|
||||
// gate to compose with this one; the repository's own check is each pull request's, not the group's.
|
||||
app, appHead := aCheckedRepository(t, map[string]string{"module.json": `{"module":"app"}`})
|
||||
repo2, head2 := aCheckedRepository(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "exit 1\n"})
|
||||
gid := fmt.Sprintf("check-group-%d", time.Now().UnixNano())
|
||||
v, err = Check(t.Context(), Command, CheckSpec{ID: gid, Repository: repo2, Ref: head2, Owner: "novox",
|
||||
Repo: "mesh-catalog", Paths: []string{"modules/gitea/module.json"}, Beside: beside, Modules: []string{"gitea"},
|
||||
Manifests: []string{"modules/gitea/module.json"}, Toolchain: goToolchain,
|
||||
Group: []GroupHead{{Owner: "novox", Repo: "app", Repository: app, Ref: appHead, Paths: []string{"module.json"}}}},
|
||||
t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate.Verdict != "pass" || v.Gate.Summary != "composed with novox/app" {
|
||||
t.Fatalf("the group's other head was not composed: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" || !strings.Contains(v.Repo.Summary, "group") {
|
||||
t.Fatalf("a group's check ran a member's own script: %+v", v.Repo)
|
||||
}
|
||||
|
||||
// A change to the controller judges itself: one that does not build fails its own gate.
|
||||
v = check(map[string]string{"go.mod": "module x\n\ngo 1.22\n", "cmd/mesh-controller/main.go": "package main\nfunc main() { nope }\n",
|
||||
"modules/gitea/module.json": "{}"}, "self")
|
||||
if v.Gate.Verdict != "fail" || !strings.Contains(v.Gate.Summary, "does not build") {
|
||||
t.Fatalf("a controller that does not build judged itself %+v", v.Gate)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildSaysWhetherItsCommitIsOnTheTrunk(t *testing.T) {
|
||||
repo, onMain := aCheckedRepository(t, map[string]string{"module.json": `{"module":"x","version":"1"}`})
|
||||
git := func(dir string, args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git(repo, "checkout", "--quiet", "-b", "feature")
|
||||
if err := os.WriteFile(filepath.Join(repo, "x"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "--quiet", "-m", "off the trunk")
|
||||
offMain := git(repo, "rev-parse", "HEAD")
|
||||
git(repo, "checkout", "--quiet", "main")
|
||||
|
||||
clone := filepath.Join(t.TempDir(), "clone")
|
||||
git(filepath.Dir(clone), "clone", "--quiet", repo, clone)
|
||||
if trunk, on := trunkOf(t.Context(), Command, clone, onMain); trunk != "main" || !on {
|
||||
t.Errorf("a commit on main reads as on %q: %v", trunk, on)
|
||||
}
|
||||
if trunk, on := trunkOf(t.Context(), Command, clone, offMain); trunk != "main" || on {
|
||||
t.Errorf("a feature branch's commit reads as on %q: %v", trunk, on)
|
||||
}
|
||||
if got := strings.Join(branchesHolding(t.Context(), Command, clone, offMain), ","); got != "feature" {
|
||||
t.Errorf("the feature branch's commit is said to be on %q", got)
|
||||
}
|
||||
if got := strings.Join(branchesHolding(t.Context(), Command, clone, onMain), ","); got != "feature,main" {
|
||||
t.Errorf("main's commit is said to be on %q", got)
|
||||
}
|
||||
// A tree that says no trunk is not known — never read as on it.
|
||||
if trunk, on := trunkOf(t.Context(), Command, repo, onMain); trunk != "" || on {
|
||||
t.Errorf("a repository with no origin reads as trunk %q, on %v", trunk, on)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARedeliveredCheckRemovesWhatItsEarlierDeliveryLeft(t *testing.T) {
|
||||
registry := checkEnvironment(t)
|
||||
repo, head := aCheckedRepository(t, map[string]string{CheckScript: "echo checked\n"})
|
||||
id := fmt.Sprintf("check-again-%d", time.Now().UnixNano())
|
||||
if out, err := exec.Command("docker", "run", "-d", "--rm", "--label", BuildLabel+"="+id, "--name", id+"-store",
|
||||
"postgres:17-alpine", "sleep", "300").CombinedOutput(); err != nil {
|
||||
t.Skipf("no container for the earlier delivery: %v %s", err, out)
|
||||
}
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "hq", Number: 7, Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a redelivered check could not run: %v", err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" {
|
||||
t.Errorf("the repository's check answered %+v", v.Repo)
|
||||
}
|
||||
if left := labelled(id); len(left) > 0 {
|
||||
t.Errorf("the check left %d container(s) behind", len(left))
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryInTwoLanguagesIsCheckedInBoth(t *testing.T) {
|
||||
script := "#!/bin/sh\n# mesh-check-toolchain: typescript\n# mesh-check-also: go replays/merge-check.sh\nnpm test\n"
|
||||
parts := ScriptParts([]byte(script))
|
||||
if len(parts) != 2 || parts[0] != (ScriptPart{"typescript", CheckScript}) ||
|
||||
parts[1] != (ScriptPart{"go", "replays/merge-check.sh"}) {
|
||||
t.Fatalf("the parts read as %+v", parts)
|
||||
}
|
||||
if got := ScriptParts([]byte("#!/bin/sh\nset -eu\n")); len(got) != 1 || got[0] != (ScriptPart{"go", CheckScript}) {
|
||||
t.Fatalf("a script declaring nothing reads as %+v", got)
|
||||
}
|
||||
|
||||
// Each part run where its toolchain is: here, the image the part was given is what the command says.
|
||||
held := CheckSpec{Toolchain: "go-image", Toolchains: map[string]string{"typescript": "ts-image", "go": "go-image"}}
|
||||
run := func(t *testing.T, spec CheckSpec, files map[string]string, parts []ScriptPart) (*Layer, []string) {
|
||||
t.Helper()
|
||||
tree := t.TempDir()
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(tree, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(tree, name), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
var images []string
|
||||
var out tail
|
||||
layer := ownCheck(t.Context(), spec, parts, tree, &out, func() bool { return false },
|
||||
func(image, script string) *exec.Cmd {
|
||||
images = append(images, image+" "+script)
|
||||
cmd := exec.CommandContext(t.Context(), "sh", script)
|
||||
cmd.Dir = tree
|
||||
return cmd
|
||||
}, func(string, string, ...any) {})
|
||||
return layer, images
|
||||
}
|
||||
twoParts := []ScriptPart{{"typescript", CheckScript}, {"go", "replays/merge-check.sh"}}
|
||||
|
||||
layer, images := run(t, held, map[string]string{CheckScript: "exit 0\n", "replays/merge-check.sh": "exit 0\n"}, twoParts)
|
||||
if layer.Verdict != "pass" || !strings.Contains(layer.Summary, "replays/merge-check.sh (go)") ||
|
||||
strings.Join(images, ", ") != "ts-image merge-check.sh, go-image replays/merge-check.sh" {
|
||||
t.Errorf("both parts passing answered %+v, ran %v", layer, images)
|
||||
}
|
||||
layer, _ = run(t, held, map[string]string{CheckScript: "exit 0\n",
|
||||
"replays/merge-check.sh": "echo 'not gofmt'\"'\"'d:'; echo register.go; exit 1\n"}, twoParts)
|
||||
if layer.Verdict != "fail" || !strings.Contains(layer.Summary, "replays/merge-check.sh failed") ||
|
||||
!strings.Contains(layer.Summary, "register.go") || !strings.Contains(layer.Summary, "merge-check.sh (typescript) passed") {
|
||||
t.Errorf("a failing Go part answered %+v", layer)
|
||||
}
|
||||
layer, images = run(t, held, map[string]string{CheckScript: "exit 2\n", "replays/merge-check.sh": "exit 0\n"}, twoParts)
|
||||
if layer.Verdict != "fail" || len(images) != 1 {
|
||||
t.Errorf("a failing first part answered %+v and ran %v", layer, images)
|
||||
}
|
||||
layer, _ = run(t, held, map[string]string{CheckScript: "exit 0\n"}, twoParts)
|
||||
if layer.Verdict != "fail" || !strings.Contains(layer.Summary, "does not hold") {
|
||||
t.Errorf("a declared part the repository lacks answered %+v", layer)
|
||||
}
|
||||
layer, _ = run(t, held, map[string]string{CheckScript: "exit 0\n"}, []ScriptPart{{"typescript", CheckScript},
|
||||
{"go", "../elsewhere.sh"}})
|
||||
if layer.Verdict != "fail" || !strings.Contains(layer.Summary, "not a path inside") {
|
||||
t.Errorf("a part outside the repository answered %+v", layer)
|
||||
}
|
||||
layer, _ = run(t, CheckSpec{Toolchains: map[string]string{"typescript": "ts-image"}},
|
||||
map[string]string{CheckScript: "exit 0\n", "replays/merge-check.sh": "exit 0\n"}, twoParts)
|
||||
if layer.Verdict != "error" || !strings.Contains(layer.Summary, "go toolchain") {
|
||||
t.Errorf("a part in a toolchain the mesh does not hold answered %+v — never a pass", layer)
|
||||
}
|
||||
}
|
||||
@@ -1,115 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// **A merge check judges the merge, not the head** (novox/hq ADR 0323, issue 341).
|
||||
//
|
||||
// On 2026-10-09 a node-engine pull request was checked green at its head and broke the trunk when it merged:
|
||||
// a merge an hour earlier had added a field under the same JSON key on other lines, git merged the two edits
|
||||
// without a conflict, and the result did not compile. The check had judged the head alone. So a check now
|
||||
// takes the trunk the pull request merges into, its head at the moment the check clones it as the check's
|
||||
// **trunk commit**, and makes in its own clone the merge of the head into that commit, as the forge's
|
||||
// default merge would. The gate and the repository's own check judge that merge.
|
||||
//
|
||||
// **The merge is only checked** (ADR 0323 §2, ADR 0238 §1): it is a commit off the trunk, made in the
|
||||
// check's own clone, never pushed to the forge and never registered; the clone is removed when the check
|
||||
// ends. Nothing here writes to a remote.
|
||||
//
|
||||
// The forge keeps no merge reference for a pull request, only its head, so the builder makes it itself.
|
||||
|
||||
// TrunkMerge is the merge a check made: the head and the trunk commit it was merged into, whole, and the merge
|
||||
// commit; or the files in conflict when the head does not merge cleanly.
|
||||
type TrunkMerge struct {
|
||||
Head, Trunk, Merge string
|
||||
Conflicts []string
|
||||
}
|
||||
|
||||
// JudgedHead is one head a check judged, with the trunk commit it merged that head into, each as twelve
|
||||
// characters (ADR 0323 §3, §6).
|
||||
type JudgedHead struct {
|
||||
Owner, Repo string
|
||||
Head, Trunk string
|
||||
}
|
||||
|
||||
// mergeIdentity is who a check's merge is made as: it is never pushed, so it names nobody's account. No hook of the
|
||||
// clone's or the build machine's git configuration runs, and no setting there changes the merge.
|
||||
var mergeIdentity = []string{"-c", "user.name=mesh merge check", "-c", "user.email=merge-check@mesh.invalid",
|
||||
"-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "-c", "merge.verifySignatures=false"}
|
||||
|
||||
// mergeIntoTrunk makes, in the clone at tree, the merge of head into the head of origin/<base> as it stands in
|
||||
// that fresh clone: the trunk commit. A head that does not merge cleanly answers the files in conflict and leaves
|
||||
// the tree at the trunk commit; an error is that the merge could not be tried at all.
|
||||
func mergeIntoTrunk(ctx context.Context, run Runner, tree, base, head string) (TrunkMerge, error) {
|
||||
var m TrunkMerge
|
||||
revParse := func(ref string) (string, error) {
|
||||
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", ref+"^{commit}")
|
||||
return strings.TrimSpace(out), err
|
||||
}
|
||||
var err error
|
||||
if m.Head, err = revParse(head); err != nil || m.Head == "" {
|
||||
return m, fmt.Errorf("the head %s cannot be read in the check's clone: %v", head, err)
|
||||
}
|
||||
if m.Trunk, err = revParse("refs/remotes/origin/" + base); err != nil || m.Trunk == "" {
|
||||
return m, fmt.Errorf("the trunk %s cannot be read, and a check judges the head merged into it "+
|
||||
"(novox/hq ADR 0323): %v", base, err)
|
||||
}
|
||||
if _, err := run(ctx, tree, "git", "checkout", "--quiet", "--detach", m.Trunk); err != nil {
|
||||
return m, fmt.Errorf("the trunk commit %s cannot be checked out: %w", twelve(m.Trunk), err)
|
||||
}
|
||||
message := fmt.Sprintf("Check the merge of %s into %s at %s (made for a merge check, never pushed)",
|
||||
twelve(m.Head), base, twelve(m.Trunk))
|
||||
args := append(append([]string{}, mergeIdentity...), "merge", "--no-ff", "--no-edit", "--no-verify", "--quiet", "-m", message, m.Head)
|
||||
if _, err := run(ctx, tree, "git", args...); err != nil {
|
||||
out, _ := run(ctx, tree, "git", "diff", "--name-only", "--diff-filter=U")
|
||||
m.Conflicts = strings.Fields(out)
|
||||
_, _ = run(ctx, tree, "git", "merge", "--abort")
|
||||
if len(m.Conflicts) == 0 {
|
||||
return m, fmt.Errorf("the head %s could not be merged into trunk %s: %w", twelve(m.Head), twelve(m.Trunk), err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
if m.Merge, err = revParse("HEAD"); err != nil {
|
||||
return m, fmt.Errorf("the merge made for the check cannot be read: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// changedByMerge is every path the merge changes against the trunk commit: what the gate's delivery plan is
|
||||
// computed from (ADR 0323 §1). Nil when it cannot be read; the caller keeps the pull request's own list.
|
||||
func changedByMerge(ctx context.Context, run Runner, tree string, m TrunkMerge) []string {
|
||||
if m.Merge == "" || m.Trunk == "" {
|
||||
return nil
|
||||
}
|
||||
out, err := run(ctx, tree, "git", "diff", "--name-only", "--no-renames", m.Trunk, m.Merge)
|
||||
if err != nil {
|
||||
return nil // empty-on-error: the caller keeps the pull request's own list of changed files, the safe one
|
||||
}
|
||||
return strings.Fields(out)
|
||||
}
|
||||
|
||||
// conflictWords is a head that does not merge cleanly, said as its check's failure.
|
||||
func conflictWords(m TrunkMerge) string {
|
||||
return fmt.Sprintf("head %s does not merge cleanly into trunk %s: in conflict %s", twelve(m.Head), twelve(m.Trunk),
|
||||
strings.Join(m.Conflicts, ", "))
|
||||
}
|
||||
|
||||
// headAt is the commit a clone is checked out at, as twelve characters; empty when it cannot be read.
|
||||
func headAt(ctx context.Context, run Runner, tree string) string {
|
||||
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", "HEAD^{commit}")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return twelve(strings.TrimSpace(out))
|
||||
}
|
||||
|
||||
// twelve is a commit as a verdict names it: its first twelve characters.
|
||||
func twelve(commit string) string {
|
||||
if len(commit) > 12 {
|
||||
return commit[:12]
|
||||
}
|
||||
return commit
|
||||
}
|
||||
@@ -1,264 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// **A merge check judges the merge, not the head** (novox/hq ADR 0323, issue 341): the head merged into the trunk
|
||||
// commit in the check's own clone, never pushed; a head that does not merge cleanly fails, naming the files; the
|
||||
// verdict names the head, the trunk commit and the commits of the repositories read beside it.
|
||||
|
||||
// aTrunkAndABranch is a repository whose main moved after the branch was cut: main holds trunkFiles over the
|
||||
// common files, the branch headFiles. It answers the repository, the branch's head and main's head.
|
||||
func aTrunkAndABranch(t *testing.T, common, trunkFiles, headFiles map[string]string) (string, string, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
git := func(args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
write := func(files map[string]string) {
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
git("add", "-A")
|
||||
}
|
||||
git("init", "--quiet", "-b", "main")
|
||||
write(common)
|
||||
git("commit", "--quiet", "-m", "common")
|
||||
git("checkout", "--quiet", "-b", "change")
|
||||
write(headFiles)
|
||||
git("commit", "--quiet", "-m", "the change")
|
||||
head := git("rev-parse", "HEAD")
|
||||
git("checkout", "--quiet", "main")
|
||||
write(trunkFiles)
|
||||
git("commit", "--quiet", "-m", "the trunk moved")
|
||||
return dir, head, git("rev-parse", "HEAD")
|
||||
}
|
||||
|
||||
func TestACheckJudgesTheHeadMergedIntoTheTrunk(t *testing.T) {
|
||||
// main's script passes only in a tree holding both the trunk's file and the head's: the merge.
|
||||
repo, head, trunk := aTrunkAndABranch(t,
|
||||
map[string]string{CheckScript: "test -f trunk-only && test -f head-only || { echo 'FAIL: not the merge'; exit 1; }\n"},
|
||||
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
|
||||
beside, besideHead := aCheckedRepository(t, map[string]string{"README": "the catalogue"})
|
||||
registry := aFactsRegistry(t)
|
||||
aRuntimeOnThisMachine(t)
|
||||
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-lab", Number: 1, Base: "main", Toolchain: "golang",
|
||||
Beside: map[string]Beside{"mesh-catalog": {Repository: beside, Ref: besideHead}}}, t.TempDir(), registry,
|
||||
GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" {
|
||||
t.Fatalf("the check did not judge the head merged into the trunk: %+v\n%s", v.Repo, v.Report)
|
||||
}
|
||||
if v.Head != head[:12] || v.Trunk != trunk[:12] {
|
||||
t.Fatalf("the verdict names head %q and trunk %q, not %s and %s", v.Head, v.Trunk, head[:12], trunk[:12])
|
||||
}
|
||||
if v.Beside["mesh-catalog"] != besideHead[:12] {
|
||||
t.Fatalf("the verdict names the catalogue read beside it as %q, not %s: %v", v.Beside["mesh-catalog"],
|
||||
besideHead[:12], v.Beside)
|
||||
}
|
||||
if v.Merge == "" || v.Merge == head || v.Merge == trunk {
|
||||
t.Fatalf("no merge of the head into the trunk was made for the check: %q", v.Merge)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeadThatDoesNotMergeCleanlyFailsNamingTheFiles(t *testing.T) {
|
||||
repo, head, trunk := aTrunkAndABranch(t,
|
||||
map[string]string{"store.go": "package store\n\nconst Declared = 1\n", "other.go": "package store\n"},
|
||||
map[string]string{"store.go": "package store\n\nconst Declared = 2\n"},
|
||||
map[string]string{"store.go": "package store\n\nconst Declared = 3\n"})
|
||||
|
||||
// No artifact store and no container runtime: a head that does not merge is judged before either is needed.
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-conflict", Repository: repo, Ref: head,
|
||||
Owner: "novox", Repo: "mesh-lab", Number: 2, Base: "main", Toolchain: "golang"}, t.TempDir(), "",
|
||||
GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a head that does not merge cleanly was not judged: %v", err)
|
||||
}
|
||||
want := "does not merge cleanly into trunk " + trunk[:12]
|
||||
for name, l := range map[string]*Layer{"gate": v.Gate, "repository": v.Repo} {
|
||||
if l == nil || l.Verdict != "fail" || !strings.Contains(l.Summary, want) || !strings.Contains(l.Summary, "store.go") {
|
||||
t.Fatalf("its %s layer said %+v, not a failure saying %q and naming store.go", name, l, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(v.Summary, "other.go") {
|
||||
t.Fatalf("a file not in conflict is named: %s", v.Summary)
|
||||
}
|
||||
if v.Head != head[:12] || v.Trunk != trunk[:12] {
|
||||
t.Fatalf("the failure names head %q and trunk %q", v.Head, v.Trunk)
|
||||
}
|
||||
}
|
||||
|
||||
// **The merge made for a check is only checked** (ADR 0323 §2): the forge never receives it.
|
||||
func TestTheMergeMadeForACheckIsNeverPushed(t *testing.T) {
|
||||
repo, head, trunk := aTrunkAndABranch(t, map[string]string{CheckScript: "true\n"},
|
||||
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
|
||||
registry := aFactsRegistry(t)
|
||||
aRuntimeOnThisMachine(t)
|
||||
refsBefore := gitOut(t, repo, "for-each-ref")
|
||||
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-push", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-lab", Number: 3, Base: "main", Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Merge == "" {
|
||||
t.Fatal("the check made no merge of the head into the trunk")
|
||||
}
|
||||
if err := exec.Command("git", "-C", repo, "cat-file", "-e", v.Merge+"^{commit}").Run(); err == nil {
|
||||
t.Fatalf("the merge %s made for the check reached the forge's repository", v.Merge)
|
||||
}
|
||||
if after := gitOut(t, repo, "for-each-ref"); after != refsBefore {
|
||||
t.Fatalf("the check moved a ref of the forge's repository:\nbefore %s\nafter %s", refsBefore, after)
|
||||
}
|
||||
if got := gitOut(t, repo, "rev-parse", "main"); got != trunk {
|
||||
t.Fatalf("the trunk moved to %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A delivery group's composed check merges each member's head into its own trunk's head (ADR 0323 §6), and names
|
||||
// both for each; a member that does not merge cleanly fails the composed check, naming it.
|
||||
func TestAGroupsComposedCheckMergesEachMemberIntoItsTrunk(t *testing.T) {
|
||||
repo, head, trunk := aTrunkAndABranch(t, map[string]string{CheckScript: "true\n"},
|
||||
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
|
||||
member, memberHead, memberTrunk := aTrunkAndABranch(t, map[string]string{"a": "1"},
|
||||
map[string]string{"b": "trunk"}, map[string]string{"c": "head"})
|
||||
registry := aFactsRegistry(t)
|
||||
aRuntimeOnThisMachine(t)
|
||||
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-341-group", Repository: repo, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-controller", Number: 4, Base: "main", Toolchain: "golang", Group: []GroupHead{{Owner: "novox",
|
||||
Repo: "mesh-host", Repository: member, Ref: memberHead, Base: "main"}}}, t.TempDir(), registry,
|
||||
GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Head != head[:12] || v.Trunk != trunk[:12] {
|
||||
t.Fatalf("the composed check names its first head %q into %q", v.Head, v.Trunk)
|
||||
}
|
||||
if len(v.Members) != 1 || v.Members[0].Head != memberHead[:12] || v.Members[0].Trunk != memberTrunk[:12] {
|
||||
t.Fatalf("the composed check names its members %+v, not mesh-host %s into %s", v.Members, memberHead[:12],
|
||||
memberTrunk[:12])
|
||||
}
|
||||
|
||||
conflicting, conflictHead, _ := aTrunkAndABranch(t, map[string]string{"a": "1"}, map[string]string{"a": "2"},
|
||||
map[string]string{"a": "3"})
|
||||
v, err = Check(t.Context(), Command, CheckSpec{ID: "check-341-group-conflict", Repository: repo, Ref: head,
|
||||
Owner: "novox", Repo: "mesh-controller", Number: 4, Base: "main", Toolchain: "golang", Group: []GroupHead{{
|
||||
Owner: "novox", Repo: "mesh-host", Repository: conflicting, Ref: conflictHead, Base: "main"}}}, t.TempDir(),
|
||||
registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Verdict != "fail" || !strings.Contains(v.Summary, "novox/mesh-host") ||
|
||||
!strings.Contains(v.Summary, "does not merge cleanly") {
|
||||
t.Fatalf("a member that does not merge cleanly was answered %s: %s", v.Verdict, v.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
func gitOut(t *testing.T, dir string, args ...string) string {
|
||||
t.Helper()
|
||||
out, err := exec.Command("git", append([]string{"-C", dir}, args...)...).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
|
||||
// No hook runs when a check makes its merge: not one in the clone, whatever the build machine's git configuration says.
|
||||
func TestACheckMergeRunsNoHook(t *testing.T) {
|
||||
repo, head, _ := aTrunkAndABranch(t, map[string]string{"a": "1"}, map[string]string{"b": "trunk"},
|
||||
map[string]string{"c": "head"})
|
||||
clone := filepath.Join(t.TempDir(), "clone")
|
||||
if out, err := exec.Command("git", "clone", "--quiet", repo, clone).CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
ran := filepath.Join(t.TempDir(), "ran")
|
||||
for _, hook := range []string{"pre-merge-commit", "commit-msg", "prepare-commit-msg", "post-merge"} {
|
||||
if err := os.WriteFile(filepath.Join(clone, ".git", "hooks", hook),
|
||||
[]byte("#!/bin/sh\necho "+hook+" >> "+ran+"\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
m, err := mergeIntoTrunk(t.Context(), Command, clone, "main", head)
|
||||
if err != nil || m.Merge == "" {
|
||||
t.Fatalf("the merge was not made: %+v, %v", m, err)
|
||||
}
|
||||
if said, err := os.ReadFile(ran); err == nil {
|
||||
t.Fatalf("a hook ran when the check made its merge: %s", said)
|
||||
}
|
||||
}
|
||||
|
||||
// What a check's tests change in the workspace never reaches what it builds for the merge (novox/hq issue 703): the
|
||||
// builds are made from a copy of the merge, outside the workspace, cloned before anything of the change runs.
|
||||
func TestWhatTheTestsChangeInTheWorkspaceNeverReachesTheBuild(t *testing.T) {
|
||||
repo, head, _ := aTrunkAndABranch(t, map[string]string{"main.go": "package main // the merge\n"},
|
||||
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
|
||||
workspace := t.TempDir()
|
||||
tree := filepath.Join(workspace, "check", "checked")
|
||||
if out, err := exec.Command("git", "clone", "--quiet", repo, tree).CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
dir, cloned, err := checkoutForTheBuilds(t.Context(), Command, tree, head, workspace)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
if withinDir(workspace, cloned) {
|
||||
t.Fatalf("the builds' checkout %s is inside the workspace %s the tests mount", cloned, workspace)
|
||||
}
|
||||
|
||||
// The tests rewrite the tree, its branch and every object it holds.
|
||||
if err := os.WriteFile(filepath.Join(tree, "main.go"), []byte("package main // planted\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := filepath.WalkDir(filepath.Join(tree, ".git", "objects"), func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
_ = os.Chmod(path, 0o644)
|
||||
return os.WriteFile(path, []byte("planted"), 0o644)
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// The build clones the merge from its own checkout, as buildForTheMerge does, and reads what the merge holds.
|
||||
build := filepath.Join(t.TempDir(), "source")
|
||||
if out, err := exec.Command("git", "clone", "--quiet", cloned, build).CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
if out, err := exec.Command("git", "-C", build, "checkout", "--quiet", head).CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
body, err := os.ReadFile(filepath.Join(build, "main.go"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(body) != "package main // the merge\n" {
|
||||
t.Fatalf("the build read %q: what the tests wrote reached it", body)
|
||||
}
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Issue 1391, AC5: what a check says names the validator as mesh-sdk's at the controller's pin, with the version from the
|
||||
// judge's own go.mod.
|
||||
func TestTheValidatorIsNamedAsMeshSDKsAtTheControllersPin(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
mod := "module example.test/c\n\ngo 1.25\n\nrequire (\n\tgit.novox.be/novox/mesh-sdk/go v0.1.14-0.20261020085642-ab0c16ddad26\n)\n"
|
||||
if err := os.WriteFile(filepath.Join(dir, "go.mod"), []byte(mod), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, want := validatorAt(dir), "mesh-sdk's validator at v0.1.14-0.20261020085642-ab0c16ddad26"; got != want {
|
||||
t.Fatalf("the check says %q, not %q", got, want)
|
||||
}
|
||||
if got, want := pinNote(dir), " (v0.1.14-0.20261020085642-ab0c16ddad26)"; got != want {
|
||||
t.Fatalf("the pin note is %q, not %q", got, want)
|
||||
}
|
||||
if got := validatorAt(t.TempDir()); got != "mesh-sdk's validator at the controller's pin" {
|
||||
t.Fatalf("a controller with no go.mod is said as %q", got)
|
||||
}
|
||||
|
||||
// A placeholder or a replaced pin is no version: it is said as a local mesh-sdk.
|
||||
for name, mod := range map[string]string{
|
||||
"placeholder": "module example.test/c\n\nrequire git.novox.be/novox/mesh-sdk/go v0.0.0\n",
|
||||
"replaced": "module example.test/c\n\nrequire git.novox.be/novox/mesh-sdk/go v0.1.14\n\n" +
|
||||
"replace git.novox.be/novox/mesh-sdk/go => ../mesh-sdk/go\n",
|
||||
} {
|
||||
d := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(d, "go.mod"), []byte(mod), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := validatorAt(d); !strings.Contains(got, "a local mesh-sdk") || !strings.Contains(got, "replaced") {
|
||||
t.Fatalf("%s: the check says %q, which does not say the mesh-sdk is local", name, got)
|
||||
}
|
||||
if got := pinNote(d); !strings.HasPrefix(got, " (a local mesh-sdk, v") {
|
||||
t.Fatalf("%s: the pin note is %q", name, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,227 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/facts"
|
||||
"github.com/novox/mesh-controller/internal/artifacts"
|
||||
)
|
||||
|
||||
// A merge check on the build seat (novox/hq to-be 45 §9), against a real container runtime and a real
|
||||
// registry: the repository's own merge-check.sh runs with the facts the controller keeps, beside a
|
||||
// throwaway store and bus of **the versions the mesh runs**, and everything raised is removed.
|
||||
//
|
||||
// MESH_TEST_DOCKER=1 MESH_TEST_REGISTRY=127.0.0.1:15000 go test ./internal/builder -run Check
|
||||
|
||||
func TestTheStoreAndBusAChecksStandsOnAreTheOnesTheMeshRuns(t *testing.T) {
|
||||
if got := StoreImage("17.11"); got != "postgres:17-alpine" {
|
||||
t.Errorf("a store at 17.11 is checked against %s", got)
|
||||
}
|
||||
if got := StoreImage("16.4 (Debian 16.4-1.pgdg120+1)"); got != "postgres:16-alpine" {
|
||||
t.Errorf("a store at 16.4 is checked against %s", got)
|
||||
}
|
||||
if got := BusImage("2.11.17"); got != "nats:2.11.17-alpine" {
|
||||
t.Errorf("a bus at 2.11.17 is checked against %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 306**: the throwaway store is raised without durability — it lives for one check — so a suite
|
||||
// that makes a database per test does not wait on the disk of a busy machine for every flush.
|
||||
|
||||
// aRepository is a git repository holding these files, committed, and its head.
|
||||
|
||||
func checkEnvironment(t *testing.T) string {
|
||||
t.Helper()
|
||||
if os.Getenv("MESH_TEST_DOCKER") != "1" || os.Getenv("MESH_TEST_REGISTRY") == "" {
|
||||
t.Skip("MESH_TEST_DOCKER=1 and MESH_TEST_REGISTRY: a check raises containers and reads the registry")
|
||||
}
|
||||
registry := os.Getenv("MESH_TEST_REGISTRY")
|
||||
body, err := json.Marshal(facts.Facts{Format: facts.Format, Taken: time.Now().UTC(),
|
||||
Versions: facts.Versions{Bus: "2.11.17", Store: "17.11"},
|
||||
Machines: []facts.Machine{{Name: "abcdef", Length: 6}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := (artifacts.Store{Address: registry}).PutTagged(t.Context(), facts.Repository, facts.Tag,
|
||||
facts.MediaType, body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return registry
|
||||
}
|
||||
|
||||
// goToolchain is the Go image a check's script runs in here: the base the controller's own image is built on.
|
||||
const goToolchain = "golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"
|
||||
|
||||
// A repository that touches nothing of the graph and has no script of its own runs nothing, and says
|
||||
// both: the gate a pass that names why, the repository a warning — never silent.
|
||||
|
||||
// A script declares its toolchain among its first lines; go when it declares none.
|
||||
func TestAScriptDeclaresItsToolchain(t *testing.T) {
|
||||
for script, want := range map[string]string{
|
||||
"#!/bin/sh\nset -eu\n": "go",
|
||||
"#!/bin/sh\n# mesh-check-toolchain: typescript\nnpm test\n": "typescript",
|
||||
"#!/bin/sh\n#mesh-check-toolchain:go\n": "go",
|
||||
"#!/bin/sh\necho '# mesh-check-toolchain: python'\n": "go",
|
||||
} {
|
||||
if got := ScriptToolchain([]byte(script)); got != want {
|
||||
t.Errorf("%q declares %q, read as %q", script, want, got)
|
||||
}
|
||||
}
|
||||
held := map[string]string{"mesh-tools/build": "reg/mesh-tools-build@sha256:a", "mesh-tools-go/build": "reg/go@sha256:b"}
|
||||
chains := ToolchainsOf(held)
|
||||
if chains["typescript"] != "reg/mesh-tools-build@sha256:a" || chains["go"] != "reg/go@sha256:b" || ToolchainOf(held) != chains["go"] {
|
||||
t.Fatalf("the toolchains held read as %v", chains)
|
||||
}
|
||||
if _, held := chains["python"]; held {
|
||||
t.Error("a toolchain the mesh does not hold read as held")
|
||||
}
|
||||
}
|
||||
|
||||
// aJudge is a controller that judges as told: `merge-gate` answers a warning, `module check` refuses a
|
||||
// manifest that says it is broken. What the gate layer is tested against without building the real one.
|
||||
var aJudge = map[string]string{
|
||||
"go.mod": "module example.org/judge\n\ngo 1.22\n",
|
||||
"cmd/mesh-controller/main.go": `package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func main() {
|
||||
switch {
|
||||
case len(os.Args) == 2 && os.Args[1] == "merge-gate":
|
||||
fmt.Println("usage: merge-gate --facts <file> --store <postgres>")
|
||||
os.Exit(2)
|
||||
case len(os.Args) > 2 && os.Args[1] == "module":
|
||||
for _, m := range os.Args[3:] {
|
||||
body, _ := os.ReadFile(m)
|
||||
if strings.Contains(string(body), "broken") {
|
||||
fmt.Println(m + ": refused, it says it is broken")
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println(m + ": ok")
|
||||
}
|
||||
case os.Args[1] == "merge-gate":
|
||||
for i, a := range os.Args {
|
||||
if a == "--group" && i+1 < len(os.Args) {
|
||||
body, _ := os.ReadFile(os.Args[i+1])
|
||||
var heads []struct {
|
||||
Repository string ` + "`json:\"repository\"`" + `
|
||||
Tree string ` + "`json:\"tree\"`" + `
|
||||
}
|
||||
_ = json.Unmarshal(body, &heads)
|
||||
var said []string
|
||||
for _, h := range heads {
|
||||
if _, err := os.Stat(h.Tree + "/module.json"); err == nil {
|
||||
said = append(said, h.Repository)
|
||||
}
|
||||
}
|
||||
fmt.Printf("{\"verdict\":\"pass\",\"summary\":\"composed with %s\"}\n", strings.Join(said, ","))
|
||||
return
|
||||
}
|
||||
}
|
||||
fmt.Println(` + "`" + `{"verdict":"warning","summary":"a merge rebuilds 14 module(s)"}` + "`" + `)
|
||||
}
|
||||
}
|
||||
`,
|
||||
}
|
||||
|
||||
// **Only a commit on the trunk is published** (novox/hq ADR 0238): the build seat reads, from the clone it
|
||||
// just made, the branch the forge names its default and whether the commit built is on it.
|
||||
|
||||
// **Issue 286**: a failed merge-check.sh was said by its last line — a bare "FAIL", or the name of a file
|
||||
// gofmt listed — which named nothing a reader could act on. The status says what failed.
|
||||
func TestAFailedScriptIsSaidByWhatFailed(t *testing.T) {
|
||||
for out, want := range map[string]string{
|
||||
"not gofmt'd:\ninternal/bootstrap/publish_test.go\n": "not gofmt'd by the toolchain's gofmt: internal/bootstrap/publish_test.go",
|
||||
"ok \tx/a\t1s\n--- FAIL: TestTheInstallersFirstUserList (0.59s)\n t.go:37: refused\nFAIL\nFAIL\tx/b\t1s\nFAIL\n": "--- FAIL: TestTheInstallersFirstUserList (0.59s)",
|
||||
"ok \tx/a\t1s\nFAIL\tx/b [build failed]\nFAIL\n": "FAIL\tx/b [build failed]",
|
||||
"npm ERR! missing script: typecheck\n": "npm ERR! missing script: typecheck",
|
||||
} {
|
||||
if got := whatFailed(out); got != want {
|
||||
t.Errorf("%q is said as %q, not %q", out, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 285**: a judge from before the rule passed "every machine composes with the change as it did
|
||||
// without (0 of 4 compose)". The seat reads the machines itself: a machine the mesh composes that did not
|
||||
// compose in the gate's store makes the gate an error, whatever judged it.
|
||||
func TestTheSeatCallsAGateThatRaisedNoMachineAnError(t *testing.T) {
|
||||
old := "bus users without a credential: controller\n" + `{"verdict":"pass","summary":"every machine composes with the change as it did without (0 of 2 compose)",
|
||||
"machines":[{"described":"the hub","live-composes":true,"base":{"composes":false}},
|
||||
{"described":"a machine","live-composes":true,"base":{"composes":false}}]}`
|
||||
v, ok := readGateVerdict([]byte(old))
|
||||
if !ok {
|
||||
t.Fatal("a verdict after a line of composition's was not read")
|
||||
}
|
||||
verdict, summary, raised := gateRaisedTheMesh(v)
|
||||
if raised || verdict != "error" || !strings.Contains(summary, "2 of 2 machines") || !strings.Contains(summary, "the hub") {
|
||||
t.Errorf("0 of 2 composing is %q %q", verdict, summary)
|
||||
}
|
||||
good := `{"verdict":"pass","summary":"(2 of 2 compose)","machines":[{"described":"the hub","live-composes":true,"base":{"composes":true}},
|
||||
{"described":"a laptop","live-composes":false,"base":{"composes":false}}]}`
|
||||
v, _ = readGateVerdict([]byte(good))
|
||||
if _, _, raised := gateRaisedTheMesh(v); !raised {
|
||||
t.Error("a machine that does not compose on the mesh either was read as the gate's failure")
|
||||
}
|
||||
failed := `{"verdict":"fail","summary":"x","machines":[{"described":"the hub","live-composes":true,"base":{"composes":false}}]}`
|
||||
v, _ = readGateVerdict([]byte(failed))
|
||||
if _, _, raised := gateRaisedTheMesh(v); !raised {
|
||||
t.Error("a failing verdict is the change's, and stands")
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 285**: an ask redelivered after its holder stopped mid-check found its own throwaway store still
|
||||
// there under its name, and the check said it could not run. What an earlier delivery left goes first.
|
||||
|
||||
// **A repository in two languages checks both, each in its own toolchain** (novox/hq issue 302): the lab's
|
||||
// merge-check.sh runs in the TypeScript toolchain, and its Go replays were said "NOT CHECKED HERE" on every
|
||||
// pull request, so a register that did not compile would have merged green. Its script declares the Go
|
||||
// part; the check runs it in the Go toolchain's container after the script, and the layer passes only
|
||||
// when both do.
|
||||
|
||||
// A branch tried before its pull request may narrow what the repository's script runs (novox/hq issue 686); a
|
||||
// pull request's check is handed nothing, and runs whole.
|
||||
func TestOnlyATrysSelectionReachesTheScript(t *testing.T) {
|
||||
if env := selection(CheckSpec{Number: 7}); len(env) != 0 {
|
||||
t.Fatalf("a pull request's check was handed a selection: %v", env)
|
||||
}
|
||||
env := selection(CheckSpec{Tests: "^TestX$", StoreOnly: true})
|
||||
if len(env) != 2 || env[0] != "MESH_CHECK_TESTS=^TestX$" || env[1] != "MESH_CHECK_STORE_ONLY=1" {
|
||||
t.Fatalf("a try's selection: %v", env)
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 1310**: the controller's module check prints its warning counts after its refusal; a failed check
|
||||
// is said by the refusal, never by the count printed last.
|
||||
func TestAFailedModuleCheckIsSaidByItsRefusalNotItsWarningCount(t *testing.T) {
|
||||
log := "mesh-controller: 3 problem(s) in 139 manifest(s)\n" +
|
||||
"mesh-files claims node-files and says it serves upload, download, which that seat's protocol does not promise\n" +
|
||||
"minio claims the seat \"object-store\", which no module declares and the mesh does not define\n" +
|
||||
"only-office claims the seat \"document-converter\", which no module declares and the mesh does not define\n" +
|
||||
"adwaita: ok, 4 tool(s)\nlong-running resources without health: 59\n" +
|
||||
"files asking for a setting without saying whether it is trusted: 36\n"
|
||||
// 1310/AC1
|
||||
want := "mesh-controller: 3 problem(s) in 139 manifest(s): mesh-files claims node-files and says it serves upload, " +
|
||||
"download, which that seat's protocol does not promise"
|
||||
if got := whatFailed(log); got != want {
|
||||
t.Errorf("1310/AC1: the captured log is said as %q, not %q", got, want)
|
||||
}
|
||||
// 1310/AC2
|
||||
if got, want := whatFailed("ok \tx/a\t1s\n--- FAIL: TestX (0.01s)\n x_test.go:12: got 1\nFAIL\nlong-running resources without health: 59\n"),
|
||||
"--- FAIL: TestX (0.01s)"; got != want {
|
||||
t.Errorf("1310/AC2: a failing test is said as %q, not %q", got, want)
|
||||
}
|
||||
// 1310/AC3
|
||||
if got, want := whatFailed("starting\nsomething odd happened\n"), "something odd happened"; got != want {
|
||||
t.Errorf("1310/AC3: a log with no failure line is said as %q, not %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -1,114 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A clone from the forge waits out a resolver that did not answer, for a bounded time (novox/hq issue 1773).
|
||||
//
|
||||
// **Why.** A build or a check failed now and then on "Could not resolve host: novox.internal" while cloning, between
|
||||
// clones that resolved the same name a second before and after, and the whole delivery failed with it. The lookup
|
||||
// failing is the resolver's passing fault, not the repository's: nothing reached the forge, so cloning again cannot
|
||||
// do anything twice. This is issue 457's shape (a build waits out a registry held still) applied to the name lookup.
|
||||
//
|
||||
// **Only the lookup.** Any other failure of git — authentication, a repository or a ref that is not there, a 404 —
|
||||
// fails at once, as it always did: trying it again would only make a real failure slower.
|
||||
//
|
||||
// **Bounded, and loud.** Each retry is said in the build's log as the resolver's failure, and past lookupWait the
|
||||
// build fails saying the resolver did not answer for that long.
|
||||
|
||||
var (
|
||||
// lookupWait is how long a clone waits for a resolver that does not answer.
|
||||
lookupWait = 60 * time.Second
|
||||
// lookupFirstPause is the first pause between tries; each pause doubles, up to lookupMostPause.
|
||||
lookupFirstPause = time.Second
|
||||
)
|
||||
|
||||
// lookupMostPause is the longest pause between two tries.
|
||||
const lookupMostPause = 10 * time.Second
|
||||
|
||||
// lookupFailures are what git (through curl) and Go's resolver say when a name did not resolve, in lower case.
|
||||
var lookupFailures = []string{
|
||||
"could not resolve host",
|
||||
"temporary failure in name resolution",
|
||||
}
|
||||
|
||||
// failedLookup is whether an error is a name lookup that failed: git's "Could not resolve host", the system's
|
||||
// "Temporary failure in name resolution", or a lookup that timed out.
|
||||
func failedLookup(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
said := strings.ToLower(err.Error())
|
||||
for _, f := range lookupFailures {
|
||||
if strings.Contains(said, f) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return strings.Contains(said, "lookup ") && strings.Contains(said, "i/o timeout")
|
||||
}
|
||||
|
||||
// cloneFromForge runs `git clone` of url into dest (git's arguments in args, run from dir), and while it fails on the
|
||||
// name lookup, clones again with a growing pause until lookupWait has passed. What a failed try left at dest is
|
||||
// removed before the next. Any other failure is returned at once.
|
||||
func cloneFromForge(ctx context.Context, run Runner, dir, url, dest string, args []string,
|
||||
say func(step, format string, args ...any)) error {
|
||||
_, err := run(ctx, dir, "git", args...)
|
||||
if !failedLookup(err) {
|
||||
return err
|
||||
}
|
||||
started := time.Now()
|
||||
pause := lookupFirstPause
|
||||
for {
|
||||
elapsed := time.Since(started)
|
||||
left := lookupWait - elapsed
|
||||
if left <= 0 {
|
||||
say("clone", "the resolver did not answer for %s, cloning %s, for %s; the clone fails",
|
||||
urlHost(url), url, elapsed.Round(time.Second))
|
||||
return fmt.Errorf("the resolver did not answer a lookup of %s for %s, cloning %s: %w",
|
||||
urlHost(url), elapsed.Round(time.Millisecond), url, err)
|
||||
}
|
||||
wait := min(pause, left)
|
||||
say("clone", "the resolver did not answer for %s, cloning %s; trying again in %s, for up to %s "+
|
||||
"(novox/hq issue 1773)", urlHost(url), url, wait, lookupWait)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("stopped while waiting for the resolver to answer for %s: %w (last: %v)",
|
||||
urlHost(url), ctx.Err(), err)
|
||||
case <-time.After(wait):
|
||||
}
|
||||
pause = min(pause*2, lookupMostPause)
|
||||
if err := os.RemoveAll(dest); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = run(ctx, dir, "git", args...); !failedLookup(err) {
|
||||
if err == nil {
|
||||
say("clone", "the resolver answered for %s after %s; the clone worked", urlHost(url),
|
||||
time.Since(started).Round(time.Millisecond))
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// urlHost is the host (and port) a URL names, or the URL itself when it names none.
|
||||
func urlHost(url string) string {
|
||||
rest := url
|
||||
if _, after, ok := strings.Cut(url, "://"); ok {
|
||||
rest = after
|
||||
}
|
||||
if i := strings.IndexAny(rest, "/?#"); i >= 0 {
|
||||
rest = rest[:i]
|
||||
}
|
||||
if i := strings.LastIndex(rest, "@"); i >= 0 {
|
||||
rest = rest[i+1:]
|
||||
}
|
||||
if rest == "" {
|
||||
return url
|
||||
}
|
||||
return rest
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Issue 1773: a build or a check failed now and then on "Could not resolve host: novox.internal" while cloning from
|
||||
// the forge, between clones that resolved the same name, and its whole delivery failed with it. A clone that fails on
|
||||
// the name lookup is tried again within a bound, says each retry in the build's log as the resolver's failure, and
|
||||
// only past the bound fails, saying the resolver did not answer. Any other failure of git fails at once, as before.
|
||||
|
||||
const lookupFailure = "exit status 128\nfatal: unable to access 'http://novox.internal:20000/novox/x.git/': " +
|
||||
"Could not resolve host: novox.internal"
|
||||
|
||||
// failingClones is a runner whose first n forge clones fail with failure, then clone as r does; n < 0 fails every one.
|
||||
type failingClones struct {
|
||||
r *recorded
|
||||
n int
|
||||
failure string
|
||||
clones int
|
||||
}
|
||||
|
||||
func (f *failingClones) run(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "git" && isACloneOf(args) {
|
||||
f.clones++
|
||||
if f.n < 0 || f.clones <= f.n {
|
||||
return "", fmt.Errorf("git %s: %s", strings.Join(args, " "), f.failure)
|
||||
}
|
||||
}
|
||||
return f.r.run(ctx, dir, name, args...)
|
||||
}
|
||||
|
||||
func isACloneOf(args []string) bool {
|
||||
for _, a := range args {
|
||||
if a == "clone" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// buildLog collects what a build says in its own log.
|
||||
func buildLog() (Log, func() []string) {
|
||||
var mu sync.Mutex
|
||||
var lines []string
|
||||
return func(step, message string) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
lines = append(lines, step+": "+message)
|
||||
}, func() []string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return append([]string(nil), lines...)
|
||||
}
|
||||
}
|
||||
|
||||
const nothingToBuild = `{"module":"x","version":"1"}`
|
||||
|
||||
// 1773/AC1: a clone failing twice on the lookup, then working, is tried again, each retry said, and the build goes on.
|
||||
func TestACloneThatFailsOnTheLookupIsTriedAgainAndTheBuildGoesOn(t *testing.T) {
|
||||
quickLookups(t)
|
||||
r, workspace := aRepository(t, nothingToBuild, nil)
|
||||
f := &failingClones{r: r, n: 2, failure: lookupFailure}
|
||||
log, said := buildLog()
|
||||
_, err := Build(context.Background(), f.run, r, "http://novox.internal:20000/novox/x.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, log)
|
||||
if err != nil {
|
||||
t.Fatalf("a clone the resolver failed twice, then answered, ended the build: %v", err)
|
||||
}
|
||||
if f.clones != 3 {
|
||||
t.Errorf("the clone ran %d times, want 3 (two lookups failed, the third worked)", f.clones)
|
||||
}
|
||||
retries := 0
|
||||
for _, line := range said() {
|
||||
if strings.Contains(line, "the resolver did not answer") && strings.Contains(line, "novox.internal") {
|
||||
retries++
|
||||
}
|
||||
}
|
||||
if retries != 2 {
|
||||
t.Errorf("the build's log said %d retries as the resolver's failure, want 2:\n%s", retries,
|
||||
strings.Join(said(), "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
// 1773/AC2: a lookup that keeps failing past the bound fails the build, saying the resolver did not answer.
|
||||
func TestACloneWhoseLookupNeverAnswersFailsSayingTheResolverDidNotAnswer(t *testing.T) {
|
||||
quickLookups(t)
|
||||
r, workspace := aRepository(t, nothingToBuild, nil)
|
||||
f := &failingClones{r: r, n: -1, failure: lookupFailure}
|
||||
_, err := Build(context.Background(), f.run, r, "http://novox.internal:20000/novox/x.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a clone whose lookup never answered built")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "the resolver did not answer") ||
|
||||
!strings.Contains(err.Error(), "http://novox.internal:20000/novox/x.git") {
|
||||
t.Errorf("the failure does not say the resolver did not answer for the forge's URL: %v", err)
|
||||
}
|
||||
if f.clones < 2 {
|
||||
t.Errorf("the clone ran %d times; a failed lookup is tried again within the bound", f.clones)
|
||||
}
|
||||
}
|
||||
|
||||
// 1773/AC3: any other failure of git (authentication, a missing repository) is not tried again and fails at once.
|
||||
func TestACloneFailingForAnotherReasonIsNotTriedAgain(t *testing.T) {
|
||||
quickLookups(t)
|
||||
for _, failure := range []string{
|
||||
"exit status 128\nfatal: Authentication failed for 'http://novox.internal:20000/novox/x.git/'",
|
||||
"exit status 128\nremote: Not found.\nfatal: repository 'http://novox.internal:20000/novox/x.git/' not found",
|
||||
"exit status 128\nfatal: unable to access 'http://novox.internal:20000/novox/x.git/': The requested URL returned error: 404",
|
||||
} {
|
||||
r, workspace := aRepository(t, nothingToBuild, nil)
|
||||
f := &failingClones{r: r, n: -1, failure: failure}
|
||||
_, err := Build(context.Background(), f.run, r, "http://novox.internal:20000/novox/x.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "cannot clone") {
|
||||
t.Errorf("a clone failing with %q said %v", failure, err)
|
||||
}
|
||||
if f.clones != 1 {
|
||||
t.Errorf("a clone failing with %q ran %d times; it is not a lookup and is not tried again", failure, f.clones)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// quickLookups shortens the bound and the pauses, so a test waits for milliseconds.
|
||||
func quickLookups(t *testing.T) {
|
||||
t.Helper()
|
||||
wasBound, wasFirst := lookupWait, lookupFirstPause
|
||||
lookupWait, lookupFirstPause = 200*time.Millisecond, 10*time.Millisecond
|
||||
t.Cleanup(func() { lookupWait, lookupFirstPause = wasBound, wasFirst })
|
||||
}
|
||||
|
||||
// 1773/AC1, 1773/AC3: a check's clones are the same: a failed lookup is tried again, and the failure after it (here
|
||||
// an authentication refused) ends the check at once, not tried again.
|
||||
func TestACheckCloneTriesAFailedLookupAgain(t *testing.T) {
|
||||
quickLookups(t)
|
||||
clones := 0
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
if name == "git" && isACloneOf(args) {
|
||||
clones++
|
||||
if clones <= 2 {
|
||||
return "", fmt.Errorf("git clone: %s", lookupFailure)
|
||||
}
|
||||
return "", fmt.Errorf("git clone: exit status 128\nfatal: Authentication failed")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
_, err := Check(context.Background(), run, CheckSpec{ID: "build-1", Owner: "novox", Repo: "x", Number: 1,
|
||||
Repository: "http://novox.internal:20000/novox/x.git", Ref: "abc"}, t.TempDir(), "registry.invalid",
|
||||
GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "cannot clone") || !strings.Contains(err.Error(), "Authentication failed") {
|
||||
t.Fatalf("the check said %v", err)
|
||||
}
|
||||
if clones != 3 {
|
||||
t.Errorf("the check's clone ran %d times, want 3: two failed lookups, then the refusal, not tried again", clones)
|
||||
}
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A context on a seat is cloned from the base the mesh sent, joined to the repository's path; a
|
||||
// context by URL is itself; a seat the mesh sent no base for is refused by name (novox/hq ADR 0155).
|
||||
func TestAContextOnASeatIsClonedFromTheBaseTheMeshSent(t *testing.T) {
|
||||
seats := map[string]string{"git": "http://forge.example.tld:3000"}
|
||||
got, err := contextURL(catalogue.ArtifactContext{Seat: "git", Repository: "org/controller"}, seats)
|
||||
if err != nil || got != "http://forge.example.tld:3000/org/controller.git" {
|
||||
t.Fatalf("got %q, %v", got, err)
|
||||
}
|
||||
got, err = contextURL(catalogue.ArtifactContext{Repository: "https://elsewhere.example/x.git"}, seats)
|
||||
if err != nil || got != "https://elsewhere.example/x.git" {
|
||||
t.Fatalf("a URL context was changed: %q, %v", got, err)
|
||||
}
|
||||
_, err = contextURL(catalogue.ArtifactContext{Seat: "git", Repository: "org/controller"}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "git seat") {
|
||||
t.Fatalf("a seat with no base was not refused by name: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,147 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A module's own packages, installed before its bundle is compiled, so the bundler inlines them.
|
||||
//
|
||||
// **A bundle could only import what the toolchain happened to carry.** The compiler and the bundler
|
||||
// resolve an import by walking up from the module's source: the module's own directory first, then
|
||||
// the toolchain image's node_modules. Nothing ever put anything in the first, so a module needing a
|
||||
// database driver (`pg`, `mongodb`, `mssql`) could not be a bundle at all, and kept a container whose
|
||||
// recipe installed it by hand (novox/hq ADR 0198 §4: "the backend's own driver inside the bundle").
|
||||
// Now the module's `package.json` says what it depends on, as any Node package does, and the build
|
||||
// installs exactly that into the module's own directory before compiling.
|
||||
//
|
||||
// **The SDK the toolchain carries is the one a bundle is built with, whatever the module says**
|
||||
// (novox/hq issue 212: the toolchain is rebuilt on every SDK release and every bundle after it). A
|
||||
// module's `package.json` names `@novox/mesh-sdk` with a range — it has to, to type-check on a
|
||||
// workstation — and installing that range would shadow the toolchain's copy for this module alone:
|
||||
// one module compiled against an older SDK than its neighbours, chosen by a caret nobody re-reads.
|
||||
// So the SDK is taken out of what is installed (and never fetched), and any copy something else
|
||||
// pulls in is removed afterwards; every import of it resolves past the module's node_modules to the
|
||||
// toolchain's. A module therefore cannot pin a different SDK, by design: the toolchain is the pin.
|
||||
//
|
||||
// **Correctness before speed.** Every build installs afresh into a fresh clone, from the lockfile
|
||||
// when the module has one (`npm ci`, exact) and from its ranges otherwise; nothing installed is kept
|
||||
// between builds. What is shared is npm's own download cache, a named volume, which is
|
||||
// content-addressed and verified by integrity on every read — it saves the network, never the
|
||||
// install. Install scripts do not run: the build node runs nobody's postinstall, and what a script
|
||||
// would build natively could not be inlined into one file anyway.
|
||||
|
||||
// sdkPackage is the package a TypeScript bundle's launcher serves through, and the one package a
|
||||
// module's own dependencies never supply (above).
|
||||
const sdkPackage = "@novox/mesh-sdk"
|
||||
|
||||
// npmCache is the named volume npm's download cache lives in across builds on one build node.
|
||||
const npmCache = "mesh-builder-npm-cache"
|
||||
|
||||
// ownDependencies is what a module's package.json depends on beyond the SDK, sorted; nothing when
|
||||
// the module has no package.json or depends on nothing else — which builds exactly as before.
|
||||
func ownDependencies(tree string) ([]string, error) {
|
||||
raw, err := os.ReadFile(filepath.Join(tree, "package.json"))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var p struct {
|
||||
Dependencies map[string]string `json:"dependencies"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &p); err != nil {
|
||||
return nil, fmt.Errorf("the module's package.json is not JSON: %w", err)
|
||||
}
|
||||
var names []string
|
||||
for name := range p.Dependencies {
|
||||
if name != sdkPackage {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names, nil
|
||||
}
|
||||
|
||||
// installSteps is the script run inside the toolchain image, from the module's own directory ($0).
|
||||
// It works in a scratch copy so the module's package.json and lockfile are never rewritten, takes
|
||||
// the SDK out of what is installed, installs production dependencies only, removes any copy of the
|
||||
// SDK something pulled in, and puts the result at the module's node_modules.
|
||||
const installSteps = `set -e
|
||||
work="$(mktemp -d)"
|
||||
cp "$0/package.json" "$work/"
|
||||
if [ -f "$0/package-lock.json" ]; then cp "$0/package-lock.json" "$work/"; fi
|
||||
cd "$work"
|
||||
node -e '
|
||||
const fs = require("fs"), sdk = process.argv[1];
|
||||
const p = JSON.parse(fs.readFileSync("package.json", "utf8"));
|
||||
for (const k of ["dependencies", "peerDependencies", "optionalDependencies"]) if (p[k]) delete p[k][sdk];
|
||||
delete p.devDependencies; delete p.scripts;
|
||||
fs.writeFileSync("package.json", JSON.stringify(p));
|
||||
' "$1"
|
||||
shift
|
||||
if [ -f package-lock.json ]; then
|
||||
npm ci --omit=dev --omit=peer --ignore-scripts --no-audit --no-fund "$@"
|
||||
else
|
||||
npm install --omit=dev --omit=peer --ignore-scripts --no-audit --no-fund --no-package-lock "$@"
|
||||
fi
|
||||
find node_modules -depth -type d -path "*/node_modules/@novox/mesh-sdk" -exec rm -rf {} +
|
||||
rm -rf "$0/node_modules"
|
||||
cp -a node_modules "$0/node_modules"
|
||||
`
|
||||
|
||||
// installOwn installs a TypeScript module's own production dependencies into its directory, in the
|
||||
// toolchain image, before the compile — or does nothing at all for a module that has none.
|
||||
func installOwn(ctx context.Context, run Runner, tree string, chain Toolchain, base string,
|
||||
registry Npmrc, say func(step, format string, args ...any)) error {
|
||||
if chain.Language != "typescript" {
|
||||
return nil
|
||||
}
|
||||
deps, err := ownDependencies(tree)
|
||||
if err != nil || len(deps) == 0 {
|
||||
return err
|
||||
}
|
||||
scoped := strings.TrimSpace(registry.Scope)
|
||||
if !registry.Enabled() {
|
||||
// **No registry, no scoped package.** Without the mesh's registry a scoped name resolves on
|
||||
// the public one, where anybody may have published it: a dependency that installs is not
|
||||
// the dependency the module meant.
|
||||
for _, d := range deps {
|
||||
if strings.HasPrefix(d, "@novox/") {
|
||||
return fmt.Errorf("the module depends on %s, and this build knows no package registry "+
|
||||
"for its scope; it would resolve from the public registry, which is not where the "+
|
||||
"mesh publishes it", d)
|
||||
}
|
||||
}
|
||||
}
|
||||
const within = "/app/modules/module"
|
||||
invocation := []string{"run", "--rm",
|
||||
"--volume", tree + ":" + within,
|
||||
"--volume", npmCache + ":/root/.npm",
|
||||
"--workdir", within}
|
||||
var flags []string
|
||||
if registry.Enabled() {
|
||||
// The registry is reached where the binding says it is, which may be this machine's own
|
||||
// loopback — the reason an image build that resolves packages runs on the host network too.
|
||||
invocation = append(invocation, "--network", "host")
|
||||
reg := strings.TrimSpace(registry.Registry)
|
||||
if !strings.HasSuffix(reg, "/") {
|
||||
reg += "/"
|
||||
}
|
||||
flags = append(flags, "--"+scoped+":registry="+reg)
|
||||
}
|
||||
invocation = append(invocation, base, "sh", "-c", installSteps, within, sdkPackage)
|
||||
invocation = append(invocation, flags...)
|
||||
say("bundle", "installing the module's own packages: %s", strings.Join(deps, ", "))
|
||||
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
|
||||
return fmt.Errorf("installing the module's own packages (%s): %w", strings.Join(deps, ", "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,133 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A module's own packages (dependencies.go): installed into its own directory, in the toolchain,
|
||||
// before the compile, so the bundler inlines them — the SDK always the toolchain's.
|
||||
|
||||
func buildWithPackageJSON(t *testing.T, pkg string, extra map[string]string, registry Npmrc) (*recorded, error) {
|
||||
t.Helper()
|
||||
files := map[string]string{"index.ts": "console.log(1)"}
|
||||
if pkg != "" {
|
||||
files["package.json"] = pkg
|
||||
}
|
||||
for k, v := range extra {
|
||||
files[k] = v
|
||||
}
|
||||
r, workspace := aRepository(t, aBundle, files)
|
||||
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
|
||||
_, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/greeter.git", "", "", workspace, held, registry, GitCredential{}, nil)
|
||||
return r, err
|
||||
}
|
||||
|
||||
func installs(r *recorded) []string {
|
||||
var out []string
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "npm ci") {
|
||||
out = append(out, line)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func compileIndex(r *recorded) int {
|
||||
for i, line := range r.ran {
|
||||
if strings.Contains(line, "--outDir") {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
func TestAModulesOwnPackagesAreInstalledInTheToolchainBeforeTheCompile(t *testing.T) {
|
||||
r, err := buildWithPackageJSON(t, `{"type":"module","dependencies":{"@novox/mesh-sdk":"^0.1.0","pg":"^8"},"devDependencies":{"typescript":"^5"}}`,
|
||||
nil, Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := installs(r)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("want one install of the module's own packages:\n%s", strings.Join(r.ran, "\n"))
|
||||
}
|
||||
line := got[0]
|
||||
for _, want := range []string{
|
||||
"mesh-tools/build@sha256:", // in the toolchain image
|
||||
":/app/modules/module", // into the module's own directory
|
||||
"--workdir /app/modules/module", //
|
||||
npmCache + ":/root/.npm", // npm's verified download cache, and only that
|
||||
"--omit=dev", "--ignore-scripts", // production packages, no build-node scripts
|
||||
"npm ci", "npm install", "--no-package-lock", // the lockfile when there is one, else the ranges
|
||||
"--@novox:registry=https://forge.invalid/api/packages/novox/npm/", // the scope from the mesh's registry
|
||||
"--network host",
|
||||
"@novox/mesh-sdk", // named, to be taken out of what is installed
|
||||
} {
|
||||
if !strings.Contains(line, want) {
|
||||
t.Errorf("the install lacks %q:\n%s", want, line)
|
||||
}
|
||||
}
|
||||
// The SDK is the toolchain's: never installed from the module's range, and any copy removed.
|
||||
if !strings.Contains(line, `delete p[k][sdk]`) || !strings.Contains(line, `-path "*/node_modules/@novox/mesh-sdk" -exec rm -rf`) {
|
||||
t.Errorf("the module's own SDK range could shadow the toolchain's SDK:\n%s", line)
|
||||
}
|
||||
if i, c := strings.Index(strings.Join(r.ran, "\n"), "npm ci"), compileIndex(r); c < 0 ||
|
||||
i > strings.Index(strings.Join(r.ran, "\n"), "--outDir") {
|
||||
t.Fatalf("the install did not run before the compile:\n%s", strings.Join(r.ran, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
// **A module with nothing beyond the SDK builds exactly as before**: the same commands, no install.
|
||||
func TestAModuleDependingOnlyOnTheSDKBuildsExactlyAsBefore(t *testing.T) {
|
||||
without, err := buildWithPackageJSON(t, "", nil, Npmrc{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, pkg := range []string{
|
||||
`{"type":"module","dependencies":{"@novox/mesh-sdk":"^0.1.0"},"devDependencies":{"typescript":"^5"}}`,
|
||||
`{"type":"module"}`,
|
||||
} {
|
||||
with, err := buildWithPackageJSON(t, pkg, map[string]string{"package-lock.json": "{}"}, Npmrc{Scope: "@novox", Registry: "https://forge.invalid/npm/"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(strings.Join(with.ran, "\n"), "npm ") {
|
||||
t.Fatalf("a module depending on nothing but the SDK ran npm:\n%s", strings.Join(with.ran, "\n"))
|
||||
}
|
||||
if len(with.ran) != len(without.ran) {
|
||||
t.Fatalf("a module depending only on the SDK built differently from one with no package.json:\n%s\n---\n%s",
|
||||
strings.Join(with.ran, "\n"), strings.Join(without.ran, "\n"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Without the mesh's registry a scoped package would resolve on the public one: refused by name.
|
||||
func TestAScopedPackageWithNoRegistryIsRefused(t *testing.T) {
|
||||
r, err := buildWithPackageJSON(t, `{"dependencies":{"@novox/mesh-sdk":"^0.1.0","@novox/other":"^1"}}`, nil, Npmrc{})
|
||||
if err == nil || !strings.Contains(err.Error(), "@novox/other") {
|
||||
t.Fatalf("a scoped package was installed with no registry for its scope: %v", err)
|
||||
}
|
||||
if strings.Contains(strings.Join(r.ran, "\n"), "--outDir") {
|
||||
t.Fatal("the compile ran after the refusal")
|
||||
}
|
||||
// A public package installs without one, from the public registry and nothing else.
|
||||
r, err = buildWithPackageJSON(t, `{"dependencies":{"mssql":"^11"}}`, nil, Npmrc{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := installs(r); len(got) != 1 || strings.Contains(got[0], ":registry=") || strings.Contains(got[0], "--network host") {
|
||||
t.Fatalf("a public package's install: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnreadablePackageJSONIsRefusedByName(t *testing.T) {
|
||||
_, err := buildWithPackageJSON(t, `{"dependencies":`, nil, Npmrc{})
|
||||
if err == nil || !strings.Contains(err.Error(), "package.json") {
|
||||
t.Fatalf("a broken package.json was not refused by name: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Issue 1834: a build node's root disk filled and every check there ended "the store never became ready". A node
|
||||
// frees what each build and check leaves, refuses to take work under a floor of free space, and a store that does
|
||||
// not come up says why.
|
||||
|
||||
func init() {
|
||||
// The disk the tests run on is none of their business; the tests of the floor say their own.
|
||||
FreeBytes = func(string) (uint64, error) { return 500 << 30, nil }
|
||||
}
|
||||
|
||||
func withFree(t *testing.T, bytes uint64) {
|
||||
t.Helper()
|
||||
was := FreeBytes
|
||||
FreeBytes = func(string) (uint64, error) { return bytes, nil }
|
||||
t.Cleanup(func() { FreeBytes = was })
|
||||
}
|
||||
|
||||
// 1834/AC2: given 3 GiB free, when a check or a build is asked, then it refuses at once, saying so, running nothing.
|
||||
func TestACheckUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
|
||||
withFree(t, 3<<30)
|
||||
var ran []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
return "", nil
|
||||
}
|
||||
_, err := Check(context.Background(), run, CheckSpec{ID: "build-1"}, t.TempDir(), "registry.invalid", GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "the build node has 3 GiB free on /, under the floor of 20 GiB") {
|
||||
t.Fatalf("a check on a full disk said %v", err)
|
||||
}
|
||||
if len(ran) != 0 {
|
||||
t.Errorf("a refused check ran %v", ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
|
||||
withFree(t, 3<<30)
|
||||
r := &recorded{}
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "under the floor of 20 GiB") {
|
||||
t.Fatalf("a build on a full disk said %v", err)
|
||||
}
|
||||
if len(r.ran) != 0 {
|
||||
t.Errorf("a refused build ran %v", r.ran)
|
||||
}
|
||||
}
|
||||
|
||||
// The workspace's filesystem is read too, and a workspace not made yet is read at its nearest folder.
|
||||
func TestTheWorkspacesFilesystemIsReadToo(t *testing.T) {
|
||||
was := FreeBytes
|
||||
t.Cleanup(func() { FreeBytes = was })
|
||||
missing := filepath.Join(t.TempDir(), "not", "yet")
|
||||
FreeBytes = func(p string) (uint64, error) {
|
||||
if p == "/" {
|
||||
return 100 << 30, nil
|
||||
}
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Errorf("read the free space of %s, which is not there", p)
|
||||
}
|
||||
return 1 << 30, nil
|
||||
}
|
||||
err := RoomToWork("/", missing)
|
||||
if err == nil || !strings.Contains(err.Error(), "1 GiB free on "+missing) {
|
||||
t.Errorf("the workspace's disk, under the floor, said %v", err)
|
||||
}
|
||||
FreeBytes = func(string) (uint64, error) { return 0, errors.New("no") }
|
||||
if RoomToWork("/") == nil {
|
||||
t.Error("a disk that cannot be read was taken as having room")
|
||||
}
|
||||
}
|
||||
|
||||
// 1834/AC3: given a store that stops at once, when the check waits for it, then its exit code and last log lines are
|
||||
// in the error and the container is removed afterwards, not by --rm before anyone could read it.
|
||||
func TestAStoreThatDiesSaysWhyItDied(t *testing.T) {
|
||||
var ran []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
line := name + " " + strings.Join(args, " ")
|
||||
ran = append(ran, line)
|
||||
switch {
|
||||
case args[0] == "exec":
|
||||
return "", errors.New("Error response from daemon: container is not running")
|
||||
case args[0] == "inspect":
|
||||
return "exited exit code 1\n", nil
|
||||
case args[0] == "logs":
|
||||
return "initdb: error: could not create directory: No space left on device\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
err := waitReady(context.Background(), run, "build-1-store", []string{"pg_isready"}, 1)
|
||||
for _, want := range []string{"build-1-store never became ready", "exit code 1", "No space left on device"} {
|
||||
if err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("the verdict %q lacks %q", err, want)
|
||||
}
|
||||
}
|
||||
logs := ""
|
||||
for _, l := range ran {
|
||||
if strings.HasPrefix(l, "docker logs") {
|
||||
logs = l
|
||||
}
|
||||
}
|
||||
if logs != "docker logs --tail 20 build-1-store" {
|
||||
t.Errorf("the log was read as %q", logs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheThrowawayStoreIsNotStartedWithRm(t *testing.T) {
|
||||
var started []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
if args[0] == "run" {
|
||||
started = args
|
||||
}
|
||||
return "127.0.0.1:40000\n", nil
|
||||
}
|
||||
if _, err := throwaway(context.Background(), run, run, "build-1-store", "postgres:17-alpine", 5432, nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, a := range started {
|
||||
if a == "--rm" {
|
||||
t.Errorf("started with --rm, which erases why it died: %v", started)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 1834/AC1: given an image built for a check or a walk and published to the registry, when the build ends, then the
|
||||
// local image tag, the narrowed source and the context clone it left are gone; an image published nowhere (a mesh
|
||||
// being raised) is kept, since it is the artifact.
|
||||
// pushedNamed is a publisher that tags the image as <registry>/<repository> to push it, as the registry one does.
|
||||
type pushedNamed struct{ *recorded }
|
||||
|
||||
func (pushedNamed) PushedNames(repository string) []string {
|
||||
return []string{"registry.invalid/" + repository}
|
||||
}
|
||||
|
||||
func TestABuildRemovesWhatItLeftOnTheNode(t *testing.T) {
|
||||
r := &recorded{
|
||||
contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"},
|
||||
secondary: map[string]map[string]string{"https://forge.invalid/ctl.git": {"go.mod": "module x\n"}},
|
||||
}
|
||||
workspace := t.TempDir()
|
||||
if _, err := Build(context.Background(), r.run, pushedNamed{r}, "https://forge.invalid/c.git", "modules/m", "", workspace, nil, Npmrc{},
|
||||
GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Both names go in one rmi: the layers stay while either is left (the push tagged the image with the registry's).
|
||||
if !strings.Contains(strings.Join(r.ran, "\n"), "docker rmi m-server:c0ffeec0 registry.invalid/m/server") {
|
||||
t.Errorf("the image built for it and its pushed name were not removed: %v", r.ran)
|
||||
}
|
||||
left, _ := filepath.Glob(filepath.Join(workspace, "*"))
|
||||
if len(left) != 0 {
|
||||
t.Errorf("the build left %v in its workspace", left)
|
||||
}
|
||||
}
|
||||
|
||||
const anImageWithAContext = `{"module":"m","version":"1",
|
||||
"build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile",
|
||||
"context":{"repository":"https://forge.invalid/ctl.git","ref":"main"}}]}}`
|
||||
|
||||
func TestAnImageKeptLocallyIsNotRemoved(t *testing.T) {
|
||||
r := &recorded{contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"}}
|
||||
local := Local{Run: func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if args[0] == "image" {
|
||||
return "sha256:" + strings.Repeat("a", 64) + "\n", nil
|
||||
}
|
||||
return r.run(ctx, dir, name, args...)
|
||||
}}
|
||||
if _, err := Build(context.Background(), local.Run, local, "https://forge.invalid/c.git", "modules/m", "", t.TempDir(), nil, Npmrc{},
|
||||
GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(strings.Join(r.ran, "\n"), "docker rmi") {
|
||||
t.Errorf("the artifact of a mesh being raised was removed: %v", r.ran)
|
||||
}
|
||||
}
|
||||
|
||||
// The held image the comparison pulls to read is let go of once read.
|
||||
func TestTheHeldImageAComparisonPullsIsRemovedAfterReading(t *testing.T) {
|
||||
var ran []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
if args[0] == "create" {
|
||||
return "abcdef0123456789\n", nil
|
||||
}
|
||||
if args[0] == "image" {
|
||||
return "", errors.New("stop here")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
_, _ = imageContent(context.Background(), run, t.TempDir(), "reg.invalid/m/server@sha256:"+strings.Repeat("a", 64))
|
||||
if got := ran[len(ran)-1]; !strings.HasPrefix(got, "docker rmi reg.invalid/m/server@sha256:") {
|
||||
t.Errorf("the last command was %q, not the removal of the pulled image: %v", got, ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnImageThatCannotBeRemovedIsNotSaidRemoved(t *testing.T) {
|
||||
var said []string
|
||||
run := func(context.Context, string, string, ...string) (string, error) {
|
||||
return "", errors.New("image is being used")
|
||||
}
|
||||
releaseImage(context.Background(), run, []string{"a:1"}, func(step, format string, args ...any) {
|
||||
said = append(said, format)
|
||||
})
|
||||
for _, s := range said {
|
||||
if strings.Contains(s, "is removed") {
|
||||
t.Errorf("said removed: %q", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,285 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// **What failed is picked from the whole of a check's output, as it streams** (novox/hq issue 460).
|
||||
//
|
||||
// A repository's own check kept only the last reportLines lines of what it printed, and named what failed
|
||||
// from those. A long store-backed run prints its logs after the failure: the `--- FAIL:` line of mesh-controller
|
||||
// #218 was pushed out of the tail by the package's own log lines, and the verdict said only
|
||||
// "FAIL <package> 268.072s" — the test that failed could not be named, and the fix was a guess. #220's failure
|
||||
// came a few dozen lines before the end, inside the tail, and was named. The length of what came after the
|
||||
// failure decided it, nothing else.
|
||||
//
|
||||
// So every line is looked at as it is written, and the lines that name what failed are kept whole, wherever
|
||||
// in the output they are: each `--- FAIL:` line with its indented message lines, a `panic:` with its first
|
||||
// frames, a `WARNING: DATA RACE` report, each `FAIL\t<package>` line, a build error with its package, and
|
||||
// the files gofmt lists. The rest is cut as before.
|
||||
|
||||
// Bounds on what is picked, so a run that fails everywhere still travels in a verdict: the controller cuts a
|
||||
// report to its last 60 KiB (maxCheckReport), and what failed is said at the report's end, so it stays inside.
|
||||
const (
|
||||
// failureLines and failureBytes bound everything picked from one run.
|
||||
failureLines = 400
|
||||
failureBytes = 32 << 10
|
||||
// failureLineBytes bounds one picked line: a message that pastes a whole document is cut, said.
|
||||
failureLineBytes = 1 << 10
|
||||
// The lines kept after the line that opens each kind of block.
|
||||
failMessageLines = 30 // a --- FAIL's messages
|
||||
panicLines = 40 // a panic's first frames
|
||||
raceLines = 80 // a data race report, to its closing rule
|
||||
buildLines = 40 // a package's build errors
|
||||
gofmtLines = 50 // the files gofmt lists
|
||||
// failedNames bounds how many further failing tests the summary names after the first.
|
||||
failedNames = 5
|
||||
)
|
||||
|
||||
// goError is a compiler's or vet's line: a Go file, a line, and what is wrong there.
|
||||
var goError = regexp.MustCompile(`^\S+\.go:\d+(:\d+)?: `)
|
||||
|
||||
// raceRule is the line the race detector opens and closes its report with.
|
||||
const raceRule = "=================="
|
||||
|
||||
type failureBlock int
|
||||
|
||||
const (
|
||||
noBlock failureBlock = iota
|
||||
inFail
|
||||
inPanic
|
||||
inRace
|
||||
inBuild
|
||||
inGofmt
|
||||
)
|
||||
|
||||
// failures keeps the lines of a check's output that name what failed. It is an io.Writer, fed the same
|
||||
// bytes as the report's tail.
|
||||
type failures struct {
|
||||
partial []byte
|
||||
kept []string
|
||||
size int
|
||||
dropped int
|
||||
|
||||
block failureBlock
|
||||
indent int // a --- FAIL line's indentation: its messages are indented deeper
|
||||
left int // lines the open block may still keep
|
||||
headers []string
|
||||
tests []string // every failing test's --- FAIL line, in order
|
||||
lastRule bool // the line before was the race detector's rule, which opens its report
|
||||
}
|
||||
|
||||
func (f *failures) Write(p []byte) (int, error) {
|
||||
f.partial = append(f.partial, p...)
|
||||
for {
|
||||
i := bytes.IndexByte(f.partial, '\n')
|
||||
if i < 0 {
|
||||
break
|
||||
}
|
||||
f.line(strings.TrimRight(string(f.partial[:i]), "\r"))
|
||||
f.partial = f.partial[i+1:]
|
||||
}
|
||||
// A line with no end, longer than any line is kept, is judged by its start.
|
||||
if len(f.partial) > failureLineBytes*4 {
|
||||
f.line(string(f.partial))
|
||||
f.partial = nil
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// flush judges what is left of a last line with no newline.
|
||||
func (f *failures) flush() {
|
||||
if len(f.partial) > 0 {
|
||||
f.line(strings.TrimRight(string(f.partial), "\r"))
|
||||
f.partial = nil
|
||||
}
|
||||
}
|
||||
|
||||
func indentOf(line string) int {
|
||||
return len(line) - len(strings.TrimLeft(line, " \t"))
|
||||
}
|
||||
|
||||
func (f *failures) keep(line string) {
|
||||
if f.size >= failureBytes || len(f.kept) >= failureLines {
|
||||
f.dropped++
|
||||
return
|
||||
}
|
||||
if len(line) > failureLineBytes {
|
||||
line = line[:failureLineBytes] + fmt.Sprintf(" … (%d bytes more)", len(line)-failureLineBytes)
|
||||
}
|
||||
f.kept = append(f.kept, line)
|
||||
f.size += len(line) + 1
|
||||
}
|
||||
|
||||
func (f *failures) open(block failureBlock, lines int) {
|
||||
f.block, f.left = block, lines
|
||||
}
|
||||
|
||||
func (f *failures) line(line string) {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
rule := trimmed == raceRule
|
||||
|
||||
// A line that opens a block ends whatever block was open.
|
||||
if f.opens(line, trimmed) {
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
switch f.block {
|
||||
case inFail:
|
||||
if trimmed != "" && indentOf(line) > f.indent && f.left > 0 {
|
||||
f.left--
|
||||
f.keep(line)
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
case inPanic:
|
||||
if f.left > 0 && !strings.HasPrefix(line, "FAIL") && !strings.HasPrefix(line, "ok \t") &&
|
||||
!strings.HasPrefix(line, "exit status") {
|
||||
f.left--
|
||||
f.keep(line)
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
case inRace:
|
||||
if f.left > 0 {
|
||||
f.left--
|
||||
f.keep(line)
|
||||
if rule {
|
||||
f.block = noBlock
|
||||
}
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
case inBuild:
|
||||
if f.left > 0 && trimmed != "" && !strings.HasPrefix(line, "FAIL") && !strings.HasPrefix(line, "ok \t") &&
|
||||
!strings.HasPrefix(line, "? \t") {
|
||||
f.left--
|
||||
f.keep(line)
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
case inGofmt:
|
||||
if f.left > 0 && trimmed != "" {
|
||||
f.left--
|
||||
f.keep(line)
|
||||
f.lastRule = rule
|
||||
return
|
||||
}
|
||||
}
|
||||
f.block = noBlock
|
||||
|
||||
switch {
|
||||
case strings.HasPrefix(line, "# "):
|
||||
// A package's name before its build errors — kept only when an error follows it.
|
||||
f.headers = append(f.headers, line)
|
||||
if len(f.headers) > 4 {
|
||||
f.headers = f.headers[1:]
|
||||
}
|
||||
case len(f.headers) > 0 && goError.MatchString(line):
|
||||
for _, h := range f.headers {
|
||||
f.keep(h)
|
||||
}
|
||||
f.headers = nil
|
||||
f.keep(line)
|
||||
f.open(inBuild, buildLines)
|
||||
case strings.HasPrefix(line, "FAIL\t"):
|
||||
f.headers = nil
|
||||
f.keep(line)
|
||||
default:
|
||||
f.headers = nil
|
||||
}
|
||||
f.lastRule = rule
|
||||
}
|
||||
|
||||
// opens keeps a line that opens a block of what failed, and opens the block. False for any other line.
|
||||
func (f *failures) opens(line, trimmed string) bool {
|
||||
switch {
|
||||
case strings.HasPrefix(trimmed, "--- FAIL:"):
|
||||
f.headers = nil
|
||||
f.keep(line)
|
||||
f.tests = append(f.tests, trimmed)
|
||||
f.indent = indentOf(line)
|
||||
f.open(inFail, failMessageLines)
|
||||
case strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:"):
|
||||
f.headers = nil
|
||||
f.keep(line)
|
||||
f.open(inPanic, panicLines)
|
||||
case trimmed == "WARNING: DATA RACE":
|
||||
f.headers = nil
|
||||
if f.lastRule {
|
||||
f.keep(raceRule)
|
||||
}
|
||||
f.keep(line)
|
||||
f.open(inRace, raceLines)
|
||||
case strings.HasPrefix(trimmed, "not gofmt'd:"):
|
||||
f.headers = nil
|
||||
f.keep(line)
|
||||
f.open(inGofmt, gofmtLines)
|
||||
default:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// String is every line picked, in the order the run printed them, and how many more were left out.
|
||||
func (f *failures) String() string {
|
||||
f.flush()
|
||||
s := strings.Join(f.kept, "\n")
|
||||
if f.dropped > 0 {
|
||||
s += fmt.Sprintf("\n… %d more line(s) naming what failed, past the %d lines or %d KiB kept", f.dropped,
|
||||
failureLines, failureBytes>>10)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// said is what the repository layer's summary says failed: the first thing that failed, whole, then the
|
||||
// further failing tests by name. Empty when nothing was picked.
|
||||
func (f *failures) said() string {
|
||||
picked := f.String()
|
||||
if picked == "" {
|
||||
return ""
|
||||
}
|
||||
first := whatFailed(picked)
|
||||
var more []string
|
||||
for _, t := range f.tests {
|
||||
if t == first {
|
||||
continue
|
||||
}
|
||||
// A subtest's parent fails with it; its name is in the subtest's.
|
||||
name := strings.Fields(strings.TrimPrefix(t, "--- FAIL:"))
|
||||
if len(name) > 0 {
|
||||
more = append(more, name[0])
|
||||
}
|
||||
}
|
||||
more = withoutParents(more)
|
||||
switch {
|
||||
case len(more) == 0:
|
||||
return first
|
||||
case len(more) > failedNames:
|
||||
return fmt.Sprintf("%s; and %s and %d more", first, strings.Join(more[:failedNames], ", "),
|
||||
len(more)-failedNames)
|
||||
default:
|
||||
return first + "; and " + strings.Join(more, ", ")
|
||||
}
|
||||
}
|
||||
|
||||
// withoutParents drops a test whose subtest is also named.
|
||||
func withoutParents(names []string) []string {
|
||||
var out []string
|
||||
for _, n := range names {
|
||||
parent := false
|
||||
for _, m := range names {
|
||||
if strings.HasPrefix(m, n+"/") {
|
||||
parent = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !parent {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -1,152 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func ownCheckOf(t *testing.T, printed string) (*Layer, []string) {
|
||||
t.Helper()
|
||||
tree := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(tree, "printed.txt"), []byte(printed), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(tree, CheckScript), []byte("cat printed.txt\nexit 1\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out tail
|
||||
var logged []string
|
||||
layer := ownCheck(t.Context(), CheckSpec{Toolchain: "go-image"}, []ScriptPart{{"go", CheckScript}}, tree, &out,
|
||||
func() bool { return false },
|
||||
func(_, script string) *exec.Cmd {
|
||||
cmd := exec.CommandContext(t.Context(), "sh", script)
|
||||
cmd.Dir = tree
|
||||
return cmd
|
||||
}, func(step, format string, args ...any) {
|
||||
if step == "output" {
|
||||
logged = append(logged, fmt.Sprintf(format, args...))
|
||||
}
|
||||
})
|
||||
if layer == nil || layer.Verdict != "fail" {
|
||||
t.Fatalf("a failing check answered %+v", layer)
|
||||
}
|
||||
return layer, logged
|
||||
}
|
||||
|
||||
func TestAFailureEarlyInALongRunIsStillNamed(t *testing.T) {
|
||||
layer, _ := ownCheckOf(t, captured(t, "a-long-run-failing-early.txt"))
|
||||
if !strings.Contains(layer.Summary, "failed: --- FAIL: TestTheEnvelopeIsPinned (0.00s)") {
|
||||
t.Errorf("a failure 600 lines from the end is said as %q", layer.Summary)
|
||||
}
|
||||
if !strings.Contains(layer.Summary, "TestSubtests/the_hub") {
|
||||
t.Errorf("the second failing test is not named: %q", layer.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARaceRunOfEveryPackageNamesItsFirstFailure(t *testing.T) {
|
||||
layer, _ := ownCheckOf(t, captured(t, "a-race-run-of-every-package.txt"))
|
||||
if !strings.Contains(layer.Summary, "failed: --- FAIL: TestTheEnvelopeIsPinned (0.00s)") ||
|
||||
!strings.Contains(layer.Summary, "TestAMapIsNil") || !strings.Contains(layer.Summary, "TestACounterIsShared") {
|
||||
t.Errorf("a run failing in four packages is said as %q", layer.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildErrorIsNamedByTheError(t *testing.T) {
|
||||
layer, _ := ownCheckOf(t, captured(t, "a-build-error.txt"))
|
||||
if !strings.HasSuffix(layer.Summary, "failed: broken/broken.go:3:28: undefined: undefinedThing") {
|
||||
t.Errorf("a build error is said as %q", layer.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheWholeOutputIsInTheBuildsLog(t *testing.T) {
|
||||
printed := captured(t, "a-long-run-failing-early.txt")
|
||||
_, logged := ownCheckOf(t, printed)
|
||||
want := strings.Split(strings.TrimRight(printed, "\n"), "\n")
|
||||
if len(logged) != len(want) {
|
||||
t.Fatalf("the build's log holds %d line(s) of the %d printed", len(logged), len(want))
|
||||
}
|
||||
for i := range want {
|
||||
if logged[i] != want[i] {
|
||||
t.Fatalf("line %d is logged as %q, printed as %q", i+1, logged[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheVerdictKeepsWhatFailedWhole(t *testing.T) {
|
||||
layer, _ := ownCheckOf(t, captured(t, "a-long-run-failing-early.txt"))
|
||||
want := "--- FAIL: TestTheEnvelopeIsPinned (0.00s)\n" +
|
||||
" early_test.go:9: the envelope's subject is \"mesh.a\", not \"mesh.b\"\n" +
|
||||
" early_test.go:10: a second line of the same failure\n" +
|
||||
"--- FAIL: TestSubtests (0.00s)\n" +
|
||||
" --- FAIL: TestSubtests/the_hub (0.00s)\n" +
|
||||
" early_test.go:14: the hub does not compose\n" +
|
||||
"FAIL\texample.com/cap/early\t"
|
||||
if !strings.HasPrefix(layer.Failed, want) || strings.Contains(layer.Failed, "kept what home-server says") {
|
||||
t.Errorf("the early failure is kept as:\n%s", layer.Failed)
|
||||
}
|
||||
|
||||
layer, _ = ownCheckOf(t, captured(t, "a-panic.txt"))
|
||||
for _, line := range []string{"--- FAIL: TestAMapIsNil (0.00s)", "panic: assignment to entry in nil map",
|
||||
"[running]:", "example.com/cap/panics.TestAMapIsNil(", "/src/cap/panics/panics_test.go:7",
|
||||
"FAIL\texample.com/cap/panics\t"} {
|
||||
if !strings.Contains(layer.Failed, line) {
|
||||
t.Errorf("a panic is kept without %q:\n%s", line, layer.Failed)
|
||||
}
|
||||
}
|
||||
|
||||
layer, _ = ownCheckOf(t, captured(t, "a-data-race.txt"))
|
||||
race := captured(t, "a-data-race.txt")
|
||||
report := race[:strings.Index(race, "--- FAIL:")]
|
||||
if !strings.HasPrefix(layer.Failed, report) ||
|
||||
!strings.Contains(layer.Failed, "--- FAIL: TestACounterIsShared (0.00s)\n testing.go:1865: race detected") {
|
||||
t.Errorf("a data race is kept as:\n%s", layer.Failed)
|
||||
}
|
||||
|
||||
layer, _ = ownCheckOf(t, captured(t, "a-build-error.txt"))
|
||||
if layer.Failed != "# example.com/cap/broken\nbroken/broken.go:3:28: undefined: undefinedThing\n"+
|
||||
"FAIL\texample.com/cap/broken [build failed]" {
|
||||
t.Errorf("a build error is kept as:\n%s", layer.Failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatFailedIsBoundedAndSurvivesTheReportsCut(t *testing.T) {
|
||||
var b strings.Builder
|
||||
for i := range 3000 {
|
||||
fmt.Fprintf(&b, "--- FAIL: TestNumber%d (0.00s)\n x_test.go:1: %s\n", i, strings.Repeat("why ", 600))
|
||||
fmt.Fprintf(&b, "2026/10/11 01:30:03 a log line %d\n", i)
|
||||
}
|
||||
b.WriteString("FAIL\tx/everything\t1s\nFAIL\n")
|
||||
layer, logged := ownCheckOf(t, b.String())
|
||||
if len(layer.Failed) > failureBytes+2*failureLineBytes || !strings.Contains(layer.Failed, "more line(s) naming what failed") {
|
||||
t.Errorf("what failed in a run that fails everywhere is %d bytes, ending %q", len(layer.Failed),
|
||||
layer.Failed[max(0, len(layer.Failed)-200):])
|
||||
}
|
||||
if !strings.Contains(layer.Summary, "--- FAIL: TestNumber0 (0.00s); and TestNumber1, ") ||
|
||||
!strings.Contains(layer.Summary, "and 2994 more") {
|
||||
t.Errorf("a run failing 3000 tests is said as %q", layer.Summary)
|
||||
}
|
||||
|
||||
var out tail
|
||||
out.Write([]byte(b.String()))
|
||||
report := withWhatFailed(out.String(), layer)
|
||||
const maxCheckReport = 60 << 10 // as the controller cuts it, from the front
|
||||
if len(report) > maxCheckReport {
|
||||
report = report[len(report)-maxCheckReport:]
|
||||
}
|
||||
if !strings.Contains(report, "--- what failed") || !strings.Contains(report, "--- FAIL: TestNumber0 (0.00s)") {
|
||||
t.Error("what failed did not survive the controller's cut of the report")
|
||||
}
|
||||
|
||||
// The build's log keeps its first logLines lines, says how many it left out, then what failed.
|
||||
if len(logged) < logLines+2 || logged[logLines] != "… 1002 more line(s) of its output are not in this log, which keeps its first 8000" ||
|
||||
logged[logLines+1] != "--- what failed, picked from the whole of its output" ||
|
||||
logged[logLines+2] != "--- FAIL: TestNumber0 (0.00s)" {
|
||||
t.Errorf("the log past its bound says %q", logged[logLines:min(len(logged), logLines+3)])
|
||||
}
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// ownCheckOf runs a repository's own check whose script prints a captured output and fails, as the build
|
||||
// seat runs it, and answers the layer and every line it said in the build's log.
|
||||
|
||||
func captured(t *testing.T, name string) string {
|
||||
t.Helper()
|
||||
body, err := os.ReadFile(filepath.Join("testdata", "issue460", name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(body)
|
||||
}
|
||||
|
||||
// **novox/hq issue 460**: mesh-controller #218's check failed with "FAIL <package> 268.072s" and no test
|
||||
// name: the package's own logs, printed after its `--- FAIL:` lines, pushed them out of the last 200 lines,
|
||||
// which was all the summary was named from. The first failing test is named, and the others after it,
|
||||
// however much came after them.
|
||||
|
||||
// One -race run of every package: the first failure is the earliest, not the last within the tail.
|
||||
|
||||
// A package that does not build is named by its error, not by its bare "[build failed]".
|
||||
|
||||
// The whole of what the check printed is in the build's log, line by line — where before none of it was.
|
||||
|
||||
// What failed is kept whole in the verdict: each --- FAIL with its messages, a panic with its first frames,
|
||||
// a data race report from rule to rule — and none of the log lines around them.
|
||||
|
||||
// A run that fails everywhere is bounded in what it keeps, and what it keeps survives the controller's cut
|
||||
// of a report to its last 60 KiB, said at the report's end.
|
||||
|
||||
// A passing check says nothing of what failed.
|
||||
func TestAPassingCheckReportsAsBefore(t *testing.T) {
|
||||
if got := withWhatFailed("ok\tx\t1s", &Layer{Verdict: "pass"}); got != "ok\tx\t1s" {
|
||||
t.Errorf("a passing report became %q", got)
|
||||
}
|
||||
if got := withWhatFailed("r", nil); got != "r" {
|
||||
t.Errorf("no repository layer became %q", got)
|
||||
}
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// FreeSpaceFloor is the free space a build node must have, on / and on its workspace's filesystem, before it takes
|
||||
// a build or a check (novox/hq issue 1834): under it, a throwaway store cannot initialise its data directory and
|
||||
// dies at once, and the check ended "never became ready" instead of saying the disk was full.
|
||||
const FreeSpaceFloor = 20 << 30
|
||||
|
||||
// FreeBytes reads the bytes an unprivileged writer can still use on the filesystem holding path. A variable so a
|
||||
// test answers for a disk it does not have.
|
||||
var FreeBytes = func(path string) (uint64, error) {
|
||||
var s syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &s); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return s.Bavail * uint64(s.Bsize), nil
|
||||
}
|
||||
|
||||
// RoomToWork refuses, saying how much is free, when any of the paths' filesystems is under FreeSpaceFloor. A path
|
||||
// that does not exist yet (a workspace not made) is read at its nearest folder that does.
|
||||
func RoomToWork(paths ...string) error {
|
||||
for _, p := range paths {
|
||||
at := p
|
||||
for {
|
||||
if _, err := os.Stat(at); err == nil || filepath.Dir(at) == at {
|
||||
break
|
||||
}
|
||||
at = filepath.Dir(at)
|
||||
}
|
||||
free, err := FreeBytes(at)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the free space on %s cannot be read: %w", p, err)
|
||||
}
|
||||
if free < FreeSpaceFloor {
|
||||
return fmt.Errorf("the build node has %d GiB free on %s, under the floor of %d GiB",
|
||||
free>>30, p, FreeSpaceFloor>>30)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,71 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Nothing could compile the mesh's own components, which is why nothing delivers the host
|
||||
// (novox/hq 04-ISSUES/142, and ADR 0141's own insight naming it). The toolchain list was a closed
|
||||
// set of typescript and python, and two things in the path beyond it assumed TypeScript.
|
||||
|
||||
func TestTheMeshCanCompileGo(t *testing.T) {
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Named, not pinned: the mesh answers with the copy it holds, so moving compiler is a build
|
||||
// rather than an edit to this source (ADR 0044, 0142).
|
||||
if chain.Base != "mesh-tools-go" || chain.Artifact != "build" {
|
||||
t.Fatalf("the go toolchain is based on %s/%s", chain.Base, chain.Artifact)
|
||||
}
|
||||
joined := strings.Join(chain.Compile, " ")
|
||||
// Static, because what a machine holds is a file and not a container: a binary needing a libc
|
||||
// it did not bring is a delivery that works until a machine differs.
|
||||
if !strings.Contains(joined, "CGO_ENABLED=0") {
|
||||
t.Fatalf("the go toolchain does not build statically: %q", joined)
|
||||
}
|
||||
// Reproducible: a version comes from where a component sits, not from its linker (ADR 0142),
|
||||
// so two builds of one commit should produce the same bytes.
|
||||
if !strings.Contains(joined, "-trimpath") {
|
||||
t.Fatalf("the go toolchain leaves build paths in the binary: %q", joined)
|
||||
}
|
||||
if chain.Unit != UnitPackage {
|
||||
t.Fatalf("the go toolchain compiles %q, wanted a package", chain.Unit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryToolchainSaysWhatItIsPointedAt(t *testing.T) {
|
||||
// The field exists because the compile path used to assume one language. A toolchain that says
|
||||
// nothing would fall through to the entrypoint branch and compile a file list, which for a
|
||||
// compiled language builds a program out of exactly those files and ignores the rest of the
|
||||
// package — a missing symbol rather than a legible refusal.
|
||||
for _, chain := range toolchains {
|
||||
switch chain.Unit {
|
||||
case UnitPackage:
|
||||
case UnitSources:
|
||||
if chain.SourceExt == "" {
|
||||
t.Fatalf("%s compiles a file list and names no source extension", chain.Language)
|
||||
}
|
||||
if !strings.HasPrefix(chain.SourceExt, ".") {
|
||||
t.Fatalf("%s's source extension %q is not an extension", chain.Language, chain.SourceExt)
|
||||
}
|
||||
default:
|
||||
t.Fatalf("%s says it is pointed at %q, which is neither sources nor a package",
|
||||
chain.Language, chain.Unit)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEntrypointBecomesASourceInItsOwnLanguage(t *testing.T) {
|
||||
// It used to become a `.ts` whatever the language was.
|
||||
out := Out("build")
|
||||
got := sourcesFor([]string{out + "/tools/index.js"}, out, ".ts")
|
||||
if len(got) != 1 || got[0] != "tools/index.ts" {
|
||||
t.Fatalf("a typescript entrypoint became %v", got)
|
||||
}
|
||||
got = sourcesFor([]string{out + "/tools/index.js"}, out, ".py")
|
||||
if len(got) != 1 || got[0] != "tools/index.py" {
|
||||
t.Fatalf("a python entrypoint became %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,584 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Helpers of the judge-build tests: repositories with a branch, a fake container runtime, and the planted workspace.
|
||||
|
||||
const releaseBranch = "feat/57-config-and-provisioning"
|
||||
|
||||
// gitInRepo runs git in a repository made for a test.
|
||||
func gitInRepo(t *testing.T, dir string, args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
|
||||
// commitFiles writes files into a repository and commits them on the branch checked out.
|
||||
func commitFiles(t *testing.T, dir string, files map[string]string) string {
|
||||
t.Helper()
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
gitInRepo(t, dir, "add", "-A")
|
||||
gitInRepo(t, dir, "commit", "--quiet", "-m", "x")
|
||||
return gitInRepo(t, dir, "rev-parse", "HEAD")
|
||||
}
|
||||
|
||||
// aRepositoryWithARelease is a repository whose main holds mainFiles and, when releaseFiles is not nil, a release
|
||||
// branch over them holding releaseFiles. It answers the repository, main's head and the release branch's head.
|
||||
func aRepositoryWithARelease(t *testing.T, mainFiles, releaseFiles map[string]string) (string, string, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
gitInRepo(t, dir, "init", "--quiet", "-b", "main")
|
||||
main := commitFiles(t, dir, mainFiles)
|
||||
if releaseFiles == nil {
|
||||
return dir, main, ""
|
||||
}
|
||||
gitInRepo(t, dir, "checkout", "--quiet", "-b", releaseBranch)
|
||||
head := commitFiles(t, dir, releaseFiles)
|
||||
gitInRepo(t, dir, "checkout", "--quiet", "main")
|
||||
return dir, main, head
|
||||
}
|
||||
|
||||
// aJudgeOf is a controller that judges as told: its gate passes, saying who it is and the word of the mesh-sdk
|
||||
// validator it vendors — the one it was built with.
|
||||
func aJudgeOf(who string) map[string]string {
|
||||
return map[string]string{
|
||||
"go.mod": "module example.org/judge\n\ngo 1.22\n\nrequire git.novox.be/novox/mesh-sdk/go v0.0.0\n",
|
||||
"vendor/modules.txt": "# git.novox.be/novox/mesh-sdk/go v0.0.0\n## explicit; go 1.22\ngit.novox.be/novox/mesh-sdk/go/validate\n",
|
||||
"vendor/git.novox.be/novox/mesh-sdk/go/validate/v.go": "package validate\n\nconst Word = \"the vendored validator\"\n",
|
||||
"cmd/mesh-controller/main.go": `package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/validate"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) == 2 && os.Args[1] == "merge-gate" {
|
||||
fmt.Println("usage: merge-gate --facts <file> --store <postgres>")
|
||||
os.Exit(2)
|
||||
}
|
||||
fmt.Printf("{\"verdict\":\"pass\",\"summary\":\"%s with %s\"}\n", "` + who + `", validate.Word)
|
||||
}
|
||||
`,
|
||||
}
|
||||
}
|
||||
|
||||
// aHostSaying is the node-engine's validator, saying a word.
|
||||
func aHostSaying(word string) map[string]string {
|
||||
return map[string]string{
|
||||
"validate/v.go": "package validate\n\nconst Word = \"" + word + "\"\n",
|
||||
"internal/declaration/d.go": "package declaration\n",
|
||||
"validate/v_test.go": "package validate\n",
|
||||
"internal/declaration/x_test.go": "package declaration\n",
|
||||
}
|
||||
}
|
||||
|
||||
// aRuntimeThatBuildsTheJudge puts a container runtime on PATH that runs every container's command on this machine,
|
||||
// in the directory and with the environment the check gives it — the judge's build too — and a store and a bus that
|
||||
// answer. It answers the file each `go` the check runs is written to.
|
||||
func aRuntimeThatBuildsTheJudge(t *testing.T) string {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("go"); err != nil {
|
||||
t.Skip("no Go toolchain on this machine to build a judge with")
|
||||
}
|
||||
bus, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { bus.Close() })
|
||||
go func() {
|
||||
for {
|
||||
c, err := bus.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
c.Close()
|
||||
}
|
||||
}()
|
||||
dir := t.TempDir()
|
||||
built := filepath.Join(dir, "built")
|
||||
runtime := `#!/bin/sh
|
||||
cmd="$1"; shift
|
||||
case "$cmd" in
|
||||
ps|rm|exec) exit 0 ;;
|
||||
volume)
|
||||
if [ -n "$VOLUMES" ]; then echo "$*" >> "$VOLUMES"; fi
|
||||
exit 0 ;;
|
||||
port) echo "` + bus.Addr().String() + `"; exit 0 ;;
|
||||
run)
|
||||
if [ -n "$RUNS" ]; then echo "$*" >> "$RUNS"; fi
|
||||
workdir=""; detached=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-d) detached=1; shift ;;
|
||||
--rm) shift ;;
|
||||
--workdir) workdir="$2"; shift 2 ;;
|
||||
--env) export "$2"; shift 2 ;;
|
||||
--network|--volume|--user|--label|--name|-p|-e) shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
if [ -n "$detached" ]; then echo a-container; exit 0; fi
|
||||
shift
|
||||
if [ "$1" = go ]; then echo "GOCACHE=$GOCACHE GOENV=$GOENV $*" >> "` + built + `"; fi
|
||||
if [ "$1" = go ] && [ "$2" = build ] && [ -n "$JUDGE_HOMES" ]; then echo "$HOME" >> "$JUDGE_HOMES"; fi
|
||||
if [ "$1" = go ] && [ "$2" = test ] && [ -n "$REPLAYS_CANNOT_START" ]; then
|
||||
echo "docker: Error response from daemon: invalid mount config for type bind" >&2; exit 125
|
||||
fi
|
||||
if [ "$1" = go ] && [ "$2" = test ] && [ -n "$REPLAYS" ]; then
|
||||
echo "HOME=$HOME GOCACHE=$GOCACHE GOMODCACHE=$GOMODCACHE GOENV=$GOENV GOFLAGS=$GOFLAGS DIR=$workdir" >> "$REPLAYS"
|
||||
fi
|
||||
if [ "$1" = go ] && [ -n "$WARM_GO_CACHE" ] && [ -n "$GOCACHE" ] && [ -z "$(ls -A "$GOCACHE" 2>/dev/null)" ]; then
|
||||
mkdir -p "$GOCACHE" && cp -R "$WARM_GO_CACHE/." "$GOCACHE/"
|
||||
fi
|
||||
cd "$workdir" && exec "$@" ;;
|
||||
esac
|
||||
exit 1
|
||||
`
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(runtime), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
t.Setenv("GOTOOLCHAIN", "local")
|
||||
// An empty Go cache the check names is filled from a warm one before `go` runs (warm_go_cache_test.go).
|
||||
t.Setenv("WARM_GO_CACHE", warmGoCache(t))
|
||||
return built
|
||||
}
|
||||
|
||||
// builds is how many times the check ran go to build a judge.
|
||||
func builds(t *testing.T, file string) int {
|
||||
t.Helper()
|
||||
body, err := os.ReadFile(file)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return strings.Count(string(body), "go build")
|
||||
}
|
||||
|
||||
// theReleasedMesh is the core beside a release-branch check: a controller whose main and release branch judge
|
||||
// differently, a node-engine with a validator of each, and a lab with no release branch.
|
||||
func theReleasedMesh(t *testing.T) (map[string]Beside, string, string, string) {
|
||||
t.Helper()
|
||||
controller, _, controllerRelease := aRepositoryWithARelease(t, aJudgeOf("main's controller"),
|
||||
aJudgeOf("the release branch's controller"))
|
||||
host, _, hostRelease := aRepositoryWithARelease(t, aHostSaying("main's validator"),
|
||||
aHostSaying("the release branch's validator"))
|
||||
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{"README": "the lab"}, nil)
|
||||
return map[string]Beside{
|
||||
"mesh-controller": {Repository: controller, Ref: "main"},
|
||||
"mesh-controller-main": {Repository: controller, Ref: "main"},
|
||||
"mesh-host": {Repository: host, Ref: "main"},
|
||||
"mesh-lab": {Repository: lab, Ref: "main"},
|
||||
}, controllerRelease, hostRelease, labMain
|
||||
}
|
||||
|
||||
// aPullRequestInto is a repository whose pull request's head is cut from base, with a change.
|
||||
func aPullRequestInto(t *testing.T, files map[string]string, base string, change map[string]string) (string, string) {
|
||||
t.Helper()
|
||||
repo, _, _ := aRepositoryWithARelease(t, files, map[string]string{"released": "r"})
|
||||
gitInRepo(t, repo, "checkout", "--quiet", "-b", "change", base)
|
||||
head := commitFiles(t, repo, change)
|
||||
gitInRepo(t, repo, "checkout", "--quiet", "main")
|
||||
return repo, head
|
||||
}
|
||||
|
||||
// aPlantedWorkspace is a workspace a pull request's own script has written into, as it can: a git configuration in its
|
||||
// HOME whose fsmonitor hook says each directory git runs in, a git configuration under XDG_CONFIG_HOME doing the same,
|
||||
// and a Go environment file that would hand the judge's build a tool of its own. It answers the workspace and the
|
||||
// file the hook writes to.
|
||||
func aPlantedWorkspace(t *testing.T) (string, string) {
|
||||
t.Helper()
|
||||
workspace, logs := t.TempDir(), t.TempDir()
|
||||
ran := filepath.Join(logs, "hook-ran")
|
||||
hook := filepath.Join(workspace, "hook.sh")
|
||||
files := map[string]string{
|
||||
hook: "#!/bin/sh\necho \"$PWD\" >> " + ran + "\n",
|
||||
filepath.Join(workspace, ".gitconfig"): "[core]\n\tfsmonitor = " + hook + "\n",
|
||||
filepath.Join(workspace, ".config", "git", "config"): "[core]\n\tfsmonitor = " + hook + "\n",
|
||||
filepath.Join(workspace, ".config", "go", "env"): "GOFLAGS=-toolexec=" + hook + "\n",
|
||||
}
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(name), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(name, []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return workspace, ran
|
||||
}
|
||||
|
||||
// judgeRanTheHook says whether the planted hook ran in a judge's tree: the controller beside the check, or the
|
||||
// change's own when it judges itself.
|
||||
func judgeRanTheHook(t *testing.T, ran string) string {
|
||||
t.Helper()
|
||||
body, _ := os.ReadFile(ran)
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
if strings.HasSuffix(line, "/check/mesh-controller") {
|
||||
return string(body)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// **Every judge is built where no pull request reaches** (issue 482, ADR 0358 review): a git configuration with a
|
||||
// hook, and a Go environment file, planted in the workspace by an earlier pull request's script, do not run during
|
||||
// the judge's build — while the pull request's own script, in the workspace, still meets them.
|
||||
func TestAHookPlantedInTheWorkspaceDoesNotRunDuringTheJudgesBuild(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
homes := filepath.Join(t.TempDir(), "homes")
|
||||
t.Setenv("JUDGE_HOMES", homes)
|
||||
registry := aFactsRegistry(t)
|
||||
workspace, ran := aPlantedWorkspace(t)
|
||||
beside, _, _, _ := theReleasedMesh(t)
|
||||
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "git status >/dev/null 2>&1; true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-planted", Repository: catalogue, Ref: head,
|
||||
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch,
|
||||
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
|
||||
Toolchain: "golang"}, workspace, registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" {
|
||||
t.Fatalf("the check did not judge: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
if body, _ := os.ReadFile(ran); !strings.Contains(string(body), "/check/mesh-catalog") {
|
||||
t.Fatalf("the planted hook never ran, so this test proves nothing: %q", body)
|
||||
}
|
||||
if said := judgeRanTheHook(t, ran); said != "" {
|
||||
t.Fatalf("the hook planted in the workspace ran during the judge's build:\n%s", said)
|
||||
}
|
||||
if body, _ := os.ReadFile(homes); len(body) == 0 || strings.Contains(string(body), workspace) {
|
||||
t.Fatalf("the judge was built with a HOME in the workspace: %q", body)
|
||||
}
|
||||
}
|
||||
|
||||
// **Judges are built warm and where no pull request reaches** (ADR 0358 §6): every judge build uses the judge cache's
|
||||
// one Go cache, kept between checks; its home is made fresh under the judge cache and removed after; nothing is left in
|
||||
// a temporary directory of the host; and no container a check runs for the pull request mounts the judge cache.
|
||||
func TestJudgesAreBuiltWarmFromAJudgeCacheNoCheckContainerMounts(t *testing.T) {
|
||||
built := aRuntimeThatBuildsTheJudge(t)
|
||||
// Not filled from the test's warm cache: what the first judge's build leaves in the judge cache is what is proved.
|
||||
t.Setenv("WARM_GO_CACHE", "")
|
||||
runs := filepath.Join(t.TempDir(), "runs")
|
||||
t.Setenv("RUNS", runs)
|
||||
tmp := t.TempDir()
|
||||
t.Setenv("TMPDIR", tmp)
|
||||
registry := aFactsRegistry(t)
|
||||
cache := t.TempDir()
|
||||
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
|
||||
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
|
||||
for i := range 2 {
|
||||
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": fmt.Sprint(i)})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-704-warm-%d", i), Repository: catalogue,
|
||||
Ref: head, Owner: "novox", Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache,
|
||||
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang", Beside: beside},
|
||||
t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Summary != "the controller the mesh runs with the vendored validator" {
|
||||
t.Fatalf("check %d was not judged: %+v\n%s", i, v.Gate, v.Report)
|
||||
}
|
||||
if i == 0 {
|
||||
if entries, _ := os.ReadDir(filepath.Join(cache, "go-cache")); len(entries) == 0 {
|
||||
t.Fatal("the judge's build left nothing in the judge cache's Go cache to build the next judge warm")
|
||||
}
|
||||
}
|
||||
}
|
||||
// Both judges built with the one Go cache of the judge cache.
|
||||
said, _ := os.ReadFile(built)
|
||||
if n := strings.Count(string(said), "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build"); n != 2 {
|
||||
t.Fatalf("%d of 2 judge builds used the judge cache's Go cache:\n%s", n, said)
|
||||
}
|
||||
// Nothing left behind: no home under the judge cache, nothing in the host's temporary directory.
|
||||
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
|
||||
t.Fatalf("judge builds left %d home(s) under the judge cache", len(left))
|
||||
}
|
||||
if left, _ := os.ReadDir(tmp); len(left) != 0 {
|
||||
t.Fatalf("a check left %d entr(ies) in the temporary directory, e.g. %s", len(left), left[0].Name())
|
||||
}
|
||||
// Only a judge build mounts the judge cache: no container the check runs for the pull request does.
|
||||
body, _ := os.ReadFile(runs)
|
||||
judgeBuilds := 0
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(body)), "\n") {
|
||||
judging := strings.Contains(line, " go build ")
|
||||
if judging {
|
||||
judgeBuilds++
|
||||
if !strings.Contains(line, "--volume "+filepath.Join(cache, "go-cache")+":") {
|
||||
t.Fatalf("a judge build did not mount the judge cache's Go cache: %s", line)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.Contains(line, cache) {
|
||||
t.Fatalf("a container the check runs for the pull request names the judge cache: %s", line)
|
||||
}
|
||||
}
|
||||
if judgeBuilds != 2 {
|
||||
t.Fatalf("%d judge builds were run, not 2", judgeBuilds)
|
||||
}
|
||||
}
|
||||
|
||||
// Without a judge cache, a judge is built cold under the check's own root — never in a temporary directory of the
|
||||
// host — and a judge cache inside the workspace is refused.
|
||||
func TestWithoutAJudgeCacheAJudgeIsBuiltColdUnderTheChecksRoot(t *testing.T) {
|
||||
built := aRuntimeThatBuildsTheJudge(t)
|
||||
tmp := t.TempDir()
|
||||
t.Setenv("TMPDIR", tmp)
|
||||
registry := aFactsRegistry(t)
|
||||
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
|
||||
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
|
||||
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
|
||||
workspace := t.TempDir()
|
||||
spec := CheckSpec{ID: "check-704-cold", Repository: catalogue, Ref: head, Owner: "novox", Repo: "mesh-catalog",
|
||||
Number: 1, Base: "main", Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"},
|
||||
Toolchain: "golang", Beside: beside}
|
||||
v, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" {
|
||||
t.Fatalf("the check was not judged: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
said, _ := os.ReadFile(built)
|
||||
if !strings.Contains(string(said), "GOCACHE="+filepath.Join(workspace, "check", "judge-go-cache")+" GOENV=off go build") {
|
||||
t.Fatalf("without a judge cache the judge was not built under the check's root:\n%s", said)
|
||||
}
|
||||
if left, _ := os.ReadDir(tmp); len(left) != 0 {
|
||||
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
|
||||
}
|
||||
spec.JudgeCache = filepath.Join(workspace, "judge-cache")
|
||||
if _, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil); err == nil ||
|
||||
!strings.Contains(err.Error(), "inside the workspace") {
|
||||
t.Fatalf("a judge cache inside the workspace was taken: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// aLabWithReplays is mesh-lab with replays: a Go module of its own whose one test passes, needing nothing fetched.
|
||||
func aLabWithReplays(t *testing.T) (string, string) {
|
||||
t.Helper()
|
||||
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{
|
||||
"replays/go.mod": "module example.org/replays\n\ngo 1.22\n",
|
||||
"replays/replay_test.go": "package replays\n\nimport \"testing\"\n\nfunc TestAReplay(t *testing.T) {}\n",
|
||||
}, nil)
|
||||
return lab, labMain
|
||||
}
|
||||
|
||||
// **mesh-lab's replays run where no pull request reaches** (novox/hq issue 709, ADR 0358 §6): they hold the container
|
||||
// runtime's socket, so a Go environment file with a -toolexec hook and a Go cache and module cache planted in the
|
||||
// workspace by a pull request's script are neither run nor read — the replays run from a copy of mesh-lab outside
|
||||
// the workspace with a fresh home and the judge cache's caches, all removed or kept outside every check container.
|
||||
func TestTheReplaysRunWhereNoPullRequestReaches(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
replays := filepath.Join(t.TempDir(), "replays")
|
||||
t.Setenv("REPLAYS", replays)
|
||||
registry := aFactsRegistry(t)
|
||||
workspace, ran := aPlantedWorkspace(t)
|
||||
for _, poisoned := range []string{filepath.Join(workspace, "go-cache", "README"),
|
||||
filepath.Join(workspace, "go-modules", "cache", "download", "planted")} {
|
||||
if err := os.MkdirAll(filepath.Dir(poisoned), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(poisoned, []byte("planted by a pull request"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
cache := t.TempDir()
|
||||
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
|
||||
lab, labMain := aLabWithReplays(t)
|
||||
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709", Repository: catalogue, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache, Paths: []string{"modules/gitea/index.ts"},
|
||||
Modules: []string{"gitea"}, Toolchain: "golang", Beside: map[string]Beside{
|
||||
"mesh-controller": {Repository: controller, Ref: controllerMain}, "mesh-lab": {Repository: lab, Ref: labMain}}},
|
||||
workspace, registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
|
||||
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
body, _ := os.ReadFile(replays)
|
||||
said := strings.TrimSpace(string(body))
|
||||
if said == "" || strings.Count(said, "\n") != 0 {
|
||||
t.Fatalf("the replays ran %q, not once", said)
|
||||
}
|
||||
for _, want := range []string{"GOCACHE=" + filepath.Join(cache, "go-cache") + " ",
|
||||
"GOMODCACHE=" + filepath.Join(cache, "go-modules") + " ", "GOENV=off ", "GOFLAGS=-buildvcs=false ",
|
||||
"HOME=" + filepath.Join(cache, "homes", "check-709-replays", "home") + " ",
|
||||
"DIR=" + filepath.Join(cache, "homes", "check-709-replays", "mesh-lab", "replays")} {
|
||||
if !strings.Contains(said, want) {
|
||||
t.Fatalf("the replays did not run with %q: %s", want, said)
|
||||
}
|
||||
}
|
||||
if strings.Contains(said, workspace) {
|
||||
t.Fatalf("the replays ran with something of the workspace: %s", said)
|
||||
}
|
||||
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
|
||||
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
|
||||
}
|
||||
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
|
||||
t.Fatalf("the replays left %d entr(ies) under the judge cache's homes", len(left))
|
||||
}
|
||||
}
|
||||
|
||||
// aSelfJudgingControllerThatWritesIntoTheReplays is a controller pull request whose own judge, while the gate runs it,
|
||||
// writes a test into the replays beside the check — the reviewer's case for issue 709: code of the pull request's,
|
||||
// run before the replays, reaching what they run.
|
||||
func aSelfJudgingControllerThatWritesIntoTheReplays(t *testing.T, marker string) (string, string) {
|
||||
t.Helper()
|
||||
files := aJudgeOf("the change's controller")
|
||||
evil := "package replays\n\nimport (\n\t\"os\"\n\t\"testing\"\n)\n\nfunc TestEvil(t *testing.T) {\n\t_ = os.WriteFile(" +
|
||||
fmt.Sprintf("%q", marker) + ", []byte(\"ran\"), 0o644)\n}\n"
|
||||
files["cmd/mesh-controller/main.go"] = strings.Replace(files["cmd/mesh-controller/main.go"], "func main() {",
|
||||
"func main() {\n\tif len(os.Args) > 2 && os.Args[1] == \"merge-gate\" {\n\t\t_ = os.WriteFile(os.Getenv(\"MESH_CHECK_BESIDE\")+"+
|
||||
"\"/mesh-lab/replays/evil_test.go\", []byte("+fmt.Sprintf("%q", evil)+"), 0o644)\n\t\t_ = os.WriteFile("+
|
||||
fmt.Sprintf("%q", marker+".wrote")+", nil, 0o644)\n\t}", 1)
|
||||
return aPullRequestInto(t, files, "main", map[string]string{"cmd/x.txt": "x"})
|
||||
}
|
||||
|
||||
// **The replays never read what a pull request wrote** (issue 709, reviews 12303 and 12308): with a judge cache they
|
||||
// run from a copy under it, and a self-judging controller pull request writing a test into the replays beside the
|
||||
// check, a planted GOFLAGS hook and a poisoned cache entry in the workspace neither run nor are read. Without one the
|
||||
// replays are refused — the gate an error that says why, never a pass, never the workspace — they never start, and
|
||||
// nothing is made on the host.
|
||||
func TestTheReplaysNeverReadWhatAPullRequestWroteAndNeedAJudgeCache(t *testing.T) {
|
||||
for _, withCache := range []bool{true, false} {
|
||||
t.Run(fmt.Sprintf("judge cache %v", withCache), func(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
replays, volumes := filepath.Join(t.TempDir(), "replays"), filepath.Join(t.TempDir(), "volumes")
|
||||
t.Setenv("REPLAYS", replays)
|
||||
t.Setenv("VOLUMES", volumes)
|
||||
tmp := t.TempDir()
|
||||
t.Setenv("TMPDIR", tmp)
|
||||
registry := aFactsRegistry(t)
|
||||
workspace, ran := aPlantedWorkspace(t)
|
||||
if err := os.MkdirAll(filepath.Join(workspace, "go-cache"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(workspace, "go-cache", "README"), []byte("planted"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
evilRan := filepath.Join(t.TempDir(), "evil-ran")
|
||||
controller, head := aSelfJudgingControllerThatWritesIntoTheReplays(t, evilRan)
|
||||
lab, labMain := aLabWithReplays(t)
|
||||
cache := ""
|
||||
if withCache {
|
||||
cache = t.TempDir()
|
||||
}
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709-self", Repository: controller, Ref: head,
|
||||
Owner: "novox", Repo: "mesh-controller", Number: 330, Base: "main", Judge: "self", JudgeCache: cache,
|
||||
Paths: []string{"cmd/x.txt"}, Modules: []string{"mesh-controller"}, Toolchain: "golang",
|
||||
Beside: map[string]Beside{"mesh-lab": {Repository: lab, Ref: labMain}}}, workspace, registry,
|
||||
GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(evilRan + ".wrote"); err != nil {
|
||||
t.Fatalf("the self-judging pull request did not write into the replays beside it, so this proves nothing: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(evilRan); err == nil {
|
||||
t.Fatal("the test the pull request wrote into the replays ran with the socket")
|
||||
}
|
||||
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
|
||||
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
|
||||
}
|
||||
body, _ := os.ReadFile(replays)
|
||||
said := strings.TrimSpace(string(body))
|
||||
if vols, _ := os.ReadFile(volumes); len(vols) != 0 {
|
||||
t.Fatalf("the check made a container volume: %q", vols)
|
||||
}
|
||||
if left, _ := os.ReadDir(tmp); len(left) != 0 {
|
||||
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
|
||||
}
|
||||
if !withCache {
|
||||
if v.Gate == nil || v.Gate.Verdict != "error" || !strings.Contains(v.Gate.Summary, replaysNeedTheJudgeCache) ||
|
||||
v.Verdict != "error" {
|
||||
t.Fatalf("without a judge cache the gate said %+v, not an error saying the replays need it", v.Gate)
|
||||
}
|
||||
if said != "" {
|
||||
t.Fatalf("without a judge cache the replays started: %s", said)
|
||||
}
|
||||
return
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
|
||||
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
base := filepath.Join(cache, "homes", "check-709-self-replays")
|
||||
if said == "" || strings.Contains(said, workspace) || !strings.Contains(said, "GOENV=off ") ||
|
||||
!strings.Contains(said, "DIR="+filepath.Join(base, "mesh-lab", "replays")) {
|
||||
t.Fatalf("the replays did not run from the copy under %s, apart from the workspace: %q", base, said)
|
||||
}
|
||||
if _, err := os.Stat(base); err == nil {
|
||||
t.Fatalf("the replays' place %s was left after the check", base)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// **Replays the container runtime could not start are the check's error, never the change's failure**, said in the
|
||||
// build's log (review 12308); and a check whose id cannot name a place in the judge cache is refused, never given a
|
||||
// shared one.
|
||||
func TestReplaysThatCannotStartAreAnErrorAndAnUnsafeIDIsRefused(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
t.Setenv("REPLAYS_CANNOT_START", "1")
|
||||
registry := aFactsRegistry(t)
|
||||
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
|
||||
lab, labMain := aLabWithReplays(t)
|
||||
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
|
||||
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
|
||||
spec := CheckSpec{ID: "check-709-start", Repository: catalogue, Ref: head, Owner: "novox", Repo: "mesh-catalog",
|
||||
Number: 1, Base: "main", JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"},
|
||||
Modules: []string{"gitea"}, Toolchain: "golang", Beside: map[string]Beside{
|
||||
"mesh-controller": {Repository: controller, Ref: controllerMain}, "mesh-lab": {Repository: lab, Ref: labMain}}}
|
||||
var logged []string
|
||||
v, err := Check(t.Context(), Command, spec, t.TempDir(), registry, GitCredential{}, func(step, message string) {
|
||||
logged = append(logged, message)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "error" || !strings.Contains(v.Gate.Summary, "could not be started") ||
|
||||
!strings.Contains(v.Gate.Summary, "invalid mount config") {
|
||||
t.Fatalf("replays that could not start were said %+v, not an error naming why", v.Gate)
|
||||
}
|
||||
if !strings.Contains(strings.Join(logged, "\n"), "the replays could not be started") {
|
||||
t.Fatalf("the build's log does not say what failed:\n%s", strings.Join(logged, "\n"))
|
||||
}
|
||||
|
||||
spec.ID = "Check 1/../../x"
|
||||
if _, err := Check(t.Context(), Command, spec, t.TempDir(), registry, GitCredential{}, nil); err == nil ||
|
||||
!strings.Contains(err.Error(), "cannot name its place in the judge cache") {
|
||||
t.Fatalf("a check with an unsafe id was taken: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,170 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A build killed by hand (novox/hq ADR 0219).
|
||||
//
|
||||
// A build is a tree of commands — git, then docker, and docker's own children — and a container the
|
||||
// docker client started keeps running when the client dies. So ending one by hand is three things:
|
||||
// the build's context is cancelled, which kills each command's whole process group rather than the
|
||||
// one process the context knows; every container the build started carries the build's id as a
|
||||
// label, so what outlived its client is found and removed by that label; and the holder announces
|
||||
// the outcome itself, since nothing else will.
|
||||
|
||||
// BuildLabel is the label every container a build starts carries, valued with the build's id.
|
||||
const BuildLabel = "mesh.build"
|
||||
|
||||
// KillWait is how long a command killed with its build may take to let go of its output before it
|
||||
// is abandoned: a grandchild holding the pipe open must not hold the build open with it.
|
||||
const KillWait = 10 * time.Second
|
||||
|
||||
// inItsOwnGroup makes a command the leader of its own process group, killed as a group when its
|
||||
// context ends.
|
||||
func inItsOwnGroup(cmd *exec.Cmd) {
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
if cmd.Process == nil {
|
||||
return nil
|
||||
}
|
||||
// The negative pid is the group: the command and everything it started.
|
||||
if err := syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL); err != nil {
|
||||
return cmd.Process.Kill()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
cmd.WaitDelay = KillWait
|
||||
}
|
||||
|
||||
// CgroupParent is the slice every container a build or check starts joins (novox/hq issue 1798), so the node's own
|
||||
// weights and cap bind it: the build-agent module declares the slice with the machine's CPU weight, IO weight and
|
||||
// memory limits, and names it in MESH_BUILD_CGROUP_PARENT. Empty, a container joins the runtime's default.
|
||||
var CgroupParent string
|
||||
|
||||
// Labelled is a Runner that marks every container a build starts with the build's id, so a kill
|
||||
// finds what outlived the docker client (novox/hq ADR 0219). Applied at the one place every command
|
||||
// passes rather than at each `docker run` the builder composes: a run added later is labelled too.
|
||||
func Labelled(run Runner, id string) Runner {
|
||||
return func(ctx context.Context, dir string, name string, args ...string) (string, error) {
|
||||
return run(ctx, dir, name, LabelledArgs(name, args, id)...)
|
||||
}
|
||||
}
|
||||
|
||||
// LabelledArgs is a command's arguments with the build's label added, when it is a `docker run`.
|
||||
func LabelledArgs(name string, args []string, id string) []string {
|
||||
if name != "docker" || len(args) == 0 || args[0] != "run" || id == "" {
|
||||
return args
|
||||
}
|
||||
out := make([]string, 0, len(args)+4)
|
||||
out = append(out, "run", "--label", BuildLabel+"="+id)
|
||||
if CgroupParent != "" {
|
||||
out = append(out, "--cgroup-parent", CgroupParent)
|
||||
}
|
||||
return append(out, args[1:]...)
|
||||
}
|
||||
|
||||
// RemoveContainersOf removes every container labelled with the build's id, running or not, and
|
||||
// says how many. Run with a context of its own: the build's is the one that was just cancelled.
|
||||
func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error) {
|
||||
out, err := run(ctx, "", "docker", "ps", "-aq", "--filter", "label="+BuildLabel+"="+id)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
ids := strings.Fields(out)
|
||||
if len(ids) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if err := removeWithVolumes(ctx, run, ids); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(ids), nil
|
||||
}
|
||||
|
||||
// MountedVolumes is the template `docker inspect --format` reads the names of a container's volumes with.
|
||||
const MountedVolumes = `{{range .Mounts}}{{if eq .Type "volume"}}{{println .Name}}{{end}}{{end}}`
|
||||
|
||||
// anonymousVolume is the name docker gives a volume it makes for an image's VOLUME line: 64 hex digits.
|
||||
var anonymousVolume = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||
|
||||
// removeWithVolumes removes containers together with their anonymous volumes (novox/hq issue 1772). A throwaway
|
||||
// store declares its data directory a VOLUME, docker makes an anonymous volume for it, and `docker rm -f`
|
||||
// without -v leaves that volume for ever: 1,629 of them, 107 GB, on one build node. So the volumes the
|
||||
// containers mount are read first, the containers are removed with -v, and an anonymous volume still there
|
||||
// afterwards is removed by name. Only a volume one of these containers mounted, and only one with an anonymous
|
||||
// volume's name, is touched: a named volume — an operator's own data — never is. A volume already gone is
|
||||
// no failure.
|
||||
func removeWithVolumes(ctx context.Context, run Runner, ids []string) error {
|
||||
var volumes []string
|
||||
if out, err := run(ctx, "", "docker", append([]string{"inspect", "--format", MountedVolumes}, ids...)...); err == nil {
|
||||
seen := map[string]bool{}
|
||||
for _, name := range strings.Fields(out) {
|
||||
if anonymousVolume.MatchString(name) && !seen[name] {
|
||||
seen[name] = true
|
||||
volumes = append(volumes, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f", "-v"}, ids...)...); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(volumes) > 0 {
|
||||
_, _ = run(ctx, "", "docker", append([]string{"volume", "rm"}, volumes...)...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// A holder that stops mid-build leaves its containers running (novox/hq issue 306).
|
||||
//
|
||||
// A build's containers are removed by the build itself, by a kill, or by the next delivery of the same
|
||||
// ask — but only on the machine that delivery reaches. A holder recreated by a rollout while a check
|
||||
// ran left the check's throwaway store and bus running, the ask went to another machine, and nothing
|
||||
// on the first one ever looked at them again: eleven pairs across four machines in two days, each a
|
||||
// postgres that had written its share of the disk. A holder starting has taken nothing yet, so every
|
||||
// container labelled with an ask of the seat is one an earlier holder here left, and is removed then.
|
||||
|
||||
// RemoveLeftBehind removes every container on this machine labelled with an ask of the build seat —
|
||||
// an id of the seat's shape, `build-<n>` — and says how many. Called by a holder before it takes
|
||||
// anything, so none of them can be a build it runs. A container labelled with any other id — a check
|
||||
// a person runs by hand (`check-here-…`), a test's — is not the seat's, and is left.
|
||||
func RemoveLeftBehind(ctx context.Context, run Runner) (int, error) {
|
||||
out, err := run(ctx, "", "docker", "ps", "-a", "--filter", "label="+BuildLabel,
|
||||
"--format", `{{.ID}} {{.Label "`+BuildLabel+`"}}`)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var ids []string
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) == 2 && anAskOfTheSeat(fields[1]) {
|
||||
ids = append(ids, fields[0])
|
||||
}
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if err := removeWithVolumes(ctx, run, ids); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(ids), nil
|
||||
}
|
||||
|
||||
// anAskOfTheSeat is whether an id is of the shape the controller gives the seat's asks: `build-` and the
|
||||
// moment it was asked, in nanoseconds.
|
||||
func anAskOfTheSeat(id string) bool {
|
||||
n, ok := strings.CutPrefix(id, "build-")
|
||||
if !ok || n == "" {
|
||||
return false
|
||||
}
|
||||
for _, c := range n {
|
||||
if c < '0' || c > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAKilledBuildEndsItsWholeProcessGroup(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
child := filepath.Join(dir, "child")
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
began := time.Now()
|
||||
go func() {
|
||||
// A shell that starts a grandchild and waits on it: killing the shell alone would leave the
|
||||
// grandchild running, holding the output open.
|
||||
_, err := Command(ctx, dir, "sh", "-c", `sleep 60 & echo $! > child; wait`)
|
||||
done <- err
|
||||
}()
|
||||
var pid int
|
||||
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); time.Sleep(20 * time.Millisecond) {
|
||||
if raw, err := os.ReadFile(child); err == nil && strings.TrimSpace(string(raw)) != "" {
|
||||
pid, _ = strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
break
|
||||
}
|
||||
}
|
||||
if pid == 0 {
|
||||
t.Fatal("the command never started its child")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err == nil {
|
||||
t.Fatal("a killed command reported success")
|
||||
}
|
||||
case <-time.After(KillWait + 5*time.Second):
|
||||
t.Fatal("the killed command never returned")
|
||||
}
|
||||
if took := time.Since(began); took > KillWait {
|
||||
t.Errorf("ending the command took %s: the group was not killed, the wait ran out", took)
|
||||
}
|
||||
for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); time.Sleep(20 * time.Millisecond) {
|
||||
if err := syscall.Kill(pid, 0); errors.Is(err, syscall.ESRCH) {
|
||||
return
|
||||
}
|
||||
}
|
||||
_ = syscall.Kill(pid, syscall.SIGKILL)
|
||||
t.Fatalf("the grandchild %d outlived the kill", pid)
|
||||
}
|
||||
@@ -1,166 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A build killed by hand (novox/hq ADR 0219) ends every command it started: the context's end kills
|
||||
// the command's whole process group, not the one process the context knows about.
|
||||
|
||||
// Every `docker run` a build starts carries its id as a label; nothing else is touched.
|
||||
func TestEveryContainerABuildStartsCarriesItsID(t *testing.T) {
|
||||
got := LabelledArgs("docker", []string{"run", "--rm", "img", "sh"}, "build-1")
|
||||
if want := []string{"run", "--label", "mesh.build=build-1", "--rm", "img", "sh"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("docker run became %v", got)
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
args []string
|
||||
}{{"docker", []string{"build", "."}}, {"git", []string{"run"}}, {"docker", nil}} {
|
||||
if got := LabelledArgs(c.name, c.args, "build-1"); !reflect.DeepEqual(got, c.args) {
|
||||
t.Errorf("%s %v became %v", c.name, c.args, got)
|
||||
}
|
||||
}
|
||||
var ran [][]string
|
||||
run := Labelled(func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, append([]string{name}, args...))
|
||||
return "", nil
|
||||
}, "build-2")
|
||||
_, _ = run(context.Background(), "", "docker", "run", "img")
|
||||
if want := [][]string{{"docker", "run", "--label", "mesh.build=build-2", "img"}}; !reflect.DeepEqual(ran, want) {
|
||||
t.Errorf("ran %v", ran)
|
||||
}
|
||||
}
|
||||
|
||||
// What a kill removes is found by the label, and only what it finds.
|
||||
func TestAKillRemovesTheContainersLabelledWithTheBuild(t *testing.T) {
|
||||
var ran []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
if args[0] == "ps" {
|
||||
return "c1\nc2\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
n, err := RemoveContainersOf(context.Background(), run, "build-3")
|
||||
if err != nil || n != 2 {
|
||||
t.Fatalf("%d %v", n, err)
|
||||
}
|
||||
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker inspect --format " + MountedVolumes + " c1 c2",
|
||||
"docker rm -f -v c1 c2"}; !reflect.DeepEqual(ran, want) {
|
||||
t.Errorf("ran %v", ran)
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 306**: a holder starting removes what earlier holders on its machine left — every container
|
||||
// labelled with an ask of the seat — and nothing labelled otherwise.
|
||||
func TestAStartingHolderRemovesWhatEarlierHoldersLeftAndNothingElse(t *testing.T) {
|
||||
var ran [][]string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, append([]string{name}, args...))
|
||||
if args[0] == "ps" {
|
||||
return "c1 build-1791418000320743455\nc2 build-1791418000320743455\n" +
|
||||
"c3 check-here-1791418000320743455\nc4 check-test-17\nc5 build-x\nc6 build-\n\nc7 build-1791319895953076871\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
n, err := RemoveLeftBehind(context.Background(), run)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n != 3 {
|
||||
t.Errorf("removed %d", n)
|
||||
}
|
||||
if len(ran) != 3 || !reflect.DeepEqual(ran[2], []string{"docker", "rm", "-f", "-v", "c1", "c2", "c7"}) {
|
||||
t.Fatalf("ran %v", ran)
|
||||
}
|
||||
if !strings.Contains(strings.Join(ran[0], " "), "--filter label=mesh.build") {
|
||||
t.Errorf("listed with %v", ran[0])
|
||||
}
|
||||
|
||||
// Nothing left behind: nothing removed, and no rm asked.
|
||||
ran = nil
|
||||
quiet := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, append([]string{name}, args...))
|
||||
return "c3 check-here-1\n", nil
|
||||
}
|
||||
if n, err := RemoveLeftBehind(context.Background(), quiet); err != nil || n != 0 || len(ran) != 1 {
|
||||
t.Errorf("with nothing of the seat's: %d %v %v", n, err, ran)
|
||||
}
|
||||
}
|
||||
|
||||
// **Issue 1772**: a throwaway store declares its data directory a VOLUME, so docker makes an anonymous volume for
|
||||
// it, and `docker rm -f` without -v leaves that volume behind (1,629 of them, 107 GB, on one build node). Every
|
||||
// container a build or check removes is removed with its anonymous volumes (-v), and any volume the container
|
||||
// mounted that is still there afterwards is removed by name — those, and only those, never a named volume.
|
||||
func TestARemovedContainerTakesItsAnonymousVolumesWithIt(t *testing.T) {
|
||||
anon := strings.Repeat("a", 64)
|
||||
other := strings.Repeat("b", 64)
|
||||
var ran []string
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
switch args[0] {
|
||||
case "ps":
|
||||
return "c1\n", nil
|
||||
case "inspect":
|
||||
return anon + "\n" + "compose_data\n" + other + "\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
if n, err := RemoveContainersOf(context.Background(), run, "build-4"); err != nil || n != 1 {
|
||||
t.Fatalf("%d %v", n, err)
|
||||
}
|
||||
last := ran[len(ran)-1]
|
||||
if want := "docker volume rm " + anon + " " + other; last != want {
|
||||
t.Errorf("last command %q, want %q", last, want)
|
||||
}
|
||||
if !strings.Contains(strings.Join(ran, "\n"), "docker rm -f -v c1") {
|
||||
t.Errorf("ran %v", ran)
|
||||
}
|
||||
if strings.Contains(last, "compose_data") {
|
||||
t.Errorf("a named volume was removed: %q", last)
|
||||
}
|
||||
}
|
||||
|
||||
// A volume already gone when the sweep looks (rm -v took it) is no failure.
|
||||
func TestAVolumeAlreadyGoneIsNoFailure(t *testing.T) {
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
switch args[0] {
|
||||
case "ps":
|
||||
return "c1\n", nil
|
||||
case "inspect":
|
||||
return strings.Repeat("c", 64) + "\n", nil
|
||||
case "volume":
|
||||
return "", errors.New("no such volume")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
if n, err := RemoveContainersOf(context.Background(), run, "build-5"); err != nil || n != 1 {
|
||||
t.Fatalf("%d %v", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A node's build slice limits everything a build or check starts (novox/hq issue 1798): with a parent
|
||||
// set, every `docker run` joins it, so the node's CPU and IO weights and memory cap bind the container;
|
||||
// with none set the arguments are as they always were.
|
||||
func TestEveryContainerJoinsTheNodesBuildSlice(t *testing.T) {
|
||||
was := CgroupParent
|
||||
defer func() { CgroupParent = was }()
|
||||
CgroupParent = "meshbuild.slice"
|
||||
got := LabelledArgs("docker", []string{"run", "--rm", "img", "sh"}, "build-1")
|
||||
want := []string{"run", "--label", "mesh.build=build-1", "--cgroup-parent", "meshbuild.slice", "--rm", "img", "sh"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("1798/AC1: docker run became %v", got)
|
||||
}
|
||||
if got := LabelledArgs("docker", []string{"build", "."}, "build-1"); !reflect.DeepEqual(got, []string{"build", "."}) {
|
||||
t.Errorf("a docker build became %v", got)
|
||||
}
|
||||
CgroupParent = ""
|
||||
if got := LabelledArgs("docker", []string{"run", "img"}, "build-1"); !reflect.DeepEqual(got, []string{"run", "--label", "mesh.build=build-1", "img"}) {
|
||||
t.Errorf("1798/AC1: with no slice, docker run became %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// novox/hq ADR 0193: every entrypoint the runtime may serve is executable, and the runtime knows no
|
||||
// language — so a TypeScript bundle carries a launcher beside each entrypoint.
|
||||
func TestATypeScriptBundleCarriesAnExecutableLauncherBesideEachEntrypoint(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
chain, err := ToolchainFor("typescript")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := writeLaunchers(root, chain, catalogue.Artifact{Name: "tools", Kind: catalogue.ArtifactBundle,
|
||||
Language: "typescript", Entrypoints: []string{"tools/index.js", "index.js"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got["tools/index.js"] != "tools/index.serve.mjs" || got["index.js"] != "index.serve.mjs" {
|
||||
t.Fatalf("launchers: %v", got)
|
||||
}
|
||||
path := filepath.Join(root, "tools", "index.serve.mjs")
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if info.Mode().Perm() != 0o755 {
|
||||
t.Errorf("the launcher is %v, not executable 0755", info.Mode().Perm())
|
||||
}
|
||||
body, _ := os.ReadFile(path)
|
||||
for _, want := range []string{"#!/usr/bin/env node\n", `from "@novox/mesh-sdk/stdio"`, `await import("./index.js")`, "serveRegisteredOverStdio()"} {
|
||||
if !strings.Contains(string(body), want) {
|
||||
t.Errorf("the launcher lacks %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
|
||||
// A compiled language's build is executable already: no launcher.
|
||||
goChain, _ := ToolchainFor("go")
|
||||
none, err := writeLaunchers(t.TempDir(), goChain, catalogue.Artifact{Name: "tools", Kind: catalogue.ArtifactBundle, Language: "go"})
|
||||
if err != nil || len(none) != 0 {
|
||||
t.Errorf("a Go bundle was given launchers: %v %v", none, err)
|
||||
}
|
||||
}
|
||||
@@ -1,178 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// **A built image's bases are linked into its repository before it is pushed** (novox/hq issue 694).
|
||||
//
|
||||
// A push reads every blob of the image out of this machine's store, unless the repository it goes to
|
||||
// already holds that blob: the runtime asks the registry first, blob by blob, and reads locally only
|
||||
// what the registry lacks. A runtime keeping its images in containerd's store — the default for a
|
||||
// fresh install from docker 29 on — does not always hold the compressed layers of the base an image
|
||||
// was built FROM: a layer whose unpacked form it already had is never fetched. The push of such an
|
||||
// image then fails on a layer this machine never had ("not all of them are available locally"), while
|
||||
// the very bytes sit in this registry, under the repository the base was mirrored into.
|
||||
//
|
||||
// So the base's blobs are mounted into the image's repository first. A mount moves no bytes, and once
|
||||
// the repository holds a blob the push asks this machine for nothing of it — whichever store the
|
||||
// runtime keeps. A runtime that holds every layer (the classic store) is not changed by this: its push
|
||||
// finds the same blobs already there.
|
||||
|
||||
// BaseLender links what a build's bases hold into the repository its image is about to be pushed to.
|
||||
type BaseLender interface {
|
||||
// LendBases mounts every blob of each base this registry holds — for the platform this machine
|
||||
// builds — into `repository`, and answers how many it mounted.
|
||||
LendBases(ctx context.Context, repository string, bases []string) (int, error)
|
||||
}
|
||||
|
||||
// platformed is an index's entry: a manifest and the platform it is for.
|
||||
type platformed struct {
|
||||
descriptor
|
||||
Platform *struct {
|
||||
OS string `json:"os"`
|
||||
Architecture string `json:"architecture"`
|
||||
} `json:"platform"`
|
||||
}
|
||||
|
||||
// LendBases mounts each base's blobs into the repository. A base this registry does not hold under its
|
||||
// repository — named by tag, or held somewhere else — lends nothing and is no error: the push reads it
|
||||
// from this machine's store as before.
|
||||
func (r Registry) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
|
||||
mounted := 0
|
||||
for _, base := range bases {
|
||||
name, digest, pinned := strings.Cut(base, "@")
|
||||
if !pinned || !strings.HasPrefix(digest, "sha256:") {
|
||||
continue
|
||||
}
|
||||
where, err := parseReference(name + "@" + digest)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
from := where.repository
|
||||
if from == repository {
|
||||
continue
|
||||
}
|
||||
blobs, held, err := r.blobsOf(ctx, from, digest)
|
||||
if err != nil {
|
||||
return mounted, fmt.Errorf("reading %s: %w", base, err)
|
||||
}
|
||||
if !held {
|
||||
continue
|
||||
}
|
||||
for _, blob := range blobs {
|
||||
did, err := r.mount(ctx, repository, from, blob)
|
||||
if err != nil {
|
||||
return mounted, fmt.Errorf("linking %s of %s into %s: %w", blob, base, repository, err)
|
||||
}
|
||||
if did {
|
||||
mounted++
|
||||
}
|
||||
}
|
||||
}
|
||||
return mounted, nil
|
||||
}
|
||||
|
||||
// blobsOf is the config and layers of the manifest under repository@digest — for an index, of each
|
||||
// manifest it names for this machine's platform. False when the registry holds no such manifest there.
|
||||
func (r Registry) blobsOf(ctx context.Context, repository, digest string) ([]string, bool, error) {
|
||||
body, held, err := r.manifest(ctx, repository, digest)
|
||||
if err != nil || !held {
|
||||
return nil, held, err
|
||||
}
|
||||
var document struct {
|
||||
Manifests []platformed `json:"manifests"`
|
||||
Config *descriptor `json:"config"`
|
||||
Layers []descriptor `json:"layers"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &document); err != nil {
|
||||
return nil, false, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
|
||||
}
|
||||
var blobs []string
|
||||
if document.Config != nil {
|
||||
blobs = append(blobs, document.Config.Digest)
|
||||
}
|
||||
for _, l := range document.Layers {
|
||||
blobs = append(blobs, l.Digest)
|
||||
}
|
||||
for _, m := range document.Manifests {
|
||||
if m.Platform == nil || m.Platform.OS != runtime.GOOS || m.Platform.Architecture != runtime.GOARCH {
|
||||
continue
|
||||
}
|
||||
more, held, err := r.blobsOf(ctx, repository, m.Digest)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if held {
|
||||
blobs = append(blobs, more...)
|
||||
}
|
||||
}
|
||||
return blobs, true, nil
|
||||
}
|
||||
|
||||
// manifest reads one manifest document from this registry; false when it is not there.
|
||||
func (r Registry) manifest(ctx context.Context, repository, digest string) ([]byte, bool, error) {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet,
|
||||
"http://"+r.Address+"/v2/"+repository+"/manifests/"+digest, nil)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
request.Header.Set("Accept", manifestAccept)
|
||||
response, err := r.client().Do(request)
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode == http.StatusNotFound {
|
||||
return nil, false, nil
|
||||
}
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return nil, false, fmt.Errorf("%s/%s@%s: %s", r.Address, repository, digest, response.Status)
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(response.Body, 4<<20))
|
||||
return body, err == nil, err
|
||||
}
|
||||
|
||||
// mount links one blob from one repository into another; false when the repository already held it.
|
||||
func (r Registry) mount(ctx context.Context, repository, from, digest string) (bool, error) {
|
||||
base := "http://" + r.Address + "/v2/" + repository
|
||||
if there, err := r.has(ctx, base+"/blobs/"+digest); err != nil {
|
||||
return false, err
|
||||
} else if there {
|
||||
return false, nil
|
||||
}
|
||||
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
|
||||
base+"/blobs/uploads/?mount="+digest+"&from="+from, nil)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
begun, err := r.client().Do(start)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
|
||||
}
|
||||
begun.Body.Close()
|
||||
if begun.StatusCode == http.StatusCreated {
|
||||
return true, nil
|
||||
}
|
||||
if begun.StatusCode == http.StatusAccepted {
|
||||
// Not mounted, and an upload begun in its place: let go of it, so nothing is left half-open.
|
||||
if location := begun.Header.Get("Location"); location != "" {
|
||||
if strings.HasPrefix(location, "/") {
|
||||
location = "http://" + r.Address + location
|
||||
}
|
||||
if cancel, err := http.NewRequestWithContext(ctx, http.MethodDelete, location, nil); err == nil {
|
||||
if done, err := r.client().Do(cancel); err == nil {
|
||||
done.Body.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("%s did not mount it from %s", r.Address, from)
|
||||
}
|
||||
return false, fmt.Errorf("%s answered %s when asked to mount it from %s", r.Address, begun.Status, from)
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAnImageIsPushedOnlyAfterItsBasesAreLent(t *testing.T) {
|
||||
r, workspace := aRepository(t, anImage, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
|
||||
r.contents["modules/bus/module.json"] = anImage
|
||||
l := &lending{mirroring: &mirroring{recorded: r, at: "127.0.0.1:5000"}}
|
||||
if _, err := Build(context.Background(), r.run, l, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
|
||||
nil, Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{
|
||||
"lend bus/server 127.0.0.1:5000/upstream/docker.io/vendor/server@sha256:" + strings.Repeat("1", 64),
|
||||
"push bus/server",
|
||||
}
|
||||
if strings.Join(l.order, "\n") != strings.Join(want, "\n") {
|
||||
t.Fatalf("the build did:\n%s\nwant:\n%s", strings.Join(l.order, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
}
|
||||
@@ -1,123 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A built image's bases are linked into its repository before it is pushed (novox/hq issue 694): a
|
||||
// runtime keeping containerd's image store may not hold a base layer it never had to fetch, and its push
|
||||
// asks this machine only for what the repository lacks.
|
||||
|
||||
// aBaseForThisMachine puts an index into the registry under `repository`: one manifest for this
|
||||
// machine's platform, one for another, and answers the index's digest and each manifest's blobs.
|
||||
func aBaseForThisMachine(t *testing.T, dst *aRegistryOfRepositories, repository string) (string, []string, []string) {
|
||||
t.Helper()
|
||||
put := func(arch string) (string, []string) {
|
||||
config := []byte(`{"architecture":"` + arch + `"}`)
|
||||
layer := []byte(arch + " layer bytes")
|
||||
for _, b := range [][]byte{config, layer} {
|
||||
dst.blobs[digestOf(b)] = b
|
||||
dst.link(repository, digestOf(b))
|
||||
}
|
||||
m, _ := json.Marshal(map[string]any{
|
||||
"schemaVersion": 2, "mediaType": mediaManifestOCI,
|
||||
"config": map[string]any{"mediaType": "application/vnd.oci.image.config.v1+json", "digest": digestOf(config), "size": len(config)},
|
||||
"layers": []map[string]any{{"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": digestOf(layer), "size": len(layer)}},
|
||||
})
|
||||
dst.manifests[repository+"@"+digestOf(m)] = m
|
||||
return digestOf(m), []string{digestOf(config), digestOf(layer)}
|
||||
}
|
||||
other := "s390x"
|
||||
if runtime.GOARCH == other {
|
||||
other = "amd64"
|
||||
}
|
||||
mine, mineBlobs := put(runtime.GOARCH)
|
||||
theirs, theirBlobs := put(other)
|
||||
index, _ := json.Marshal(map[string]any{
|
||||
"schemaVersion": 2, "mediaType": mediaIndexOCI,
|
||||
"manifests": []map[string]any{
|
||||
{"mediaType": mediaManifestOCI, "digest": mine, "size": 1, "platform": map[string]string{"os": runtime.GOOS, "architecture": runtime.GOARCH}},
|
||||
{"mediaType": mediaManifestOCI, "digest": theirs, "size": 1, "platform": map[string]string{"os": "linux", "architecture": other}},
|
||||
},
|
||||
})
|
||||
dst.manifests[repository+"@"+digestOf(index)] = index
|
||||
return digestOf(index), mineBlobs, theirBlobs
|
||||
}
|
||||
|
||||
func TestABuildsBaseIsLinkedIntoItsRepositoryBeforeThePush(t *testing.T) {
|
||||
dst := newRegistryOfRepositories()
|
||||
server := httptest.NewServer(dst.handler())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
r := Registry{Address: address}
|
||||
|
||||
const from = "upstream/docker.io/library/golang"
|
||||
index, mine, theirs := aBaseForThisMachine(t, dst, from)
|
||||
bases := []string{
|
||||
address + "/" + from + "@" + index,
|
||||
// Not bases this registry holds: a package version, a tag, an image held nowhere here.
|
||||
"@novox/mesh-sdk@1.2.3",
|
||||
"golang:1-alpine",
|
||||
"elsewhere.invalid/thing@sha256:" + strings.Repeat("e", 64),
|
||||
}
|
||||
n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Every blob of this machine's platform is in the image's repository, so a push asks this machine's
|
||||
// store for none of them; another platform's are not, since nothing built here stands on them.
|
||||
for _, blob := range mine {
|
||||
if !dst.links["mesh-tools-go/build"][blob] {
|
||||
t.Fatalf("the base's blob %s is not in the image's repository", blob)
|
||||
}
|
||||
}
|
||||
for _, blob := range theirs {
|
||||
if dst.links["mesh-tools-go/build"][blob] {
|
||||
t.Fatalf("another platform's blob %s was linked", blob)
|
||||
}
|
||||
}
|
||||
if n != len(mine) || dst.mounts != len(mine) || dst.uploads != 0 {
|
||||
t.Fatalf("%d linked, %d mounts, %d uploads; want %d mounts and no bytes moved", n, dst.mounts, dst.uploads, len(mine))
|
||||
}
|
||||
|
||||
// Asked again, nothing is mounted twice.
|
||||
if n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases); err != nil || n != 0 {
|
||||
t.Fatalf("a second lending mounted %d: %v", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A registry that will not mount is said, not hidden: the push is still tried, from the store.
|
||||
func TestABaseTheRegistryWillNotMountIsAnError(t *testing.T) {
|
||||
dst := newRegistryOfRepositories()
|
||||
dst.noMount = true
|
||||
server := httptest.NewServer(dst.handler())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
index, _, _ := aBaseForThisMachine(t, dst, "upstream/thing")
|
||||
_, err := Registry{Address: address}.LendBases(context.Background(), "hello/server",
|
||||
[]string{address + "/upstream/thing@" + index})
|
||||
if err == nil || !strings.Contains(err.Error(), "did not mount") {
|
||||
t.Fatalf("a refused mount passed silently: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// lending records what a build lent, and when, against the push.
|
||||
type lending struct {
|
||||
*mirroring
|
||||
order []string
|
||||
}
|
||||
|
||||
func (l *lending) LendBases(_ context.Context, repository string, bases []string) (int, error) {
|
||||
l.order = append(l.order, "lend "+repository+" "+strings.Join(bases, " "))
|
||||
return len(bases), nil
|
||||
}
|
||||
|
||||
func (l *lending) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
|
||||
l.order = append(l.order, "push "+repository)
|
||||
return l.mirroring.PublishImage(ctx, localTag, repository)
|
||||
}
|
||||
@@ -1,61 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Local is what a build publishes into when there is nowhere to publish yet.
|
||||
//
|
||||
// **This exists for exactly one moment: raising a mesh** (novox/hq ADR 0073). The installer builds
|
||||
// the control plane on the machine that is about to run it, and at that moment there is no registry
|
||||
// — the registry is installed afterwards, by the control plane this build produces. So the artifact
|
||||
// stays where the build left it: in the machine's own container runtime.
|
||||
//
|
||||
// That is not a weaker kind of pinning. An image held locally is named by the digest of its own
|
||||
// configuration, which is content-addressed and unforgeable and requires nothing to have served it
|
||||
// — the same identity the installer has always used for the image it carried. What changes when a
|
||||
// registry exists is not that the artifact becomes exact, but that something other than this
|
||||
// machine can fetch it.
|
||||
//
|
||||
// It is deliberately unable to publish an archive. An archive has no local identity to fall back
|
||||
// on: it is bytes that only mean something once something serves them at a URL. A build that
|
||||
// produces one before there is anywhere to put it has produced nothing usable, and saying so is
|
||||
// better than returning a path on a disk that no other machine can read.
|
||||
type Local struct {
|
||||
// Run is how docker is invoked, so a test does not need one.
|
||||
Run Runner
|
||||
}
|
||||
|
||||
// PublishImage leaves the image where the build put it, and names it by its own configuration.
|
||||
//
|
||||
// The local tag is not returned: a tag is a name somebody can move, and every other reference in a
|
||||
// resolved manifest is exact. The digest is read back from the runtime rather than computed, for
|
||||
// the same reason the registry publisher reads it back from the registry — what matters is what
|
||||
// will be served for that reference, and only the thing serving it can say.
|
||||
func (l Local) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
|
||||
out, err := l.Run(ctx, "", "docker", "image", "inspect", "--format", "{{.Id}}", localTag)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot read back the image just built as %s: %w", localTag, err)
|
||||
}
|
||||
id := strings.TrimSpace(out)
|
||||
if !strings.HasPrefix(id, "sha256:") || len(id) != len("sha256:")+64 {
|
||||
// Refused rather than passed on. A bundle naming an image by anything a person could move
|
||||
// is refused by the machine applying it, and a value that is not an identity would fail
|
||||
// there instead — one step further from the thing that could explain it.
|
||||
return "", fmt.Errorf(
|
||||
"the container runtime named the image just built %q, which is not an image id: "+
|
||||
"sha256 and sixty-four hex characters", id)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// PublishArchive refuses, and says why rather than inventing somewhere to put it.
|
||||
func (l Local) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
|
||||
return "", fmt.Errorf(
|
||||
"%s declares an archive, and this build has nowhere to publish one. An image can stay in "+
|
||||
"the machine's own runtime and still be named exactly; an archive is bytes that mean "+
|
||||
"nothing until something serves them. Build this once the mesh has a registry",
|
||||
repository)
|
||||
}
|
||||
@@ -1,449 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// An upstream image is copied between registries, never through a machine's image store
|
||||
// (novox/hq 04-ISSUES/046, ADR 0096).
|
||||
//
|
||||
// A published image is ordinarily an index over several architectures. Pulling it leaves the index
|
||||
// in the runtime's store, and pushing one platform out of that store is what the runtime refuses —
|
||||
// every variant of pull-then-push was tried and failed the same way. A copy never needs a platform:
|
||||
// it moves what is there. Read the index, read each manifest it names, put each blob by digest,
|
||||
// put the manifests, put the index under the module's repository — the registry API is enough, and
|
||||
// the runtime's image store is never involved.
|
||||
|
||||
// Mirrorer copies an upstream image into the mesh's own registry, whole.
|
||||
type Mirrorer interface {
|
||||
MirrorImage(ctx context.Context, from, repository string) (string, error)
|
||||
}
|
||||
|
||||
// BaseMirrorer copies a base a build stands on into the one repository the mesh keeps for that
|
||||
// upstream image, taking what an earlier copy under `former` already holds rather than asking
|
||||
// upstream for it again (novox/hq ADR 0257).
|
||||
type BaseMirrorer interface {
|
||||
MirrorBase(ctx context.Context, from, repository, former string) (string, error)
|
||||
}
|
||||
|
||||
// MirrorPrefix is the namespace of the repositories a base is mirrored into: `upstream/<host>/<path>`,
|
||||
// one repository per upstream image, whichever modules stand on it (novox/hq ADR 0257).
|
||||
const MirrorPrefix = "upstream/"
|
||||
|
||||
// MirrorRepository is the repository the mesh keeps an upstream image's copy in: the image's own
|
||||
// host and path under MirrorPrefix, so two modules standing on one image stand on one copy, and two
|
||||
// images that share a path on different hosts are never confused. Lower case and without a port's
|
||||
// colon, because a repository name allows neither.
|
||||
func MirrorRepository(from string) (string, error) {
|
||||
where, err := parseReference(from)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
host := strings.TrimPrefix(strings.TrimPrefix(where.base, "https://"), "http://")
|
||||
if host == "registry-1.docker.io" {
|
||||
host = "docker.io"
|
||||
}
|
||||
host = strings.ReplaceAll(host, ":", "-")
|
||||
return strings.ToLower(MirrorPrefix + host + "/" + where.repository), nil
|
||||
}
|
||||
|
||||
// FormerMirrorRepository is where a module's base was copied before ADR 0257: under the module's own
|
||||
// repository, one copy per module (ADR 0097). Read only, as a source of what is already held.
|
||||
func FormerMirrorRepository(module, arg string) string {
|
||||
return module + "/on-" + strings.ToLower(arg)
|
||||
}
|
||||
|
||||
const (
|
||||
mediaIndexOCI = "application/vnd.oci.image.index.v1+json"
|
||||
mediaIndexDocker = "application/vnd.docker.distribution.manifest.list.v2+json"
|
||||
mediaManifestOCI = "application/vnd.oci.image.manifest.v1+json"
|
||||
mediaManifestDocker = "application/vnd.docker.distribution.manifest.v2+json"
|
||||
)
|
||||
|
||||
var manifestAccept = strings.Join([]string{mediaIndexOCI, mediaIndexDocker, mediaManifestOCI, mediaManifestDocker}, ", ")
|
||||
|
||||
// upstream is where an image lives, as the registry API addresses it.
|
||||
type upstream struct {
|
||||
// base is the scheme and host, e.g. https://registry-1.docker.io.
|
||||
base string
|
||||
// repository is the path under /v2/, e.g. library/alpine.
|
||||
repository string
|
||||
// reference is a tag or a digest.
|
||||
reference string
|
||||
}
|
||||
|
||||
// parseReference splits `[host/]repo[:tag][@digest]` the way a container runtime does: no host
|
||||
// means the public hub, and a single-segment repository there lives under `library/`.
|
||||
func parseReference(ref string) (upstream, error) {
|
||||
name, reference := ref, "latest"
|
||||
if at := strings.Index(ref, "@"); at >= 0 {
|
||||
name, reference = ref[:at], ref[at+1:]
|
||||
// `repo:tag@digest` is what a runtime prints; the digest names the image and the tag is
|
||||
// only what it was called. The tag is not part of the repository.
|
||||
if colon := strings.LastIndex(name, ":"); colon > strings.LastIndex(name, "/") {
|
||||
name = name[:colon]
|
||||
}
|
||||
} else if colon := strings.LastIndex(ref, ":"); colon > strings.LastIndex(ref, "/") {
|
||||
name, reference = ref[:colon], ref[colon+1:]
|
||||
}
|
||||
if name == "" || reference == "" {
|
||||
return upstream{}, fmt.Errorf("%q is not an image reference", ref)
|
||||
}
|
||||
host, repository := "docker.io", name
|
||||
if slash := strings.Index(name, "/"); slash >= 0 && strings.ContainsAny(name[:slash], ".:") {
|
||||
host, repository = name[:slash], name[slash+1:]
|
||||
} else if slash >= 0 && name[:slash] == "localhost" {
|
||||
host, repository = name[:slash], name[slash+1:]
|
||||
}
|
||||
if host == "docker.io" {
|
||||
host = "registry-1.docker.io"
|
||||
if !strings.Contains(repository, "/") {
|
||||
repository = "library/" + repository
|
||||
}
|
||||
}
|
||||
scheme := "https://"
|
||||
if strings.HasPrefix(host, "localhost") || strings.HasPrefix(host, "127.") {
|
||||
scheme = "http://"
|
||||
}
|
||||
return upstream{base: scheme + host, repository: repository, reference: reference}, nil
|
||||
}
|
||||
|
||||
// source reads from one upstream registry, taking a bearer token where the registry asks for one.
|
||||
type source struct {
|
||||
client *http.Client
|
||||
token string
|
||||
// mountFrom is a repository of the mesh's own registry the source is, so a blob is mounted from
|
||||
// it rather than read and written again.
|
||||
mountFrom string
|
||||
}
|
||||
|
||||
// get fetches a registry URL, answering a bearer challenge once with an anonymous token — which is
|
||||
// how the public hub serves public images, and every registry the catalogue names does the same.
|
||||
func (s *source) get(ctx context.Context, url, accept string) (*http.Response, error) {
|
||||
for attempt := 0; attempt < 2; attempt++ {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if accept != "" {
|
||||
request.Header.Set("Accept", accept)
|
||||
}
|
||||
if s.token != "" {
|
||||
request.Header.Set("Authorization", "Bearer "+s.token)
|
||||
}
|
||||
response, err := s.client.Do(request)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.StatusCode != http.StatusUnauthorized || attempt == 1 {
|
||||
return response, nil
|
||||
}
|
||||
challenge := response.Header.Get("WWW-Authenticate")
|
||||
response.Body.Close()
|
||||
token, err := s.tokenFor(ctx, challenge)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.token = token
|
||||
}
|
||||
return nil, fmt.Errorf("unreachable")
|
||||
}
|
||||
|
||||
// tokenFor answers `Bearer realm="…",service="…",scope="…"` with an anonymous token request.
|
||||
func (s *source) tokenFor(ctx context.Context, challenge string) (string, error) {
|
||||
if !strings.HasPrefix(challenge, "Bearer ") {
|
||||
return "", fmt.Errorf("the registry asks for %q, and this copies public images anonymously", challenge)
|
||||
}
|
||||
fields := map[string]string{}
|
||||
for _, part := range strings.Split(challenge[len("Bearer "):], ",") {
|
||||
key, value, found := strings.Cut(strings.TrimSpace(part), "=")
|
||||
if found {
|
||||
fields[key] = strings.Trim(value, `"`)
|
||||
}
|
||||
}
|
||||
realm := fields["realm"]
|
||||
if realm == "" {
|
||||
return "", fmt.Errorf("the registry's challenge names no realm: %q", challenge)
|
||||
}
|
||||
url := realm + "?service=" + fields["service"]
|
||||
if scope := fields["scope"]; scope != "" {
|
||||
url += "&scope=" + scope
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
response, err := s.client.Do(request)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot get a token from %s: %w", realm, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
var issued struct {
|
||||
Token string `json:"token"`
|
||||
AccessToken string `json:"access_token"`
|
||||
}
|
||||
if err := json.NewDecoder(response.Body).Decode(&issued); err != nil {
|
||||
return "", fmt.Errorf("%s answered with something that is not a token: %w", realm, err)
|
||||
}
|
||||
if issued.Token == "" {
|
||||
issued.Token = issued.AccessToken
|
||||
}
|
||||
if issued.Token == "" {
|
||||
return "", fmt.Errorf("%s issued no token", realm)
|
||||
}
|
||||
return issued.Token, nil
|
||||
}
|
||||
|
||||
// descriptor is what an index or a manifest names: a blob or another manifest, by digest.
|
||||
type descriptor struct {
|
||||
MediaType string `json:"mediaType"`
|
||||
Digest string `json:"digest"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
|
||||
// MirrorImage copies `from` — an index or a single manifest, by tag or digest — into this registry
|
||||
// under `repository`, and returns the reference the mesh will pin: this registry, the repository,
|
||||
// and the digest of the document that was put last, which is the index where there is one.
|
||||
func (r Registry) MirrorImage(ctx context.Context, from, repository string) (string, error) {
|
||||
return r.MirrorBase(ctx, from, repository, "")
|
||||
}
|
||||
|
||||
// MirrorBase is MirrorImage, and where `former` — a repository of this registry — already holds the
|
||||
// image by its digest, the copy is made from there: manifests read from this registry, blobs mounted
|
||||
// across rather than moved (novox/hq ADR 0257). Upstream is asked only for what no repository here
|
||||
// holds, which is what kept the public hub's anonymous pull limit out of reach when every module's
|
||||
// base moved into one shared repository.
|
||||
func (r Registry) MirrorBase(ctx context.Context, from, repository, former string) (string, error) {
|
||||
where, err := parseReference(from)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// **Already held is already mirrored.** A base is named by digest, and a digest this registry
|
||||
// holds under the module's repository is the same bytes whatever upstream would say — so
|
||||
// upstream is not asked. Asked every build, the public hub's anonymous pull limit was reached
|
||||
// on the first merge that rebuilt a whole catalogue (2026-09-28), and every module whose base
|
||||
// lives there failed on a copy it did not need.
|
||||
if strings.HasPrefix(where.reference, "sha256:") {
|
||||
// **Held whole, not only its index** (novox/hq ADR 0257): every platform manifest the index
|
||||
// names is asked about too. A sweep that stopped half-way, or that ran while this build was
|
||||
// copying, may have left an index naming a platform the store no longer holds; such a copy
|
||||
// is made again, and the copy puts back what is missing.
|
||||
held, err := r.holdsWhole(ctx, repository, where.reference)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("asking %s whether it holds %s: %w", r.Address, from, err)
|
||||
}
|
||||
if held {
|
||||
return r.Address + "/" + repository + "@" + where.reference, nil
|
||||
}
|
||||
}
|
||||
src := &source{client: r.client()}
|
||||
if former != "" && former != repository && strings.HasPrefix(where.reference, "sha256:") {
|
||||
held, err := r.holdsWhole(ctx, former, where.reference)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("asking %s whether %s holds %s: %w", r.Address, former, from, err)
|
||||
}
|
||||
if held {
|
||||
// The same bytes, already here: copied from this registry, each blob mounted.
|
||||
where = upstream{base: "http://" + r.Address, repository: former, reference: where.reference}
|
||||
src.mountFrom = former
|
||||
}
|
||||
}
|
||||
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("copying %s into %s/%s: %w", from, r.Address, repository, err)
|
||||
}
|
||||
return r.Address + "/" + repository + "@" + digest, nil
|
||||
}
|
||||
|
||||
// holdsWhole is whether this registry holds the manifest under the repository and, for an index, every
|
||||
// manifest it names.
|
||||
func (r Registry) holdsWhole(ctx context.Context, repository, digest string) (bool, error) {
|
||||
url := "http://" + r.Address + "/v2/" + repository + "/manifests/"
|
||||
held, err := r.has(ctx, url+digest, manifestAccept)
|
||||
if err != nil || !held {
|
||||
return false, err
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url+digest, nil)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
request.Header.Set("Accept", manifestAccept)
|
||||
response, err := r.client().Do(request)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return false, nil
|
||||
}
|
||||
var document struct {
|
||||
Manifests []descriptor `json:"manifests"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&document); err != nil {
|
||||
return false, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
|
||||
}
|
||||
for _, m := range document.Manifests {
|
||||
held, err := r.has(ctx, url+m.Digest, manifestAccept)
|
||||
if err != nil || !held {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// copyManifest copies one manifest document and everything it names, and returns its digest. An
|
||||
// index is copied by copying each manifest it names first, so the index never points at something
|
||||
// the registry does not hold yet.
|
||||
func (r Registry) copyManifest(ctx context.Context, src *source, where upstream, reference, repository string) (string, error) {
|
||||
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/manifests/"+reference, manifestAccept)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
said, _ := io.ReadAll(io.LimitReader(response.Body, 2048))
|
||||
return "", fmt.Errorf("%s/%s@%s: %s %s", where.base, where.repository, reference, response.Status, strings.TrimSpace(string(said)))
|
||||
}
|
||||
body, err := io.ReadAll(response.Body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
mediaType := response.Header.Get("Content-Type")
|
||||
if semi := strings.Index(mediaType, ";"); semi >= 0 {
|
||||
mediaType = mediaType[:semi]
|
||||
}
|
||||
var document struct {
|
||||
MediaType string `json:"mediaType"`
|
||||
Manifests []descriptor `json:"manifests"`
|
||||
Config *descriptor `json:"config"`
|
||||
Layers []descriptor `json:"layers"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &document); err != nil {
|
||||
return "", fmt.Errorf("%s is not a manifest: %w", reference, err)
|
||||
}
|
||||
if mediaType == "" || mediaType == "application/json" {
|
||||
mediaType = document.MediaType
|
||||
}
|
||||
|
||||
switch mediaType {
|
||||
case mediaIndexOCI, mediaIndexDocker:
|
||||
// The manifests first, each by its digest; the index that names them last.
|
||||
for _, m := range document.Manifests {
|
||||
if _, err := r.copyManifest(ctx, src, where, m.Digest, repository); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
case mediaManifestOCI, mediaManifestDocker:
|
||||
blobs := append([]descriptor{}, document.Layers...)
|
||||
if document.Config != nil {
|
||||
blobs = append(blobs, *document.Config)
|
||||
}
|
||||
for _, b := range blobs {
|
||||
if err := r.copyBlob(ctx, src, where, b.Digest, repository); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
default:
|
||||
return "", fmt.Errorf("%s is a %q, which is neither an image index nor an image manifest", reference, mediaType)
|
||||
}
|
||||
|
||||
digest := "sha256:" + hexOf(sha256.Sum256(body))
|
||||
put, err := http.NewRequestWithContext(ctx, http.MethodPut,
|
||||
"http://"+r.Address+"/v2/"+repository+"/manifests/"+digest, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
put.Header.Set("Content-Type", mediaType)
|
||||
done, err := r.client().Do(put)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot put a manifest into %s: %w", r.Address, err)
|
||||
}
|
||||
defer done.Body.Close()
|
||||
if done.StatusCode != http.StatusCreated {
|
||||
said, _ := io.ReadAll(io.LimitReader(done.Body, 2048))
|
||||
return "", fmt.Errorf("%s refused the manifest %s: %s %s", r.Address, digest, done.Status, strings.TrimSpace(string(said)))
|
||||
}
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// copyBlob moves one blob by digest, unless the registry already holds it — blobs are immutable
|
||||
// and content-named, so "already there" is the whole check.
|
||||
func (r Registry) copyBlob(ctx context.Context, src *source, where upstream, digest, repository string) error {
|
||||
base := "http://" + r.Address + "/v2/" + repository
|
||||
if there, err := r.has(ctx, base+"/blobs/"+digest); err != nil {
|
||||
return err
|
||||
} else if there {
|
||||
return nil
|
||||
}
|
||||
// **Mounted where this registry already holds it** (novox/hq ADR 0257): a blob is stored once
|
||||
// whichever repositories link it, so a mount moves no bytes. A registry that cannot mount answers
|
||||
// with an ordinary upload's location, and the blob is moved as before.
|
||||
uploads := base + "/blobs/uploads/"
|
||||
if src.mountFrom != "" {
|
||||
uploads += "?mount=" + digest + "&from=" + src.mountFrom
|
||||
}
|
||||
start, err := http.NewRequestWithContext(ctx, http.MethodPost, uploads, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
begun, err := r.client().Do(start)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot start an upload to %s: %w", base, err)
|
||||
}
|
||||
begun.Body.Close()
|
||||
if begun.StatusCode == http.StatusCreated && src.mountFrom != "" {
|
||||
return nil
|
||||
}
|
||||
if begun.StatusCode != http.StatusAccepted {
|
||||
return fmt.Errorf("%s answered %s when asked where to put a blob", base, begun.Status)
|
||||
}
|
||||
|
||||
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/blobs/"+digest, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("%s/%s: blob %s: %s", where.base, where.repository, digest, response.Status)
|
||||
}
|
||||
location := begun.Header.Get("Location")
|
||||
if location == "" {
|
||||
return fmt.Errorf("%s accepted an upload and said nowhere to put it", base)
|
||||
}
|
||||
if strings.HasPrefix(location, "/") {
|
||||
location = "http://" + r.Address + location
|
||||
}
|
||||
put, err := http.NewRequestWithContext(ctx, http.MethodPut, location+separator(location)+"digest="+digest, response.Body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
put.Header.Set("Content-Type", "application/octet-stream")
|
||||
if response.ContentLength > 0 {
|
||||
put.ContentLength = response.ContentLength
|
||||
}
|
||||
// **Not waited for if refused** (novox/hq issue 457): the body streams from upstream and cannot be
|
||||
// read twice, so a registry held still between the POST above and this PUT fails the copy with
|
||||
// "its body cannot be read twice" rather than waiting. Accepted: the POST a moment before already
|
||||
// waited the registry out, so the window is the length of one upstream fetch, and the build fails
|
||||
// loudly, to be asked again, rather than buffering every base blob in memory.
|
||||
done, err := r.client().Do(put)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot upload blob %s: %w", digest, err)
|
||||
}
|
||||
defer done.Body.Close()
|
||||
if done.StatusCode != http.StatusCreated {
|
||||
said, _ := io.ReadAll(io.LimitReader(done.Body, 2048))
|
||||
return fmt.Errorf("%s refused blob %s: %s %s", base, digest, done.Status, strings.TrimSpace(string(said)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func hexOf(sum [32]byte) string { return hex.EncodeToString(sum[:]) }
|
||||
@@ -1,40 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestABuildSaysWhatItMirroredEvenWhenItFails(t *testing.T) {
|
||||
// A recipe that reaches for an image nobody declared is refused — after the base was copied.
|
||||
r, workspace := aRepository(t, anImage, map[string]string{
|
||||
"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}\nCOPY --from=vendor/other:1 /x /x"})
|
||||
r.contents["modules/bus/module.json"] = anImage
|
||||
r.tree = "aaaa"
|
||||
m := &mirroring{recorded: r, at: "registry-a:5000"}
|
||||
got, err := Build(context.Background(), r.run, m, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
|
||||
nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a recipe fetching an undeclared image was built")
|
||||
}
|
||||
want := "registry-a:5000/upstream/docker.io/vendor/server@sha256:" + strings.Repeat("1", 64)
|
||||
if len(got.Mirrored) != 1 || got.Mirrored[0] != want {
|
||||
t.Fatalf("a failed build said it mirrored %v, want [%s]", got.Mirrored, want)
|
||||
}
|
||||
|
||||
// And a build that works says it too.
|
||||
r2, workspace2 := aRepository(t, anImage, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
|
||||
r2.contents["modules/bus/module.json"] = anImage
|
||||
r2.tree = "aaaa"
|
||||
ok, err := Build(context.Background(), r2.run, &mirroring{recorded: r2, at: "registry-a:5000"},
|
||||
"https://forge.invalid/catalogue.git", "modules/bus", "", workspace2, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(ok.Mirrored) != 1 || ok.Mirrored[0] != want {
|
||||
t.Fatalf("a build said it mirrored %v, want [%s]", ok.Mirrored, want)
|
||||
}
|
||||
}
|
||||
@@ -1,345 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// One copy per upstream image, whichever modules stand on it (novox/hq ADR 0257).
|
||||
|
||||
func TestAnUpstreamImageHasOneRepositoryNamedForIt(t *testing.T) {
|
||||
for from, want := range map[string]string{
|
||||
"golang@sha256:abc": "upstream/docker.io/library/golang",
|
||||
"n8nio/n8n@sha256:abc": "upstream/docker.io/n8nio/n8n",
|
||||
"quay.io/minio/mc@sha256:abc": "upstream/quay.io/minio/mc",
|
||||
"ghcr.io/Mailu/Admin@sha256:abc": "upstream/ghcr.io/mailu/admin",
|
||||
"localhost:5000/x/y@sha256:abc": "upstream/localhost-5000/x/y",
|
||||
"docker.io/library/alpine:3.20@sha256:ab": "upstream/docker.io/library/alpine",
|
||||
} {
|
||||
got, err := MirrorRepository(from)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", from, err)
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("%s: got %q want %q", from, got, want)
|
||||
}
|
||||
}
|
||||
if got := FormerMirrorRepository("route-proxy", "GO_BASE"); got != "route-proxy/on-go_base" {
|
||||
t.Fatalf("the former repository is %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// aRegistryOfRepositories is a registry the way the real one is: blobs stored once, and each
|
||||
// repository linking the blobs and manifests it holds. It mounts a blob across repositories.
|
||||
type aRegistryOfRepositories struct {
|
||||
mu sync.Mutex
|
||||
blobs map[string][]byte // digest → bytes, stored once
|
||||
links map[string]map[string]bool // repository → blob digests it links
|
||||
manifests map[string][]byte // repository@digest → document
|
||||
uploads int
|
||||
mounts int
|
||||
gets int
|
||||
// noMount answers every mount with an ordinary upload's location, as a registry that cannot
|
||||
// mount across repositories does.
|
||||
noMount bool
|
||||
}
|
||||
|
||||
func newRegistryOfRepositories() *aRegistryOfRepositories {
|
||||
return &aRegistryOfRepositories{blobs: map[string][]byte{}, links: map[string]map[string]bool{},
|
||||
manifests: map[string][]byte{}}
|
||||
}
|
||||
|
||||
func (m *aRegistryOfRepositories) link(repository, digest string) {
|
||||
if m.links[repository] == nil {
|
||||
m.links[repository] = map[string]bool{}
|
||||
}
|
||||
m.links[repository][digest] = true
|
||||
}
|
||||
|
||||
// split reads /v2/<repository>/<kind>/<rest> with a repository of any depth.
|
||||
func splitPath(path string) (repository, kind, rest string) {
|
||||
path = strings.TrimPrefix(path, "/v2/")
|
||||
for _, k := range []string{"/manifests/", "/blobs/uploads/", "/blobs/"} {
|
||||
if i := strings.Index(path, k); i >= 0 {
|
||||
return path[:i], strings.Trim(k, "/"), path[i+len(k):]
|
||||
}
|
||||
}
|
||||
return "", "", ""
|
||||
}
|
||||
|
||||
func (m *aRegistryOfRepositories) handler() http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
repository, kind, rest := splitPath(r.URL.Path)
|
||||
switch {
|
||||
case kind == "manifests" && (r.Method == http.MethodHead || r.Method == http.MethodGet):
|
||||
body, ok := m.manifests[repository+"@"+rest]
|
||||
if !ok {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodGet {
|
||||
m.gets++
|
||||
w.Header().Set("Content-Type", mediaTypeOf(body))
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
case kind == "manifests" && r.Method == http.MethodPut:
|
||||
body, _ := readAll(r)
|
||||
m.manifests[repository+"@"+digestOf(body)] = body
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
case kind == "blobs" && (r.Method == http.MethodHead || r.Method == http.MethodGet):
|
||||
if !m.links[repository][rest] {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodGet {
|
||||
m.gets++
|
||||
_, _ = w.Write(m.blobs[rest])
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
case kind == "manifests" && r.Method == http.MethodDelete:
|
||||
delete(m.manifests, repository+"@"+rest)
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
case kind == "blobs/uploads" && r.Method == http.MethodPost:
|
||||
if digest, from := r.URL.Query().Get("mount"), r.URL.Query().Get("from"); digest != "" && m.links[from][digest] && !m.noMount {
|
||||
m.link(repository, digest)
|
||||
m.mounts++
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Location", "/v2/"+repository+"/blobs/uploads/one")
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
case kind == "blobs/uploads" && r.Method == http.MethodPut:
|
||||
body, _ := readAll(r)
|
||||
digest := r.URL.Query().Get("digest")
|
||||
if digestOf(body) != digest {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
m.blobs[digest] = body
|
||||
m.link(repository, digest)
|
||||
m.uploads++
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func mediaTypeOf(body []byte) string {
|
||||
switch {
|
||||
case strings.Contains(string(body), mediaIndexOCI):
|
||||
return mediaIndexOCI
|
||||
default:
|
||||
return mediaManifestOCI
|
||||
}
|
||||
}
|
||||
|
||||
// A base a module's own repository already holds is copied into the image's repository from there:
|
||||
// every blob mounted, nothing asked of upstream — which may be gone, or rationing anonymous pulls.
|
||||
func TestABaseHeldUnderTheModulesFormerRepositoryIsMountedNotFetchedAgain(t *testing.T) {
|
||||
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
|
||||
dst := newRegistryOfRepositories()
|
||||
dstServer := httptest.NewServer(dst.handler())
|
||||
defer dstServer.Close()
|
||||
address := strings.TrimPrefix(dstServer.URL, "http://")
|
||||
r := Registry{Address: address, HTTP: src.Client()}
|
||||
host := strings.TrimPrefix(src.URL, "http://")
|
||||
|
||||
// Before: copied the old way, under the module's repository.
|
||||
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/on-thing_base"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
uploaded := dst.uploads
|
||||
if uploaded != len(srcBlobs) {
|
||||
t.Fatalf("the first copy uploaded %d of %d blobs", uploaded, len(srcBlobs))
|
||||
}
|
||||
src.Close()
|
||||
|
||||
from := host + "/library/thing@" + indexDigest
|
||||
repository, err := MirrorRepository(from)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reference, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base")
|
||||
if err != nil {
|
||||
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
|
||||
}
|
||||
if reference != address+"/"+repository+"@"+indexDigest {
|
||||
t.Fatalf("pinned as %q", reference)
|
||||
}
|
||||
if dst.uploads != uploaded {
|
||||
t.Fatalf("the move to one repository uploaded %d blobs again", dst.uploads-uploaded)
|
||||
}
|
||||
if dst.mounts != len(srcBlobs) {
|
||||
t.Fatalf("%d of %d blobs mounted", dst.mounts, len(srcBlobs))
|
||||
}
|
||||
// The index and both platform manifests are in the image's repository, and every blob is linked.
|
||||
for key := range dst.manifests {
|
||||
if strings.HasPrefix(key, "hello-web/") {
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(key, repository+"@") {
|
||||
t.Fatalf("a manifest went to %s", key)
|
||||
}
|
||||
}
|
||||
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
|
||||
t.Fatalf("%d manifests in %s, want the index and two platforms", n, repository)
|
||||
}
|
||||
for digest := range srcBlobs {
|
||||
if !dst.links[repository][digest] {
|
||||
t.Fatalf("blob %s is not linked into %s", digest, repository)
|
||||
}
|
||||
}
|
||||
|
||||
// A second module standing on the same image: already held, nothing copied, the same reference.
|
||||
again, err := r.MirrorBase(context.Background(), from, repository, "other-module/on-thing_base")
|
||||
if err != nil || again != reference {
|
||||
t.Fatalf("a second module's copy: %q %v", again, err)
|
||||
}
|
||||
}
|
||||
|
||||
func countPrefix(m map[string][]byte, prefix string) int {
|
||||
n := 0
|
||||
for k := range m {
|
||||
if strings.HasPrefix(k, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// What a build copied is said whether it worked or not: the copy is in the store either way, and a
|
||||
// build that failed after copying is the one record that it is there.
|
||||
|
||||
// held puts one image in a repository the old way and answers its index digest and the registry.
|
||||
func heldUnderAModule(t *testing.T) (*aRegistryOfRepositories, Registry, string, string, map[string][]byte, func()) {
|
||||
t.Helper()
|
||||
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
|
||||
t.Cleanup(src.Close)
|
||||
dst := newRegistryOfRepositories()
|
||||
dstServer := httptest.NewServer(dst.handler())
|
||||
t.Cleanup(dstServer.Close)
|
||||
r := Registry{Address: strings.TrimPrefix(dstServer.URL, "http://"), HTTP: src.Client()}
|
||||
host := strings.TrimPrefix(src.URL, "http://")
|
||||
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/on-thing_base"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return dst, r, host + "/library/thing@" + indexDigest, indexDigest, srcBlobs, src.Close
|
||||
}
|
||||
|
||||
// An index whose platform manifests are not all held is not "already held": it is copied again, and the
|
||||
// copy puts back what is missing — what a sweep that stopped half-way, or ran while a build was copying,
|
||||
// leaves behind.
|
||||
func TestAnIndexMissingAPlatformIsCopiedAgain(t *testing.T) {
|
||||
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
|
||||
repository, _ := MirrorRepository(from)
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A sweep lets one platform go, between this build's copy and the next.
|
||||
var platform string
|
||||
for key := range dst.manifests {
|
||||
if strings.HasPrefix(key, repository+"@") && !strings.HasSuffix(key, indexDigest) {
|
||||
platform = key
|
||||
break
|
||||
}
|
||||
}
|
||||
delete(dst.manifests, platform)
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, back := dst.manifests[platform]; !back {
|
||||
t.Fatalf("an index missing %s was taken as held", platform)
|
||||
}
|
||||
}
|
||||
|
||||
// The same, for the module's former copy: a former copy missing a platform is not a source; upstream
|
||||
// is asked instead.
|
||||
func TestAFormerCopyMissingAPlatformIsNotTheSource(t *testing.T) {
|
||||
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
|
||||
for key := range dst.manifests {
|
||||
if strings.HasPrefix(key, "hello-web/on-thing_base@") && !strings.HasSuffix(key, indexDigest) {
|
||||
delete(dst.manifests, key)
|
||||
break
|
||||
}
|
||||
}
|
||||
repository, _ := MirrorRepository(from)
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dst.mounts != 0 {
|
||||
t.Fatalf("a former copy missing a platform was mounted from (%d mounts)", dst.mounts)
|
||||
}
|
||||
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
|
||||
t.Fatalf("%d manifests copied from upstream, want 3", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A registry that answers a mount with an upload's location (202) gets the blob moved as before.
|
||||
func TestAMountRefusedFallsBackToAnUpload(t *testing.T) {
|
||||
dst, r, from, _, srcBlobs, _ := heldUnderAModule(t)
|
||||
dst.noMount = true
|
||||
uploaded := dst.uploads
|
||||
repository, _ := MirrorRepository(from)
|
||||
reference, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(reference, repository+"@"+strings.SplitN(from, "@", 2)[1]) {
|
||||
t.Fatalf("pinned as %s", reference)
|
||||
}
|
||||
if dst.mounts != 0 || dst.uploads-uploaded != len(srcBlobs) {
|
||||
t.Fatalf("%d mounts, %d uploads; want every blob uploaded from the former copy", dst.mounts, dst.uploads-uploaded)
|
||||
}
|
||||
for digest := range srcBlobs {
|
||||
if !dst.links[repository][digest] {
|
||||
t.Fatalf("blob %s is not linked into %s", digest, repository)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A sweep and a build at once: while builds copy the base, platforms are let go of under them. Each
|
||||
// build that returns has the copy whole.
|
||||
func TestABuildCopyingWhileASweepLetsGoEndsWhole(t *testing.T) {
|
||||
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
|
||||
repository, _ := MirrorRepository(from)
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for i := 0; i < 50; i++ {
|
||||
dst.mu.Lock()
|
||||
for key := range dst.manifests {
|
||||
if strings.HasPrefix(key, repository+"@") && !strings.HasSuffix(key, indexDigest) {
|
||||
delete(dst.manifests, key)
|
||||
break
|
||||
}
|
||||
}
|
||||
dst.mu.Unlock()
|
||||
}
|
||||
}()
|
||||
for i := 0; i < 20; i++ {
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
<-done
|
||||
// The sweep is over; the next build finds what it left and makes the copy whole.
|
||||
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
|
||||
t.Fatalf("after the sweep and the builds, %d manifests in %s, want 3", n, repository)
|
||||
}
|
||||
}
|
||||
@@ -1,266 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// An upstream image is copied between registries, never through a machine's image store
|
||||
// (novox/hq 04-ISSUES/046, ADR 0096): the index, every manifest it names, every blob — moved by
|
||||
// digest, and the index put last under the module's repository.
|
||||
|
||||
func digestOf(b []byte) string {
|
||||
sum := sha256.Sum256(b)
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// anUpstreamRegistry serves one image as an index over two platforms, behind an anonymous bearer
|
||||
// challenge the way the public hub does, and records what was fetched.
|
||||
func anUpstreamRegistry(t *testing.T) (*httptest.Server, string, map[string][]byte) {
|
||||
t.Helper()
|
||||
blobs := map[string][]byte{}
|
||||
manifests := map[string][]byte{}
|
||||
put := func(kind, mediaType string, layer []byte) string {
|
||||
config := []byte(`{"architecture":"` + kind + `"}`)
|
||||
blobs[digestOf(config)] = config
|
||||
blobs[digestOf(layer)] = layer
|
||||
m, _ := json.Marshal(map[string]any{
|
||||
"schemaVersion": 2, "mediaType": mediaType,
|
||||
"config": map[string]any{"mediaType": "application/vnd.oci.image.config.v1+json", "digest": digestOf(config), "size": len(config)},
|
||||
"layers": []map[string]any{{"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": digestOf(layer), "size": len(layer)}},
|
||||
})
|
||||
manifests[digestOf(m)] = m
|
||||
return digestOf(m)
|
||||
}
|
||||
amd := put("amd64", mediaManifestOCI, []byte("amd64 layer bytes"))
|
||||
arm := put("arm64", mediaManifestOCI, []byte("arm64 layer bytes"))
|
||||
index, _ := json.Marshal(map[string]any{
|
||||
"schemaVersion": 2, "mediaType": mediaIndexOCI,
|
||||
"manifests": []map[string]any{
|
||||
{"mediaType": mediaManifestOCI, "digest": amd, "size": len(manifests[amd]), "platform": map[string]string{"os": "linux", "architecture": "amd64"}},
|
||||
{"mediaType": mediaManifestOCI, "digest": arm, "size": len(manifests[arm]), "platform": map[string]string{"os": "linux", "architecture": "arm64"}},
|
||||
},
|
||||
})
|
||||
manifests["latest"] = index
|
||||
manifests[digestOf(index)] = index
|
||||
|
||||
var server *httptest.Server
|
||||
server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/token" {
|
||||
_, _ = w.Write([]byte(`{"token":"anonymous-token"}`))
|
||||
return
|
||||
}
|
||||
if r.Header.Get("Authorization") != "Bearer anonymous-token" {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="`+server.URL+`/token",service="test",scope="repository:library/thing:pull"`)
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(r.URL.Path, "/v2/library/thing/manifests/"):
|
||||
ref := strings.TrimPrefix(r.URL.Path, "/v2/library/thing/manifests/")
|
||||
body, ok := manifests[ref]
|
||||
if !ok {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
var typed struct {
|
||||
MediaType string `json:"mediaType"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &typed)
|
||||
w.Header().Set("Content-Type", typed.MediaType)
|
||||
_, _ = w.Write(body)
|
||||
case strings.HasPrefix(r.URL.Path, "/v2/library/thing/blobs/"):
|
||||
body, ok := blobs[strings.TrimPrefix(r.URL.Path, "/v2/library/thing/blobs/")]
|
||||
if !ok {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(body)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
return server, digestOf(index), blobs
|
||||
}
|
||||
|
||||
// theMeshsRegistry accepts blobs and manifests the way a registry does, and remembers them.
|
||||
type theMeshsRegistry struct {
|
||||
mu sync.Mutex
|
||||
blobs map[string][]byte
|
||||
manifests map[string][]byte
|
||||
uploads int
|
||||
}
|
||||
|
||||
func (m *theMeshsRegistry) handler() http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
switch {
|
||||
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/manifests/"):
|
||||
// **As strictly as a real registry.** A manifest is answered only in a media type the
|
||||
// caller named; a request with no Accept is answered as if nothing were there. The fake
|
||||
// used to answer regardless, which is why it could not catch a check that asked without
|
||||
// one — and the mesh copied every base again (2026-09-28).
|
||||
if !strings.Contains(r.Header.Get("Accept"), "manifest") && !strings.Contains(r.Header.Get("Accept"), "index") {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if _, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
} else {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/manifests/"):
|
||||
body, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]
|
||||
if !ok {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(body)
|
||||
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
|
||||
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
} else {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/blobs/uploads/"):
|
||||
w.Header().Set("Location", strings.TrimSuffix(r.URL.Path, "/")+"/one")
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
case r.Method == http.MethodPut && strings.Contains(r.URL.Path, "/blobs/uploads/"):
|
||||
body, _ := readAll(r)
|
||||
digest := r.URL.Query().Get("digest")
|
||||
if digestOf(body) != digest {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
m.blobs[digest] = body
|
||||
m.uploads++
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
case r.Method == http.MethodPut && strings.Contains(r.URL.Path, "/manifests/"):
|
||||
body, _ := readAll(r)
|
||||
m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]] = body
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func readAll(r *http.Request) ([]byte, error) {
|
||||
var buf strings.Builder
|
||||
b := make([]byte, 4096)
|
||||
for {
|
||||
n, err := r.Body.Read(b)
|
||||
buf.Write(b[:n])
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
return []byte(buf.String()), nil
|
||||
}
|
||||
|
||||
func TestAnUpstreamIndexIsCopiedWholeIntoTheMeshsRegistry(t *testing.T) {
|
||||
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
|
||||
defer src.Close()
|
||||
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
|
||||
dstServer := httptest.NewServer(dst.handler())
|
||||
defer dstServer.Close()
|
||||
|
||||
address := strings.TrimPrefix(dstServer.URL, "http://")
|
||||
r := Registry{Address: address, HTTP: src.Client()}
|
||||
from := strings.TrimPrefix(src.URL, "http://") + "/library/thing:latest"
|
||||
reference, err := r.MirrorImage(context.Background(), from, "hello-web/server")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Pinned by the INDEX's digest under the module's own repository: what a machine fetches is
|
||||
// the whole image, whatever its architecture.
|
||||
if reference != address+"/hello-web/server@"+indexDigest {
|
||||
t.Fatalf("pinned as %q, not the index under the module's repository", reference)
|
||||
}
|
||||
// Every blob of both platforms, moved by digest, and each only once.
|
||||
if len(dst.blobs) != len(srcBlobs) || dst.uploads != len(srcBlobs) {
|
||||
t.Fatalf("%d of %d blobs arrived in %d uploads", len(dst.blobs), len(srcBlobs), dst.uploads)
|
||||
}
|
||||
for digest, body := range srcBlobs {
|
||||
if string(dst.blobs[digest]) != string(body) {
|
||||
t.Fatalf("blob %s did not arrive intact", digest)
|
||||
}
|
||||
}
|
||||
// Two manifests and the index, each under its digest.
|
||||
if len(dst.manifests) != 3 {
|
||||
t.Fatalf("expected two manifests and an index, got %d: %v", len(dst.manifests), dst.manifests)
|
||||
}
|
||||
if _, ok := dst.manifests[indexDigest]; !ok {
|
||||
t.Fatal("the index was not put under its digest")
|
||||
}
|
||||
|
||||
// Copied again, nothing is uploaded twice: blobs are content-named and already there.
|
||||
if _, err := r.MirrorImage(context.Background(), from, "hello-web/server"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dst.uploads != len(srcBlobs) {
|
||||
t.Fatalf("a second copy uploaded blobs the registry already held: %d uploads", dst.uploads)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReferenceIsReadTheWayARuntimeReadsIt(t *testing.T) {
|
||||
for ref, want := range map[string]upstream{
|
||||
"alpine": {base: "https://registry-1.docker.io", repository: "library/alpine", reference: "latest"},
|
||||
"alpine@sha256:abc": {base: "https://registry-1.docker.io", repository: "library/alpine", reference: "sha256:abc"},
|
||||
"minio/minio:RELEASE.2025": {base: "https://registry-1.docker.io", repository: "minio/minio", reference: "RELEASE.2025"},
|
||||
"quay.io/minio/mc@sha256:def": {base: "https://quay.io", repository: "minio/mc", reference: "sha256:def"},
|
||||
"lscr.io/linuxserver/sonarr:4": {base: "https://lscr.io", repository: "linuxserver/sonarr", reference: "4"},
|
||||
"localhost:5000/x/y:1": {base: "http://localhost:5000", repository: "x/y", reference: "1"},
|
||||
} {
|
||||
got, err := parseReference(ref)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", ref, err)
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("%s: got %+v want %+v", ref, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// `repo:tag@digest` is what a runtime prints; the tag is not part of the repository (review C6).
|
||||
func TestATagBeforeTheDigestIsNotPartOfTheRepository(t *testing.T) {
|
||||
got, err := parseReference("quay.io/minio/mc:RELEASE.2025@sha256:abc")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.repository != "minio/mc" || got.reference != "sha256:abc" {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A base this registry already holds by digest is not asked of upstream at all: the public hub
|
||||
// limits anonymous pulls, and a catalogue rebuilt on one merge asked it once per module.
|
||||
func TestABaseAlreadyHeldIsNotAskedOfUpstream(t *testing.T) {
|
||||
src, indexDigest, _ := anUpstreamRegistry(t)
|
||||
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
|
||||
dstServer := httptest.NewServer(dst.handler())
|
||||
defer dstServer.Close()
|
||||
address := strings.TrimPrefix(dstServer.URL, "http://")
|
||||
r := Registry{Address: address, HTTP: src.Client()}
|
||||
host := strings.TrimPrefix(src.URL, "http://")
|
||||
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/server"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Upstream gone: the pinned base is answered from what the mesh holds.
|
||||
src.Close()
|
||||
reference, err := r.MirrorImage(context.Background(), host+"/library/thing@"+indexDigest, "hello-web/server")
|
||||
if err != nil {
|
||||
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
|
||||
}
|
||||
if reference != address+"/hello-web/server@"+indexDigest {
|
||||
t.Fatalf("pinned as %q", reference)
|
||||
}
|
||||
}
|
||||
@@ -1,233 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// Npmrc is what a build needs to resolve the mesh's own shared library — and any other package the
|
||||
// mesh publishes — from the mesh's package registry rather than from a git URL (novox/hq ADR 0076,
|
||||
// issue 053).
|
||||
//
|
||||
// It is a build-time credential, not a runtime one. A module compiled inside the toolchain image
|
||||
// resolves the SDK there, once, when that image is built; the running container never speaks to the
|
||||
// package registry. So this is given to the *builder*, the way the artifact store is
|
||||
// (`whereToPublish`), and reaches a build as a secret rather than a layer — see Secret.
|
||||
//
|
||||
// The registry's exact URL shape is the provider's business, not the builder's: it arrives whole,
|
||||
// either from the binding the mesh writes (a package-registry provider's `serves` facts) or from the
|
||||
// environment when a person runs a build by hand. Nothing here knows gitea from verdaccio.
|
||||
type Npmrc struct {
|
||||
// Scope is the npm scope the registry answers for, e.g. "@novox". A build resolves only this
|
||||
// scope from the mesh; everything else resolves the ordinary way, so a mesh with no internet
|
||||
// still cannot pull the public registry's version of a name the mesh also publishes.
|
||||
Scope string
|
||||
// Registry is the full base URL a client uses for this scope, e.g.
|
||||
// "https://<forge>/api/packages/<owner>/npm/". Trailing slash tolerated either way.
|
||||
Registry string
|
||||
// Token authenticates to the registry as a bearer token, when a provider mints one. Left empty
|
||||
// when the mesh authenticates the ordinary way it authenticates everything — a generated
|
||||
// password it applies and seals — for which see Username and Password.
|
||||
Token string
|
||||
// Username and Password authenticate by basic auth, which is what gitea and verdaccio both
|
||||
// accept and what lets the credential be a mesh-generated password the provider's provisioner
|
||||
// applies and the mesh seals to the consumer — the same shape a database password takes. The
|
||||
// username is the consumer's mesh identity. Ignored when Token is set.
|
||||
Username string
|
||||
Password string
|
||||
|
||||
// From says where the build agent resolved this registry from (novox/hq ADR 0336 §4): its
|
||||
// npm-package-registry binding, the genesis document the installer sealed to it, or the
|
||||
// environment of a builder run by a person. Empty when it resolved none.
|
||||
From string
|
||||
// Machine is the machine the build agent runs on, named when a build that needs the registry is
|
||||
// refused for having none (ADR 0336 §6).
|
||||
Machine string
|
||||
}
|
||||
|
||||
// Where a build agent resolves the mesh's package registry from (novox/hq ADR 0336 §3, §4).
|
||||
const (
|
||||
// FromBinding is the npm-package-registry binding the mesh wrote, with the credential it sealed.
|
||||
FromBinding = "the npm-package-registry binding"
|
||||
// FromGenesis is the genesis document the installer sealed at genesis's packages step, read only
|
||||
// while no binding is present.
|
||||
FromGenesis = "the genesis document"
|
||||
// FromEnvironment is the environment of a builder run by a person.
|
||||
FromEnvironment = "the environment"
|
||||
)
|
||||
|
||||
// RegistryRequirement is the build agent's optional requirement the registry is bound through, and
|
||||
// RegistryProvider the catalogue module that provides it (novox/hq ADR 0336 §1, §6): named when a
|
||||
// build that needs the registry is refused, so the outcome says what to assign. The controller's
|
||||
// catalogue test holds the provider to the catalogue's.
|
||||
const (
|
||||
RegistryRequirement = "npm-package-registry"
|
||||
RegistryProvider = "gitea"
|
||||
)
|
||||
|
||||
// Enabled reports whether there is a registry to resolve against at all. A bootstrap build that
|
||||
// runs before any package registry exists has none, and must still build whatever needs no
|
||||
// mesh-published dependency.
|
||||
func (n Npmrc) Enabled() bool {
|
||||
return strings.TrimSpace(n.Scope) != "" && strings.TrimSpace(n.Registry) != ""
|
||||
}
|
||||
|
||||
// File renders the .npmrc a build mounts. Two lines: which registry answers for the scope, and the
|
||||
// token to present to it. The auth line is keyed by the registry URL with its scheme removed, which
|
||||
// is how npm matches a stored credential to a request.
|
||||
//
|
||||
// It returns an error rather than a malformed file, because an .npmrc that npm parses but points
|
||||
// nowhere fails much later, inside a build, as a package that cannot be found.
|
||||
func (n Npmrc) File() (string, error) {
|
||||
scope := strings.TrimSpace(n.Scope)
|
||||
if !strings.HasPrefix(scope, "@") {
|
||||
return "", fmt.Errorf("a package-registry scope is written with its leading @, not %q", scope)
|
||||
}
|
||||
reg := strings.TrimSpace(n.Registry)
|
||||
if !strings.HasPrefix(reg, "http://") && !strings.HasPrefix(reg, "https://") {
|
||||
return "", fmt.Errorf("a package registry is reached over http(s), and %q is neither", reg)
|
||||
}
|
||||
if !strings.HasSuffix(reg, "/") {
|
||||
// npm's per-scope registry key is matched by prefix, and the auth key below is derived from
|
||||
// it; a missing trailing slash makes the two disagree and the token is never sent.
|
||||
reg += "/"
|
||||
}
|
||||
parsed, err := url.Parse(reg)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%q is not a usable registry URL: %w", reg, err)
|
||||
}
|
||||
// The auth key is the URL without its scheme, e.g. "//host/api/packages/owner/npm/".
|
||||
authKey := "//" + parsed.Host + parsed.EscapedPath()
|
||||
|
||||
var auth string
|
||||
switch {
|
||||
case strings.TrimSpace(n.Token) != "":
|
||||
auth = fmt.Sprintf("%s:_authToken=%s\n", authKey, strings.TrimSpace(n.Token))
|
||||
case strings.TrimSpace(n.Username) != "" && n.Password != "":
|
||||
// npm reads the password base64-encoded, and always-auth so it presents the credential to
|
||||
// reads as well as writes — a private registry answers neither without it.
|
||||
enc := base64.StdEncoding.EncodeToString([]byte(n.Password))
|
||||
auth = fmt.Sprintf("%s:username=%s\n%s:_password=%s\n%s:always-auth=true\n",
|
||||
authKey, strings.TrimSpace(n.Username), authKey, enc, authKey)
|
||||
default:
|
||||
return "", fmt.Errorf(
|
||||
"the package registry at %s was given neither a token nor a username and password", reg)
|
||||
}
|
||||
return fmt.Sprintf("%s:registry=%s\n%s", scope, reg, auth), nil
|
||||
}
|
||||
|
||||
// needsRegistry says why this module's build needs the mesh's package registry, or "" when it needs
|
||||
// none (novox/hq ADR 0336 §5): a `package` artifact, an image whose Dockerfile names .npmrc, or a
|
||||
// TypeScript artifact with an `@novox/` dependency. Every Go artifact, every image that does not name
|
||||
// .npmrc, and every image pulled upstream builds without it.
|
||||
func needsRegistry(manifest catalogue.Manifest, within string) (string, error) {
|
||||
if manifest.Build == nil {
|
||||
return "", nil
|
||||
}
|
||||
for _, a := range manifest.Build.Artifacts {
|
||||
switch {
|
||||
case a.Kind == catalogue.ArtifactPackage:
|
||||
return fmt.Sprintf("%s is a package, published to the registry", a.Name), nil
|
||||
case a.Kind == catalogue.ArtifactImage:
|
||||
raw, err := os.ReadFile(filepath.Join(within, a.From))
|
||||
if err == nil && strings.Contains(string(raw), ".npmrc") {
|
||||
return fmt.Sprintf("%s's Dockerfile names .npmrc", a.Name), nil
|
||||
}
|
||||
case a.Language == "typescript":
|
||||
scoped, err := scopedDependencies(within)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(scoped) > 0 {
|
||||
return fmt.Sprintf("%s is TypeScript and depends on %s", a.Name, strings.Join(scoped, ", ")), nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// scopedDependencies is every `@novox/` package the module's package.json depends on, the SDK
|
||||
// included.
|
||||
func scopedDependencies(tree string) ([]string, error) {
|
||||
raw, err := os.ReadFile(filepath.Join(tree, "package.json"))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var p struct {
|
||||
Dependencies map[string]string `json:"dependencies"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &p); err != nil {
|
||||
return nil, fmt.Errorf("the module's package.json is not JSON: %w", err)
|
||||
}
|
||||
var names []string
|
||||
for name := range p.Dependencies {
|
||||
if strings.HasPrefix(name, "@novox/") {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names, nil
|
||||
}
|
||||
|
||||
// unbound is the refusal of a build that needs the registry on a build agent that has none (novox/hq
|
||||
// ADR 0336 §6): it names the machine, the requirement left unbound there and the module that provides
|
||||
// it. Nothing is built, nothing is published, and the `@novox` scope is never resolved from the
|
||||
// public registry instead.
|
||||
func (n Npmrc) unbound(module, why string) error {
|
||||
machine := n.Machine
|
||||
if machine == "" {
|
||||
machine = "this machine"
|
||||
}
|
||||
return fmt.Errorf("%s needs the mesh's package registry (%s), and %s is unbound on %s: the build "+
|
||||
"agent there has neither its binding nor the genesis document. It is provided by the %s module; "+
|
||||
"once %s is assigned the requirement binds (novox/hq ADR 0336 §6)",
|
||||
module, why, RegistryRequirement, machine, RegistryProvider, RegistryProvider)
|
||||
}
|
||||
|
||||
// packageRecipe is how a `package` artifact is built and published: on a PUBLIC base image, never
|
||||
// the mesh toolchain, because the toolchain is built from the package this produces (the SDK). The
|
||||
// script builds the module, then publishes it to the mesh's package registry unless that exact
|
||||
// version is already there — so a re-run of genesis, which must be safe, does not fail on a version
|
||||
// it published a moment ago.
|
||||
type packageRecipe struct {
|
||||
Base string
|
||||
Script string
|
||||
}
|
||||
|
||||
var packageRecipes = map[string]packageRecipe{
|
||||
"typescript": {
|
||||
Base: "node:22-bookworm-slim",
|
||||
Script: `set -e
|
||||
npm install --no-audit --no-fund
|
||||
npm run build
|
||||
name="$(node -p "require('./package.json').name")"
|
||||
ver="$(node -p "require('./package.json').version")"
|
||||
if npm view "$name@$ver" version >/dev/null 2>&1; then
|
||||
echo "mesh-builder: $name@$ver is already published, leaving it"
|
||||
else
|
||||
npm publish
|
||||
fi`,
|
||||
},
|
||||
}
|
||||
|
||||
// PackageLanguages is the languages a package artifact can be written in, for a manifest check that
|
||||
// wants to refuse one it cannot build before a build starts.
|
||||
func PackageLanguages() []string {
|
||||
out := make([]string, 0, len(packageRecipes))
|
||||
for l := range packageRecipes {
|
||||
out = append(out, l)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -1,135 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAnImageBuildGetsTheCredentialInTheContextAndHostNetwork(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch\nCOPY .npmrc ./", "files/x": "y"})
|
||||
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
|
||||
npmrc := filepath.Join(workspace, "source", ".npmrc")
|
||||
inContext := false
|
||||
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "build" {
|
||||
_, err := os.Stat(npmrc)
|
||||
inContext = err == nil
|
||||
}
|
||||
return r.run(ctx, dir, name, args...)
|
||||
}
|
||||
if _, err := Build(context.Background(), run, r,
|
||||
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, GitCredential{}, nil); err != nil {
|
||||
t.Fatalf("the build failed: %v", err)
|
||||
}
|
||||
|
||||
var build string
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build") {
|
||||
build = line
|
||||
}
|
||||
}
|
||||
if build == "" {
|
||||
t.Fatal("no docker build ran")
|
||||
}
|
||||
if strings.Contains(build, "--secret") {
|
||||
t.Fatalf("the build used a buildkit secret, which this path avoids: %s", build)
|
||||
}
|
||||
if !strings.Contains(build, "--network host") {
|
||||
t.Fatalf("the build was not given the host network to reach the registry: %s", build)
|
||||
}
|
||||
// The .npmrc is written into the build context (the source tree), where a Dockerfile COPYs it, and
|
||||
// removed with the tree when the build ends: a later check's container mounts the workspace (novox/hq
|
||||
// issue 462).
|
||||
if !inContext {
|
||||
t.Fatal("the credential was not in the build context when the image was built")
|
||||
}
|
||||
if _, err := os.Stat(npmrc); !os.IsNotExist(err) {
|
||||
t.Fatalf("the credential outlives the build in the workspace: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnImageBuildWithoutACredentialGetsNoHostNetwork(t *testing.T) {
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch", "files/x": "y"})
|
||||
if _, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatalf("the build failed: %v", err)
|
||||
}
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build") && (strings.Contains(line, "--secret") || strings.Contains(line, "--network host")) {
|
||||
t.Fatalf("a build with no credential was still given build-network or a secret: %s", line)
|
||||
}
|
||||
}
|
||||
tree := filepath.Join(workspace, "source")
|
||||
if _, err := os.Stat(filepath.Join(tree, ".npmrc")); err == nil {
|
||||
t.Fatal("an .npmrc was written into a build that has no credential")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPackageIsBuiltOnAPublicBaseAndPublishedByVersion(t *testing.T) {
|
||||
r, workspace := aRepository(t, aPackage, map[string]string{
|
||||
"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.0"}`,
|
||||
})
|
||||
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
|
||||
got, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, n, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("the package did not build: %v", err)
|
||||
}
|
||||
if len(got.Built) != 1 || got.Built[0].Reference != "@novox/mesh-sdk@0.1.0" {
|
||||
t.Fatalf("a package is published by name and version, got %+v", got.Built)
|
||||
}
|
||||
if len(r.images) != 0 || len(r.archives) != 0 {
|
||||
t.Fatal("a package was pushed to the artifact store, which is not where packages live")
|
||||
}
|
||||
var ran string
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker run") {
|
||||
ran = line
|
||||
}
|
||||
}
|
||||
if ran == "" {
|
||||
t.Fatal("nothing ran to build the package")
|
||||
}
|
||||
if !strings.Contains(ran, "node:22-bookworm-slim") {
|
||||
t.Fatalf("a package was not built on a public base: %s", ran)
|
||||
}
|
||||
if !strings.Contains(ran, ":/root/.npmrc:ro") {
|
||||
t.Fatalf("the credential was not mounted read-only for the publish: %s", ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPackageWithNoRegistryIsRefused(t *testing.T) {
|
||||
r, workspace := aRepository(t, aPackage, map[string]string{
|
||||
"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.0"}`,
|
||||
})
|
||||
_, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("a package built with no registry to publish to, silently")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnImageThatDoesNotAskForTheCredentialDoesNotGetIt(t *testing.T) {
|
||||
// A Dockerfile with no .npmrc reference (like the control plane's) must build clean: no .npmrc
|
||||
// in its context, no host network — so its image stays deterministic and the credential does not
|
||||
// leak into a build that never resolves a mesh package.
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch\nCOPY . .", "files/x": "y"})
|
||||
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
|
||||
if _, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, GitCredential{}, nil); err != nil {
|
||||
t.Fatalf("the build failed: %v", err)
|
||||
}
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build") && strings.Contains(line, "--network host") {
|
||||
t.Fatalf("a build that does not ask for the credential got the host network: %s", line)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(workspace, "source", ".npmrc")); err == nil {
|
||||
t.Fatal("an .npmrc was written into a build that does not reference it")
|
||||
}
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestNpmrcRendersRegistryAndTokenForTheScope(t *testing.T) {
|
||||
n := Npmrc{
|
||||
Scope: "@novox",
|
||||
Registry: "https://forge.invalid/api/packages/novox/npm/",
|
||||
Token: "a-token",
|
||||
}
|
||||
got, err := n.File()
|
||||
if err != nil {
|
||||
t.Fatalf("a complete credential did not render: %v", err)
|
||||
}
|
||||
if !strings.Contains(got, "@novox:registry=https://forge.invalid/api/packages/novox/npm/") {
|
||||
t.Fatalf("the scope's registry line is missing:\n%s", got)
|
||||
}
|
||||
// The auth line is keyed by the URL without its scheme, or npm never sends the token.
|
||||
if !strings.Contains(got, "//forge.invalid/api/packages/novox/npm/:_authToken=a-token") {
|
||||
t.Fatalf("the auth line does not match the registry key:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNpmrcAddsATrailingSlashSoTheAuthKeyMatches(t *testing.T) {
|
||||
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm", Token: "t"}
|
||||
got, err := n.File()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(got, "registry=https://forge.invalid/api/packages/novox/npm/\n") {
|
||||
t.Fatalf("a missing trailing slash was not normalised:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNpmrcRefusesTheHalfConfigured(t *testing.T) {
|
||||
cases := map[string]Npmrc{
|
||||
"scope without @": {Scope: "novox", Registry: "https://x.invalid/", Token: "t"},
|
||||
"registry not http": {Scope: "@novox", Registry: "ftp://x.invalid/", Token: "t"},
|
||||
"no token": {Scope: "@novox", Registry: "https://x.invalid/", Token: ""},
|
||||
}
|
||||
for name, n := range cases {
|
||||
if _, err := n.File(); err == nil {
|
||||
t.Fatalf("%s rendered an .npmrc rather than refusing", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNpmrcDisabledUntilThereIsARegistry(t *testing.T) {
|
||||
if (Npmrc{}).Enabled() {
|
||||
t.Fatal("an empty credential reported itself usable")
|
||||
}
|
||||
if (Npmrc{Scope: "@novox"}).Enabled() {
|
||||
t.Fatal("a scope with no registry reported itself usable")
|
||||
}
|
||||
if !(Npmrc{Scope: "@novox", Registry: "https://x.invalid/"}).Enabled() {
|
||||
t.Fatal("a scope and a registry did not count as usable")
|
||||
}
|
||||
}
|
||||
|
||||
// The credential reaches an image build as an .npmrc inside the build context — for a Dockerfile to
|
||||
// COPY in a stage it does not publish — and the build runs on the host network so a RUN resolving the
|
||||
// registry reaches it where the binding says (novox/hq ADR 0076). Not a buildkit secret, because this
|
||||
// machine's docker may carry no buildx.
|
||||
|
||||
const aPackage = `{"module":"mesh-sdk","version":"1",
|
||||
"build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript"}]},
|
||||
"resources":[]}`
|
||||
|
||||
// A package is compiled on a public base and published to the mesh's package registry by version,
|
||||
// with nothing pushed to the artifact store and the credential mounted, not baked (novox/hq ADR 0076).
|
||||
|
||||
func TestNpmrcRendersBasicAuthWhenGivenAUserAndPassword(t *testing.T) {
|
||||
n := Npmrc{
|
||||
Scope: "@novox",
|
||||
Registry: "http://forge.invalid:3000/api/packages/novox/npm/",
|
||||
Username: "mesh_anchor_builder",
|
||||
Password: "s3cret",
|
||||
}
|
||||
got, err := n.File()
|
||||
if err != nil {
|
||||
t.Fatalf("basic-auth credential did not render: %v", err)
|
||||
}
|
||||
key := "//forge.invalid:3000/api/packages/novox/npm/"
|
||||
if !strings.Contains(got, key+":username=mesh_anchor_builder\n") {
|
||||
t.Fatalf("username line missing:\n%s", got)
|
||||
}
|
||||
// npm reads the password base64-encoded.
|
||||
if !strings.Contains(got, key+":_password=czNjcmV0\n") {
|
||||
t.Fatalf("base64 password line missing or wrong:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, key+":always-auth=true\n") {
|
||||
t.Fatalf("always-auth missing, so reads would go unauthenticated:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "_authToken") {
|
||||
t.Fatalf("a token line was rendered for a basic-auth credential:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNpmrcRefusesWhenGivenNeitherTokenNorPassword(t *testing.T) {
|
||||
n := Npmrc{Scope: "@novox", Registry: "http://x.invalid/npm/", Username: "u"}
|
||||
if _, err := n.File(); err == nil {
|
||||
t.Fatal("a username with no password rendered an .npmrc")
|
||||
}
|
||||
}
|
||||
@@ -1,108 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A build reads the list of predefined keys at the root of the repository at the commit it builds (novox/hq ADR
|
||||
// 0348 §7, design 57 package K2, issue 661): it refuses a list that is wrong, judges the manifest against it, and
|
||||
// carries it with the commit's generation for the controller to publish.
|
||||
|
||||
const theFirstKeys = `{
|
||||
"font-size": {"type": "number", "description": "the size text is shown in, in points", "minimum": 6, "maximum": 72},
|
||||
"agent-managed-settings": {"type": "object", "description": "the managed settings of every agent session, a JSON object", "trusted": true}
|
||||
}`
|
||||
|
||||
// fallingBack is a module of the catalogue whose font size falls back to the predefined key named.
|
||||
func fallingBack(key, kind string) string {
|
||||
def := "10"
|
||||
if kind == "string" {
|
||||
def = `"10"`
|
||||
}
|
||||
return `{"module":"dunst","version":"1",
|
||||
"build":{"artifacts":[{"name":"look","kind":"archive","from":"files"}]},
|
||||
"config":{"font-size":{"type":"` + kind + `","description":"the notification's font size, in points",
|
||||
"default":` + def + `,"predefined":"` + key + `"}},
|
||||
"resources":[
|
||||
{"id":"theme","type":"archive","path":"/opt/dunst","artifact":"look"},
|
||||
{"id":"rc","type":"file","path":"/etc/xdg/dunst/dunstrc","content":"font = Inter ${config:font-size}\n"}]}`
|
||||
}
|
||||
|
||||
func buildInTheCatalogue(t *testing.T, manifest string, list *string) (Result, error) {
|
||||
t.Helper()
|
||||
r := &recorded{contents: map[string]string{"modules/dunst/" + ManifestName: manifest,
|
||||
"modules/dunst/files/theme.conf": "dark", "README.md": "the catalogue"}}
|
||||
if list != nil {
|
||||
r.contents[catalogue.PredefinedFile] = *list
|
||||
}
|
||||
return Build(context.Background(), onTrunk{recorded: r}.run, r, "https://forge.invalid/mesh-catalog.git",
|
||||
"modules/dunst", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
|
||||
}
|
||||
|
||||
func TestABuildNamingAPredefinedKeyIsJudgedAgainstItsTreesListAndCarriesIt(t *testing.T) {
|
||||
list := theFirstKeys
|
||||
got, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), &list)
|
||||
if err != nil {
|
||||
t.Fatalf("a module falling back to font-size did not build: %v", err)
|
||||
}
|
||||
if got.Predefined == nil || got.Predefined.Generation != 42 {
|
||||
t.Fatalf("the list was not carried with its generation: %+v", got.Predefined)
|
||||
}
|
||||
carried, err := catalogue.ParsePredefined(got.Predefined.Keys)
|
||||
if err != nil || len(carried) != 2 || carried["font-size"].Type != catalogue.ConfigNumber {
|
||||
t.Fatalf("the list carried is %s (%v)", got.Predefined.Keys, err)
|
||||
}
|
||||
// A change to the list is a change to the module that was judged against it.
|
||||
if len(got.Sources) == 0 || !slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
|
||||
t.Fatalf("its build source leaves the list out: %+v", got.Sources)
|
||||
}
|
||||
|
||||
for _, c := range []struct{ manifest, says string }{
|
||||
{fallingBack("font-scale", "number"), `"font-scale", which is not on the list`},
|
||||
{fallingBack("font-size", "string"), "which is a number, and this value is a string"},
|
||||
} {
|
||||
if _, err := buildInTheCatalogue(t, c.manifest, &list); err == nil || !strings.Contains(err.Error(), c.says) {
|
||||
t.Errorf("wanted the build refused saying %q, got %v", c.says, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildRefusesAWrongList(t *testing.T) {
|
||||
wrong := `{"font-size": {"type": "number", "description": "points", "unit": "pt"}}`
|
||||
if _, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), &wrong); err == nil ||
|
||||
!strings.Contains(err.Error(), `unknown field "unit"`) {
|
||||
t.Fatalf("a wrong list was built with: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A repository without the list builds as before, carrying none; a value naming a key there is refused.
|
||||
func TestABuildWithoutAListCarriesNone(t *testing.T) {
|
||||
plain := strings.Replace(fallingBack("font-size", "number"), `,"predefined":"font-size"`, "", 1)
|
||||
got, err := buildInTheCatalogue(t, plain, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Predefined != nil {
|
||||
t.Fatalf("a list was carried from a tree holding none: %+v", got.Predefined)
|
||||
}
|
||||
if slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
|
||||
t.Fatalf("the build source names a list: %+v", got.Sources)
|
||||
}
|
||||
if _, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), nil); err == nil ||
|
||||
!strings.Contains(err.Error(), "is not present") {
|
||||
t.Fatalf("a key named where no list is: %v", err)
|
||||
}
|
||||
// A list present and a module naming no key: carried, and not the module's build source.
|
||||
list := theFirstKeys
|
||||
got, err = buildInTheCatalogue(t, plain, &list)
|
||||
if err != nil || got.Predefined == nil || slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
|
||||
t.Fatalf("a module naming no key: %+v %+v %v", got.Predefined, got.Sources, err)
|
||||
}
|
||||
}
|
||||
@@ -1,191 +0,0 @@
|
||||
package builder
|
||||
|
||||
// **Every line of a check's output is redacted before it is kept** (novox/hq issue 462).
|
||||
//
|
||||
// A repository's own check prints into the build's log, which the bus keeps for days and anyone who may read
|
||||
// its events reads, and into the verdict, which the forge shows on the pull request. Software prints what it
|
||||
// was given — a URL carrying a password, a token in a flag — and a pull request may print on purpose.
|
||||
// So a line is said only after every secret the builder knows (the forge credential's password) and every
|
||||
// value whose shape says it is one is replaced by a mark naming what was there, as the journal verb does.
|
||||
//
|
||||
// Copied from the journal tool's redactor (mesh-catalog, modules/systemd/cmd/systemd-tools/secrets.go,
|
||||
// itself a copy of the docker module's), narrowed to a line's shapes, with the token shapes a check's output
|
||||
// may carry added. A third copy: sharing them through mesh-sdk is novox/hq issue 471.
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// secretName is a variable name that says its value is a secret.
|
||||
var secretName = regexp.MustCompile(`(?i)(pass(word|wd|phrase)?|secret|token|api_?key|private_?key|access_?key|credential|auth)`)
|
||||
|
||||
// notAValue is a name that says its value is where a secret is, not the secret: a file or a path.
|
||||
var notAValue = regexp.MustCompile(`(?i)(_FILE|FILE|_PATH|_DIR)$`)
|
||||
|
||||
// uriPassword is a URI carrying a password in its userinfo: scheme://user:password@.
|
||||
var uriPassword = regexp.MustCompile(`[A-Za-z][A-Za-z0-9+.-]*://[^\s/:@'"]*:([^\s/@'"]+)@`)
|
||||
|
||||
// tokenShaped are tokens recognised by their own prefix, whatever surrounds them: a forge's or a host's
|
||||
// access token, a JSON web token, a NATS seed.
|
||||
var tokenShaped = []struct {
|
||||
name string
|
||||
re *regexp.Regexp
|
||||
}{
|
||||
{"an access token", regexp.MustCompile(`\b(gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,}|glpat-[A-Za-z0-9_-]{20,}|xox[abpr]-[A-Za-z0-9-]{10,}|sk-ant-[A-Za-z0-9_-]{20,})`)},
|
||||
{"a JSON web token", regexp.MustCompile(`\beyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]+`)},
|
||||
{"a NATS seed", regexp.MustCompile(`\bS[ACNOU][A-Z2-7]{56}\b`)},
|
||||
}
|
||||
|
||||
// masked is a password a program already hid: ***, xxx, <redacted>, [REDACTED].
|
||||
var masked = regexp.MustCompile(`^(\*+|x+|X+|<[^>]*>|\[[^\]]*\]|%2A+)$`)
|
||||
|
||||
// ordinary is a value under a secret's name that is not one: a path, an address, a number, a switch.
|
||||
var ordinary = regexp.MustCompile(`^(/.*|[A-Za-z][A-Za-z0-9+.-]*://.*|[0-9.]+[a-z]?|(?i:true|false|yes|no|on|off|none|null))$`)
|
||||
|
||||
// leastSecret is the shortest value compared as a secret: a shorter one matches ordinary words.
|
||||
const leastSecret = 6
|
||||
|
||||
// passwordFlags take a secret as their next word, or after `=`, whatever the program.
|
||||
var passwordFlags = map[string]bool{
|
||||
"-P": true, "--password": true, "--pass": true, "--passwd": true, "--secret": true, "--secret-key": true,
|
||||
"--token": true, "--api-key": true, "--apikey": true, "--auth": true,
|
||||
}
|
||||
|
||||
// knownSecret is one value the builder holds, by the name it is said under.
|
||||
type knownSecret struct {
|
||||
Name string
|
||||
Value string
|
||||
}
|
||||
|
||||
// redactor hides the secrets it knows and those a line's shapes say are secrets.
|
||||
type redactor struct{ known []knownSecret }
|
||||
|
||||
// redactorFor knows the forge credential's password, and its user's name with it, in every form git or a
|
||||
// program may print them.
|
||||
func redactorFor(forge GitCredential) redactor {
|
||||
var r redactor
|
||||
if forge.URL == "" {
|
||||
return r
|
||||
}
|
||||
for _, m := range uriPassword.FindAllStringSubmatch(forge.URL, -1) {
|
||||
r.add("the forge credential", m[1])
|
||||
if dec, err := url.PathUnescape(m[1]); err == nil && dec != m[1] {
|
||||
r.add("the forge credential", dec)
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *redactor) add(name, value string) {
|
||||
if len(value) < leastSecret || masked.MatchString(value) {
|
||||
return
|
||||
}
|
||||
for _, k := range r.known {
|
||||
if k.Value == value {
|
||||
return
|
||||
}
|
||||
}
|
||||
r.known = append(r.known, knownSecret{name, value})
|
||||
}
|
||||
|
||||
// redact is a text with every known secret, every value its shape says is one, and every password inside a
|
||||
// URI replaced by a mark naming what was there. Line by line: a shape is judged within its line.
|
||||
func (r redactor) redact(text string) string {
|
||||
if !strings.ContainsAny(text, "\n") {
|
||||
return r.line(text)
|
||||
}
|
||||
lines := strings.Split(text, "\n")
|
||||
for i, l := range lines {
|
||||
lines[i] = r.line(l)
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
func (r redactor) line(line string) string {
|
||||
replace := func(s knownSecret) {
|
||||
for _, f := range forms(s.Value) {
|
||||
line = strings.ReplaceAll(line, f, "[redacted: "+s.Name+"]")
|
||||
}
|
||||
}
|
||||
for _, s := range r.known {
|
||||
replace(s)
|
||||
}
|
||||
for _, s := range shaped(line) {
|
||||
replace(s)
|
||||
}
|
||||
line = uriPassword.ReplaceAllStringFunc(line, func(m string) string {
|
||||
sub := uriPassword.FindStringSubmatch(m)
|
||||
if masked.MatchString(sub[1]) || strings.HasPrefix(sub[1], "[redacted") {
|
||||
return m
|
||||
}
|
||||
return strings.TrimSuffix(m, sub[1]+"@") + "[redacted: a password in a URI]@"
|
||||
})
|
||||
for _, t := range tokenShaped {
|
||||
line = t.re.ReplaceAllString(line, "[redacted: "+t.name+"]")
|
||||
}
|
||||
return line
|
||||
}
|
||||
|
||||
// forms are the ways a value may appear printed: as given, and URL-encoded.
|
||||
func forms(value string) []string {
|
||||
out := []string{value}
|
||||
for _, f := range []string{url.QueryEscape(value), url.PathEscape(value)} {
|
||||
if f != value && !hasString(out, f) {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func hasString(list []string, s string) bool {
|
||||
for _, x := range list {
|
||||
if x == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// shaped are the values a line carries by their shape: the word after a password flag, or the value of one
|
||||
// given with `=`, and a NAME=value whose name says secret.
|
||||
func shaped(line string) []knownSecret {
|
||||
var out []knownSecret
|
||||
add := func(name, value string) {
|
||||
value = strings.Trim(value, `"',;`)
|
||||
if len(value) < leastSecret || masked.MatchString(value) || ordinary.MatchString(value) ||
|
||||
strings.HasPrefix(value, "[redacted") {
|
||||
return
|
||||
}
|
||||
out = append(out, knownSecret{name, value})
|
||||
}
|
||||
words := strings.Fields(line)
|
||||
for i, w := range words {
|
||||
if flag, value, ok := strings.Cut(w, "="); ok && strings.HasPrefix(flag, "-") {
|
||||
if passwordFlags[flag] {
|
||||
add("the value of "+flag, value)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if name, value, ok := strings.Cut(w, "="); ok && name != "" && secretName.MatchString(name) &&
|
||||
!notAValue.MatchString(name) && !strings.ContainsAny(name, "/:") {
|
||||
add("the value of "+name, value)
|
||||
continue
|
||||
}
|
||||
if i+1 < len(words) && passwordFlags[w] {
|
||||
add("the word after "+w, words[i+1])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// withoutUserinfo is a URL with its userinfo left out, and whether it carried any.
|
||||
func withoutUserinfo(raw string) (string, bool) {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || u.User == nil {
|
||||
return raw, false
|
||||
}
|
||||
u.User = nil
|
||||
return u.String(), true
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/novox/mesh-controller/internal/artifacts"
|
||||
)
|
||||
|
||||
// Where built artifacts go.
|
||||
//
|
||||
// **One store, and it is the registry the bootstrap already pulls from.** An OCI registry is a
|
||||
// content-addressed blob store that happens to also understand images: `PUT` a blob and it is
|
||||
// retrievable at `/v2/<name>/blobs/sha256:…` for ever, by digest, over plain HTTP. An archive is
|
||||
// a content-addressed blob. So it goes there.
|
||||
//
|
||||
// The alternative considered was a second store beside it — S3-shaped, buckets, signed URLs. It
|
||||
// is the right answer for objects that are *mutable*, or need per-reader access, or are not build
|
||||
// output: somebody's uploads, a backup, a thing with a lifecycle. None of that describes a
|
||||
// digest-pinned archive, and standing up a second service to hold one kind of immutable blob
|
||||
// means two things to run, two things to back up and two ways for an artifact to be missing.
|
||||
//
|
||||
// **This is a decision that can be overturned by reading**: if something needs an object store
|
||||
// for reasons other than build artifacts, it should have one, and archives can move to it without
|
||||
// anything else changing — the manifest carries a URL and a digest, and neither says what served
|
||||
// it.
|
||||
|
||||
// Registry publishes to an OCI registry over plain HTTP.
|
||||
//
|
||||
// Plain HTTP because the registry the mesh runs is reached over the private network, which is
|
||||
// already the encrypted, authenticated thing. A second layer inside it would be certificates to
|
||||
// issue and rotate for no property the first does not have.
|
||||
type Registry struct {
|
||||
// Address is host:port, as a machine will fetch from.
|
||||
Address string
|
||||
// Run is how docker is invoked, so a test does not need one.
|
||||
Run Runner
|
||||
// HTTP is the client used for blobs.
|
||||
HTTP *http.Client
|
||||
}
|
||||
|
||||
// PushedNames are the names PublishImage leaves the image under on the node.
|
||||
func (r Registry) PushedNames(repository string) []string {
|
||||
return []string{r.Address + "/" + repository}
|
||||
}
|
||||
|
||||
// PublishImage pushes a locally built image and returns a reference pinned by digest.
|
||||
//
|
||||
// The digest is read back from the registry's own answer rather than computed here. What matters
|
||||
// is what the registry will serve for that reference, and only it can say.
|
||||
func (r Registry) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
|
||||
remote := r.Address + "/" + repository
|
||||
if _, err := r.Run(ctx, "", "docker", "tag", localTag, remote); err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The push waits for a registry held still, as every request to it does (novox/hq issue 457).
|
||||
if err := waitForRegistry(ctx, r.Address, "docker push "+remote, func() error {
|
||||
_, err := r.Run(ctx, "", "docker", "push", remote)
|
||||
return err
|
||||
}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
// Every digest the daemon lists for the image, one per line: the image may have been pushed under other names before
|
||||
// (a check's scratch copy, another registry), and the first of them is not the one this push made (novox/hq issue
|
||||
// 1742: a build recorded a digest the store never held).
|
||||
out, err := r.Run(ctx, "", "docker", "inspect", "--format", "{{range .RepoDigests}}{{println .}}{{end}}", remote)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pushed %s and cannot read back what it was pinned as: %w", remote, err)
|
||||
}
|
||||
first, pinned := "", ""
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if first == "" {
|
||||
first = line
|
||||
}
|
||||
if strings.HasPrefix(line, remote+"@sha256:") {
|
||||
pinned = line
|
||||
break
|
||||
}
|
||||
}
|
||||
if pinned == "" {
|
||||
if strings.Contains(first, "@sha256:") {
|
||||
return "", fmt.Errorf("pushed %s, but the daemon lists no digest for that name (only %q): the digest of another "+
|
||||
"repository is not what this registry serves", remote, first)
|
||||
}
|
||||
// A tag is not a pin. It can be made to point at something else, and this file is applied
|
||||
// on machines with no mesh to ask about anything (novox/hq ADR 0006).
|
||||
return "", fmt.Errorf("%s came back as %q, which is not pinned by digest", remote, first)
|
||||
}
|
||||
return pinned, nil
|
||||
}
|
||||
|
||||
// PublishArchive stores bytes as a blob, holds it by a manifest, and returns where to fetch them
|
||||
// from.
|
||||
//
|
||||
// Two steps for the blob, which is the registry's own protocol: ask for somewhere to put it, then
|
||||
// put it there naming the digest. The registry verifies the digest itself, so a blob that arrived
|
||||
// corrupted is refused by the thing storing it rather than by the machine unpacking it a week
|
||||
// later.
|
||||
//
|
||||
// **Then a manifest that names it** (novox/hq issue 253, ADR 0189). The store's own collector
|
||||
// marks only from manifests, and a blob no manifest names is collected however much the mesh
|
||||
// means to keep it — so an archive published bare is an archive the first nightly collection
|
||||
// deletes. The holder is composed from the digest and size alone (artifacts.Holder), which is
|
||||
// what lets the sweep recompute it to backfill or let go without anything being recorded here.
|
||||
// What a machine is told to fetch is the blob, exactly as before.
|
||||
func (r Registry) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
|
||||
base := "http://" + r.Address + "/v2/" + repository
|
||||
final := base + "/blobs/" + digest
|
||||
|
||||
// Already there. Blobs are immutable and named by their content, so this is not an
|
||||
// optimisation — re-uploading would be asking the registry to store what it already has under
|
||||
// the name it already has. It is still held: a blob published before holders existed is
|
||||
// exactly the one a rebuild of the same source finds already there.
|
||||
if there, err := r.has(ctx, final); err != nil {
|
||||
return "", err
|
||||
} else if there {
|
||||
return final, r.hold(ctx, repository, digest, len(body))
|
||||
}
|
||||
|
||||
start, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/blobs/uploads/", nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
begun, err := r.client().Do(start)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot start an upload to %s: %w", base, err)
|
||||
}
|
||||
defer begun.Body.Close()
|
||||
if begun.StatusCode != http.StatusAccepted {
|
||||
return "", fmt.Errorf("%s answered %s when asked where to put a blob", base, begun.Status)
|
||||
}
|
||||
where := begun.Header.Get("Location")
|
||||
if where == "" {
|
||||
return "", fmt.Errorf("%s accepted an upload and said nowhere to put it", base)
|
||||
}
|
||||
if strings.HasPrefix(where, "/") {
|
||||
where = "http://" + r.Address + where
|
||||
}
|
||||
|
||||
put, err := http.NewRequestWithContext(ctx, http.MethodPut,
|
||||
where+separator(where)+"digest="+digest, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
put.Header.Set("Content-Type", "application/octet-stream")
|
||||
done, err := r.client().Do(put)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer done.Body.Close()
|
||||
if done.StatusCode != http.StatusCreated {
|
||||
said, _ := io.ReadAll(io.LimitReader(done.Body, 4096))
|
||||
return "", fmt.Errorf("%s refused the blob: %s %s", base, done.Status, strings.TrimSpace(string(said)))
|
||||
}
|
||||
return final, r.hold(ctx, repository, digest, len(body))
|
||||
}
|
||||
|
||||
// hold puts the manifest holding an archive beside it. A build whose archive could not be held is
|
||||
// a failed build: recorded as published, it would be an archive the store's collector takes.
|
||||
func (r Registry) hold(ctx context.Context, repository, digest string, size int) error {
|
||||
store := artifacts.Store{Address: r.Address, HTTP: r.client()}
|
||||
if _, err := store.HoldBlob(ctx, repository, digest, int64(size)); err != nil {
|
||||
return fmt.Errorf("published %s/blobs/%s and could not hold it by a manifest: %w", repository, digest, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// has is whether this registry already holds what is at that URL.
|
||||
//
|
||||
// **A manifest HEAD must say what it accepts.** A registry answers a manifest request only in a media
|
||||
// type the caller named, and a bare HEAD — no Accept at all — is answered 404 for a manifest it holds
|
||||
// perfectly well. Measured against the mesh's own registry (2026-09-28): the same digest answered 200
|
||||
// with the manifest media types and 404 without them, so a check written without them concluded the
|
||||
// registry held nothing, copied every base again, and exhausted the public hub's pull limit. A blob
|
||||
// needs no Accept, which is why this went unnoticed: the same helper was right for blobs and wrong
|
||||
// for manifests.
|
||||
func (r Registry) has(ctx context.Context, url string, accept ...string) (bool, error) {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
for _, media := range accept {
|
||||
request.Header.Add("Accept", media)
|
||||
}
|
||||
response, err := r.client().Do(request)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
return response.StatusCode == http.StatusOK, nil
|
||||
}
|
||||
|
||||
// client is the client for the registry and for upstream, whose requests to the registry wait out a
|
||||
// registry held still (novox/hq issue 457).
|
||||
func (r Registry) client() *http.Client {
|
||||
if r.HTTP != nil {
|
||||
return waiting(r.HTTP, r.Address)
|
||||
}
|
||||
return waiting(http.DefaultClient, r.Address)
|
||||
}
|
||||
|
||||
// separator is whether the upload location already carries a query.
|
||||
func separator(where string) string {
|
||||
if strings.Contains(where, "?") {
|
||||
return "&"
|
||||
}
|
||||
return "?"
|
||||
}
|
||||
@@ -1,283 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/novox/mesh-controller/internal/artifacts"
|
||||
)
|
||||
|
||||
// An OCI registry as a content-addressed blob store, which is what it is.
|
||||
//
|
||||
// Against a stand-in rather than a real registry because what is under test is this side of the
|
||||
// protocol — that the two steps happen in order, that the digest travels, that a blob already
|
||||
// there is not sent again, and that a tag is never accepted as a pin.
|
||||
|
||||
type fakeRegistry struct {
|
||||
blobs map[string][]byte
|
||||
manifests map[string][]byte // "<repository>@<digest>"
|
||||
puts map[string]int // blob uploads, by digest
|
||||
location string
|
||||
}
|
||||
|
||||
func (f *fakeRegistry) serve(t *testing.T) *httptest.Server {
|
||||
t.Helper()
|
||||
if f.blobs == nil {
|
||||
f.blobs = map[string][]byte{}
|
||||
}
|
||||
f.manifests = map[string][]byte{}
|
||||
f.puts = map[string]int{}
|
||||
mux := http.NewServeMux()
|
||||
server := httptest.NewServer(mux)
|
||||
mux.HandleFunc("/v2/", func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/sha256:"):
|
||||
digest := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
|
||||
if _, ok := f.blobs[digest]; ok {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
case strings.Contains(r.URL.Path, "/manifests/"):
|
||||
// The archive's holder (novox/hq issue 253): asked for with an Accept, put by digest.
|
||||
repository, digest, _ := strings.Cut(strings.TrimPrefix(r.URL.Path, "/v2/"), "/manifests/")
|
||||
switch r.Method {
|
||||
case http.MethodHead:
|
||||
if _, ok := f.manifests[repository+"@"+digest]; ok &&
|
||||
strings.Contains(r.Header.Get("Accept"), "application/vnd.oci.image.manifest.v1+json") {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
case http.MethodPut:
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
sum := sha256.Sum256(body)
|
||||
if digest != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
f.manifests[repository+"@"+digest] = body
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
}
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/blobs/uploads/"):
|
||||
where := f.location
|
||||
if where == "" {
|
||||
where = "/v2/upload/" + hex.EncodeToString([]byte("session"))
|
||||
}
|
||||
w.Header().Set("Location", where)
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
case r.Method == http.MethodPut:
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
digest := r.URL.Query().Get("digest")
|
||||
sum := sha256.Sum256(body)
|
||||
if digest != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||
// A registry verifies what it is given, which is why a corrupt blob is refused
|
||||
// here rather than by a machine unpacking it a week later.
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
f.blobs[digest] = body
|
||||
f.puts[digest]++
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
})
|
||||
t.Cleanup(server.Close)
|
||||
return server
|
||||
}
|
||||
|
||||
func registryFor(t *testing.T, f *fakeRegistry) Registry {
|
||||
t.Helper()
|
||||
server := f.serve(t)
|
||||
return Registry{Address: strings.TrimPrefix(server.URL, "http://")}
|
||||
}
|
||||
|
||||
func TestAnArchiveIsStoredAndFetchableByItsDigest(t *testing.T) {
|
||||
f := &fakeRegistry{}
|
||||
r := registryFor(t, f)
|
||||
body := []byte("a theme")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
|
||||
where, err := r.PublishArchive(context.Background(), "shell/config", body, digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(where, "/blobs/"+digest) {
|
||||
t.Fatalf("what a machine is told to fetch is %q, which does not name the digest", where)
|
||||
}
|
||||
if string(f.blobs[digest]) != "a theme" {
|
||||
t.Fatalf("the registry holds %q", f.blobs[digest])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArchiveIsPublishedWithAManifestHoldingIt(t *testing.T) {
|
||||
// The store's collector marks only from manifests, so a bare blob is one the first collection
|
||||
// deletes, kept or not (novox/hq issue 253, ADR 0189). Every archive goes out held, by the
|
||||
// holder the sweep can compute for itself from the digest and size.
|
||||
f := &fakeRegistry{}
|
||||
r := registryFor(t, f)
|
||||
body := []byte("a theme")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
|
||||
where, err := r.PublishArchive(context.Background(), "shell/config", body, digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(where, "/v2/shell/config/blobs/"+digest) {
|
||||
t.Fatalf("a machine is told to fetch %q; the blob is still what is fetched", where)
|
||||
}
|
||||
manifest, holder := artifacts.Holder(digest, int64(len(body)))
|
||||
if got := f.manifests["shell/config@"+holder]; string(got) != string(manifest) {
|
||||
t.Fatalf("the store holds %v; want the holder %s in the archive's own repository", f.manifests, holder)
|
||||
}
|
||||
if !strings.Contains(string(manifest), `"digest":"`+digest+`"`) {
|
||||
t.Fatalf("the holder does not name the archive: %s", manifest)
|
||||
}
|
||||
empty := sha256.Sum256([]byte("{}"))
|
||||
if _, ok := f.blobs["sha256:"+hex.EncodeToString(empty[:])]; !ok {
|
||||
t.Fatal("the holder's empty config was never put, and a registry refuses a manifest without it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArchiveAlreadyThereIsStillHeld(t *testing.T) {
|
||||
// A rebuild of the same source finds its archive already there — often one published bare,
|
||||
// before holders. It is not sent again, and it is held.
|
||||
f := &fakeRegistry{}
|
||||
r := registryFor(t, f)
|
||||
body := []byte("published bare")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
f.blobs[digest] = body
|
||||
|
||||
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if f.puts[digest] != 0 {
|
||||
t.Fatal("a blob already there was sent again")
|
||||
}
|
||||
if _, holder := artifacts.Holder(digest, int64(len(body))); f.manifests["shell/config@"+holder] == nil {
|
||||
t.Fatal("a blob already there was left bare")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABlobAlreadyThereIsNotSentAgain(t *testing.T) {
|
||||
// Not an optimisation: blobs are named by their content, so re-uploading is asking the
|
||||
// registry to store what it already has under the name it already has.
|
||||
f := &fakeRegistry{}
|
||||
r := registryFor(t, f)
|
||||
body := []byte("a theme")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
|
||||
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if f.puts[digest] != 1 {
|
||||
t.Fatalf("the blob was uploaded %d times", f.puts[digest])
|
||||
}
|
||||
if len(f.manifests) != 1 {
|
||||
t.Fatalf("publishing twice left %d holders; want the one", len(f.manifests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAbsoluteUploadLocationIsFollowed(t *testing.T) {
|
||||
// A registry may answer with a full URL or with a path. Both happen in the wild, and a client
|
||||
// that handles one produces a confusing failure against the other.
|
||||
f := &fakeRegistry{}
|
||||
server := f.serve(t)
|
||||
f.location = server.URL + "/v2/upload/session?state=abc"
|
||||
r := Registry{Address: strings.TrimPrefix(server.URL, "http://")}
|
||||
|
||||
body := []byte("x")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
if _, err := r.PublishArchive(context.Background(), "a/b", body, digest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := f.blobs[digest]; !ok {
|
||||
t.Fatal("the blob did not arrive when the location carried a query")
|
||||
}
|
||||
}
|
||||
|
||||
func TestATagIsNeverAcceptedAsAPin(t *testing.T) {
|
||||
// A tag can be made to point at something else, and a bundle is applied on machines with no
|
||||
// mesh to ask about anything.
|
||||
r := Registry{Address: "registry.invalid", Run: func(
|
||||
_ context.Context, _, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
|
||||
return "registry.invalid/shell/server:latest\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}}
|
||||
_, err := r.PublishImage(context.Background(), "local", "shell/server")
|
||||
if err == nil {
|
||||
t.Fatal("an image referred to by tag was accepted")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "pinned by digest") {
|
||||
t.Fatalf("refused for the wrong reason: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPushedImageComesBackPinned(t *testing.T) {
|
||||
pinned := "registry.invalid/shell/server@sha256:" + strings.Repeat("a", 64)
|
||||
var ran []string
|
||||
r := Registry{Address: "registry.invalid", Run: func(
|
||||
_ context.Context, _, name string, args ...string) (string, error) {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
|
||||
return pinned + "\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}}
|
||||
got, err := r.PublishImage(context.Background(), "local", "shell/server")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != pinned {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if len(ran) < 2 || !strings.HasPrefix(ran[0], "docker tag") || !strings.HasPrefix(ran[1], "docker push") {
|
||||
t.Fatalf("it did not tag and then push: %v", ran)
|
||||
}
|
||||
}
|
||||
|
||||
// 1742: an image pushed under other names before lists their digests too, and the digest recorded is the one of the name
|
||||
// this push made — never the first listed, which may be another repository's.
|
||||
func TestThePinnedDigestIsTheOneOfTheNamePushed(t *testing.T) {
|
||||
want := "registry.invalid/shell/server@sha256:" + strings.Repeat("b", 64)
|
||||
earlier := "registry.invalid/scratch/shell/server@sha256:" + strings.Repeat("a", 64)
|
||||
r := Registry{Address: "registry.invalid", Run: func(
|
||||
_ context.Context, _, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
|
||||
return earlier + "\n" + want + "\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}}
|
||||
got, err := r.PublishImage(context.Background(), "local", "shell/server")
|
||||
if err != nil || got != want {
|
||||
t.Fatalf("pinned %q (%v); want %q", got, err, want)
|
||||
}
|
||||
// Only another repository's digest is listed: the push is not taken for pinned.
|
||||
r.Run = func(_ context.Context, _, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
|
||||
return earlier + "\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
if _, err := r.PublishImage(context.Background(), "local", "shell/server"); err == nil ||
|
||||
!strings.Contains(err.Error(), "no digest for that name") {
|
||||
t.Fatalf("a digest of another repository was taken for the push's own: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestABuildNeedingTheRegistryIsRefusedOnAnAgentWithoutOne(t *testing.T) {
|
||||
none := Npmrc{Machine: "workstation"}
|
||||
cases := map[string]struct {
|
||||
manifest string
|
||||
files map[string]string
|
||||
held map[string]string
|
||||
}{
|
||||
"a package": {aPackage, map[string]string{"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.0"}`}, nil},
|
||||
"an image naming .npmrc": {withBoth, map[string]string{
|
||||
"Dockerfile": "FROM scratch\nCOPY .npmrc /root/.npmrc\n", "files/x": "y"}, nil},
|
||||
"a TypeScript bundle depending on the SDK": {aBundle, map[string]string{
|
||||
"index.ts": "console.log(1)", "package.json": `{"dependencies":{"@novox/mesh-sdk":"^0.1.0"}}`},
|
||||
map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}},
|
||||
}
|
||||
for name, c := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
r, workspace := aRepository(t, c.manifest, c.files)
|
||||
var said []string
|
||||
_, err := Build(context.Background(), compiling{r}.run, r,
|
||||
"https://forge.invalid/m.git", "", "", workspace, c.held, none, GitCredential{},
|
||||
func(step, message string) { said = append(said, step+": "+message) })
|
||||
if err == nil {
|
||||
t.Fatal("built with no package registry")
|
||||
}
|
||||
for _, want := range []string{"workstation", "npm-package-registry", "unbound", "gitea"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("the refusal does not say %q: %v", want, err)
|
||||
}
|
||||
}
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build") || strings.HasPrefix(line, "docker run") ||
|
||||
strings.HasPrefix(line, "docker pull") {
|
||||
t.Errorf("something was built after the refusal: %s", line)
|
||||
}
|
||||
}
|
||||
if len(r.images) != 0 || len(r.archives) != 0 {
|
||||
t.Errorf("something was published after the refusal: %v %v", r.images, r.archives)
|
||||
}
|
||||
if !strings.Contains(strings.Join(said, "\n"), "REFUSED") {
|
||||
t.Errorf("the build's log does not say it was refused:\n%s", strings.Join(said, "\n"))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A Go artifact builds on the same agent.
|
||||
files := aSharedRepository("one", "1")
|
||||
manifest := `{"module":"ctl","version":"1","build":{"artifacts":[
|
||||
{"name":"controller","kind":"bundle","language":"go","system":"arch","from":"cmd/ctl","binary":"ctl"}]},
|
||||
"resources":[{"id":"controller","type":"process","name":"ctl","artifact":"controller","run":["./ctl"]}]}`
|
||||
r := &recorded{contents: map[string]string{ManifestName: manifest}}
|
||||
for k, v := range files {
|
||||
r.contents[k] = v
|
||||
}
|
||||
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
|
||||
if _, err := Build(context.Background(), onTrunk{recorded: r}.run, r,
|
||||
"https://forge.invalid/ctl.git", "", "", t.TempDir(), held, none, GitCredential{}, nil); err != nil {
|
||||
t.Fatalf("a Go artifact did not build on an agent without the registry: %v", err)
|
||||
}
|
||||
// And an image that does not name .npmrc.
|
||||
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch\nCOPY . .", "files/x": "y"})
|
||||
if _, err := Build(context.Background(), r.run, r,
|
||||
"https://forge.invalid/meshboard.git", "", "", workspace, nil, none, GitCredential{}, nil); err != nil {
|
||||
t.Fatalf("an image not naming .npmrc did not build on an agent without the registry: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildSaysWhichRegistryItResolvedFrom(t *testing.T) {
|
||||
for _, from := range []string{FromBinding, FromGenesis} {
|
||||
r, workspace := aRepository(t, aPackage, map[string]string{
|
||||
"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.0"}`,
|
||||
})
|
||||
n := Npmrc{Scope: "@novox", Registry: "http://127.0.0.1:3000/api/packages/novox/npm/",
|
||||
Username: "mesh-builder", Password: "p", From: from, Machine: "anchor"}
|
||||
var said []string
|
||||
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/mesh-sdk.git", "", "",
|
||||
workspace, nil, n, GitCredential{}, func(step, message string) { said = append(said, step+": "+message) }); err != nil {
|
||||
t.Fatalf("%s: %v", from, err)
|
||||
}
|
||||
if !strings.Contains(strings.Join(said, "\n"), "from "+from) {
|
||||
t.Errorf("the log does not say the registry came from %s:\n%s", from, strings.Join(said, "\n"))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"github.com/novox/mesh-controller/internal/beside"
|
||||
)
|
||||
|
||||
// **A build that needs the registry fails loudly while the build agent has none** (novox/hq ADR 0336
|
||||
// §5, §6, issue 121). On a holder with neither the npm-package-registry binding nor the genesis
|
||||
// document, a `package` artifact, an image whose Dockerfile names .npmrc and a TypeScript artifact
|
||||
// with an `@novox/` dependency each fail before anything is built or published, naming the machine,
|
||||
// the unbound requirement and the module that provides it. A Go artifact, and an image that does not
|
||||
// name .npmrc, build there as before.
|
||||
|
||||
// **The build's log says which registry it resolved from** (novox/hq ADR 0336 §4).
|
||||
|
||||
// The refusal names the module that provides the registry; it is the catalogue's provider.
|
||||
func TestTheRegistryABuildIsRefusedForNamesItsProvider(t *testing.T) {
|
||||
raw, err := os.ReadFile(filepath.Join(beside.Catalogue(t), RegistryProvider, "module.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
forge, err := catalogue.ParseManifest(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
provides := false
|
||||
for _, p := range forge.Provides {
|
||||
provides = provides || p.Name == RegistryRequirement
|
||||
}
|
||||
if !provides {
|
||||
t.Fatalf("%s does not provide %s, which a refused build names it for", RegistryProvider, RegistryRequirement)
|
||||
}
|
||||
}
|
||||
@@ -1,153 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A build waits out a registry that refuses connections, for a bounded time (novox/hq issue 457).
|
||||
//
|
||||
// **Why waiting, and why here.** The store's nightly collection holds the registry still for its run —
|
||||
// about a minute and a half, measured on 2026-10-11 — and a build that reached the registry in that
|
||||
// window failed on "connection refused", and its whole delivery plan with it: a plan failed for a
|
||||
// pause the mesh itself scheduled. The other design weighed was the collection telling the controller
|
||||
// it holds the registry, and the controller holding build asks while it runs. Waiting here is smaller
|
||||
// and covers more: it is local to the one place that talks to the registry, needs no new message
|
||||
// between modules, and also carries a build over any other short outage — a registry restarted by its
|
||||
// own update, say. A refusal is the one error waited for: nothing was sent, so trying again cannot
|
||||
// do anything twice, and it is what a registry that is stopped answers.
|
||||
//
|
||||
// **Bounded, and loud past the bound.** registryWait is longer than the collection holds the registry
|
||||
// (five minutes against about one and a half), so the pause the mesh schedules is always waited out,
|
||||
// and a registry that is really down still fails the build — saying how long it was refused — rather
|
||||
// than holding a build machine for ever. Every wait is said in the build's log, with why, and so is
|
||||
// the registry answering again.
|
||||
|
||||
var (
|
||||
// registryWait is how long a build waits for a registry that refuses, per call that found it so.
|
||||
registryWait = 5 * time.Minute
|
||||
// registryFirstPause is the first pause between tries; each pause doubles, up to registryMostPause.
|
||||
registryFirstPause = time.Second
|
||||
)
|
||||
|
||||
// registryMostPause is the longest pause between two tries: short enough that a build goes on within
|
||||
// seconds of the registry answering again.
|
||||
const registryMostPause = 10 * time.Second
|
||||
|
||||
// refused is whether an error is a connection refused: from a dial here, or as a command such as docker
|
||||
// said it in its output.
|
||||
func refused(err error) bool {
|
||||
return err != nil && (errors.Is(err, syscall.ECONNREFUSED) || strings.Contains(err.Error(), "connection refused"))
|
||||
}
|
||||
|
||||
// waitedFor is whether an error is one a build waits out: a refused connection, or a lookup of the registry's name
|
||||
// that the resolver did not answer (novox/hq issue 1748; failedLookup, from issue 1773, says which).
|
||||
func waitedFor(err error) bool { return refused(err) || failedLookup(err) }
|
||||
|
||||
// waitForRegistry runs try, and while it fails because the registry at address refuses connections, or the
|
||||
// resolver does not answer a lookup of its name, tries again with a growing pause until registryWait has passed. what names the call, for the log.
|
||||
func waitForRegistry(ctx context.Context, address, what string, try func() error) error {
|
||||
err := try()
|
||||
if !waitedFor(err) {
|
||||
return err
|
||||
}
|
||||
started := time.Now()
|
||||
pause := registryFirstPause
|
||||
if failedLookup(err) {
|
||||
tell("registry", "%s: the resolver did not answer a lookup of the registry's name (%v); the build waits, "+
|
||||
"for up to %s (novox/hq issue 1748)", what, err, registryWait)
|
||||
} else {
|
||||
tell("registry", "%s: refused; the build waits for the registry at %s, for up to %s — it is held still while "+
|
||||
"the store's nightly collection runs, about a minute and a half (novox/hq issue 457)",
|
||||
what, address, registryWait)
|
||||
}
|
||||
for {
|
||||
left := registryWait - time.Since(started)
|
||||
if left <= 0 {
|
||||
if failedLookup(err) {
|
||||
tell("registry", "%s: the resolver still did not answer a lookup of the registry at %s after %s; the build fails",
|
||||
what, address, time.Since(started).Round(time.Second))
|
||||
return fmt.Errorf("the resolver did not answer a lookup of the registry at %s for %s, so it is the "+
|
||||
"resolver, not the registry, that is down: %w",
|
||||
address, time.Since(started).Round(time.Millisecond), err)
|
||||
}
|
||||
tell("registry", "%s: the registry at %s still refuses after %s; the build fails", what, address,
|
||||
time.Since(started).Round(time.Second))
|
||||
return fmt.Errorf("the registry at %s refused every connection for %s, longer than its nightly "+
|
||||
"collection holds it still, so it is down, not paused: %w",
|
||||
address, time.Since(started).Round(time.Millisecond), err)
|
||||
}
|
||||
wait := min(pause, left)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("stopped while waiting for the registry at %s: %w (last: %v)", address, ctx.Err(), err)
|
||||
case <-time.After(wait):
|
||||
}
|
||||
pause = min(pause*2, registryMostPause)
|
||||
if err = try(); !waitedFor(err) {
|
||||
// Said as it is: the registry answering is only the build going on when the call worked.
|
||||
if err == nil {
|
||||
tell("registry", "%s: the registry at %s answers again after %s; the build goes on", what, address,
|
||||
time.Since(started).Round(time.Millisecond))
|
||||
} else {
|
||||
tell("registry", "%s: the registry at %s no longer refuses after %s, and answered with: %v", what,
|
||||
address, time.Since(started).Round(time.Millisecond), err)
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// waitingTransport waits for the registry on every request to it, and on none to anywhere else: the
|
||||
// same client copies from upstream registries, whose refusals are theirs to answer.
|
||||
type waitingTransport struct {
|
||||
base http.RoundTripper
|
||||
address string
|
||||
}
|
||||
|
||||
func (t waitingTransport) RoundTrip(request *http.Request) (*http.Response, error) {
|
||||
if request.URL.Host != t.address {
|
||||
return t.base.RoundTrip(request)
|
||||
}
|
||||
var response *http.Response
|
||||
tries := 0
|
||||
err := waitForRegistry(request.Context(), t.address, request.Method+" "+request.URL.Path, func() error {
|
||||
attempt := request
|
||||
if tries > 0 && request.Body != nil && request.Body != http.NoBody {
|
||||
// A body is sent again only when it can be read again; one that cannot is not retried.
|
||||
if request.GetBody == nil {
|
||||
return fmt.Errorf("%s %s cannot be sent again: its body cannot be read twice", request.Method, request.URL)
|
||||
}
|
||||
body, err := request.GetBody()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
attempt = request.Clone(request.Context())
|
||||
attempt.Body = body
|
||||
}
|
||||
tries++
|
||||
var err error
|
||||
response, err = t.base.RoundTrip(attempt)
|
||||
return err
|
||||
})
|
||||
return response, err
|
||||
}
|
||||
|
||||
// waiting is a client like c whose requests to the registry wait for it.
|
||||
func waiting(c *http.Client, address string) *http.Client {
|
||||
if _, already := c.Transport.(waitingTransport); already {
|
||||
return c
|
||||
}
|
||||
copied := *c
|
||||
base := c.Transport
|
||||
if base == nil {
|
||||
base = http.DefaultTransport
|
||||
}
|
||||
copied.Transport = waitingTransport{base: base, address: address}
|
||||
return &copied
|
||||
}
|
||||
@@ -1,219 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A registry held still for a while (novox/hq issue 457): the store's nightly collection stops it for
|
||||
// about a minute and a half, and a build in that window was failed, and its whole plan with it, for a
|
||||
// pause the mesh itself scheduled. A build waits it out — for a bound longer than the collection
|
||||
// holds it — says so in its log, and still fails, loudly, past the bound.
|
||||
|
||||
// heldStill is a registry address that refuses every connection until it starts answering after
|
||||
// pause, or never when pause is negative.
|
||||
func heldStill(t *testing.T, f *fakeRegistry, pause time.Duration) string {
|
||||
t.Helper()
|
||||
handler := f.serve(t).Config.Handler
|
||||
reserved, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
address := reserved.Addr().String()
|
||||
reserved.Close() // refused from here on: nothing listens
|
||||
if pause < 0 {
|
||||
return address
|
||||
}
|
||||
server := &http.Server{Handler: handler}
|
||||
go func() {
|
||||
time.Sleep(pause)
|
||||
l, err := net.Listen("tcp", address)
|
||||
if err != nil {
|
||||
t.Errorf("cannot answer at %s again: %v", address, err)
|
||||
return
|
||||
}
|
||||
_ = server.Serve(l)
|
||||
}()
|
||||
t.Cleanup(func() { _ = server.Close() })
|
||||
return address
|
||||
}
|
||||
|
||||
// saying collects what a build says, as the build machine's per-build Said does.
|
||||
func saying(t *testing.T) func() []string {
|
||||
t.Helper()
|
||||
var mu sync.Mutex
|
||||
var lines []string
|
||||
was := Said
|
||||
Said = func(step, message string) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
lines = append(lines, step+": "+message)
|
||||
}
|
||||
t.Cleanup(func() { Said = was })
|
||||
return func() []string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return append([]string(nil), lines...)
|
||||
}
|
||||
}
|
||||
|
||||
// waitingFor shortens the bound and the first pause, so a test waits for milliseconds.
|
||||
func waitingFor(t *testing.T, bound time.Duration) {
|
||||
t.Helper()
|
||||
wasBound, wasFirst := registryWait, registryFirstPause
|
||||
registryWait, registryFirstPause = bound, 20*time.Millisecond
|
||||
t.Cleanup(func() { registryWait, registryFirstPause = wasBound, wasFirst })
|
||||
}
|
||||
|
||||
func TestABuildWaitsForARegistryHeldStillAndGoesOn(t *testing.T) {
|
||||
waitingFor(t, 5*time.Second)
|
||||
said := saying(t)
|
||||
f := &fakeRegistry{}
|
||||
r := Registry{Address: heldStill(t, f, 100*time.Millisecond)}
|
||||
body := []byte("a theme")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
|
||||
if _, err := r.PublishArchive(context.Background(), "shell/config", body, digest); err != nil {
|
||||
t.Fatalf("a registry refusing for 100ms failed the build: %v", err)
|
||||
}
|
||||
if string(f.blobs[digest]) != "a theme" {
|
||||
t.Fatalf("the registry holds %q", f.blobs[digest])
|
||||
}
|
||||
log := strings.Join(said(), "\n")
|
||||
if !strings.Contains(log, "waits for the registry at "+r.Address) || !strings.Contains(log, "nightly collection") {
|
||||
t.Errorf("the build's log does not say it waited for the registry, and why:\n%s", log)
|
||||
}
|
||||
if !strings.Contains(log, "answers again") {
|
||||
t.Errorf("the build's log does not say the registry came back:\n%s", log)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildFailsLoudlyOnARegistryRefusingPastTheBound(t *testing.T) {
|
||||
waitingFor(t, 100*time.Millisecond)
|
||||
said := saying(t)
|
||||
f := &fakeRegistry{}
|
||||
r := Registry{Address: heldStill(t, f, -1)}
|
||||
body := []byte("a theme")
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
|
||||
started := time.Now()
|
||||
_, err := r.PublishArchive(context.Background(), "shell/config", body, digest)
|
||||
if err == nil {
|
||||
t.Fatal("a registry that never answered published the archive")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "refused every connection for") || !strings.Contains(err.Error(), "connection refused") {
|
||||
t.Errorf("the failure does not say it waited and was refused throughout: %v", err)
|
||||
}
|
||||
if waited := time.Since(started); waited < 100*time.Millisecond {
|
||||
t.Errorf("failed after %s, before the bound", waited)
|
||||
}
|
||||
if log := strings.Join(said(), "\n"); !strings.Contains(log, "waits for the registry") {
|
||||
t.Errorf("the build's log does not say it waited:\n%s", log)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnImagePushWaitsForARegistryHeldStill(t *testing.T) {
|
||||
waitingFor(t, 5*time.Second)
|
||||
said := saying(t)
|
||||
pushes := 0
|
||||
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
switch args[0] {
|
||||
case "push":
|
||||
pushes++
|
||||
if pushes < 3 {
|
||||
return "", errorString("docker push: dial tcp 127.0.0.1:5000: connect: connection refused")
|
||||
}
|
||||
case "inspect":
|
||||
return "127.0.0.1:5000/m/server@sha256:abc\n", nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
r := Registry{Address: "127.0.0.1:5000", Run: run}
|
||||
if _, err := r.PublishImage(context.Background(), "local", "m/server"); err != nil {
|
||||
t.Fatalf("a push refused twice failed the build: %v", err)
|
||||
}
|
||||
if pushes != 3 {
|
||||
t.Errorf("pushed %d times, want 3", pushes)
|
||||
}
|
||||
if log := strings.Join(said(), "\n"); !strings.Contains(log, "waits for the registry") {
|
||||
t.Errorf("the build's log does not say it waited:\n%s", log)
|
||||
}
|
||||
}
|
||||
|
||||
type errorString string
|
||||
|
||||
func (e errorString) Error() string { return string(e) }
|
||||
|
||||
// A resolver that does not answer for a moment (novox/hq issue 1748): a lookup of the registry's name that
|
||||
// timed out failed the build at once, though the same name resolved a second later.
|
||||
|
||||
const lookupTimeout = "Head \"http://novox.internal:5100/v2/x/blobs/sha256:1\": dial tcp: lookup novox.internal on 10.10.0.3:53: " +
|
||||
"read udp 10.10.0.3:44790->10.10.0.3:53: i/o timeout"
|
||||
|
||||
type lookupTimedOut struct{}
|
||||
|
||||
func (lookupTimedOut) Error() string { return lookupTimeout }
|
||||
|
||||
func TestABuildWaitsOutAResolverThatDoesNotAnswerAndGoesOn(t *testing.T) { // 1748/AC1
|
||||
waitingFor(t, 5*time.Second)
|
||||
said := saying(t)
|
||||
calls := 0
|
||||
err := waitForRegistry(context.Background(), "novox.internal:5100", "HEAD /v2/x", func() error {
|
||||
calls++
|
||||
if calls < 3 {
|
||||
return lookupTimedOut{}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("a lookup timing out twice failed the build: %v", err)
|
||||
}
|
||||
if calls != 3 {
|
||||
t.Errorf("tried %d times, want 3", calls)
|
||||
}
|
||||
log := strings.Join(said(), "\n")
|
||||
if !strings.Contains(log, "resolver did not answer") || !strings.Contains(log, "answers again") {
|
||||
t.Errorf("the build's log does not say the resolver was waited for, and that it came back:\n%s", log)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildFailsNamingTheResolverWhenItDoesNotAnswerPastTheBound(t *testing.T) { // 1748/AC2
|
||||
waitingFor(t, 100*time.Millisecond)
|
||||
saying(t)
|
||||
started := time.Now()
|
||||
err := waitForRegistry(context.Background(), "novox.internal:5100", "HEAD /v2/x", func() error { return lookupTimedOut{} })
|
||||
if err == nil {
|
||||
t.Fatal("a resolver that never answered let the build go on")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "resolver did not answer") || !strings.Contains(err.Error(), "i/o timeout") {
|
||||
t.Errorf("the failure does not name the resolver: %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "refused every connection") {
|
||||
t.Errorf("the failure blames a refusal: %v", err)
|
||||
}
|
||||
if waited := time.Since(started); waited < 100*time.Millisecond {
|
||||
t.Errorf("failed after %s, before the bound", waited)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnErrorThatIsNeitherARefusalNorALookupFailsAtOnce(t *testing.T) { // 1748/AC3
|
||||
waitingFor(t, 5*time.Second)
|
||||
calls := 0
|
||||
err := waitForRegistry(context.Background(), "novox.internal:5100", "HEAD /v2/x", func() error {
|
||||
calls++
|
||||
return errors.New("401 unauthorized")
|
||||
})
|
||||
if err == nil || calls != 1 {
|
||||
t.Fatalf("err %v after %d calls, want a failure at once", err, calls)
|
||||
}
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReplay310 is the replay of novox/hq issue 310, written with only what the build agent had before its
|
||||
// fix, so mesh-lab's prover can lay it over the commit before: a pull request whose branch holds no
|
||||
// merge-check.sh, into a main that holds one. Before the fix the check read the head's tree alone, found no
|
||||
// script, and answered the repository's layer a warning — none of its tests run, and the forge took it.
|
||||
// After it, main's script judges the change: with no artifact store to read the facts from here, the check
|
||||
// goes on to run it and says it cannot, and never answers the warning.
|
||||
func TestReplay310(t *testing.T) {
|
||||
origin := t.TempDir()
|
||||
git := func(args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = origin
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git("init", "--quiet", "-b", "main")
|
||||
if err := os.WriteFile(filepath.Join(origin, CheckScript), []byte("echo 'FAIL: the suite'; exit 1\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git("add", "-A")
|
||||
git("commit", "--quiet", "-m", "main holds its check")
|
||||
git("checkout", "--quiet", "-b", "change")
|
||||
git("rm", "--quiet", CheckScript)
|
||||
git("commit", "--quiet", "-m", "the change holds none")
|
||||
head := git("rev-parse", "HEAD")
|
||||
git("checkout", "--quiet", "main")
|
||||
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "replay-310", Repository: origin, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-media-catalog", Number: 11, Base: "main"}, t.TempDir(), "", GitCredential{}, nil)
|
||||
if err == nil && (v.Repo == nil || v.Repo.Verdict != "fail") {
|
||||
t.Fatalf("a pull request without main's %s was answered %+v: none of its tests ran, and main's script "+
|
||||
"did not judge it", CheckScript, v.Repo)
|
||||
}
|
||||
if err != nil && !strings.Contains(err.Error(), "artifact store") {
|
||||
t.Fatalf("the check stopped before main's script could run: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,141 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReplay341 is the replay of novox/hq issue 341, written with only what the build agent had before its fix,
|
||||
// so mesh-lab's prover can lay it over the commit before. On 2026-10-09 node-engine #64/#65 merged at 00:15 and
|
||||
// added Applied.Declared, under the JSON key `declared`, to the store record. #66, checked at 00:49 at its head,
|
||||
// had added its own Applied.Declared under the same key on other lines. Its check passed; git merged the two
|
||||
// edits without a conflict, and the trunk no longer compiled: "Declared redeclared". Before the fix the check
|
||||
// judged the head alone and passed. After it (novox/hq ADR 0323), the check judges the head merged into the
|
||||
// trunk commit that held the other field, and fails saying the trunk does not compile.
|
||||
//
|
||||
// It runs the check whole, with a container runtime that runs each container's command on this machine
|
||||
// (aRuntimeOnThisMachine): the repository's own check is `go build ./...`.
|
||||
func TestReplay341(t *testing.T) {
|
||||
origin, head := theNinthOfOctober(t)
|
||||
registry := aFactsRegistry(t)
|
||||
aRuntimeOnThisMachine(t)
|
||||
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "replay-341", Repository: origin, Ref: head, Owner: "novox",
|
||||
Repo: "mesh-host", Number: 66, Base: "main", Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("the check could not run: %v", err)
|
||||
}
|
||||
if v.Repo == nil || v.Repo.Verdict != "fail" {
|
||||
t.Fatalf("node-engine #66 at its head was answered %+v: the merge it makes into the trunk that holds #65's "+
|
||||
"field does not compile, and the check let it merge", v.Repo)
|
||||
}
|
||||
if said := v.Repo.Summary + "\n" + v.Repo.Failed + "\n" + v.Report; !strings.Contains(said, "Declared redeclared") {
|
||||
t.Fatalf("the check failed without saying the trunk does not compile:\n%s", said)
|
||||
}
|
||||
}
|
||||
|
||||
// theNinthOfOctober is the node-engine's repository as it stood at 00:49 on 2026-10-09, in miniature: a trunk
|
||||
// holding #65's Applied.Declared (a string, key `declared`), and #66's head, branched before it, holding its own
|
||||
// Applied.Declared (a *FoundUnit, the same key) on other lines. Each compiles alone; their merge does not.
|
||||
func theNinthOfOctober(t *testing.T) (string, string) {
|
||||
t.Helper()
|
||||
origin := t.TempDir()
|
||||
git := func(args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = origin
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
|
||||
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v\n%s", args, err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(origin, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(origin, name), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
store := func(declared65, declared66 string) string {
|
||||
return "package store\n\n// FoundUnit is a unit found on the machine.\ntype FoundUnit struct{ Name string }\n\n" +
|
||||
"// Applied is what the node-engine applied.\ntype Applied struct {\n\tGeneration int `json:\"generation\"`\n" +
|
||||
declared66 + "\tModules []string `json:\"modules\"`\n\tFiles []string `json:\"files\"`\n" +
|
||||
"\tUnits []string `json:\"units\"`\n\tAt string `json:\"at\"`\n" + declared65 + "}\n"
|
||||
}
|
||||
git("init", "--quiet", "-b", "main")
|
||||
write("go.mod", "module example.org/mesh-host\n\ngo 1.21\n")
|
||||
write("internal/store/store.go", store("", ""))
|
||||
write(CheckScript, "# mesh-check-toolchain: go\nset -e\nGOFLAGS= GOWORK=off GOTOOLCHAIN=local GOPROXY=off go build ./...\n")
|
||||
git("add", "-A")
|
||||
git("commit", "--quiet", "-m", "#63: the store record")
|
||||
git("checkout", "--quiet", "-b", "adr-0269")
|
||||
write("internal/store/store.go", store("", "\tDeclared *FoundUnit `json:\"declared\"`\n"))
|
||||
git("commit", "--quiet", "-am", "#66: the unit a declaration names")
|
||||
head := git("rev-parse", "HEAD")
|
||||
git("checkout", "--quiet", "main")
|
||||
write("internal/store/store.go", store("\tDeclared string `json:\"declared\"`\n", ""))
|
||||
git("commit", "--quiet", "-am", "#65: the declaration applied")
|
||||
return origin, head
|
||||
}
|
||||
|
||||
// aRuntimeOnThisMachine puts a container runtime on PATH that runs a container's command on this machine, in the
|
||||
// directory and with the environment the check gives it: a store and a bus that answer, a judge that is never
|
||||
// built, and the repository's own check run as written. Only for a check whose code is the test's own.
|
||||
func aRuntimeOnThisMachine(t *testing.T) {
|
||||
t.Helper()
|
||||
bus, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { bus.Close() })
|
||||
go func() {
|
||||
for {
|
||||
c, err := bus.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
c.Close()
|
||||
}
|
||||
}()
|
||||
dir := t.TempDir()
|
||||
runtime := `#!/bin/sh
|
||||
cmd="$1"; shift
|
||||
case "$cmd" in
|
||||
ps|rm|exec) exit 0 ;;
|
||||
port) echo "` + bus.Addr().String() + `"; exit 0 ;;
|
||||
run)
|
||||
workdir=""; detached=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-d) detached=1; shift ;;
|
||||
--rm) shift ;;
|
||||
--workdir) workdir="$2"; shift 2 ;;
|
||||
--env) export "$2"; shift 2 ;;
|
||||
--network|--volume|--user|--label|--name|-p|-e) shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
if [ -n "$detached" ]; then echo replay-container; exit 0; fi
|
||||
shift
|
||||
if [ "$1" = "sh" ] && [ -n "$workdir" ]; then cd "$workdir" && exec "$@"; fi
|
||||
echo "no such image on this machine" >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
exit 1
|
||||
`
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(runtime), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
}
|
||||
@@ -1,154 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReplay363 is the replay of novox/hq issue 363's built part (ADR 0267 rule 6), written with only what the
|
||||
// builder had before its fix (mesh-controller #292), so mesh-lab's prover can lay it over the commit before.
|
||||
//
|
||||
// A Go module built from its repository's root — the controller, the node-engine — is compiled with the clone's
|
||||
// `.git` in view, and Go stamped the commit, its time and whether the tree was modified into the binary. On
|
||||
// 2026-10-13 the node-engine running on the laptop carried `vcs.revision=7a41e120…` in its build info, so a merge
|
||||
// that changed only its README made a new digest of an unchanged program. Before the fix the two builds below give
|
||||
// two digests; after it (`-buildvcs=false` in the Go toolchain) they give one.
|
||||
//
|
||||
// It runs the builder whole, Build to the published bundle, with a container runtime that runs the toolchain's
|
||||
// command on this machine: the module's directory mounted where the toolchain would see it, the real git and the
|
||||
// real Go compiler. Two builds of one module at two commits that differ only in README.md, outside its build source.
|
||||
func TestReplay363(t *testing.T) {
|
||||
for _, tool := range []string{"go", "git"} {
|
||||
if _, err := exec.LookPath(tool); err != nil {
|
||||
t.Skipf("%s is not on this machine: %v", tool, err)
|
||||
}
|
||||
}
|
||||
root := t.TempDir()
|
||||
origin := filepath.Join(root, "origin")
|
||||
files := map[string]string{
|
||||
"module.json": `{"module":"ctl","version":"1","build":{"artifacts":[` +
|
||||
`{"name":"runtime","kind":"bundle","language":"go","system":"arch","from":"cmd/ctl","binary":"ctl"}]},` +
|
||||
`"resources":[{"id":"runtime","type":"process","name":"ctl","artifact":"runtime","run":["./ctl"]}]}`,
|
||||
"go.mod": "module example.com/ctl\n\ngo 1.22\n",
|
||||
"cmd/ctl/main.go": "package main\n\nfunc main() { println(\"ctl\") }\n",
|
||||
"README.md": "one\n",
|
||||
}
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(origin, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(origin, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
git := func(dir string, args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", append([]string{"-c", "user.name=t", "-c", "user.email=t@example.invalid",
|
||||
"-c", "commit.gpgsign=false", "-c", "init.defaultBranch=main"}, args...)...)
|
||||
cmd.Dir = dir
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git(origin, "init", "-q")
|
||||
git(origin, "add", ".")
|
||||
git(origin, "commit", "-q", "-m", "one")
|
||||
first := git(origin, "rev-parse", "HEAD")
|
||||
if err := os.WriteFile(filepath.Join(origin, "README.md"), []byte("two\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git(origin, "commit", "-q", "-am", "only the README")
|
||||
second := git(origin, "rev-parse", "HEAD")
|
||||
|
||||
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
|
||||
build := func(commit, at string) string {
|
||||
t.Helper()
|
||||
got, err := Build(context.Background(), replay363Runtime(t), replay363Store{}, origin, "", commit,
|
||||
filepath.Join(root, at), held, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("the build at %s: %v", commit[:12], err)
|
||||
}
|
||||
if len(got.Built) != 1 || got.Built[0].Digest == "" {
|
||||
t.Fatalf("the build at %s made %+v", commit[:12], got.Built)
|
||||
}
|
||||
return got.Built[0].Digest
|
||||
}
|
||||
a, b := build(first, "one"), build(second, "elsewhere/two")
|
||||
if a != b {
|
||||
t.Fatalf("one build source, two digests: %s at %s, %s at %s — a README-only merge is a new program",
|
||||
a, first[:12], b, second[:12])
|
||||
}
|
||||
}
|
||||
|
||||
// replay363Store publishes nothing: the digest is the builder's own, over the packed bundle.
|
||||
type replay363Store struct{}
|
||||
|
||||
func (replay363Store) PublishImage(_ context.Context, local, repository string) (string, error) {
|
||||
return "registry.invalid/" + repository + "@sha256:" + strings.Repeat("c", 64), nil
|
||||
}
|
||||
|
||||
func (replay363Store) PublishArchive(_ context.Context, repository string, _ []byte, digest string) (string, error) {
|
||||
return "artifact-store://" + repository + "@" + digest, nil
|
||||
}
|
||||
|
||||
// replay363Runtime runs every command on this machine, and a toolchain container's command in the directory
|
||||
// mounted for it, as the container would see it: nothing above the mounted directory.
|
||||
func replay363Runtime(t *testing.T) Runner {
|
||||
t.Helper()
|
||||
return func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "run" {
|
||||
mounts := map[string]string{}
|
||||
workdir := ""
|
||||
i := 1
|
||||
for i < len(args) {
|
||||
switch args[i] {
|
||||
case "--rm":
|
||||
i++
|
||||
continue
|
||||
case "--volume":
|
||||
host, inside, _ := strings.Cut(args[i+1], ":")
|
||||
mounts[inside] = host
|
||||
i += 2
|
||||
continue
|
||||
case "--workdir":
|
||||
workdir = args[i+1]
|
||||
i += 2
|
||||
continue
|
||||
case "--network", "--user", "--env", "-e", "--label", "--name":
|
||||
i += 2
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
command := args[i+1:] // the image is args[i]
|
||||
where := dir
|
||||
for inside, host := range mounts {
|
||||
if workdir == inside {
|
||||
where = host
|
||||
} else if rest, ok := strings.CutPrefix(workdir, inside+"/"); ok {
|
||||
where = filepath.Join(host, rest)
|
||||
}
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, command[0], command[1:]...)
|
||||
cmd.Dir = where
|
||||
cmd.Env = append(os.Environ(), "GOTOOLCHAIN=local", "GOPROXY=off", "GOWORK=off", "GOFLAGS=")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return string(out), &exec.ExitError{ProcessState: cmd.ProcessState, Stderr: out}
|
||||
}
|
||||
return string(out), nil
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
cmd.Dir = dir
|
||||
out, err := cmd.Output()
|
||||
return string(out), err
|
||||
}
|
||||
}
|
||||
@@ -1,127 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// novox/hq issue 713: after ADR 0350 the walk of novox/mesh-controller#331 built mesh-controller, build-agent and
|
||||
// route-proxy again, each "built: no check of novox/mesh-controller#331 built it". The check had built none: its
|
||||
// build log said "built for the merge: 0 of 3 module(s)", mesh-controller's go bundle failing with "go.mod file not
|
||||
// found". The build seat's holder runs in a container sharing only its workspace with the host, and a build's
|
||||
// toolchain container is run by the host's container engine, which mounts a path as the host names it. The check
|
||||
// built in a temporary directory of the holder's own (issue 703's copy), which the engine mounted as an empty one.
|
||||
|
||||
// aHostsEngine runs as the host's container engine does for a holder in a container: a mount of a path outside the
|
||||
// workspace the holder shares with the host is an empty directory, and a Go toolchain finds no go.mod in it.
|
||||
type aHostsEngine struct {
|
||||
onTrunk
|
||||
shared string
|
||||
}
|
||||
|
||||
func (h aHostsEngine) run(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "run" {
|
||||
for i, a := range args {
|
||||
if a != "--volume" || i+1 >= len(args) {
|
||||
continue
|
||||
}
|
||||
from, _, _ := strings.Cut(args[i+1], ":")
|
||||
if !withinDir(h.shared, from) {
|
||||
return "", errors.New("go: go.mod file not found in current directory or any parent directory")
|
||||
}
|
||||
}
|
||||
}
|
||||
return h.onTrunk.run(ctx, dir, name, args...)
|
||||
}
|
||||
|
||||
// aGoModule is a module of one Go bundle built from a repository's root.
|
||||
func aGoModule(module string) map[string]string {
|
||||
files := aSharedRepository("one", "1")
|
||||
files[ManifestName] = `{"module":"` + module + `","version":"1","build":{"artifacts":[
|
||||
{"name":"server","kind":"bundle","language":"go","system":"arch","from":"cmd/ctl","binary":"ctl"}]},
|
||||
"resources":[{"id":"server","type":"process","name":"` + module + `","artifact":"server","run":["./ctl"]}]}`
|
||||
return files
|
||||
}
|
||||
|
||||
// A controller change whose walk moves mesh-controller, build-agent and route-proxy (#331's) has its check build all
|
||||
// three, in the workspace the host's engine sees, and the walk of its merge takes each from the check by its key.
|
||||
func TestReplay713TheWalkOfAControllerChangeTakesAllThreeFromItsCheck(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
pull := PullRef{Owner: "novox", Repo: "mesh-controller", Number: 331}
|
||||
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
|
||||
repositories := map[string]string{"build-agent": "https://forge.invalid/mesh-catalog-build-agent.git",
|
||||
"route-proxy": "https://forge.invalid/mesh-catalog-route-proxy.git"}
|
||||
checked := func(shared string) (*recorded, aHostsEngine) {
|
||||
r := &recorded{tree: "tree-of-the-merge", contents: aGoModule("mesh-controller"),
|
||||
secondary: map[string]map[string]string{}}
|
||||
for module, repository := range repositories {
|
||||
r.secondary[repository] = aGoModule(module)
|
||||
}
|
||||
return r, aHostsEngine{onTrunk: onTrunk{recorded: r}, shared: shared}
|
||||
}
|
||||
spec := CheckSpec{ID: "build-331", Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number,
|
||||
Builds: []CheckBuildSpec{{Module: "mesh-controller", Merged: true, Tier: 0},
|
||||
{Module: "build-agent", Repository: repositories["build-agent"], Ref: "main", Tier: 1},
|
||||
{Module: "route-proxy", Repository: repositories["route-proxy"], Ref: "main", Tier: 2}}}
|
||||
judged := CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053", Merge: "2d22279b1d07"}
|
||||
passed := CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "pass"}}
|
||||
|
||||
// As it ran: the builds made in the holder's own temporary directory, which the host's engine does not see.
|
||||
shared, outside := t.TempDir(), t.TempDir()
|
||||
merged := filepath.Join(outside, "merged")
|
||||
r, engine := checked(shared)
|
||||
asItRan := buildForTheMerge(t.Context(), engine.run, spec, CheckBuilding{Publish: r, Scratch: newScratch(), Held: held},
|
||||
outside, merged, judged.Merge, judged, GitCredential{}, nil, logging(nil))
|
||||
if builtCount(asItRan) != 0 {
|
||||
t.Fatalf("the engine of this test sees the holder's temporary directory: %+v", asItRan)
|
||||
}
|
||||
|
||||
// The check as it builds now: every module its merge moves, kept under its key.
|
||||
scratch := newScratch()
|
||||
spec.BuildWith = &CheckBuilding{Publish: r, Scratch: scratch, Held: held}
|
||||
built := forTheMerge(t.Context(), engine.run, spec, passed, shared, merged, judged, GitCredential{}, nil,
|
||||
logging(nil))
|
||||
if len(built) != 3 || builtCount(built) != 3 {
|
||||
t.Fatalf("the check built %d of %d module(s) its merge moves: %+v", builtCount(built), len(built), built)
|
||||
}
|
||||
var names []NamedBuild
|
||||
for _, b := range built {
|
||||
n := NamedBuild{Pull: pull, Check: spec.ID, Module: b.Module, Key: b.Key, Digests: map[string]string{}}
|
||||
for _, m := range b.Made {
|
||||
d := m.Digest
|
||||
if d == "" {
|
||||
d = DigestOf(m.Reference)
|
||||
}
|
||||
n.Digests[m.Name] = d
|
||||
}
|
||||
names = append(names, n)
|
||||
}
|
||||
|
||||
// The walk of the merge commit, tier by tier: each module taken from the check, none built.
|
||||
walked := map[string]string{"mesh-controller": "https://forge.invalid/mesh-controller.git"}
|
||||
for module, repository := range repositories {
|
||||
walked[module] = repository
|
||||
}
|
||||
for _, module := range []string{"mesh-controller", "build-agent", "route-proxy"} {
|
||||
w, walk := checked(shared)
|
||||
w.secondary[walked["mesh-controller"]] = aGoModule("mesh-controller")
|
||||
got, err := BuildReusing(t.Context(), walk.run, w, Reuse{Scratch: scratch, Pulls: []PullRef{pull}, Named: names},
|
||||
walked[module], "", "main", t.TempDir(), held, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", module, err)
|
||||
}
|
||||
if got.Reused == nil || got.Why != "" {
|
||||
t.Fatalf("the walk built %s again: %s", module, got.Why)
|
||||
}
|
||||
for _, line := range w.ran {
|
||||
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "go build") {
|
||||
t.Fatalf("the walk compiled %s: %s", module, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,160 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// novox/hq issue 716: a check built the images its merge moves one after another, and only after its merge-check.sh
|
||||
// had passed — 3 to 6 minutes on a check of about 5. The modules of a tier stand on none of each other and are built
|
||||
// at the same time; with a judge cache, beside the tests; a check that does not pass lets go of them.
|
||||
|
||||
// atOnce is a fake engine whose compiles wait until every one of a tier's has started: builds one after another never
|
||||
// get there, and the first one's compile says so.
|
||||
type atOnce struct {
|
||||
mu sync.Mutex
|
||||
r *recorded
|
||||
want int
|
||||
started int
|
||||
all chan struct{}
|
||||
shared string
|
||||
}
|
||||
|
||||
func (a *atOnce) run(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "docker" && len(args) > 0 && args[0] == "run" && strings.Contains(strings.Join(args, " "), "go build") {
|
||||
a.mu.Lock()
|
||||
a.started++
|
||||
if a.started == a.want {
|
||||
close(a.all)
|
||||
}
|
||||
a.mu.Unlock()
|
||||
select {
|
||||
case <-a.all:
|
||||
case <-time.After(5 * time.Second):
|
||||
return "", errors.New("built alone: no other module of its tier was being built")
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
return onTrunk{recorded: a.r}.run(ctx, dir, name, args...)
|
||||
}
|
||||
|
||||
func (a *atOnce) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
return a.r.PublishImage(ctx, localTag, repository)
|
||||
}
|
||||
|
||||
func (a *atOnce) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
return a.r.PublishArchive(ctx, repository, body, digest)
|
||||
}
|
||||
|
||||
// aLockedScratch is the scratch namespace in memory, said to by builds at once.
|
||||
type aLockedScratch struct {
|
||||
mu sync.Mutex
|
||||
*aScratch
|
||||
discarded []CheckBuild
|
||||
}
|
||||
|
||||
func (s *aLockedScratch) Keep(ctx context.Context, p PullRef, c string, at time.Time, b []CheckBuild) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.aScratch.Keep(ctx, p, c, at, b)
|
||||
}
|
||||
|
||||
func (s *aLockedScratch) Find(ctx context.Context, module, key string) (CheckBuild, bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.aScratch.Find(ctx, module, key)
|
||||
}
|
||||
|
||||
func (s *aLockedScratch) Discard(_ context.Context, b []CheckBuild) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.discarded = append(s.discarded, b...)
|
||||
return nil
|
||||
}
|
||||
|
||||
// threeAtOnce are three Go modules in one tier, each from its own repository.
|
||||
func threeAtOnce(t *testing.T) (*atOnce, CheckSpec, map[string]string) {
|
||||
t.Helper()
|
||||
r := &recorded{tree: "tree-of-the-merge", contents: aGoModule("mesh-controller"),
|
||||
secondary: map[string]map[string]string{}}
|
||||
spec := CheckSpec{ID: "build-716", Owner: "novox", Repo: "mesh-controller", Number: 716,
|
||||
Builds: []CheckBuildSpec{{Module: "mesh-controller", Merged: true, Tier: 0}}}
|
||||
for _, module := range []string{"build-agent", "route-proxy"} {
|
||||
repository := "https://forge.invalid/" + module + ".git"
|
||||
r.secondary[repository] = aGoModule(module)
|
||||
spec.Builds = append(spec.Builds, CheckBuildSpec{Module: module, Repository: repository, Ref: "main", Tier: 0})
|
||||
}
|
||||
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
|
||||
return &atOnce{r: r, want: 3, all: make(chan struct{})}, spec, held
|
||||
}
|
||||
|
||||
// The modules of one tier are built at the same time, each in a working copy of its own.
|
||||
func TestReplay716TheModulesOfATierAreBuiltAtTheSameTime(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
engine, spec, held := threeAtOnce(t)
|
||||
scratch := &aLockedScratch{aScratch: newScratch()}
|
||||
built := buildForTheMerge(t.Context(), engine.run, spec, CheckBuilding{Publish: engine, Scratch: scratch, Held: held},
|
||||
t.TempDir(), filepath.Join(t.TempDir(), "merged"), "2d22279b1d07",
|
||||
CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053"}, GitCredential{}, nil, logging(nil))
|
||||
if builtCount(built) != 3 {
|
||||
t.Fatalf("the check built %d of 3 module(s) of one tier: %+v", builtCount(built), built)
|
||||
}
|
||||
if len(scratch.builds) != 3 {
|
||||
t.Fatalf("the check kept %d build(s), not 3", len(scratch.builds))
|
||||
}
|
||||
}
|
||||
|
||||
// With a judge cache, the builds run while the tests run, where no container of the check reaches; a check that passed
|
||||
// keeps them, and one that failed lets go of every one and names none.
|
||||
func TestReplay716TheBuildsRunBesideTheTestsAndAFailedCheckLetsGoOfThem(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
workspace := t.TempDir()
|
||||
if _, err := buildsBesideTheTests(filepath.Join(workspace, "judge-cache"), workspace, "build-716"); err == nil {
|
||||
t.Fatal("a judge cache inside the workspace was built in, where a pull request's script reaches")
|
||||
}
|
||||
for _, passed := range []bool{true, false} {
|
||||
cache := t.TempDir()
|
||||
place, err := buildsBesideTheTests(cache, workspace, "build-716")
|
||||
if err != nil || !withinDir(cache, place) || withinDir(workspace, place) {
|
||||
t.Fatalf("the builds' place %q (%v) is not the judge cache's, outside the workspace", place, err)
|
||||
}
|
||||
engine, spec, held := threeAtOnce(t)
|
||||
scratch := &aLockedScratch{aScratch: newScratch()}
|
||||
spec.BuildWith = &CheckBuilding{Publish: engine, Scratch: scratch, Held: held}
|
||||
judged := CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053", Merge: "2d22279b1d07"}
|
||||
beside := startBesideTheTests(t.Context(), engine.run, spec, place, filepath.Join(t.TempDir(), "merged"),
|
||||
judged, GitCredential{}, nil, logging(nil))
|
||||
// The tests: the builds are under way before the check has a verdict.
|
||||
select {
|
||||
case <-engine.all:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("nothing was built while the tests ran")
|
||||
}
|
||||
verdict := CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "pass"}}
|
||||
if !passed {
|
||||
verdict.Repo.Verdict = "fail"
|
||||
}
|
||||
built := beside.settle(t.Context(), verdict)
|
||||
beside.abandon()
|
||||
switch {
|
||||
case passed && (builtCount(built) != 3 || len(scratch.builds) != 3 || len(scratch.discarded) != 0):
|
||||
t.Fatalf("a check that passed: built %+v, kept %d, let go of %d", built, len(scratch.builds),
|
||||
len(scratch.discarded))
|
||||
case !passed && (built != nil || len(scratch.builds) != 0 || len(scratch.discarded) != 3):
|
||||
t.Fatalf("a check that failed: named %+v, kept %d, let go of %d", built, len(scratch.builds),
|
||||
len(scratch.discarded))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,91 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAGoBundleBuiltTwiceFromOneBuildSourceIsOneDigest(t *testing.T) {
|
||||
for _, tool := range []string{"go", "git"} {
|
||||
if _, err := exec.LookPath(tool); err != nil {
|
||||
t.Skipf("%s is not on this machine: %v", tool, err)
|
||||
}
|
||||
}
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
repo := filepath.Join(root, "origin")
|
||||
files := map[string]string{
|
||||
"go.mod": "module example.com/ctl\n\ngo 1.22\n",
|
||||
"cmd/ctl/main.go": "package main\n\nfunc main() { println(\"ctl\") }\n",
|
||||
"README.md": "one\n",
|
||||
}
|
||||
for name, body := range files {
|
||||
if err := os.MkdirAll(filepath.Dir(filepath.Join(repo, name)), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(repo, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
git := func(dir string, args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", append([]string{"-c", "user.name=t", "-c", "user.email=t@example.invalid",
|
||||
"-c", "commit.gpgsign=false", "-c", "init.defaultBranch=main"}, args...)...)
|
||||
cmd.Dir = dir
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git(repo, "init", "-q")
|
||||
git(repo, "add", ".")
|
||||
git(repo, "commit", "-q", "-m", "one")
|
||||
first := git(repo, "rev-parse", "HEAD")
|
||||
if err := os.WriteFile(filepath.Join(repo, "README.md"), []byte("two\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git(repo, "commit", "-q", "-am", "only the README")
|
||||
second := git(repo, "rev-parse", "HEAD")
|
||||
|
||||
build := func(commit, at string) string {
|
||||
t.Helper()
|
||||
clone := filepath.Join(root, at)
|
||||
git(root, "clone", "-q", repo, clone)
|
||||
git(clone, "checkout", "-q", commit)
|
||||
out := filepath.Join(clone, Out("server"), "ctl")
|
||||
line := append([]string(nil), chain.Compile...)
|
||||
line = append(line, "-ldflags", strings.Join(chain.LinkerFlags, " "), chain.OutputFlag, out, "./cmd/ctl")
|
||||
cmd := exec.Command(line[0], line[1:]...)
|
||||
cmd.Dir = clone
|
||||
cmd.Env = append(os.Environ(), "GOTOOLCHAIN=local", "GOPROXY=off", "GOWORK=off")
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("%s: %v\n%s", strings.Join(line, " "), err, b)
|
||||
}
|
||||
body, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if bytes.Contains(body, []byte(commit)) {
|
||||
t.Errorf("the binary built at %s carries the commit it was built at", commit[:12])
|
||||
}
|
||||
sum := sha256.Sum256(body)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
a, b := build(first, "a"), build(second, "elsewhere/b")
|
||||
if a != b {
|
||||
t.Fatalf("one build source, two digests: %s at %s, %s at %s (compiled with %q)",
|
||||
a[:12], first[:12], b[:12], second[:12], strings.Join(chain.Compile, " "))
|
||||
}
|
||||
}
|
||||
@@ -1,65 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// **A Go bundle does not vary with the commit it was built at** (novox/hq ADR 0267 rule 6, issue 363). A
|
||||
// module built from its repository's root — the controller — is compiled with the clone's `.git` in what
|
||||
// the toolchain sees, and Go stamps the commit, its time and whether the tree was modified into every
|
||||
// binary built there unless told not to. Two builds of one program at commits that differ only in a
|
||||
// README then made two binaries, so an unchanged build source was a new digest.
|
||||
//
|
||||
// The toolchain's own compile line is run here, as the build seat runs it inside the toolchain image: in
|
||||
// two clones at two paths, of two commits that differ outside the program. One digest is wanted.
|
||||
|
||||
// The toolchain says so itself, so a reader of the compile line sees it without running a build.
|
||||
func TestTheGoToolchainStampsNoVersionControl(t *testing.T) {
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if joined := strings.Join(chain.Compile, " "); !strings.Contains(joined, "-buildvcs=false") {
|
||||
t.Fatalf("the go toolchain stamps the commit into what it compiles: %q", joined)
|
||||
}
|
||||
}
|
||||
|
||||
// **And so does every recipe in this repository that compiles Go** (review of issue 363's fix): the
|
||||
// controller's genesis image copies the whole clone, `.git` with it, and the reference images beside it
|
||||
// are built the same way by the Makefile.
|
||||
func TestEveryGoRecipeInThisRepositoryStampsNoVersionControl(t *testing.T) {
|
||||
root := filepath.Join("..", "..")
|
||||
var recipes []string
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() && (d.Name() == "vendor" || d.Name() == "testdata" || d.Name() == ".git") {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
if !d.IsDir() && d.Name() == "Dockerfile" {
|
||||
recipes = append(recipes, path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(recipes) == 0 {
|
||||
t.Fatal("no recipe found in this repository")
|
||||
}
|
||||
for _, path := range recipes {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
if strings.Contains(line, "go build") && !strings.Contains(line, "-buildvcs=false") {
|
||||
t.Errorf("%s compiles Go and stamps the commit into it: %s", path, strings.TrimSpace(line))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,630 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"github.com/novox/mesh-controller/internal/artifacts"
|
||||
)
|
||||
|
||||
// One build per change (novox/hq ADR 0350).
|
||||
//
|
||||
// **The merge check builds what the merge would move** (§1): every artifact of every module the delivery plan
|
||||
// moves, from the merged tree it checks, into the artifact store's scratch namespace — never a module's own —
|
||||
// kept under a **reuse key**: the source fingerprint as the build takes it (the module's tree, its contexts' trees,
|
||||
// its bases and toolchains by digest; issue 280) with the builder's own version and, for an upstream artifact, the
|
||||
// image it copies. **The walk of a trunk commit takes that build** (§2): before building, it takes the reuse key
|
||||
// of the trunk commit's tree the same way, and where a check build holds the same key it copies it by digest into
|
||||
// the module's namespace and answers with that, building nothing. Otherwise it builds and says why (§3): no check
|
||||
// built it, the key differs (naming the part), or the check build is gone; a copy that fails is said and built.
|
||||
//
|
||||
// **What the key names is what a fresh build could differ by.** The tree is every file the recipe and the
|
||||
// compiler see; the contexts are the other repositories it reads, by tree; the bases and toolchains are what it
|
||||
// stands on, by digest; the builder's version is how it is built. A build whose source does not pin it — one that
|
||||
// resolves packages from the mesh's registry when it is built, or copies an upstream image named by tag — has no
|
||||
// key, and is always built.
|
||||
|
||||
// Version is this builder's own version, as the reuse key takes it: the digest of the running executable, so any
|
||||
// change to how it builds is a change of key (§1). Set once by the build seat's holder (SelfVersion); empty — no
|
||||
// key, every build its own — when it cannot be said.
|
||||
var Version string
|
||||
|
||||
// SelfVersion is the version the reuse key takes: the running executable's digest, or the version stamped at build
|
||||
// when the executable cannot be read. Empty when neither says anything ("development" says nothing: every unstamped
|
||||
// builder would read as one).
|
||||
func SelfVersion(stamped string) string {
|
||||
if exe, err := os.Executable(); err == nil {
|
||||
if f, err := os.Open(exe); err == nil {
|
||||
defer f.Close()
|
||||
sum := sha256.New()
|
||||
if _, err := io.Copy(sum, f); err == nil {
|
||||
return "exe:sha256:" + hex.EncodeToString(sum.Sum(nil))
|
||||
}
|
||||
}
|
||||
}
|
||||
if stamped == "" || stamped == "development" {
|
||||
return ""
|
||||
}
|
||||
return stamped
|
||||
}
|
||||
|
||||
// ScratchPrefix is the namespace a check builds into (novox/hq ADR 0239 §9, ADR 0350 §1): `scratch/<module>/<artifact>`.
|
||||
// Registration refuses a reference into it; only the walk's copy, in the module's own namespace, is registered.
|
||||
const ScratchPrefix = "scratch/"
|
||||
|
||||
// checksRepository holds one document per check that built: which builds it made, of which pull request, when.
|
||||
// What the holder lets go of by (§5).
|
||||
const checksRepository = ScratchPrefix + "checks"
|
||||
|
||||
// reuseTagPrefix names a check build's record under its module's scratch repository: `r1-<hex>`.
|
||||
const reuseTagPrefix = "r1-"
|
||||
|
||||
// CheckBuildsKept is how long a check build is kept in the scratch namespace (§5).
|
||||
const CheckBuildsKept = 7 * 24 * time.Hour
|
||||
|
||||
// The parts of a reuse key, as the walk names the one that differs (§3).
|
||||
const (
|
||||
partModule = "the module"
|
||||
partTree = "the build source"
|
||||
partBases = "a base image"
|
||||
partBuilder = "the builder's version"
|
||||
partContext = "a context"
|
||||
partTool = "a toolchain"
|
||||
partUpstream = "an upstream image"
|
||||
)
|
||||
|
||||
// reuseKey is a build's reuse key and the parts it was made from; Key empty, with why, when it has none.
|
||||
type reuseKey struct {
|
||||
Key string
|
||||
Parts map[string]string
|
||||
Why string
|
||||
}
|
||||
|
||||
// keyOf is the reuse key of what src says the build was made from, built by version (ADR 0350 §1).
|
||||
func keyOf(src *sourceInputs, manifest catalogue.Manifest, version string) reuseKey {
|
||||
if src == nil || src.unpinned != "" || src.tree == "" {
|
||||
why := "its source does not pin the build"
|
||||
if src != nil && src.unpinned != "" {
|
||||
why = src.unpinned
|
||||
} else if src != nil && src.tree == "" {
|
||||
why = "its tree could not be named"
|
||||
}
|
||||
return reuseKey{Why: why}
|
||||
}
|
||||
if version == "" {
|
||||
return reuseKey{Why: "the builder's version cannot be said"}
|
||||
}
|
||||
parts := map[string]string{partModule: src.module, partTree: src.tree, partBuilder: version}
|
||||
var bases []string
|
||||
seen := map[string]bool{}
|
||||
for _, b := range src.bases {
|
||||
if d := referenceDigest(b); !seen[d] {
|
||||
seen[d] = true
|
||||
bases = append(bases, d)
|
||||
}
|
||||
}
|
||||
sort.Strings(bases)
|
||||
parts[partBases] = strings.Join(bases, " ")
|
||||
for a, t := range src.contexts {
|
||||
parts[partContext+" "+a] = t
|
||||
}
|
||||
for a, t := range src.toolchains {
|
||||
parts[partTool+" "+a] = t
|
||||
}
|
||||
// An upstream artifact is copied, not built: what it copies is in module.json, and so in the tree — but a tag
|
||||
// names whatever its registry says today, so only an image pinned by digest is the same image tomorrow.
|
||||
if manifest.Build != nil {
|
||||
for _, a := range manifest.Build.Artifacts {
|
||||
if a.Kind != catalogue.ArtifactUpstream {
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(a.From, "@sha256:") {
|
||||
return reuseKey{Why: fmt.Sprintf("%s copies %s, named by tag, which can move", a.Name, a.From)}
|
||||
}
|
||||
parts[partUpstream+" "+a.Name] = a.From
|
||||
}
|
||||
}
|
||||
lines := make([]string, 0, len(parts))
|
||||
for k, v := range parts {
|
||||
lines = append(lines, k+"="+v)
|
||||
}
|
||||
sort.Strings(lines)
|
||||
sum := sha256.Sum256([]byte(strings.Join(lines, "\n")))
|
||||
return reuseKey{Key: "r1:" + hex.EncodeToString(sum[:]), Parts: parts}
|
||||
}
|
||||
|
||||
// tagOf is the tag a key's record is kept under.
|
||||
func tagOf(key string) string { return reuseTagPrefix + strings.TrimPrefix(key, "r1:") }
|
||||
|
||||
// differing names the parts of two keys that differ, in a fixed order: what the walk says when it builds (§3).
|
||||
func differing(was, now map[string]string) []string {
|
||||
names := map[string]bool{}
|
||||
for k := range was {
|
||||
names[k] = true
|
||||
}
|
||||
for k := range now {
|
||||
names[k] = true
|
||||
}
|
||||
var out []string
|
||||
for k := range names {
|
||||
if was[k] != now[k] {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// PullRef names a pull request.
|
||||
type PullRef struct {
|
||||
Owner string `json:"owner"`
|
||||
Repo string `json:"repo"`
|
||||
Number int `json:"number"`
|
||||
}
|
||||
|
||||
func (p PullRef) String() string { return fmt.Sprintf("%s/%s#%d", p.Owner, p.Repo, p.Number) }
|
||||
|
||||
// slug is the pull request as a tag's first part: owner.repo.number, every other character a dash.
|
||||
func (p PullRef) slug() string {
|
||||
clean := func(s string) string {
|
||||
return strings.Map(func(r rune) rune {
|
||||
if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '_' || r == '-' {
|
||||
return r
|
||||
}
|
||||
return '-'
|
||||
}, s)
|
||||
}
|
||||
return clean(p.Owner) + "." + clean(p.Repo) + "." + strconv.Itoa(p.Number)
|
||||
}
|
||||
|
||||
// CheckBuild is one module a check built into the scratch namespace, kept under its reuse key (§1).
|
||||
type CheckBuild struct {
|
||||
Key string `json:"key"`
|
||||
Module string `json:"module"`
|
||||
Parts map[string]string `json:"parts"`
|
||||
// Made are its artifacts, in the scratch namespace.
|
||||
Made []catalogue.Built `json:"made"`
|
||||
// Pull is the pull request checked, Head its head and Trunk the trunk commit it was merged into; Check the
|
||||
// check's ask; At when it was built.
|
||||
Pull PullRef `json:"pull"`
|
||||
Head string `json:"head,omitempty"`
|
||||
Trunk string `json:"trunk,omitempty"`
|
||||
Check string `json:"check"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Said is the check build as the walk names it: "the check of owner/repo#n at head H (trunk T)".
|
||||
func (c CheckBuild) Said() string {
|
||||
s := "the check of " + c.Pull.String()
|
||||
if c.Head != "" {
|
||||
s += " at head " + twelve(c.Head)
|
||||
}
|
||||
if c.Trunk != "" {
|
||||
s += " (merged into trunk " + twelve(c.Trunk) + ")"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// checkIndex is one check's document under checksRepository: every build it made.
|
||||
type checkIndex struct {
|
||||
Pull PullRef `json:"pull"`
|
||||
Check string `json:"check"`
|
||||
At time.Time `json:"at"`
|
||||
Builds []CheckBuild `json:"builds"`
|
||||
}
|
||||
|
||||
// Scratch is where check builds are kept and taken from: the artifact store's scratch namespace (ADR 0350). The
|
||||
// Registry is one; a test's is a map.
|
||||
type Scratch interface {
|
||||
// Keep records a check's builds: each under its module and key, and the check's own document.
|
||||
Keep(ctx context.Context, pull PullRef, check string, at time.Time, builds []CheckBuild) error
|
||||
// Find is the check build kept under a module's key; false when none is.
|
||||
Find(ctx context.Context, module, key string) (CheckBuild, bool, error)
|
||||
// OfPulls are the check builds kept for those pull requests, newest first: what the walk names a differing
|
||||
// key from.
|
||||
OfPulls(ctx context.Context, pulls []PullRef) ([]CheckBuild, error)
|
||||
// Take copies one artifact of a check build by digest into the module's own namespace, and answers it there.
|
||||
Take(ctx context.Context, module string, made catalogue.Built) (catalogue.Built, error)
|
||||
// Forget lets go of every check build of a pull request, and Expire of every check older than the bound;
|
||||
// each answers how many checks it let go of.
|
||||
Forget(ctx context.Context, pull PullRef) (int, error)
|
||||
Expire(ctx context.Context, older time.Duration, now time.Time) (int, error)
|
||||
// Discard lets go of what builds a check could not keep made: nothing would ever find them.
|
||||
Discard(ctx context.Context, builds []CheckBuild) error
|
||||
}
|
||||
|
||||
// scratchPublisher is a Publisher into the scratch namespace: what a check builds is never under a module's
|
||||
// own name (ADR 0239 §9). A base the build stands on is copied where every build copies it.
|
||||
type scratchPublisher struct{ to Publisher }
|
||||
|
||||
func (s scratchPublisher) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
|
||||
return s.to.PublishImage(ctx, localTag, ScratchPrefix+repository)
|
||||
}
|
||||
|
||||
func (s scratchPublisher) PushedNames(repository string) []string {
|
||||
if p, ok := s.to.(PushedNames); ok {
|
||||
return p.PushedNames(ScratchPrefix + repository)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s scratchPublisher) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
|
||||
return s.to.PublishArchive(ctx, ScratchPrefix+repository, body, digest)
|
||||
}
|
||||
|
||||
func (s scratchPublisher) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
|
||||
if l, can := s.to.(BaseLender); can {
|
||||
return l.LendBases(ctx, ScratchPrefix+repository, bases)
|
||||
}
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func (s scratchPublisher) MirrorImage(ctx context.Context, from, repository string) (string, error) {
|
||||
m, can := s.to.(Mirrorer)
|
||||
if !can {
|
||||
return "", errors.New("the artifact store here cannot copy an upstream image")
|
||||
}
|
||||
return m.MirrorImage(ctx, from, ScratchPrefix+repository)
|
||||
}
|
||||
|
||||
func (s scratchPublisher) MirrorBase(ctx context.Context, from, repository, former string) (string, error) {
|
||||
if m, can := s.to.(BaseMirrorer); can {
|
||||
return m.MirrorBase(ctx, from, repository, former)
|
||||
}
|
||||
if m, can := s.to.(Mirrorer); can {
|
||||
return m.MirrorImage(ctx, from, repository)
|
||||
}
|
||||
return "", errors.New("the artifact store here cannot copy a base")
|
||||
}
|
||||
|
||||
// The registry as the scratch namespace.
|
||||
|
||||
func (r Registry) store() artifacts.Store {
|
||||
return artifacts.Store{Address: r.Address, HTTP: r.client()}
|
||||
}
|
||||
|
||||
const recordMedia = "application/vnd.novox.mesh.check-build.v1+json"
|
||||
|
||||
// Keep puts each build's record under its module's scratch repository, by key, and the check's document.
|
||||
func (r Registry) Keep(ctx context.Context, pull PullRef, check string, at time.Time, builds []CheckBuild) error {
|
||||
store := r.store()
|
||||
for _, b := range builds {
|
||||
body, err := json.Marshal(b)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := store.PutTagged(ctx, ScratchPrefix+b.Module, tagOf(b.Key), recordMedia, body); err != nil {
|
||||
return fmt.Errorf("keeping %s's check build: %w", b.Module, err)
|
||||
}
|
||||
}
|
||||
body, err := json.Marshal(checkIndex{Pull: pull, Check: check, At: at, Builds: builds})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := store.PutTagged(ctx, checksRepository, pull.slug()+"."+check, recordMedia, body); err != nil {
|
||||
return fmt.Errorf("keeping which builds the check of %s made: %w", pull, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Find reads the record kept under a module's key.
|
||||
func (r Registry) Find(ctx context.Context, module, key string) (CheckBuild, bool, error) {
|
||||
body, _, err := r.store().GetTagged(ctx, ScratchPrefix+module, tagOf(key))
|
||||
if errors.Is(err, artifacts.ErrNoTag) {
|
||||
return CheckBuild{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return CheckBuild{}, false, err
|
||||
}
|
||||
var b CheckBuild
|
||||
if err := json.Unmarshal(body, &b); err != nil {
|
||||
return CheckBuild{}, false, fmt.Errorf("%s's check build under %s is not one: %w", module, tagOf(key), err)
|
||||
}
|
||||
if b.Key != key || b.Module != module {
|
||||
return CheckBuild{}, false, nil
|
||||
}
|
||||
return b, true, nil
|
||||
}
|
||||
|
||||
// checks are the check documents kept, by tag; only those of the pull requests named when any are.
|
||||
func (r Registry) checks(ctx context.Context, pulls []PullRef) (map[string]checkIndex, error) {
|
||||
store := r.store()
|
||||
tags, err := store.Tags(ctx, checksRepository)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
wanted := map[string]PullRef{}
|
||||
for _, p := range pulls {
|
||||
wanted[p.slug()+"."] = p
|
||||
}
|
||||
out := map[string]checkIndex{}
|
||||
for _, tag := range tags {
|
||||
if len(pulls) > 0 {
|
||||
match := false
|
||||
for prefix := range wanted {
|
||||
match = match || strings.HasPrefix(tag, prefix)
|
||||
}
|
||||
if !match {
|
||||
continue
|
||||
}
|
||||
}
|
||||
body, _, err := store.GetTagged(ctx, checksRepository, tag)
|
||||
if errors.Is(err, artifacts.ErrNoTag) {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var c checkIndex
|
||||
if err := json.Unmarshal(body, &c); err != nil {
|
||||
continue // not a document this wrote: left to nobody, said by nothing — it names no build
|
||||
}
|
||||
if len(pulls) > 0 {
|
||||
if p := wanted[c.Pull.slug()+"."]; p != c.Pull {
|
||||
continue // a slug two pull requests share: only the one named
|
||||
}
|
||||
}
|
||||
out[tag] = c
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// OfPulls are the check builds of those pull requests, newest first.
|
||||
func (r Registry) OfPulls(ctx context.Context, pulls []PullRef) ([]CheckBuild, error) {
|
||||
if len(pulls) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
checks, err := r.checks(ctx, pulls)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []CheckBuild
|
||||
for _, c := range checks {
|
||||
out = append(out, c.Builds...)
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].At.After(out[j].At) })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Take copies one artifact of a check build into the module's namespace by digest: an image's manifests as they
|
||||
// are, so the digest registered is the digest the check built; an archive's blob mounted across, and held.
|
||||
func (r Registry) Take(ctx context.Context, module string, made catalogue.Built) (catalogue.Built, error) {
|
||||
repository := module + "/" + made.Name
|
||||
from := ScratchPrefix + repository
|
||||
if made.Digest != "" || strings.Contains(made.Reference, "/blobs/") {
|
||||
digest := made.Digest
|
||||
if digest == "" {
|
||||
_, digest, _ = strings.Cut(made.Reference, "/blobs/")
|
||||
}
|
||||
where := upstream{base: "http://" + r.Address, repository: from, reference: digest}
|
||||
if err := r.copyBlob(ctx, &source{client: r.client(), mountFrom: from}, where, digest, repository); err != nil {
|
||||
return catalogue.Built{}, err
|
||||
}
|
||||
size, err := r.blobSize(ctx, repository, digest)
|
||||
if err != nil {
|
||||
return catalogue.Built{}, err
|
||||
}
|
||||
if err := r.hold(ctx, repository, digest, int(size)); err != nil {
|
||||
return catalogue.Built{}, err
|
||||
}
|
||||
return catalogue.Built{Name: made.Name, Kind: made.Kind, Digest: digest, Launchers: made.Launchers,
|
||||
Reference: "http://" + r.Address + "/v2/" + repository + "/blobs/" + digest}, nil
|
||||
}
|
||||
_, digest, pinned := strings.Cut(made.Reference, "@")
|
||||
if !pinned || !strings.HasPrefix(digest, "sha256:") {
|
||||
return catalogue.Built{}, fmt.Errorf("%s's check build %s is not pinned by digest", module, made.Reference)
|
||||
}
|
||||
reference, err := r.MirrorBase(ctx, r.Address+"/"+from+"@"+digest, repository, from)
|
||||
if err != nil {
|
||||
return catalogue.Built{}, err
|
||||
}
|
||||
if !strings.HasSuffix(reference, "@"+digest) {
|
||||
return catalogue.Built{}, fmt.Errorf("%s's check build was copied as %s, not as %s", module, reference, digest)
|
||||
}
|
||||
return catalogue.Built{Name: made.Name, Kind: made.Kind, Reference: reference, Launchers: made.Launchers}, nil
|
||||
}
|
||||
|
||||
// blobSize is the size the store says a blob has.
|
||||
func (r Registry) blobSize(ctx context.Context, repository, digest string) (int64, error) {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodHead, "http://"+r.Address+"/v2/"+repository+"/blobs/"+digest, nil)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
response, err := r.client().Do(request)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK || response.ContentLength < 0 {
|
||||
return 0, fmt.Errorf("%s/blobs/%s: %s", repository, digest, response.Status)
|
||||
}
|
||||
return response.ContentLength, nil
|
||||
}
|
||||
|
||||
// Forget lets go of every check build of a pull request (§5): what each check built in the scratch namespace, the
|
||||
// record under its key when it is still that check's, and the check's document.
|
||||
func (r Registry) Forget(ctx context.Context, pull PullRef) (int, error) {
|
||||
all, err := r.checks(ctx, nil)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
mine := map[string]checkIndex{}
|
||||
for tag, c := range all {
|
||||
if c.Pull == pull {
|
||||
mine[tag] = c
|
||||
}
|
||||
}
|
||||
return r.letGo(ctx, mine, all)
|
||||
}
|
||||
|
||||
// Expire lets go of every check older than the bound (§5).
|
||||
func (r Registry) Expire(ctx context.Context, older time.Duration, now time.Time) (int, error) {
|
||||
checks, err := r.checks(ctx, nil)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
old := map[string]checkIndex{}
|
||||
for tag, c := range checks {
|
||||
if now.Sub(c.At) > older {
|
||||
old[tag] = c
|
||||
}
|
||||
}
|
||||
return r.letGo(ctx, old, checks)
|
||||
}
|
||||
|
||||
// Discard lets go of builds no check document names.
|
||||
func (r Registry) Discard(ctx context.Context, builds []CheckBuild) error {
|
||||
all, err := r.checks(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = r.letGo(ctx, map[string]checkIndex{"": {Builds: builds}}, all)
|
||||
return err
|
||||
}
|
||||
|
||||
// letGo lets go of each check's builds, then its document; a check whose builds could not all be let go of keeps
|
||||
// its document, so the next sweep asks again. **A build another check still names stays**: deletion is by digest,
|
||||
// and two checks that built one digest in one repository — a check asked again, a dependent two pull requests move —
|
||||
// share it; it goes with the last check that names it. all are every check kept, the ones going among them.
|
||||
func (r Registry) letGo(ctx context.Context, checks, all map[string]checkIndex) (int, error) {
|
||||
store := r.store()
|
||||
stays := map[string]bool{}
|
||||
// recorded are the records under a key another check still names (novox/hq issue 703): a later check that kept
|
||||
// the build names it as the check's that built it, and the walk finds it by that record.
|
||||
recorded := map[string]bool{}
|
||||
for tag, c := range all {
|
||||
if _, going := checks[tag]; going {
|
||||
continue
|
||||
}
|
||||
for _, b := range c.Builds {
|
||||
recorded[b.Module+" "+b.Key+" "+b.Check] = true
|
||||
for _, m := range b.Made {
|
||||
stays[catalogue.Recorded(m.Reference)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
gone := 0
|
||||
var failed []string
|
||||
for tag, c := range checks {
|
||||
ok := true
|
||||
for _, b := range c.Builds {
|
||||
for _, m := range b.Made {
|
||||
reference := catalogue.Recorded(m.Reference)
|
||||
if path, kept := catalogue.InArtifactStore(reference); !kept || !strings.HasPrefix(path, ScratchPrefix) {
|
||||
continue // never anything outside the scratch namespace
|
||||
}
|
||||
if stays[reference] {
|
||||
continue // another check still names it
|
||||
}
|
||||
if err := store.LetGo(ctx, reference); err != nil && !errors.Is(err, artifacts.Gone) {
|
||||
ok = false
|
||||
failed = append(failed, fmt.Sprintf("%s: %v", m.Reference, err))
|
||||
}
|
||||
}
|
||||
// The record under the key, only while it is still this check's: a later check of the same key
|
||||
// replaced it, and that one is its own check's to let go of. A build a check kept is that build's
|
||||
// check's record, and goes with the last check naming it (novox/hq issue 703).
|
||||
if kept, found, err := r.Find(ctx, b.Module, b.Key); err == nil && found &&
|
||||
(kept.Check == c.Check || kept.Check == b.Check) && !recorded[b.Module+" "+b.Key+" "+kept.Check] {
|
||||
if err := store.DeleteTagged(ctx, ScratchPrefix+b.Module, tagOf(b.Key)); err != nil && !errors.Is(err, artifacts.Gone) {
|
||||
ok = false
|
||||
failed = append(failed, fmt.Sprintf("%s's record: %v", b.Module, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if tag == "" {
|
||||
gone++
|
||||
continue
|
||||
}
|
||||
if err := store.DeleteTagged(ctx, checksRepository, tag); err != nil && !errors.Is(err, artifacts.Gone) {
|
||||
failed = append(failed, fmt.Sprintf("%s: %v", tag, err))
|
||||
continue
|
||||
}
|
||||
gone++
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
sort.Strings(failed)
|
||||
return gone, fmt.Errorf("not let go of: %s", strings.Join(failed, "; "))
|
||||
}
|
||||
return gone, nil
|
||||
}
|
||||
|
||||
// Reused is what the walk took from a check instead of building (§2, §4).
|
||||
type Reused struct {
|
||||
From CheckBuild
|
||||
// Made are the copies in the module's namespace: what is registered.
|
||||
Made []catalogue.Built
|
||||
}
|
||||
|
||||
// Words is the walk's word for a module it took: "reused from <check build>", with each digest (§4).
|
||||
func (r Reused) Words() string {
|
||||
var digests []string
|
||||
for _, m := range r.Made {
|
||||
d := m.Digest
|
||||
if d == "" {
|
||||
_, d, _ = strings.Cut(m.Reference, "@")
|
||||
}
|
||||
digests = append(digests, m.Name+" "+d)
|
||||
}
|
||||
return "reused from " + r.From.Said() + ": " + strings.Join(digests, ", ")
|
||||
}
|
||||
|
||||
// NamedBuild is a build a check's verdict names (ADR 0350 §2): its module, key and each artifact's digest.
|
||||
type NamedBuild struct {
|
||||
Pull PullRef
|
||||
Check string
|
||||
Module string
|
||||
Key string
|
||||
Digests map[string]string
|
||||
}
|
||||
|
||||
// DigestOf is the digest a reference names: an image's after `@`, an archive's after `/blobs/`.
|
||||
func DigestOf(reference string) string {
|
||||
if _, d, ok := strings.Cut(reference, "/blobs/"); ok {
|
||||
return d
|
||||
}
|
||||
if _, d, ok := strings.Cut(reference, "@"); ok {
|
||||
return d
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// unnamed is why a record found under a key is not taken: no verdict of the walk's pull requests names that module,
|
||||
// key and every one of its digests. Empty when one does.
|
||||
func unnamed(named []NamedBuild, module, key string, found CheckBuild) string {
|
||||
for _, n := range named {
|
||||
if n.Module != module || n.Key != key || n.Check != found.Check || len(n.Digests) != len(found.Made) {
|
||||
continue
|
||||
}
|
||||
same := true
|
||||
for _, m := range found.Made {
|
||||
d := m.Digest
|
||||
if d == "" {
|
||||
d = DigestOf(m.Reference)
|
||||
}
|
||||
same = same && d != "" && n.Digests[m.Name] == d
|
||||
}
|
||||
if same {
|
||||
return ""
|
||||
}
|
||||
}
|
||||
// A build of another pull request, or of a head the forge did not merge, is in no verdict the walk is handed
|
||||
// (novox/hq issue 703): said with the check that made it, whose head names which head it was.
|
||||
return "a build is kept under this key, by " + found.Said() + ", and no check verdict of the walk's pull requests " +
|
||||
"at the head the forge merged names it with its digests: not taken"
|
||||
}
|
||||
@@ -1,336 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func aCheckBuilt(t *testing.T, scratch *aScratch, tree string) CheckBuilt {
|
||||
t.Helper()
|
||||
r, run, workspace := aModuleAt(t, tree)
|
||||
return aCheckBuiltBy(t, scratch, r, run, workspace)
|
||||
}
|
||||
|
||||
func aCheckBuiltBy(t *testing.T, scratch *aScratch, r *recorded, run Runner, workspace string) CheckBuilt {
|
||||
t.Helper()
|
||||
spec := CheckSpec{ID: "build-1", Owner: aPull.Owner, Repo: aPull.Repo, Number: aPull.Number,
|
||||
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
|
||||
Ref: "main"}}}
|
||||
built := buildForTheMerge(t.Context(), run, spec, CheckBuilding{Publish: r, Scratch: scratch}, workspace, "", "",
|
||||
CheckVerdict{Head: "abcdefabcdef", Trunk: "123456123456"}, GitCredential{}, nil, logging(nil))
|
||||
if len(built) != 1 || built[0].NotBuilt != "" || built[0].Key == "" {
|
||||
t.Fatalf("the check did not build the module for its merge: %+v", built)
|
||||
}
|
||||
if !dockerBuilt(r) {
|
||||
t.Fatal("the check built no image")
|
||||
}
|
||||
for repository := range r.images {
|
||||
if !strings.HasPrefix(repository, ScratchPrefix) {
|
||||
t.Fatalf("the check published %s outside the scratch namespace", repository)
|
||||
}
|
||||
}
|
||||
for repository := range r.archives {
|
||||
if !strings.HasPrefix(repository, ScratchPrefix) {
|
||||
t.Fatalf("the check published %s outside the scratch namespace", repository)
|
||||
}
|
||||
}
|
||||
return built[0]
|
||||
}
|
||||
|
||||
func TestAWalkWhoseKeyIsItsChecksBuildsNothingAndRegistersTheChecksDigest(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
checked := aCheckBuilt(t, scratch, "tree-one")
|
||||
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dockerBuilt(r) || len(r.images) > 0 || len(r.archives) > 0 {
|
||||
t.Fatalf("the walk built again: ran %v, published %v %v", r.ran, r.images, r.archives)
|
||||
}
|
||||
if got.Reused == nil || got.Why != "" {
|
||||
t.Fatalf("the walk did not say it reused the check's build: reused %+v, why %q", got.Reused, got.Why)
|
||||
}
|
||||
if got.ReuseKey != checked.Key {
|
||||
t.Fatalf("the walk's key %s is not the check's %s", got.ReuseKey, checked.Key)
|
||||
}
|
||||
if len(scratch.took) != 2 {
|
||||
t.Fatalf("the walk copied %v, not the check's two artifacts", scratch.took)
|
||||
}
|
||||
for _, made := range checked.Made {
|
||||
registered := strings.Replace(made.Reference, ScratchPrefix, "", 1)
|
||||
found := false
|
||||
for _, b := range got.Built {
|
||||
found = found || (b.Reference == registered && b.Digest == made.Digest)
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the walk registers %+v, not the check's %s copied", got.Built, made.Reference)
|
||||
}
|
||||
}
|
||||
if image, _ := got.Manifest.Resources[0]["image"].(string); image != "registry.invalid/meshboard/server@sha256:"+strings.Repeat("a", 64) {
|
||||
t.Fatalf("the manifest pins %q, not the check's digest in the module's namespace", image)
|
||||
}
|
||||
words := got.Reused.Words()
|
||||
if !strings.HasPrefix(words, "reused from the check of novox/meshboard#7 at head abcdefabcdef (merged into trunk 123456123456)") ||
|
||||
!strings.Contains(words, "sha256:") {
|
||||
t.Fatalf("the walk says %q", words)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARecordNoVerdictNamesIsRefusedAndBuilt(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
checked := aCheckBuilt(t, scratch, "tree-one")
|
||||
for name, leave := range map[string][]NamedBuild{
|
||||
"no verdict": nil,
|
||||
"another digest": func() []NamedBuild {
|
||||
n := named(checked)
|
||||
n[0].Digests["server"] = "sha256:" + strings.Repeat("e", 64)
|
||||
return n
|
||||
}(),
|
||||
"another check": func() []NamedBuild { n := named(checked); n[0].Check = "build-9"; return n }(),
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
scratch.took = nil
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: leave},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Reused != nil || len(scratch.took) > 0 || !dockerBuilt(r) {
|
||||
t.Fatalf("a record no verdict names was taken: %+v, copied %v", got.Reused, scratch.took)
|
||||
}
|
||||
if !strings.Contains(got.Why, "no check verdict of the walk's pull requests at the head the forge merged names it") ||
|
||||
!strings.Contains(got.Why, "at head abcdefabcdef") {
|
||||
t.Fatalf("the walk says %q", got.Why)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWalkWhoseKeyDiffersBuildsAndSaysWhy(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
checked := aCheckBuilt(t, scratch, "tree-one")
|
||||
|
||||
r, run, workspace := aModuleAt(t, "tree-two")
|
||||
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !dockerBuilt(r) || got.Reused != nil {
|
||||
t.Fatalf("a walk whose source differs did not build: reused %+v", got.Reused)
|
||||
}
|
||||
if !strings.Contains(got.Why, "the key differs") || !strings.Contains(got.Why, partTree) {
|
||||
t.Fatalf("the walk says %q, not which part of the key differs", got.Why)
|
||||
}
|
||||
|
||||
// And a builder of another version builds too, saying so.
|
||||
withVersion(t, "builder-2")
|
||||
r, run, workspace = aModuleAt(t, "tree-one")
|
||||
got, err = BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !dockerBuilt(r) || !strings.Contains(got.Why, partBuilder) {
|
||||
t.Fatalf("a newer builder took the older one's build, or did not say why not: %q", got.Why)
|
||||
}
|
||||
|
||||
// No check of the walk's pull requests built it: said.
|
||||
r, run, workspace = aModuleAt(t, "tree-one")
|
||||
got, _ = BuildReusing(t.Context(), run, r, Reuse{Scratch: newScratch(), Pulls: []PullRef{aPull}},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if !dockerBuilt(r) || !strings.Contains(got.Why, "no check of novox/meshboard#7 built it") {
|
||||
t.Fatalf("the walk says %q", got.Why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailedCopyOfAChecksBuildFallsBackToBuilding(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
checked := aCheckBuilt(t, scratch, "tree-one")
|
||||
scratch.failTake = true
|
||||
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("a failed copy failed the walk: %v", err)
|
||||
}
|
||||
if !dockerBuilt(r) || got.Reused != nil || len(r.images) == 0 {
|
||||
t.Fatal("a failed copy did not fall back to a build")
|
||||
}
|
||||
if !strings.Contains(got.Why, "copying meshboard's") || !strings.Contains(got.Why, "refused the mount") {
|
||||
t.Fatalf("the walk says %q", got.Why)
|
||||
}
|
||||
for repository := range r.images {
|
||||
if strings.HasPrefix(repository, ScratchPrefix) {
|
||||
t.Fatalf("the walk's own build went into the scratch namespace: %s", repository)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyAWalkOfATrunkCommitTakesAChecksBuild(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
checked := aCheckBuilt(t, scratch, "tree-one")
|
||||
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
if _, err := Build(t.Context(), run, r, "https://forge.invalid/meshboard.git", "mod", "main", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !dockerBuilt(r) || len(scratch.took) > 0 {
|
||||
t.Fatal("a build not asked to reuse took a check's build")
|
||||
}
|
||||
|
||||
r, _, workspace = aModuleAt(t, "tree-one") // no trunk said: off it
|
||||
got, err := BuildReusing(t.Context(), r.run, r, Reuse{Scratch: scratch, Named: named(checked)}, "https://forge.invalid/meshboard.git",
|
||||
"mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !dockerBuilt(r) || got.Reused != nil || !strings.Contains(got.Why, "not on its trunk") {
|
||||
t.Fatalf("a commit off the trunk took a check's build: %q", got.Why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckThatFailedBuildsNothingForItsMerge(t *testing.T) {
|
||||
if skipsTheBuilds(CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "fail"}}) == "" {
|
||||
t.Fatal("a check whose repository layer failed builds for its merge")
|
||||
}
|
||||
if skipsTheBuilds(CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "warning"}}) != "" {
|
||||
t.Fatal("a check that passed with a warning builds nothing for its merge")
|
||||
}
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
if built := buildForTheMerge(t.Context(), run, CheckSpec{ID: "build-1"}, CheckBuilding{Publish: r, Scratch: newScratch()},
|
||||
workspace, "", "", CheckVerdict{}, GitCredential{}, nil, logging(nil)); built != nil || len(r.ran) > 0 {
|
||||
t.Fatalf("a check moving no module built %v, ran %v", built, r.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckBuildsNoModuleWithoutAKeyNorWhatStandsAfterIt(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
r, run, workspace := aModuleAt(t, "tree-one")
|
||||
r.contents["mod/"+ManifestName] = `{"module":"meshboard","version":"1","build":{"artifacts":[
|
||||
{"name":"mirror","kind":"upstream","from":"docker.io/library/alpine:3"}]}}`
|
||||
spec := CheckSpec{ID: "build-1", Owner: "novox", Repo: "meshboard", Number: 7, Builds: []CheckBuildSpec{
|
||||
{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod", Tier: 0},
|
||||
{Module: "after", Repository: "https://forge.invalid/after.git", Tier: 1}}}
|
||||
built := buildForTheMerge(t.Context(), run, spec, CheckBuilding{Publish: r, Scratch: newScratch()}, workspace, "", "",
|
||||
CheckVerdict{}, GitCredential{}, nil, logging(nil))
|
||||
if len(built) != 2 || !strings.Contains(built[0].NotBuilt, "named by tag") ||
|
||||
!strings.Contains(built[1].NotBuilt, "stands after meshboard") {
|
||||
t.Fatalf("built %+v", built)
|
||||
}
|
||||
if len(r.images) > 0 {
|
||||
t.Fatalf("a module without a key was published: %v", r.images)
|
||||
}
|
||||
}
|
||||
|
||||
func aCheckAgain(t *testing.T, scratch *aScratch, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
|
||||
t.Helper()
|
||||
return aCheckOf(t, scratch, aPull, "abcdefabcdef", tree, leave)
|
||||
}
|
||||
|
||||
func aCheckOf(t *testing.T, scratch *aScratch, pull PullRef, head, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
|
||||
t.Helper()
|
||||
r, run, workspace := aModuleAt(t, tree)
|
||||
spec := CheckSpec{ID: "build-2", Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number,
|
||||
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
|
||||
Ref: "main"}}}
|
||||
built := buildForTheMerge(t.Context(), run, spec, CheckBuilding{Publish: r, Scratch: scratch, Named: leave},
|
||||
workspace, "", "", CheckVerdict{Head: head, Trunk: "654321654321"}, GitCredential{}, nil, logging(nil))
|
||||
if len(built) != 1 || built[0].NotBuilt != "" || built[0].Key == "" {
|
||||
t.Fatalf("the second check did not answer the module for its merge: %+v", built)
|
||||
}
|
||||
return built[0], r
|
||||
}
|
||||
|
||||
func TestASecondCheckWhoseKeyIsUnchangedBuildsNothingAndNamesTheEarlierBuild(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
first := aCheckBuilt(t, scratch, "tree-one")
|
||||
|
||||
again, r := aCheckAgain(t, scratch, "tree-one", named(first))
|
||||
if dockerBuilt(r) || len(r.images) > 0 || len(r.archives) > 0 {
|
||||
t.Fatalf("the second check built again: ran %v, published %v %v", r.ran, r.images, r.archives)
|
||||
}
|
||||
if again.Key != first.Key || again.Kept != "build-1" || !reflect.DeepEqual(again.Made, first.Made) {
|
||||
t.Fatalf("the second check names %+v, not the first check's build %+v", again, first)
|
||||
}
|
||||
// The walk takes it by what the second verdict names.
|
||||
leave := named(again)
|
||||
leave[0].Check = again.Kept
|
||||
w, run, workspace := aModuleAt(t, "tree-one")
|
||||
got, err := BuildReusing(t.Context(), run, w, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: leave},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil || got.Reused == nil || dockerBuilt(w) {
|
||||
t.Fatalf("the walk did not take the kept build: %v, why %q", err, got.Why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASecondCheckWhoseKeyChangedBuilds(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
first := aCheckBuilt(t, scratch, "tree-one")
|
||||
|
||||
again, r := aCheckAgain(t, scratch, "tree-two", named(first))
|
||||
if !dockerBuilt(r) || again.Kept != "" || again.Key == first.Key {
|
||||
t.Fatalf("a check whose key changed did not build: %+v", again)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASecondCheckKeepsNoRecordAVerdictDoesNotName(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
scratch := newScratch()
|
||||
first := aCheckBuilt(t, scratch, "tree-one")
|
||||
for name, leave := range map[string][]NamedBuild{
|
||||
"no verdict": nil,
|
||||
"another digest": func() []NamedBuild {
|
||||
n := named(first)
|
||||
n[0].Digests["server"] = "sha256:" + strings.Repeat("e", 64)
|
||||
return n
|
||||
}(),
|
||||
"another key": func() []NamedBuild { n := named(first); n[0].Key = "r1:" + strings.Repeat("f", 64); return n }(),
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
again, r := aCheckAgain(t, scratch, "tree-one", leave)
|
||||
if !dockerBuilt(r) || again.Kept != "" {
|
||||
t.Fatalf("a record no verdict names was kept: %+v", again)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestACheckKeepsNoBuildOfAnotherHeadOrPullRequest(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
|
||||
for name, c := range map[string]struct {
|
||||
pull PullRef
|
||||
head string
|
||||
}{
|
||||
"another head": {aPull, "999999999999"},
|
||||
"another pull request": {other, "abcdefabcdef"},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
scratch := newScratch()
|
||||
first := aCheckBuilt(t, scratch, "tree-one")
|
||||
leave := named(first)
|
||||
leave[0].Pull = c.pull // as if its own verdict named it
|
||||
again, r := aCheckOf(t, scratch, c.pull, c.head, "tree-one", leave)
|
||||
if !dockerBuilt(r) || again.Kept != "" {
|
||||
t.Fatalf("a build of %s was kept: %+v", name, again)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,478 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// One build per change (novox/hq ADR 0350): the merge check builds what the merge would move into the scratch
|
||||
// namespace under its reuse key, and the walk of the trunk commit takes that build by digest instead of building.
|
||||
|
||||
// aScratch is the scratch namespace in memory: check builds by module and key, and what the walk took.
|
||||
type aScratch struct {
|
||||
builds map[string]CheckBuild
|
||||
took []string
|
||||
failTake bool
|
||||
}
|
||||
|
||||
func newScratch() *aScratch { return &aScratch{builds: map[string]CheckBuild{}} }
|
||||
|
||||
func (s *aScratch) Keep(_ context.Context, _ PullRef, _ string, _ time.Time, builds []CheckBuild) error {
|
||||
for _, b := range builds {
|
||||
s.builds[b.Module+" "+b.Key] = b
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *aScratch) Find(_ context.Context, module, key string) (CheckBuild, bool, error) {
|
||||
b, ok := s.builds[module+" "+key]
|
||||
return b, ok, nil
|
||||
}
|
||||
|
||||
func (s *aScratch) OfPulls(_ context.Context, pulls []PullRef) ([]CheckBuild, error) {
|
||||
var out []CheckBuild
|
||||
for _, b := range s.builds {
|
||||
if slices.Contains(pulls, b.Pull) {
|
||||
out = append(out, b)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (s *aScratch) Take(_ context.Context, module string, made catalogue.Built) (catalogue.Built, error) {
|
||||
if s.failTake {
|
||||
return catalogue.Built{}, errors.New("the store refused the mount")
|
||||
}
|
||||
s.took = append(s.took, made.Reference)
|
||||
made.Reference = strings.Replace(made.Reference, ScratchPrefix, "", 1)
|
||||
return made, nil
|
||||
}
|
||||
|
||||
func (s *aScratch) Forget(context.Context, PullRef) (int, error) { return 0, nil }
|
||||
func (s *aScratch) Expire(context.Context, time.Duration, time.Time) (int, error) { return 0, nil }
|
||||
func (s *aScratch) Discard(context.Context, []CheckBuild) error { return nil }
|
||||
|
||||
const reusedModule = `{"module":"meshboard","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile"},
|
||||
{"name":"look","kind":"archive","from":"files"}]},
|
||||
"resources":[
|
||||
{"id":"svc","type":"container","name":"meshboard","artifact":"server"},
|
||||
{"id":"theme","type":"archive","path":"/opt/meshboard","artifact":"look"}]}`
|
||||
|
||||
// aModuleAt is a repository holding the module under mod/, whose directory git names as tree, on its trunk.
|
||||
func aModuleAt(t *testing.T, tree string) (*recorded, Runner, string) {
|
||||
t.Helper()
|
||||
return aModuleWith(t, tree, map[string]string{"mod/" + ManifestName: reusedModule,
|
||||
"mod/Dockerfile": "FROM scratch", "mod/files/theme.conf": "dark"})
|
||||
}
|
||||
|
||||
// aModuleWith is aModuleAt with the repository's files given.
|
||||
func aModuleWith(t *testing.T, tree string, contents map[string]string) (*recorded, Runner, string) {
|
||||
t.Helper()
|
||||
r := &recorded{tree: tree, contents: contents}
|
||||
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
if name == "git" && len(args) > 0 && args[0] == "symbolic-ref" {
|
||||
r.ran = append(r.ran, name+" "+strings.Join(args, " "))
|
||||
return "origin/main\n", nil
|
||||
}
|
||||
return r.run(ctx, dir, name, args...)
|
||||
}
|
||||
return r, run, t.TempDir()
|
||||
}
|
||||
|
||||
func dockerBuilt(r *recorded) bool {
|
||||
for _, line := range r.ran {
|
||||
if strings.HasPrefix(line, "docker build") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var aPull = PullRef{Owner: "novox", Repo: "meshboard", Number: 7}
|
||||
|
||||
// aCheckBuilt is the check of aPull building the module from a tree, into the scratch namespace.
|
||||
|
||||
// aCheckBuiltBy is aCheckBuilt of the repository given.
|
||||
|
||||
// named is the walk's leave naming what a check's verdict says it built, as the controller reads it.
|
||||
func named(c CheckBuilt) []NamedBuild {
|
||||
n := NamedBuild{Pull: aPull, Check: "build-1", Module: c.Module, Key: c.Key, Digests: map[string]string{}}
|
||||
for _, m := range c.Made {
|
||||
d := m.Digest
|
||||
if d == "" {
|
||||
d = DigestOf(m.Reference)
|
||||
}
|
||||
n.Digests[m.Name] = d
|
||||
}
|
||||
return []NamedBuild{n}
|
||||
}
|
||||
|
||||
func withVersion(t *testing.T, v string) {
|
||||
t.Helper()
|
||||
was := Version
|
||||
Version = v
|
||||
t.Cleanup(func() { Version = was })
|
||||
}
|
||||
|
||||
// A merge whose reuse key equals its check's builds nothing again: the walk copies the check's build by digest
|
||||
// and registers that digest.
|
||||
|
||||
// A record no verdict names is refused and built instead, and the walk says so: the store takes writes from anything
|
||||
// on its network, so a record planted under another merge's key is never registered.
|
||||
|
||||
// A differing key builds, and says which part differs.
|
||||
|
||||
// A copy that fails is said and falls back to a build; it never fails the walk.
|
||||
|
||||
// Only a walk takes a check's build: a plain build — a check's, a dry run's, `build --ref`'s — builds, and a commit
|
||||
// off the trunk never takes one.
|
||||
|
||||
// A check that did not pass builds nothing for its merge, and one asked to build nothing builds nothing.
|
||||
|
||||
// A module whose source does not pin its build has no key and is not built by the check; what stands after it
|
||||
// is not built either.
|
||||
|
||||
// What the key names: every input a fresh build could differ by.
|
||||
func TestTheReuseKeyNamesEveryInputOfTheBuild(t *testing.T) {
|
||||
src := newSourceInputs("m")
|
||||
src.tree = "t1"
|
||||
src.bases = []string{"store:5000/b/x@sha256:" + strings.Repeat("b", 64)}
|
||||
src.contexts["server"] = "ctx1"
|
||||
src.toolchains["tools"] = "go sha256:c 1234"
|
||||
manifest := catalogue.Manifest{Module: "m"}
|
||||
base := keyOf(src, manifest, "v1")
|
||||
if base.Key == "" {
|
||||
t.Fatalf("no key: %s", base.Why)
|
||||
}
|
||||
for name, change := range map[string]func(*sourceInputs) string{
|
||||
partTree: func(s *sourceInputs) string { s.tree = "t2"; return "v1" },
|
||||
partBases: func(s *sourceInputs) string {
|
||||
s.bases = []string{"other@sha256:" + strings.Repeat("c", 64)}
|
||||
return "v1"
|
||||
},
|
||||
partContext: func(s *sourceInputs) string { s.contexts["server"] = "ctx2"; return "v1" },
|
||||
partTool: func(s *sourceInputs) string { s.toolchains["tools"] = "go sha256:d 1234"; return "v1" },
|
||||
partBuilder: func(*sourceInputs) string { return "v2" },
|
||||
} {
|
||||
s := src.copy()
|
||||
version := change(s)
|
||||
if k := keyOf(s, manifest, version); k.Key == base.Key {
|
||||
t.Errorf("a change of %s left the key the same", name)
|
||||
}
|
||||
}
|
||||
// A base copied to another address is the same base: the key names it by digest.
|
||||
moved := src.copy()
|
||||
moved.bases = []string{"elsewhere:5000/scratch/b/x@sha256:" + strings.Repeat("b", 64)}
|
||||
if keyOf(moved, manifest, "v1").Key != base.Key {
|
||||
t.Error("the same base at another address changed the key")
|
||||
}
|
||||
if k := keyOf(src, manifest, ""); k.Key != "" {
|
||||
t.Error("a builder whose version cannot be said keyed its build")
|
||||
}
|
||||
unpinned := src.copy()
|
||||
unpinned.notPinned("it resolves packages from the mesh's registry at build time")
|
||||
if k := keyOf(unpinned, manifest, "v1"); k.Key != "" || !strings.Contains(k.Why, "registry") {
|
||||
t.Errorf("a build its source does not pin was keyed: %+v", k)
|
||||
}
|
||||
}
|
||||
|
||||
// aTaggingRegistry is the in-memory registry with tags and their listing, as the store's scratch namespace needs.
|
||||
type aTaggingRegistry struct {
|
||||
*aRegistryOfRepositories
|
||||
tags map[string]map[string]string // repository → tag → manifest digest
|
||||
}
|
||||
|
||||
func (m *aTaggingRegistry) serve() http.Handler {
|
||||
under := m.handler()
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if repository, ok := strings.CutSuffix(strings.TrimPrefix(r.URL.Path, "/v2/"), "/tags/list"); ok {
|
||||
m.mu.Lock()
|
||||
var tags []string
|
||||
for tag := range m.tags[repository] {
|
||||
tags = append(tags, tag)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if tags == nil {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"name": repository, "tags": tags})
|
||||
return
|
||||
}
|
||||
repository, kind, rest := splitPath(r.URL.Path)
|
||||
if kind == "blobs" && r.Method == http.MethodHead {
|
||||
m.mu.Lock()
|
||||
body, linked := m.blobs[rest], m.links[repository][rest]
|
||||
m.mu.Unlock()
|
||||
if !linked {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Length", strconv.Itoa(len(body)))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
if kind != "manifests" {
|
||||
under.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if !strings.HasPrefix(rest, "sha256:") {
|
||||
m.mu.Lock()
|
||||
switch r.Method {
|
||||
case http.MethodPut:
|
||||
body, _ := readAll(r)
|
||||
if m.tags[repository] == nil {
|
||||
m.tags[repository] = map[string]string{}
|
||||
}
|
||||
m.tags[repository][rest] = digestOf(body)
|
||||
m.manifests[repository+"@"+digestOf(body)] = body
|
||||
m.mu.Unlock()
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
return
|
||||
default:
|
||||
digest, ok := m.tags[repository][rest]
|
||||
body, held := m.manifests[repository+"@"+digest]
|
||||
m.mu.Unlock()
|
||||
if !ok || !held {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Docker-Content-Digest", digest)
|
||||
w.Header().Set("Content-Type", mediaTypeOf(body))
|
||||
if r.Method == http.MethodGet {
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
}
|
||||
if r.Method == http.MethodDelete {
|
||||
m.mu.Lock()
|
||||
for _, tags := range m.tags {
|
||||
for tag, digest := range tags {
|
||||
if digest == rest {
|
||||
delete(tags, tag)
|
||||
}
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
under.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// A check's builds go after seven days, and when its pull request closes or is merged; the walk's copy in the
|
||||
// module's own namespace stays.
|
||||
func TestCheckBuildsExpireAfterSevenDaysAndGoWhenTheirPullRequestCloses(t *testing.T) {
|
||||
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
|
||||
server := httptest.NewServer(m.serve())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
registry := Registry{Address: address}
|
||||
ctx := t.Context()
|
||||
now := time.Date(2026, 10, 13, 20, 0, 0, 0, time.UTC)
|
||||
|
||||
image := func(repository, body string) string {
|
||||
digest := digestOf([]byte(body))
|
||||
m.manifests[repository+"@"+digest] = []byte(body)
|
||||
return address + "/" + repository + "@" + digest
|
||||
}
|
||||
check := func(pull PullRef, id string, at time.Time, key string) string {
|
||||
reference := image("scratch/meshboard/server", `{"mediaType":"`+mediaManifestOCI+`","layers":[],"n":"`+id+`"}`)
|
||||
build := CheckBuild{Key: key, Module: "meshboard", Made: []catalogue.Built{{Name: "server", Kind: "image",
|
||||
Reference: reference}}, Pull: pull, Check: id, At: at}
|
||||
if err := registry.Keep(ctx, pull, id, at, []CheckBuild{build}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return reference
|
||||
}
|
||||
held := func(reference string) bool {
|
||||
_, digest, _ := strings.Cut(reference, "@")
|
||||
_, ok := m.manifests["scratch/meshboard/server@"+digest]
|
||||
return ok
|
||||
}
|
||||
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
|
||||
old := check(aPull, "build-1", now.Add(-8*24*time.Hour), "r1:"+strings.Repeat("1", 64))
|
||||
fresh := check(aPull, "build-2", now.Add(-time.Hour), "r1:"+strings.Repeat("2", 64))
|
||||
others := check(other, "build-3", now.Add(-time.Hour), "r1:"+strings.Repeat("3", 64))
|
||||
walked := image("meshboard/server", `{"mediaType":"`+mediaManifestOCI+`","layers":[],"n":"walk"}`)
|
||||
|
||||
if _, found, err := registry.Find(ctx, "meshboard", "r1:"+strings.Repeat("2", 64)); err != nil || !found {
|
||||
t.Fatalf("a check build kept is not found by its key: %v %v", found, err)
|
||||
}
|
||||
n, err := registry.Expire(ctx, CheckBuildsKept, now)
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("expired %d checks (%v), want the one eight days old", n, err)
|
||||
}
|
||||
if held(old) || !held(fresh) || !held(others) {
|
||||
t.Fatalf("after the week's sweep: old %v, fresh %v, another's %v", held(old), held(fresh), held(others))
|
||||
}
|
||||
if _, found, _ := registry.Find(ctx, "meshboard", "r1:"+strings.Repeat("1", 64)); found {
|
||||
t.Fatal("the expired check build's record is still found")
|
||||
}
|
||||
|
||||
n, err = registry.Forget(ctx, aPull)
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("forgot %d checks of %s (%v)", n, aPull, err)
|
||||
}
|
||||
if held(fresh) || !held(others) {
|
||||
t.Fatalf("after %s closed: its build held %v, another pull request's %v", aPull, held(fresh), held(others))
|
||||
}
|
||||
if _, ok := m.manifests["meshboard/server@"+strings.SplitN(walked, "@", 2)[1]]; !ok {
|
||||
t.Fatal("letting go of check builds took the walk's copy in the module's namespace")
|
||||
}
|
||||
if left, _ := registry.OfPulls(ctx, []PullRef{aPull}); len(left) != 0 {
|
||||
t.Fatalf("%s still has check builds: %+v", aPull, left)
|
||||
}
|
||||
}
|
||||
|
||||
// The walk copies a check build by digest: the digest registered is the digest the check built, and the copy is
|
||||
// in the module's own namespace.
|
||||
func TestAChecksBuildIsCopiedByDigestIntoTheModulesNamespace(t *testing.T) {
|
||||
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
|
||||
server := httptest.NewServer(m.serve())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
registry := Registry{Address: address}
|
||||
|
||||
layer := []byte("layer")
|
||||
m.blobs[digestOf(layer)] = layer
|
||||
m.link("scratch/meshboard/server", digestOf(layer))
|
||||
manifest := []byte(`{"mediaType":"` + mediaManifestOCI + `","layers":[{"digest":"` + digestOf(layer) + `","size":5}]}`)
|
||||
m.manifests["scratch/meshboard/server@"+digestOf(manifest)] = manifest
|
||||
archive := []byte("archive")
|
||||
m.blobs[digestOf(archive)] = archive
|
||||
m.link("scratch/meshboard/look", digestOf(archive))
|
||||
|
||||
got, err := registry.Take(t.Context(), "meshboard", catalogue.Built{Name: "server", Kind: "image",
|
||||
Reference: address + "/scratch/meshboard/server@" + digestOf(manifest)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Reference != address+"/meshboard/server@"+digestOf(manifest) {
|
||||
t.Fatalf("copied as %s", got.Reference)
|
||||
}
|
||||
if m.uploads != 0 || m.mounts != 1 {
|
||||
t.Fatalf("the image's layer was moved (%d uploads, %d mounts), not mounted", m.uploads, m.mounts)
|
||||
}
|
||||
got, err = registry.Take(t.Context(), "meshboard", catalogue.Built{Name: "look", Kind: "archive",
|
||||
Digest: digestOf(archive), Reference: "http://" + address + "/v2/scratch/meshboard/look/blobs/" + digestOf(archive)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Digest != digestOf(archive) || got.Reference != "http://"+address+"/v2/meshboard/look/blobs/"+digestOf(archive) ||
|
||||
!m.links["meshboard/look"][digestOf(archive)] {
|
||||
t.Fatalf("the archive was copied as %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Two checks that built one digest in one repository share it: letting go of one check's builds keeps it while the
|
||||
// other still names it, and the last to go takes it.
|
||||
func TestABuildTwoChecksShareGoesWithTheLastOfThem(t *testing.T) {
|
||||
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
|
||||
server := httptest.NewServer(m.serve())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
registry := Registry{Address: address}
|
||||
ctx := t.Context()
|
||||
now := time.Date(2026, 10, 13, 20, 0, 0, 0, time.UTC)
|
||||
|
||||
body := `{"mediaType":"` + mediaManifestOCI + `","layers":[]}`
|
||||
digest := digestOf([]byte(body))
|
||||
m.manifests["scratch/meshboard/server@"+digest] = []byte(body)
|
||||
reference := address + "/scratch/meshboard/server@" + digest
|
||||
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
|
||||
for i, pull := range []PullRef{aPull, other} {
|
||||
id := "build-" + strconv.Itoa(i+1)
|
||||
b := CheckBuild{Key: "r1:" + strings.Repeat(strconv.Itoa(i+1), 64), Module: "meshboard", Pull: pull, Check: id,
|
||||
At: now, Made: []catalogue.Built{{Name: "server", Kind: "image", Reference: reference}}}
|
||||
if err := registry.Keep(ctx, pull, id, now, []CheckBuild{b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
held := func() bool { _, ok := m.manifests["scratch/meshboard/server@"+digest]; return ok }
|
||||
|
||||
if n, err := registry.Forget(ctx, aPull); err != nil || n != 1 {
|
||||
t.Fatalf("forgot %d (%v)", n, err)
|
||||
}
|
||||
if !held() {
|
||||
t.Fatal("letting go of one check's builds took a build another check still names")
|
||||
}
|
||||
if n, err := registry.Forget(ctx, other); err != nil || n != 1 {
|
||||
t.Fatalf("forgot %d (%v)", n, err)
|
||||
}
|
||||
if held() {
|
||||
t.Fatal("the build stayed after the last check naming it went")
|
||||
}
|
||||
}
|
||||
|
||||
// aCheckAgain is a later check (build-2) of aPull building the module from a tree, with the builds the build seat's
|
||||
// verdicts name for it (novox/hq issue 703).
|
||||
|
||||
// aCheckOf is a later check (build-2) of a pull request at a head, the trunk moved since build-1.
|
||||
|
||||
// A check asked again when only the trunk moved, whose key is unchanged, builds nothing: it keeps the build an
|
||||
// earlier verdict names and names it by key and digest, so the walk can take it (novox/hq issue 703).
|
||||
|
||||
// A check whose key changed builds, as before.
|
||||
|
||||
// A scratch record no verdict names is never kept: the store takes writes from anything on its network, so a
|
||||
// check builds rather than name a planted record (novox/hq issue 703, ADR 0350 §2).
|
||||
|
||||
// A build a later check kept stays when the check that built it goes: its artifacts and its record, which the later
|
||||
// check's verdict names (novox/hq issue 703, ADR 0350 §5).
|
||||
func TestABuildALaterCheckKeptStaysWhenTheCheckThatBuiltItGoes(t *testing.T) {
|
||||
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
|
||||
server := httptest.NewServer(m.serve())
|
||||
defer server.Close()
|
||||
address := strings.TrimPrefix(server.URL, "http://")
|
||||
registry := Registry{Address: address}
|
||||
ctx := t.Context()
|
||||
now := time.Date(2026, 10, 14, 9, 0, 0, 0, time.UTC)
|
||||
|
||||
body := `{"mediaType":"` + mediaManifestOCI + `","layers":[]}`
|
||||
digest := digestOf([]byte(body))
|
||||
m.manifests["scratch/meshboard/server@"+digest] = []byte(body)
|
||||
key := "r1:" + strings.Repeat("1", 64)
|
||||
b := CheckBuild{Key: key, Module: "meshboard", Pull: aPull, Check: "build-1", At: now,
|
||||
Made: []catalogue.Built{{Name: "server", Kind: "image", Reference: address + "/scratch/meshboard/server@" + digest}}}
|
||||
if err := registry.Keep(ctx, aPull, "build-1", now, []CheckBuild{b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Another pull request's check keeps it: its own document names the build, which stays the first check's.
|
||||
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
|
||||
if err := registry.Keep(ctx, other, "build-2", now, []CheckBuild{b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
held := func() bool { _, ok := m.manifests["scratch/meshboard/server@"+digest]; return ok }
|
||||
found := func() bool { _, ok, _ := registry.Find(ctx, "meshboard", key); return ok }
|
||||
|
||||
if n, err := registry.Forget(ctx, aPull); err != nil || n != 1 {
|
||||
t.Fatalf("forgot %d (%v)", n, err)
|
||||
}
|
||||
if !held() || !found() {
|
||||
t.Fatalf("the build a later check names went with the check that built it: held %v, record %v", held(), found())
|
||||
}
|
||||
if n, err := registry.Forget(ctx, other); err != nil || n != 1 {
|
||||
t.Fatalf("forgot %d (%v)", n, err)
|
||||
}
|
||||
if held() || found() {
|
||||
t.Fatalf("the build stayed after the last check naming it went: held %v, record %v", held(), found())
|
||||
}
|
||||
}
|
||||
|
||||
// Only an earlier check of the same pull request at the same head is kept from: another head of it, or another pull
|
||||
// request whose verdict names the same key and digests, builds (novox/hq issue 703).
|
||||
@@ -1,183 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A module moves only when its program changes (novox/hq issue 721). A merge check builds every module the delivery
|
||||
// plan moves; a module whose every artifact comes out the same as the build the mesh holds of it — the same files,
|
||||
// the same modes and owners, the same entrypoint and environment — is said unchanged, and the walk leaves it where it
|
||||
// is. **The whole image is compared, never its main binary alone**: a module whose image differs in any file moves.
|
||||
// What is ignored is only what a rebuild cannot help changing: each file's time, the image's labels and its creation
|
||||
// time. **A comparison that cannot be made says nothing**, and the module is built and moved as before.
|
||||
|
||||
// sameAsHeld says whether every artifact made is the same as the build the mesh holds of the module: the sentence
|
||||
// that says so and, per artifact, the held reference it was compared with. Empty when any artifact differs, is not
|
||||
// an image, has no held build, or could not be compared.
|
||||
func sameAsHeld(ctx context.Context, run Runner, dir, module string, made []catalogue.Built,
|
||||
held map[string]string) (string, map[string]string, error) {
|
||||
if len(made) == 0 {
|
||||
return "", nil, nil
|
||||
}
|
||||
against := map[string]string{}
|
||||
for _, m := range made {
|
||||
if m.Kind != "image" || m.Reference == "" {
|
||||
return "", nil, nil
|
||||
}
|
||||
was := held[module+"/"+m.Name]
|
||||
if was == "" {
|
||||
return "", nil, nil
|
||||
}
|
||||
if digestPart(was) != "" && digestPart(was) == digestPart(m.Reference) {
|
||||
against[m.Name] = was
|
||||
continue
|
||||
}
|
||||
now, err := imageContent(ctx, run, dir, m.Reference)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("%s's new image: %w", m.Name, err)
|
||||
}
|
||||
before, err := imageContent(ctx, run, dir, was)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("%s's held image: %w", m.Name, err)
|
||||
}
|
||||
if now != before {
|
||||
return "", nil, nil
|
||||
}
|
||||
against[m.Name] = was
|
||||
}
|
||||
var names []string
|
||||
for name, ref := range against {
|
||||
names = append(names, name+" "+shortRef(ref))
|
||||
}
|
||||
sort.Strings(names)
|
||||
return "unchanged: its program is the same as running build " + strings.Join(names, ", "), against, nil
|
||||
}
|
||||
|
||||
// digestPart is a reference's "sha256:…", empty when it is not pinned.
|
||||
func digestPart(ref string) string {
|
||||
if i := strings.LastIndex(ref, "@"); i >= 0 {
|
||||
return ref[i+1:]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// shortRef names a reference by its digest's first twelve characters when it has one.
|
||||
func shortRef(ref string) string {
|
||||
if d := strings.TrimPrefix(digestPart(ref), "sha256:"); d != "" {
|
||||
return twelve(d)
|
||||
}
|
||||
return ref
|
||||
}
|
||||
|
||||
// imageContent is an image's content as one digest: its platform, its configuration without labels, and every file
|
||||
// of its filesystem by path, type, mode, owner, link, extended attributes and bytes — never by time.
|
||||
func imageContent(ctx context.Context, run Runner, dir, ref string) (string, error) {
|
||||
if _, err := run(ctx, dir, "docker", "pull", "--quiet", ref); err != nil {
|
||||
return "", fmt.Errorf("pulling %s: %w", ref, err)
|
||||
}
|
||||
// The image is on the node only for the reading (novox/hq issue 1834); one a container still runs is refused by
|
||||
// docker and stays.
|
||||
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rmi", ref) }()
|
||||
config, err := run(ctx, dir, "docker", "image", "inspect", "--format",
|
||||
"{{.Os}}/{{.Architecture}}/{{.Variant}} {{json .Config}}", ref)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading %s's configuration: %w", ref, err)
|
||||
}
|
||||
said, err := configContent(config)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading %s's configuration: %w", ref, err)
|
||||
}
|
||||
id, err := run(ctx, dir, "docker", "create", "--entrypoint", "/", ref)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("opening %s's filesystem: %w", ref, err)
|
||||
}
|
||||
id = strings.TrimSpace(id)
|
||||
if id == "" {
|
||||
return "", errors.New("the container engine named no container for " + ref)
|
||||
}
|
||||
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rm", "--force", "--volumes", id) }()
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
file := filepath.Join(dir, "content-"+twelve(id)+".tar")
|
||||
defer os.Remove(file)
|
||||
if _, err := run(ctx, dir, "docker", "export", "--output", file, id); err != nil {
|
||||
return "", fmt.Errorf("reading %s's filesystem: %w", ref, err)
|
||||
}
|
||||
f, err := os.Open(file)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
files, err := treeContent(f)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading %s's filesystem: %w", ref, err)
|
||||
}
|
||||
sum := sha256.Sum256([]byte(said + "\n" + files))
|
||||
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
|
||||
// configContent is an image's platform and configuration as compared: Labels, the parent's id and the hostname the
|
||||
// builder ran under are left out — the first holds the commit, the others no part of what runs.
|
||||
func configContent(inspected string) (string, error) {
|
||||
platform, raw, ok := strings.Cut(strings.TrimSpace(inspected), " ")
|
||||
if !ok {
|
||||
return "", fmt.Errorf("unexpected inspection %q", inspected)
|
||||
}
|
||||
var config map[string]json.RawMessage
|
||||
if err := json.Unmarshal([]byte(raw), &config); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, k := range []string{"Labels", "Image", "Hostname", "Domainname"} {
|
||||
delete(config, k)
|
||||
}
|
||||
out, err := json.Marshal(config) // map keys are sorted
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return platform + " " + string(out), nil
|
||||
}
|
||||
|
||||
// treeContent is a filesystem tarball's entries, sorted by path, each with everything but its times.
|
||||
func treeContent(r io.Reader) (string, error) {
|
||||
tr := tar.NewReader(r)
|
||||
var lines []string
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
sum := sha256.New()
|
||||
if _, err := io.Copy(sum, tr); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var xattrs []string
|
||||
for k, v := range h.PAXRecords {
|
||||
if strings.HasPrefix(k, "SCHILY.xattr.") {
|
||||
xattrs = append(xattrs, k+"="+hex.EncodeToString([]byte(v)))
|
||||
}
|
||||
}
|
||||
sort.Strings(xattrs)
|
||||
lines = append(lines, fmt.Sprintf("%q %c %o %d:%d %q %d,%d %d %x %s",
|
||||
strings.TrimPrefix(h.Name, "./"), h.Typeflag, h.Mode, h.Uid, h.Gid, h.Linkname, h.Devmajor, h.Devminor,
|
||||
h.Size, sum.Sum(nil), strings.Join(xattrs, ";")))
|
||||
}
|
||||
sort.Strings(lines)
|
||||
return strings.Join(lines, "\n"), nil
|
||||
}
|
||||
@@ -1,126 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// aFilesystem is a filesystem tarball of files, each written at the time given.
|
||||
func aFilesystem(t *testing.T, at time.Time, files map[string]string) []byte {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
tw := tar.NewWriter(&buf)
|
||||
for _, name := range []string{"usr/local/bin/mesh-builder", "etc/apk/world"} {
|
||||
body, ok := files[name]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), ModTime: at,
|
||||
Typeflag: tar.TypeReg}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tw.Write([]byte(body)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
// anEngine answers the container engine's commands for images by reference: each image's filesystem and labels.
|
||||
func anEngine(images map[string][]byte, labels map[string]string, failing string) Runner {
|
||||
created := map[string]string{}
|
||||
return func(_ context.Context, _ string, name string, args ...string) (string, error) {
|
||||
joined := strings.Join(args, " ")
|
||||
ref := args[len(args)-1]
|
||||
if failing != "" && strings.Contains(joined, failing) {
|
||||
return "", fmt.Errorf("the engine refused %s", joined)
|
||||
}
|
||||
switch {
|
||||
case args[0] == "pull", args[0] == "rm":
|
||||
return "", nil
|
||||
case args[0] == "image":
|
||||
return fmt.Sprintf(`linux/amd64/ {"Entrypoint":["/usr/local/bin/mesh-builder"],"Labels":{"commit":%q}}`,
|
||||
labels[ref]), nil
|
||||
case args[0] == "create":
|
||||
id := fmt.Sprintf("container%04d", len(created))
|
||||
created[id] = ref
|
||||
return id + "\n", nil
|
||||
case args[0] == "export":
|
||||
return "", os.WriteFile(args[2], images[created[ref]], 0o600)
|
||||
}
|
||||
return "", fmt.Errorf("unexpected %s %s", name, joined)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAModuleWhoseProgramDidNotChangeIsSaidUnchanged is novox/hq issue 721: mesh-controller #335 changed one sentence
|
||||
// in a package mesh-builder imports and never reaches, and build-agent moved. Built again, its image differs only in
|
||||
// the files' times and its commit label, and the check says it is the same as the build the mesh holds.
|
||||
func TestAModuleWhoseProgramDidNotChangeIsSaidUnchanged(t *testing.T) {
|
||||
held := map[string]string{"build-agent/agent": "store/build-agent@sha256:aaaa"}
|
||||
made := []catalogue.Built{{Name: "agent", Kind: "image", Reference: "scratch/build-agent@sha256:bbbb"}}
|
||||
files := map[string]string{"usr/local/bin/mesh-builder": "the same program", "etc/apk/world": "docker-cli\ngit\n"}
|
||||
images := map[string][]byte{
|
||||
"store/build-agent@sha256:aaaa": aFilesystem(t, time.Unix(1000, 0), files),
|
||||
"scratch/build-agent@sha256:bbbb": aFilesystem(t, time.Unix(2000, 0), files),
|
||||
}
|
||||
labels := map[string]string{"store/build-agent@sha256:aaaa": "a5500f73", "scratch/build-agent@sha256:bbbb": "c2ec0a60"}
|
||||
same, against, err := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent",
|
||||
made, held)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(same, "unchanged: its program is the same as running build agent") ||
|
||||
against["agent"] != held["build-agent/agent"] {
|
||||
t.Fatalf("an image the same but for its times and labels is not said unchanged: %q %v", same, against)
|
||||
}
|
||||
|
||||
// **Any file differing moves it**, not only its program.
|
||||
changed := map[string]string{"usr/local/bin/mesh-builder": "the same program", "etc/apk/world": "docker-cli\ngit\ncurl\n"}
|
||||
images["scratch/build-agent@sha256:bbbb"] = aFilesystem(t, time.Unix(2000, 0), changed)
|
||||
if same, _, err := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent", made,
|
||||
held); err != nil || same != "" {
|
||||
t.Fatalf("an image with another file is said unchanged: %q %v", same, err)
|
||||
}
|
||||
|
||||
// **A comparison that fails says nothing**, and the module moves.
|
||||
images["scratch/build-agent@sha256:bbbb"] = aFilesystem(t, time.Unix(2000, 0), files)
|
||||
if same, _, err := sameAsHeld(context.Background(), anEngine(images, labels, "export"), t.TempDir(), "build-agent",
|
||||
made, held); err == nil || same != "" {
|
||||
t.Fatalf("a comparison that failed said %q, %v", same, err)
|
||||
}
|
||||
|
||||
// No held build, or an artifact that is not an image: nothing compared.
|
||||
if same, _, _ := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent", made,
|
||||
nil); same != "" {
|
||||
t.Fatalf("a module the mesh holds no build of is said unchanged: %q", same)
|
||||
}
|
||||
archive := []catalogue.Built{{Name: "agent", Kind: "archive", Reference: "scratch/build-agent@sha256:bbbb"}}
|
||||
if same, _, _ := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent",
|
||||
archive, held); same != "" {
|
||||
t.Fatalf("an archive is said unchanged without being compared: %q", same)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnImageWithAnotherEntrypointIsNotTheSame: the configuration counts, but for its labels.
|
||||
func TestAnImageWithAnotherEntrypointIsNotTheSame(t *testing.T) {
|
||||
a, err := configContent(`linux/amd64/ {"Entrypoint":["/a"],"Labels":{"commit":"1"}}`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := configContent(`linux/amd64/ {"Labels":{"commit":"2"},"Entrypoint":["/a"]}`)
|
||||
c, _ := configContent(`linux/amd64/ {"Entrypoint":["/b"],"Labels":{"commit":"1"}}`)
|
||||
if a != b || a == c {
|
||||
t.Fatalf("labels count, or the entrypoint does not: %q %q %q", a, b, c)
|
||||
}
|
||||
}
|
||||
@@ -1,337 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867): a
|
||||
// mesh-host change and a mesh-controller change that need each other. The sibling's repository is read beside the
|
||||
// check at the sibling's head merged onto its base head, in place of the commit the mesh runs.
|
||||
|
||||
// aSiblingWorld is the mesh's two core repositories as the trunk holds them, and a pull request into each that the
|
||||
// other needs: the controller's adds a lock line the node-engine's own check reads, and builds only with the
|
||||
// node-engine's new validator word.
|
||||
type aSiblingWorld struct {
|
||||
controller, host string // repositories
|
||||
controllerHead, hostHead string // the pull requests' whole heads
|
||||
controllerMain, hostMain string // the trunks' heads
|
||||
}
|
||||
|
||||
func theSiblingWorld(t *testing.T) aSiblingWorld {
|
||||
t.Helper()
|
||||
var w aSiblingWorld
|
||||
// The controller's trunk: a judge saying who it is, its lock without the new line, and the check that judges it.
|
||||
controllerFiles := aJudgeOf("the trunk's controller")
|
||||
controllerFiles["lock.txt"] = "foundation: old\n"
|
||||
controllerFiles["module.json"] = `{"module":"mesh-controller"}`
|
||||
controllerFiles[CheckScript] = "true\n"
|
||||
var controllerRelease string
|
||||
w.controller, w.controllerMain, controllerRelease = aRepositoryWithARelease(t, controllerFiles, map[string]string{"r": "r"})
|
||||
_ = controllerRelease
|
||||
gitInRepo(t, w.controller, "checkout", "--quiet", "-b", "change", "main")
|
||||
change := aJudgeOf("the sibling controller")
|
||||
change["lock.txt"] = "foundation: new\n"
|
||||
w.controllerHead = commitFiles(t, w.controller, change)
|
||||
gitInRepo(t, w.controller, "checkout", "--quiet", "main")
|
||||
// The node-engine's trunk: a validator, and the check that reads the controller's lock beside it.
|
||||
hostFiles := aHostSaying("the trunk's validator")
|
||||
hostFiles["module.json"] = `{"module":"mesh-host"}`
|
||||
hostFiles[CheckScript] = "set -e\ngrep -q 'foundation: new' \"$MESH_CHECK_BESIDE/mesh-controller/lock.txt\"\n"
|
||||
w.host, w.hostMain, _ = aRepositoryWithARelease(t, hostFiles, map[string]string{"r": "r"})
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "-b", "change", "main")
|
||||
w.hostHead = commitFiles(t, w.host, aHostSaying("the sibling's validator"))
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "main")
|
||||
return w
|
||||
}
|
||||
|
||||
func (w aSiblingWorld) beside() map[string]Beside {
|
||||
return map[string]Beside{"mesh-controller": {Repository: w.controller, Ref: w.controllerMain},
|
||||
"mesh-host": {Repository: w.host, Ref: w.hostMain}}
|
||||
}
|
||||
|
||||
// hostCheck is the node-engine's pull request checked, with or without its sibling.
|
||||
func (w aSiblingWorld) hostCheck(t *testing.T, id, workspace, cache string, siblings ...Sibling) CheckVerdict {
|
||||
t.Helper()
|
||||
return w.hostCheckAs(t, "", id, workspace, cache, siblings...)
|
||||
}
|
||||
|
||||
// hostCheckAs is hostCheck for a request that names who judges: an older request says "validator" for a node-engine change.
|
||||
func (w aSiblingWorld) hostCheckAs(t *testing.T, judge, id, workspace, cache string, siblings ...Sibling) CheckVerdict {
|
||||
t.Helper()
|
||||
if workspace == "" {
|
||||
workspace = t.TempDir()
|
||||
}
|
||||
beside := w.beside()
|
||||
delete(beside, "mesh-host")
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: w.host, Ref: w.hostHead, Owner: "novox",
|
||||
Repo: "mesh-host", Number: 114, Base: "main", Judge: judge, JudgeCache: cache,
|
||||
Paths: []string{"validate/v.go"}, Modules: []string{"mesh-host"}, Beside: beside, Siblings: siblings,
|
||||
Toolchain: "golang"}, workspace, aFactsRegistry(t), GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// controllerCheck is the controller's pull request checked, with or without its sibling.
|
||||
func (w aSiblingWorld) controllerCheck(t *testing.T, id, cache string, siblings ...Sibling) CheckVerdict {
|
||||
t.Helper()
|
||||
beside := w.beside()
|
||||
delete(beside, "mesh-controller")
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: w.controller, Ref: w.controllerHead, Owner: "novox",
|
||||
Repo: "mesh-controller", Number: 346, Base: "main", Judge: "self", JudgeCache: cache,
|
||||
Paths: []string{"lock.txt"}, Modules: []string{"mesh-controller"}, Beside: beside, Siblings: siblings,
|
||||
Toolchain: "golang"}, t.TempDir(), aFactsRegistry(t), GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (w aSiblingWorld) controllerSibling() Sibling {
|
||||
return Sibling{Repository: "novox/mesh-controller", Number: 346, Head: w.controllerHead, Base: "main", Dir: "mesh-controller"}
|
||||
}
|
||||
|
||||
func (w aSiblingWorld) hostSibling() Sibling {
|
||||
return Sibling{Repository: "novox/mesh-host", Number: 114, Head: w.hostHead, Base: "main", Dir: "mesh-host"}
|
||||
}
|
||||
|
||||
// The node-engine's change fails its own check without its sibling (the controller's lock beside it is the trunk's),
|
||||
// and passes with it read at its merge: the check reads the lock the sibling's change brings, the judge is the
|
||||
// controller built from the sibling's merge with the validator that controller vendors (mesh-sdk's), and the verdict names both.
|
||||
func TestAHostMemberIsCheckedBesideItsControllerSibling(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
|
||||
alone := w.hostCheck(t, "check-867-host-alone", "", "")
|
||||
if alone.Repo == nil || alone.Repo.Verdict != "fail" || len(alone.Siblings) != 0 {
|
||||
t.Fatalf("with no sibling the node-engine change should fail its own check (the lock beside it is the trunk's): %+v",
|
||||
alone.Repo)
|
||||
}
|
||||
|
||||
v := w.hostCheck(t, "check-867-host", "", "", w.controllerSibling())
|
||||
if v.Repo == nil || v.Repo.Verdict != "pass" {
|
||||
t.Fatalf("beside its sibling the node-engine change should pass its own check: %+v\n%s", v.Repo, v.Report)
|
||||
}
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Gate.Summary, "the sibling controller with the vendored validator") {
|
||||
t.Fatalf("the judge was not the controller built from the sibling's merge with the validator it vendors: %+v\n%s",
|
||||
v.Gate, v.Report)
|
||||
}
|
||||
baseCommit := w.controllerMain
|
||||
if len(v.Siblings) != 1 || v.Siblings[0] != (JudgedSibling{Repository: "novox/mesh-controller", Number: 346,
|
||||
Head: w.controllerHead, Base: "main", BaseCommit: baseCommit}) {
|
||||
t.Fatalf("the verdict names siblings %+v, not mesh-controller#346 at %s onto %s", v.Siblings, w.controllerHead, baseCommit)
|
||||
}
|
||||
if v.Beside["mesh-controller"] == baseCommit[:12] || v.Beside["mesh-controller"] == w.controllerHead[:12] {
|
||||
t.Fatalf("the controller was read at %s, which is the trunk's or the head and not their merge", v.Beside["mesh-controller"])
|
||||
}
|
||||
if want := "sibling novox/mesh-controller#346 merge " + v.Beside["mesh-controller"]; v.JudgeFrom != want {
|
||||
t.Fatalf("the judge came from %q, not %q", v.JudgeFrom, want)
|
||||
}
|
||||
if want := "mesh-sdk's validator at the pin of sibling novox/mesh-controller#346 merge " + v.Beside["mesh-controller"] +
|
||||
" (a local mesh-sdk, v0.0.0, replaced)"; v.Validator != want {
|
||||
t.Fatalf("the validator was %q, not %q", v.Validator, want)
|
||||
}
|
||||
// The statuses' "beside siblings" part is the forge module's to write, from the verdict's siblings.
|
||||
if strings.Contains(v.Gate.Summary, "beside siblings") || strings.Contains(v.Repo.Summary, "beside siblings") {
|
||||
t.Fatalf("the check writes the siblings mark itself:\n%s\n%s", v.Gate.Summary, v.Repo.Summary)
|
||||
}
|
||||
// The base of the sibling is the trunk of the sibling's repository, as the trunk stood when the check cloned it.
|
||||
if v.Trunk != w.hostMain[:12] {
|
||||
t.Fatalf("the member's own trunk commit is %s, not %s", v.Trunk, w.hostMain[:12])
|
||||
}
|
||||
}
|
||||
|
||||
// The controller's change is judged by itself with the validator it vendors, whatever the node-engine sibling brings: the
|
||||
// node-engine holds no validator any more (issue 1391, novox/hq ADR 0446 clause 5). Nothing of the sibling is put into the judge.
|
||||
func TestAControllerMemberIsJudgedByItselfWithTheValidatorItVendors(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
|
||||
alone := w.controllerCheck(t, "check-867-controller-alone", "")
|
||||
v := w.controllerCheck(t, "check-867-controller", "", w.hostSibling())
|
||||
for what, got := range map[string]CheckVerdict{"alone": alone, "beside its sibling": v} {
|
||||
if got.Gate == nil || got.Gate.Verdict != "pass" || !strings.Contains(got.Gate.Summary, "the sibling controller with the vendored validator") {
|
||||
t.Fatalf("the controller change %s was not judged by itself with the validator it vendors: %+v\n%s", what, got.Gate, got.Report)
|
||||
}
|
||||
}
|
||||
if len(v.Siblings) != 1 || v.Siblings[0].Repository != "novox/mesh-host" || v.Siblings[0].BaseCommit != w.hostMain {
|
||||
t.Fatalf("the verdict names siblings %+v", v.Siblings)
|
||||
}
|
||||
if v.Validator != "" {
|
||||
t.Fatalf("the validator was %q: a controller judged by itself has the one it vendors", v.Validator)
|
||||
}
|
||||
}
|
||||
|
||||
// A node-engine member whose tree holds no validator (validate/ and internal/declaration/ deleted, as R4b deletes them) is judged
|
||||
// beside its controller sibling all the same: nothing of it is copied into the judge, so nothing is missing (issue 1391).
|
||||
func TestAHostMemberWithNoValidatorOfItsOwnDoesNotFailTheJudge(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "change")
|
||||
gitInRepo(t, w.host, "rm", "--quiet", "-r", "validate", "internal")
|
||||
gitInRepo(t, w.host, "commit", "--quiet", "-m", "the validator is mesh-sdk's")
|
||||
w.hostHead = gitInRepo(t, w.host, "rev-parse", "HEAD")
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "main")
|
||||
|
||||
v := w.hostCheckAs(t, "validator", "check-1391-host-without-validator", "", "", w.controllerSibling())
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Gate.Summary, "the sibling controller with the vendored validator") {
|
||||
t.Fatalf("a node-engine change with no validator of its own was not judged: %+v\n%s", v.Gate, v.Report)
|
||||
}
|
||||
alone := w.hostCheckAs(t, "validator", "check-1391-host-alone", "", "")
|
||||
if alone.Gate == nil || strings.Contains(alone.Gate.Summary, "could not be put in the judge") {
|
||||
t.Fatalf("the judge tried to copy the validator of a change that holds none: %+v", alone.Gate)
|
||||
}
|
||||
}
|
||||
|
||||
// A sibling head that does not merge cleanly onto its base fails the check, naming the sibling and the files.
|
||||
func TestASiblingThatDoesNotMergeCleanlyFailsTheCheckNamingIt(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
// The base moves under the sibling's head on the same line of the same file.
|
||||
gitInRepo(t, w.controller, "checkout", "--quiet", "main")
|
||||
commitFiles(t, w.controller, map[string]string{"lock.txt": "foundation: elsewhere\n"})
|
||||
v := w.hostCheck(t, "check-867-conflict", "", "", w.controllerSibling())
|
||||
if v.Verdict != "fail" || v.Gate == nil || v.Gate.Verdict != "fail" || v.Repo == nil || v.Repo.Verdict != "fail" {
|
||||
t.Fatalf("a sibling that does not merge was answered %+v / %+v", v.Gate, v.Repo)
|
||||
}
|
||||
for _, want := range []string{"novox/mesh-controller#346", "does not merge cleanly", "lock.txt", w.controllerHead[:12]} {
|
||||
if !strings.Contains(v.Summary, want) {
|
||||
t.Fatalf("the failure does not say %q: %s", want, v.Summary)
|
||||
}
|
||||
}
|
||||
if len(v.Siblings) != 1 || v.Siblings[0].Number != 346 {
|
||||
t.Fatalf("the verdict does not name the sibling: %+v", v.Siblings)
|
||||
}
|
||||
}
|
||||
|
||||
// A check that could not learn its siblings reads none and says so; one with none to read is judged as before.
|
||||
func TestACheckThatCouldNotLearnItsSiblingsSaysSo(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
beside := w.beside()
|
||||
delete(beside, "mesh-host")
|
||||
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-867-unknown", Repository: w.host, Ref: w.hostHead,
|
||||
Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Paths: []string{"validate/v.go"},
|
||||
Modules: []string{"mesh-host"}, Beside: beside, SiblingsUnknown: "mesh-delivery did not answer",
|
||||
Toolchain: "golang"}, t.TempDir(), aFactsRegistry(t), GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.SiblingsUnknown != "mesh-delivery did not answer" || len(v.Siblings) != 0 || v.JudgeFrom != "" {
|
||||
t.Fatalf("the verdict says %q, siblings %+v, judge from %q", v.SiblingsUnknown, v.Siblings, v.JudgeFrom)
|
||||
}
|
||||
if !strings.Contains(v.Gate.Summary, "the trunk's controller with the vendored validator") {
|
||||
t.Fatalf("a check without siblings was not judged as before: %+v", v.Gate)
|
||||
}
|
||||
if strings.Contains(v.Gate.Summary, "[beside siblings") {
|
||||
t.Fatalf("the statuses name siblings that were not read: %s", v.Gate.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
// The judge a group made is kept only for the same inputs: the controller's merge and the base commits it was merged
|
||||
// onto. The same inputs take the judge kept; any other builds it again.
|
||||
func TestTheGroupsJudgeIsReusedOnlyForTheSameInputs(t *testing.T) {
|
||||
built := aRuntimeThatBuildsTheJudge(t)
|
||||
w := theSiblingWorld(t)
|
||||
cache := t.TempDir()
|
||||
w.hostCheck(t, "check-867-cache-1", "", cache, w.controllerSibling())
|
||||
if n := builds(t, built); n != 1 {
|
||||
t.Fatalf("the first check built %d judge(s)", n)
|
||||
}
|
||||
v := w.hostCheck(t, "check-867-cache-2", "", cache, w.controllerSibling())
|
||||
if n := builds(t, built); n != 1 || v.Gate.Verdict != "pass" {
|
||||
t.Fatalf("the same inputs built the judge again (%d builds) or did not pass: %+v", n, v.Gate)
|
||||
}
|
||||
// The controller sibling's head moves: another merge, another judge.
|
||||
gitInRepo(t, w.controller, "checkout", "--quiet", "change")
|
||||
w.controllerHead = commitFiles(t, w.controller, map[string]string{"extra.txt": "x"})
|
||||
gitInRepo(t, w.controller, "checkout", "--quiet", "main")
|
||||
w.hostCheck(t, "check-867-cache-3", "", cache, w.controllerSibling())
|
||||
if n := builds(t, built); n != 2 {
|
||||
t.Fatalf("a moved sibling head did not build the judge again: %d builds", n)
|
||||
}
|
||||
// The controller's base moves: the merge differs again.
|
||||
commitFiles(t, w.controller, map[string]string{"base.txt": "moved"})
|
||||
w.hostCheck(t, "check-867-cache-4", "", cache, w.controllerSibling())
|
||||
if n := builds(t, built); n != 3 {
|
||||
t.Fatalf("a moved base did not build the judge again: %d builds", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The contrast the isolation test stands on: built the old way, in the workspace with its HOME and no judge-build
|
||||
// environment, the same planted Go environment file takes effect in the group's judge build (its toolexec hook runs in
|
||||
// the judge's sources). Without this, the test below could not fail.
|
||||
func TestWithThePlainEnvironmentThePlantedWorkspaceReachesTheGroupsJudgeBuild(t *testing.T) {
|
||||
aRuntimeThatBuildsTheJudge(t)
|
||||
was := judgeBuildEnv
|
||||
judgeBuildEnv = func(string, string) []string { return nil }
|
||||
t.Cleanup(func() { judgeBuildEnv = was })
|
||||
w := theSiblingWorld(t)
|
||||
workspace, ran := aPlantedWorkspace(t)
|
||||
w.hostCheck(t, "check-867-plain", workspace, "", w.controllerSibling())
|
||||
if body, _ := readFile(ran); !strings.Contains(body, "/judge-src") {
|
||||
t.Fatalf("with the plain environment the planted hook did not run in the judge's build, so the isolation test proves nothing: %q", body)
|
||||
}
|
||||
}
|
||||
|
||||
// **The group's judge is unreviewed code, built where nothing the change's script does can reach it**: a git hook, a
|
||||
// Go environment file and a Go cache planted in the workspace, and a script writing into its home, change nothing in the
|
||||
// judge; the judge cache is mounted into judge builds alone.
|
||||
func TestTheGroupsJudgeIsBuiltWhereNothingTheChangeDoesReachesIt(t *testing.T) {
|
||||
built := aRuntimeThatBuildsTheJudge(t)
|
||||
homes := filepath.Join(t.TempDir(), "homes")
|
||||
t.Setenv("JUDGE_HOMES", homes)
|
||||
runs := filepath.Join(t.TempDir(), "runs")
|
||||
t.Setenv("RUNS", runs)
|
||||
w := theSiblingWorld(t)
|
||||
// The member's own script (the base branch's) touches git in its tree and writes into its home.
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "main")
|
||||
commitFiles(t, w.host, map[string]string{CheckScript: "set -e\ngit status >/dev/null 2>&1 || true\n" +
|
||||
"echo planted > \"$HOME/.gitconfig\"\ngrep -q 'foundation: new' \"$MESH_CHECK_BESIDE/mesh-controller/lock.txt\"\n"})
|
||||
w.hostMain = gitInRepo(t, w.host, "rev-parse", "main")
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "change")
|
||||
gitInRepo(t, w.host, "rebase", "--quiet", "main")
|
||||
w.hostHead = gitInRepo(t, w.host, "rev-parse", "HEAD")
|
||||
gitInRepo(t, w.host, "checkout", "--quiet", "main")
|
||||
workspace, ran := aPlantedWorkspace(t)
|
||||
cache := t.TempDir()
|
||||
v := w.hostCheck(t, "check-867-planted", workspace, cache, w.controllerSibling())
|
||||
if v.Gate == nil || v.Gate.Verdict != "pass" || v.Repo == nil || v.Repo.Verdict != "pass" {
|
||||
t.Fatalf("the check did not pass: %+v %+v\n%s", v.Gate, v.Repo, v.Report)
|
||||
}
|
||||
if body, _ := readFile(ran); !strings.Contains(body, "/check/mesh-host") {
|
||||
t.Fatalf("the planted hook never ran in the member's tree, so this test proves nothing: %q", body)
|
||||
}
|
||||
if body, _ := readFile(ran); strings.Contains(body, "/judge-src") || judgeRanTheHook(t, ran) != "" {
|
||||
t.Fatalf("the hook planted in the workspace ran during the group's judge:\n%s", body)
|
||||
}
|
||||
if body, _ := readFile(homes); len(body) == 0 || strings.Contains(body, workspace) {
|
||||
t.Fatalf("the judge was built with a HOME in the workspace: %q", body)
|
||||
}
|
||||
said, _ := readFile(built)
|
||||
if !strings.Contains(said, "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build") {
|
||||
t.Fatalf("the judge was not built with the judge cache's Go cache and no Go environment file:\n%s", said)
|
||||
}
|
||||
// Only a judge build mounts the judge cache.
|
||||
body, _ := readFile(runs)
|
||||
for _, line := range strings.Split(strings.TrimSpace(body), "\n") {
|
||||
if !strings.Contains(line, " go build ") && strings.Contains(line, cache) {
|
||||
t.Fatalf("a container the check runs for the pull request names the judge cache: %s", line)
|
||||
}
|
||||
}
|
||||
// The sources the judge was built from are not left where the member's script could have written them.
|
||||
if _, err := os.Stat(filepath.Join(workspace, "check", "judge-src")); err == nil {
|
||||
t.Fatal("the judge's sources were left in the workspace")
|
||||
}
|
||||
}
|
||||
|
||||
func readFile(path string) (string, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
return string(body), err
|
||||
}
|
||||
@@ -1,327 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
)
|
||||
|
||||
// A build's source fingerprint: what it was made from, hashed (novox/hq issue 280).
|
||||
//
|
||||
// **An image is not byte-reproducible.** Two builds of one source make two image digests, so the rule
|
||||
// "a rebuild that made the same artifacts is no move" (novox/hq ADR 0236) held for archives and bundles
|
||||
// and never for an image: a merge that rebuilt the bus without touching it made a "new" bus build, and
|
||||
// every send to the machine running it was refused until a planned bus upgrade — for a bus nothing had
|
||||
// changed. What a build is made from is reproducible, so that is what is fingerprinted:
|
||||
//
|
||||
// - the git tree of the module's directory at the commit built — every file the build reads, its
|
||||
// module.json and its recipes among them, since the recipe and the compiler see that directory only;
|
||||
// - the git tree of each other repository an artifact's context is cloned from (ArtifactContext);
|
||||
// - each base it was handed, by digest — a module's artifact or a declared vendor image (build.on);
|
||||
// - for a bundle, the toolchain it was compiled in: the compiler image's digest and the builder's own
|
||||
// recipe for that language.
|
||||
//
|
||||
// **No fingerprint where the source does not pin the build.** A build that resolves packages from the
|
||||
// mesh's package registry at build time — a `package` artifact, a TypeScript bundle with packages of
|
||||
// its own, an image whose recipe reads the registry credential — takes whatever the registry holds
|
||||
// then, so the same source can be a different program; it records none, and only its artifacts can
|
||||
// say it is the same. A recipe that fetches from the internet without a pin is the recipe's choice
|
||||
// (novox/hq ADR 0097 refuses the unpinned bases; what a RUN step downloads is not seen here).
|
||||
|
||||
// sourcePrefix names the fingerprint's form, so a later form is never compared equal to this one.
|
||||
const sourcePrefix = delivery.SourcePrefix
|
||||
|
||||
// sourceInputs collects what one build was made from.
|
||||
type sourceInputs struct {
|
||||
module string
|
||||
// tree is the git tree of the module's directory at the commit built.
|
||||
tree string
|
||||
// bases are the digests of what the build was handed to stand on.
|
||||
bases []string
|
||||
// contexts are, per artifact, the git tree of the repository its context was cloned from.
|
||||
contexts map[string]string
|
||||
// toolchains are, per bundle artifact, the compiler image's digest and the recipe's hash.
|
||||
toolchains map[string]string
|
||||
// unpinned is why this build has no fingerprint: empty when it has one.
|
||||
unpinned string
|
||||
|
||||
// prefix is the module's directory within its repository, empty at the root.
|
||||
prefix string
|
||||
// own is the module's build source in its own repository, relative to the module's directory, and
|
||||
// whole when an artifact's is not known (novox/hq ADR 0267).
|
||||
own map[string]bool
|
||||
ownIsAll bool
|
||||
// root is the module's build source in its own repository outside its directory, relative to the
|
||||
// repository's root: the list of predefined keys a manifest naming one was judged against (ADR 0348 §7).
|
||||
root map[string]bool
|
||||
// read is, per context (repository and ref), the build source read there; nil for one read whole.
|
||||
read map[string]map[string]bool
|
||||
readAs map[string]catalogue.ArtifactContext
|
||||
// contextsAtHead is false once a context was cloned at something other than its trunk's head: its
|
||||
// closure is not the one the next merge there meets, and the build says no build source.
|
||||
contextsAtHead bool
|
||||
// onlyKey says the inputs are being taken for a reuse key (novox/hq ADR 0350 §2): each artifact goes as far
|
||||
// as what its source fingerprint reads, and nothing is built or published.
|
||||
onlyKey bool
|
||||
}
|
||||
|
||||
// keyOnly is whether these inputs are only being taken for a reuse key.
|
||||
func (s *sourceInputs) keyOnly() bool { return s != nil && s.onlyKey }
|
||||
|
||||
// copy is the inputs taken so far, apart: a walk's key is taken from them without touching the build's own.
|
||||
func (s *sourceInputs) copy() *sourceInputs {
|
||||
c := newSourceInputs(s.module)
|
||||
c.tree, c.unpinned, c.prefix, c.ownIsAll, c.contextsAtHead = s.tree, s.unpinned, s.prefix, s.ownIsAll, s.contextsAtHead
|
||||
c.bases = append([]string{}, s.bases...)
|
||||
for k, v := range s.contexts {
|
||||
c.contexts[k] = v
|
||||
}
|
||||
for k, v := range s.toolchains {
|
||||
c.toolchains[k] = v
|
||||
}
|
||||
for k, v := range s.own {
|
||||
c.own[k] = v
|
||||
}
|
||||
for k, v := range s.root {
|
||||
c.root[k] = v
|
||||
}
|
||||
for k, v := range s.read {
|
||||
if v == nil {
|
||||
c.read[k] = nil
|
||||
continue
|
||||
}
|
||||
set := map[string]bool{}
|
||||
for e := range v {
|
||||
set[e] = true
|
||||
}
|
||||
c.read[k] = set
|
||||
}
|
||||
for k, v := range s.readAs {
|
||||
c.readAs[k] = v
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// contextOffHead says a context was not its trunk's head.
|
||||
func (s *sourceInputs) contextOffHead() {
|
||||
if s != nil {
|
||||
s.contextsAtHead = false
|
||||
}
|
||||
}
|
||||
|
||||
func newSourceInputs(module string) *sourceInputs {
|
||||
return &sourceInputs{module: module, contexts: map[string]string{}, toolchains: map[string]string{},
|
||||
own: map[string]bool{}, root: map[string]bool{}, read: map[string]map[string]bool{}, readAs: map[string]catalogue.ArtifactContext{},
|
||||
contextsAtHead: true}
|
||||
}
|
||||
|
||||
// ownHas adds entries, relative to the module's directory, to its build source in its own repository.
|
||||
func (s *sourceInputs) ownHas(entries ...string) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
for _, e := range entries {
|
||||
s.own[e] = true
|
||||
}
|
||||
}
|
||||
|
||||
// rootHas adds entries, relative to the repository's root, to the module's build source in its own repository.
|
||||
func (s *sourceInputs) rootHas(entries ...string) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
for _, e := range entries {
|
||||
s.root[e] = true
|
||||
}
|
||||
}
|
||||
|
||||
// ownWhole says an artifact's build source in the module's own repository is not known: the module's
|
||||
// whole directory is its source, as it was before.
|
||||
func (s *sourceInputs) ownWhole() {
|
||||
if s != nil {
|
||||
s.ownIsAll = true
|
||||
}
|
||||
}
|
||||
|
||||
func contextKey(c catalogue.ArtifactContext) string { return c.Repository + "#" + c.Ref }
|
||||
|
||||
// readIn adds entries to the build source read in a context; one read whole stays whole.
|
||||
func (s *sourceInputs) readIn(c catalogue.ArtifactContext, entries []string) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
key := contextKey(c)
|
||||
s.readAs[key] = c
|
||||
set, known := s.read[key]
|
||||
if known && set == nil {
|
||||
return
|
||||
}
|
||||
if set == nil {
|
||||
set = map[string]bool{}
|
||||
s.read[key] = set
|
||||
}
|
||||
for _, e := range entries {
|
||||
set[e] = true
|
||||
}
|
||||
}
|
||||
|
||||
// readWhole says a context is read whole by an artifact.
|
||||
func (s *sourceInputs) readWhole(c catalogue.ArtifactContext) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
key := contextKey(c)
|
||||
s.readAs[key] = c
|
||||
s.read[key] = nil
|
||||
}
|
||||
|
||||
// buildSources is what the build says it was made from, per repository: its own (module.json always,
|
||||
// and every artifact's) unless an artifact's is not known, and each context not read whole.
|
||||
func (s *sourceInputs) buildSources() []BuildSource {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
var out []BuildSource
|
||||
if !s.ownIsAll {
|
||||
own := []string{withPrefix(s.prefix, ManifestName)}
|
||||
for e := range s.own {
|
||||
own = append(own, withPrefix(s.prefix, e))
|
||||
}
|
||||
for e := range s.root {
|
||||
own = append(own, e)
|
||||
}
|
||||
sort.Strings(own)
|
||||
out = append(out, BuildSource{Paths: compactSorted(own)})
|
||||
}
|
||||
var keys []string
|
||||
for k := range s.read {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
set := s.read[k]
|
||||
if set == nil {
|
||||
continue
|
||||
}
|
||||
var paths []string
|
||||
for e := range set {
|
||||
paths = append(paths, e)
|
||||
}
|
||||
sort.Strings(paths)
|
||||
c := s.readAs[k]
|
||||
out = append(out, BuildSource{Repository: c.Repository, Ref: c.Ref, Paths: paths})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// withPrefix is an entry relative to the module's directory made relative to its repository's root.
|
||||
func withPrefix(prefix, entry string) string {
|
||||
switch {
|
||||
case entry == "." || entry == "./":
|
||||
entry = ""
|
||||
case entry == "**" || entry == "./**" || entry == "/**":
|
||||
if prefix == "" {
|
||||
return "**"
|
||||
}
|
||||
return prefix + "/**"
|
||||
}
|
||||
entry = strings.TrimPrefix(entry, "./")
|
||||
if prefix == "" {
|
||||
if entry == "" {
|
||||
return "./"
|
||||
}
|
||||
return entry
|
||||
}
|
||||
if entry == "" {
|
||||
return prefix + "/"
|
||||
}
|
||||
return prefix + "/" + entry
|
||||
}
|
||||
|
||||
func compactSorted(in []string) []string {
|
||||
var out []string
|
||||
for i, e := range in {
|
||||
if i == 0 || e != in[i-1] {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// notPinned marks the build as one its source does not pin; the first reason stands.
|
||||
func (s *sourceInputs) notPinned(why string) {
|
||||
if s != nil && s.unpinned == "" {
|
||||
s.unpinned = why
|
||||
}
|
||||
}
|
||||
|
||||
// sourceFingerprint is the one function the controller's builder and the indexer both call (novox/hq ADR 0437 §5);
|
||||
// a variable only so a test can see it called.
|
||||
var sourceFingerprint = delivery.SourceFingerprint
|
||||
|
||||
// inputs are what the build's fingerprint is of, as the SDK names them.
|
||||
func (s *sourceInputs) inputs() delivery.SourceInputs {
|
||||
if s == nil {
|
||||
return delivery.SourceInputs{}
|
||||
}
|
||||
return delivery.SourceInputs{
|
||||
Module: s.module, Tree: s.tree, Bases: s.bases,
|
||||
Contexts: s.contexts, Toolchains: s.toolchains, Unpinned: s.unpinned,
|
||||
}
|
||||
}
|
||||
|
||||
// resultInputs are the inputs a build's result records; nil for a build with no inputs to speak of.
|
||||
func resultInputs(s *sourceInputs) *delivery.SourceInputs {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
in := s.inputs()
|
||||
return &in
|
||||
}
|
||||
|
||||
// fingerprint is the build's source fingerprint, or empty when the source does not pin the build.
|
||||
func (s *sourceInputs) fingerprint() string {
|
||||
if s == nil {
|
||||
return ""
|
||||
}
|
||||
fp, _ := sourceFingerprint(s.inputs())
|
||||
return fp
|
||||
}
|
||||
|
||||
// referenceDigest is a reference's digest — `sha256:…` — so the registry address it was copied into does not
|
||||
// enter the fingerprint; the reference itself when it carries none.
|
||||
func referenceDigest(reference string) string { return delivery.ReferenceDigest(reference) }
|
||||
|
||||
// gitTree is the git tree of a directory of a clone at its checked-out commit: the whole tree for an
|
||||
// empty path.
|
||||
func gitTree(ctx context.Context, run Runner, clone, path string) (string, error) {
|
||||
spec := "HEAD^{tree}"
|
||||
if rel := strings.Trim(filepath.ToSlash(filepath.Clean(path)), "/"); path != "" && rel != "" && rel != "." {
|
||||
spec = "HEAD:" + rel
|
||||
}
|
||||
out, err := run(ctx, clone, "git", "rev-parse", spec)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
tree := strings.TrimSpace(out)
|
||||
if tree == "" {
|
||||
return "", fmt.Errorf("git named no tree for %s", spec)
|
||||
}
|
||||
return tree, nil
|
||||
}
|
||||
|
||||
// toolchainOf is what a bundle's compile adds to its fingerprint: the compiler image by digest and
|
||||
// the builder's recipe for the language, hashed, so a builder that compiles differently is a change.
|
||||
func toolchainOf(chain Toolchain, base string) string {
|
||||
recipe, _ := json.Marshal(chain)
|
||||
sum := sha256.Sum256(recipe)
|
||||
return chain.Language + " " + referenceDigest(base) + " " + hex.EncodeToString(sum[:8])
|
||||
}
|
||||
@@ -1,44 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func fingerprintOf(t *testing.T, manifest, tree string, mirrored string) string {
|
||||
t.Helper()
|
||||
r, workspace := aRepository(t, manifest, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
|
||||
r.contents["modules/bus/module.json"] = manifest
|
||||
r.tree = tree
|
||||
m := &mirroring{recorded: r, at: mirrored}
|
||||
got, err := Build(context.Background(), r.run, m, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
|
||||
nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return got.Source
|
||||
}
|
||||
|
||||
func TestASourceFingerprintNamesWhatABuildWasMadeFrom(t *testing.T) {
|
||||
one := fingerprintOf(t, anImage, "aaaa", "registry-a:5000")
|
||||
if !strings.HasPrefix(one, sourcePrefix) {
|
||||
t.Fatalf("no fingerprint: %q", one)
|
||||
}
|
||||
// The same tree and base, the base copied to another registry address: one source.
|
||||
if again := fingerprintOf(t, anImage, "aaaa", "registry-b:5000"); again != one {
|
||||
t.Fatalf("one source has two fingerprints: %s %s", one, again)
|
||||
}
|
||||
// The module's tree changed: another source.
|
||||
if changed := fingerprintOf(t, anImage, "bbbb", "registry-a:5000"); changed == one {
|
||||
t.Fatal("a changed tree kept its fingerprint")
|
||||
}
|
||||
// The base moved: another source.
|
||||
moved := strings.Replace(anImage, "1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"2222222222222222222222222222222222222222222222222222222222222222", 1)
|
||||
if changed := fingerprintOf(t, moved, "aaaa", "registry-a:5000"); changed == one {
|
||||
t.Fatal("a moved base kept its fingerprint")
|
||||
}
|
||||
}
|
||||
@@ -1,125 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
)
|
||||
|
||||
// 1214/AC1: the inputs of three builds the live builder recorded a source fingerprint for, copied from mesh-sdk
|
||||
// go/delivery/fingerprint_test.go (TestFingerprintLiveBuilds), where each value is captured, never written by hand.
|
||||
func TestTheBuildersFingerprintEqualsWhatTheLiveBuilderRecorded(t *testing.T) {
|
||||
const ts = "typescript sha256:4b81682df5665c1f461d3f720a3a0fd7593f4aa71e4fc272dbfdc4bbd235b489 20125bd590041690"
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
in sourceInputs
|
||||
recorded string
|
||||
}{
|
||||
{"mailu", sourceInputs{
|
||||
module: "mailu",
|
||||
tree: "cfc6fa8ff5fc1e220132e51ade653b9d3f387b3e",
|
||||
bases: []string{"sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"},
|
||||
toolchains: map[string]string{"code": ts},
|
||||
}, "src1:539f2e64000d436c572ef53b40e524012455945f8c5a1c6bb173ee0707e88ef4"},
|
||||
{"radarr", sourceInputs{
|
||||
module: "radarr",
|
||||
tree: "42b9c918b5af75cba2bbc75845012e7f64e16df9",
|
||||
toolchains: map[string]string{
|
||||
"code": ts,
|
||||
"tools-go": "go sha256:2054591ca034b68323d87ec887ba34f56819cb3364034360c0777da4b314dd54 0995d85edc9a6a23",
|
||||
},
|
||||
}, "src1:b3c150d39c395b99dd68974a48490e9a42467aa5d486d0c604d6ef1e8e9c910a"},
|
||||
{"build-agent", sourceInputs{
|
||||
module: "build-agent",
|
||||
tree: "11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca",
|
||||
bases: []string{
|
||||
"sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc",
|
||||
"sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c",
|
||||
},
|
||||
contexts: map[string]string{"server": "narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},
|
||||
}, "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7"},
|
||||
} {
|
||||
if got := c.in.fingerprint(); got != c.recorded {
|
||||
t.Errorf("%s: fingerprint = %q, the live builder recorded %q", c.name, got, c.recorded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 1214/AC2: a build with no tree, or one that resolves packages when built, gets no fingerprint.
|
||||
func TestABuildWithoutATreeOrUnpinnedGetsNoFingerprint(t *testing.T) {
|
||||
if (&sourceInputs{module: "app"}).fingerprint() != "" {
|
||||
t.Error("a build with no tree has a fingerprint")
|
||||
}
|
||||
s := &sourceInputs{module: "app", tree: "aaaa"}
|
||||
s.notPinned("it resolves packages from the mesh's registry at build time")
|
||||
if got := s.fingerprint(); got != "" {
|
||||
t.Errorf("an unpinned build has a fingerprint: %s", got)
|
||||
}
|
||||
if got := orNoTree(""); got != "its tree was not named" {
|
||||
t.Errorf("the reason for no tree = %q", got)
|
||||
}
|
||||
if got := orNoTree(s.unpinned); got != s.unpinned {
|
||||
t.Errorf("the reason for unpinned = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// 1214/AC3: the builder has no hash of its own: its fingerprint is what delivery.SourceFingerprint says, whatever
|
||||
// that is (a stand-in proves it is called), and the reuse key does not move with it.
|
||||
func TestTheBuilderFingerprintsWithTheSDKFunction(t *testing.T) {
|
||||
was := sourceFingerprint
|
||||
t.Cleanup(func() { sourceFingerprint = was })
|
||||
var got delivery.SourceInputs
|
||||
sourceFingerprint = func(in delivery.SourceInputs) (string, string) {
|
||||
got = in
|
||||
return "src1:stand-in", ""
|
||||
}
|
||||
s := &sourceInputs{module: "m", tree: "t", bases: []string{"x@sha256:a"}, contexts: map[string]string{"c": "ct"}, toolchains: map[string]string{"c": "go"}}
|
||||
if fp := s.fingerprint(); fp != "src1:stand-in" {
|
||||
t.Fatalf("fingerprint = %q, not the SDK function's", fp)
|
||||
}
|
||||
if got.Module != "m" || got.Tree != "t" || len(got.Bases) != 1 || got.Contexts["c"] != "ct" || got.Toolchains["c"] != "go" {
|
||||
t.Errorf("the SDK function was given %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// 1215/AC1: the inputs a result records, read back, give the fingerprint the build recorded (the live builds of
|
||||
// 1214/AC1), through encoding/json as the build record keeps them.
|
||||
func TestTheInputsAResultRecordsGiveItsFingerprintAgain(t *testing.T) {
|
||||
s := &sourceInputs{module: "build-agent", tree: "11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca",
|
||||
bases: []string{
|
||||
"sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc",
|
||||
"sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c",
|
||||
},
|
||||
contexts: map[string]string{"server": "narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"}}
|
||||
raw, err := json.Marshal(resultInputs(s))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var back delivery.SourceInputs
|
||||
if err := json.Unmarshal(raw, &back); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7"
|
||||
if fp, none := delivery.SourceFingerprint(back); fp != want || none != "" {
|
||||
t.Errorf("the inputs read back give %q (%q), want %q", fp, none, want)
|
||||
}
|
||||
}
|
||||
|
||||
// 1215/AC3: a build that gets no fingerprint still has its inputs recorded, so the reason is visible.
|
||||
func TestABuildWithNoFingerprintStillSaysItsInputs(t *testing.T) {
|
||||
s := newSourceInputs("app")
|
||||
s.tree = "aaaa"
|
||||
s.notPinned("it resolves packages from the mesh's registry at build time")
|
||||
in := resultInputs(s)
|
||||
if in == nil || in.Module != "app" || in.Tree != "aaaa" || in.Unpinned == "" {
|
||||
t.Fatalf("the inputs of an unpinned build: %+v", in)
|
||||
}
|
||||
if fp, none := delivery.SourceFingerprint(*in); fp != "" || none != in.Unpinned {
|
||||
t.Errorf("the inputs give %q (%q)", fp, none)
|
||||
}
|
||||
noTree := resultInputs(newSourceInputs("app"))
|
||||
if noTree == nil || noTree.Module != "app" || noTree.Tree != "" {
|
||||
t.Errorf("the inputs of a build with no tree: %+v", noTree)
|
||||
}
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A build's source fingerprint (novox/hq issue 280): what it was made from, so a rebuild of an unchanged
|
||||
// source is one build however the image digest it made differs.
|
||||
|
||||
const anImage = `{"module":"bus","version":"1",
|
||||
"build":{"on":[{"arg":"BASE","image":"vendor/server@sha256:` + "1111111111111111111111111111111111111111111111111111111111111111" + `"}],
|
||||
"artifacts":[{"name":"server","kind":"image","from":"Dockerfile"}]},
|
||||
"resources":[{"id":"svc","type":"container","name":"bus","artifact":"server"}]}`
|
||||
|
||||
// mirroring is a store that copies a vendor's image into the mesh's registry, at an address a test says.
|
||||
type mirroring struct {
|
||||
*recorded
|
||||
at string
|
||||
}
|
||||
|
||||
func (m *mirroring) MirrorImage(_ context.Context, from, repository string) (string, error) {
|
||||
_, digest, _ := strings.Cut(from, "@")
|
||||
return m.at + "/" + repository + "@" + digest, nil
|
||||
}
|
||||
|
||||
func TestABuildTheRegistryDecidesHasNoFingerprint(t *testing.T) {
|
||||
s := newSourceInputs("app")
|
||||
s.tree = "aaaa"
|
||||
if s.fingerprint() == "" {
|
||||
t.Fatal("a pinned source has no fingerprint")
|
||||
}
|
||||
s.notPinned("it resolves packages from the mesh's registry at build time")
|
||||
if got := s.fingerprint(); got != "" {
|
||||
t.Fatalf("a build the registry decides has a fingerprint: %s", got)
|
||||
}
|
||||
if (&sourceInputs{module: "app"}).fingerprint() != "" {
|
||||
t.Fatal("a build whose tree was not named has a fingerprint")
|
||||
}
|
||||
}
|
||||
@@ -1,225 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A module naming a base the mesh has not built is refused, and the refusal names what is missing.
|
||||
//
|
||||
// **This is the whole point of naming a base rather than pinning one** (novox/hq issue 044). A
|
||||
// recipe with a fingerprint typed into it fails inside a container runtime, on its first line, with
|
||||
// a message about an image nobody can look up. This fails before anything is built, saying which
|
||||
// module has to exist first.
|
||||
func TestABaseTheMeshHasNotBuiltIsRefused(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "postgres",
|
||||
Build: &catalogue.Build{
|
||||
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
|
||||
},
|
||||
}
|
||||
_, _, err := standingOn(context.Background(), manifest, map[string]string{}, noMirror)
|
||||
if err == nil {
|
||||
t.Fatal("a base nothing has built was accepted; the build would have failed on its first line")
|
||||
}
|
||||
for _, want := range []string{"mesh-tools", "postgres"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("the refusal does not name %s: %v", want, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// And one the mesh holds becomes the argument the recipe reads it from.
|
||||
func TestABaseTheMeshHoldsBecomesABuildArgument(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "postgres",
|
||||
Build: &catalogue.Build{
|
||||
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
|
||||
},
|
||||
}
|
||||
held := map[string]string{"mesh-tools/runtime": "127.0.0.1:5000/mesh-tools/runtime@sha256:" + strings.Repeat("a", 64)}
|
||||
args, _, err := standingOn(context.Background(), manifest, held, noMirror)
|
||||
if err != nil {
|
||||
t.Fatalf("a base this mesh holds was refused: %v", err)
|
||||
}
|
||||
want := []string{"--build-arg", "RUNTIME_BASE=" + held["mesh-tools/runtime"]}
|
||||
if len(args) != len(want) || args[0] != want[0] || args[1] != want[1] {
|
||||
t.Fatalf("the build was invoked with %v, not %v", args, want)
|
||||
}
|
||||
}
|
||||
|
||||
// A module naming no base asks for nothing, which is most modules.
|
||||
func TestAModuleNamingNoBaseAddsNoArguments(t *testing.T) {
|
||||
args, _, err := standingOn(context.Background(), catalogue.Manifest{Module: "hello-web", Build: &catalogue.Build{}}, nil, noMirror)
|
||||
if err != nil || args != nil {
|
||||
t.Fatalf("a module naming no base produced %v, %v", args, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Half a base is refused rather than half-applied.
|
||||
func TestAnIncompleteBaseIsRefused(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "postgres",
|
||||
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Module: "mesh-tools", Artifact: "runtime"}}},
|
||||
}
|
||||
if _, _, err := standingOn(context.Background(), manifest, map[string]string{"mesh-tools/runtime": "x"}, noMirror); err == nil {
|
||||
t.Fatal("a base with no build argument was accepted; nothing would have read it")
|
||||
}
|
||||
}
|
||||
|
||||
// noMirror is a mirror for tests whose bases are all the mesh's own.
|
||||
func noMirror(context.Context, string, string, string) (string, error) {
|
||||
return "", fmt.Errorf("nothing to copy in this test")
|
||||
}
|
||||
|
||||
// A build may stand on an image published elsewhere, declared and pinned (novox/hq 04-ISSUES/064,
|
||||
// ADR 0097): it is copied into the mesh's registry first and the recipe is handed the copy.
|
||||
func TestADeclaredVendorImageIsCopiedInAndHandedToTheRecipe(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "minio",
|
||||
Build: &catalogue.Build{
|
||||
On: []catalogue.BuildsOn{{Arg: "MC_BASE", Image: "quay.io/minio/mc@sha256:" + strings.Repeat("c", 64)}},
|
||||
},
|
||||
}
|
||||
var asked []string
|
||||
args, _, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository, former string) (string, error) {
|
||||
asked = append(asked, from+" -> "+repository+" (from "+former+")")
|
||||
return "127.0.0.1:5000/" + repository + "@sha256:" + strings.Repeat("d", 64), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// One repository per upstream image, named for the image (novox/hq ADR 0257); the module's
|
||||
// repository of before offered as where it may already be held.
|
||||
if len(asked) != 1 || asked[0] != "quay.io/minio/mc@sha256:"+strings.Repeat("c", 64)+
|
||||
" -> upstream/quay.io/minio/mc (from minio/on-mc_base)" {
|
||||
t.Fatalf("the image was not copied into the image's own repository: %v", asked)
|
||||
}
|
||||
if strings.Join(args, " ") != "--build-arg MC_BASE=127.0.0.1:5000/upstream/quay.io/minio/mc@sha256:"+strings.Repeat("d", 64) {
|
||||
t.Fatalf("the recipe was not handed the copy: %v", args)
|
||||
}
|
||||
// Unpinned, it is refused: a tag is what somebody else can move.
|
||||
manifest.Build.On[0].Image = "quay.io/minio/mc:latest"
|
||||
if _, _, err := standingOn(context.Background(), manifest, nil, noMirror); err == nil || !strings.Contains(err.Error(), "not pinned") {
|
||||
t.Fatalf("an unpinned vendor image was accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A recipe reaching for an image the manifest did not declare is named, and its own stages,
|
||||
// declared arguments and scratch are not.
|
||||
func TestARecipeFetchingWhatTheManifestDidNotDeclareIsNamed(t *testing.T) {
|
||||
recipe := `
|
||||
ARG RUNTIME_BASE
|
||||
ARG MC_BASE
|
||||
FROM ${RUNTIME_BASE} AS build
|
||||
COPY --from=${MC_BASE} /usr/bin/mc /usr/local/bin/mc
|
||||
COPY --from=build /out /out
|
||||
COPY --from=0 /x /x
|
||||
FROM scratch
|
||||
COPY --from=vendor/tool:latest /tool /tool
|
||||
FROM golang:1.25-alpine AS go
|
||||
`
|
||||
bases, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true})
|
||||
if strings.Join(copies, "|") != "${MC_BASE} (a build argument the manifest does not declare)|vendor/tool:latest" {
|
||||
t.Fatalf("copies out of undeclared images: %v", copies)
|
||||
}
|
||||
// A base fetched on its own is named apart, and refused like a copy (ADR 0097).
|
||||
if strings.Join(bases, "|") != "golang:1.25-alpine" {
|
||||
t.Fatalf("undeclared bases: %v", bases)
|
||||
}
|
||||
if _, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true, "MC_BASE": true}); len(copies) != 1 {
|
||||
t.Fatalf("declared arguments are not fetches: %v", copies)
|
||||
}
|
||||
}
|
||||
|
||||
// Continued lines are one instruction, heredoc bodies are not instructions, and a RUN --mount reaches
|
||||
// for an image as a COPY --from does (review C4, C5).
|
||||
func TestARecipeIsReadAsInstructions(t *testing.T) {
|
||||
recipe := "ARG RUNTIME_BASE\n" +
|
||||
"FROM ${RUNTIME_BASE} \\\n AS build\n" +
|
||||
"COPY \\\n --from=docker.io/vendor/one:latest /a /a\n" +
|
||||
"COPY --from=build /out /out\n" +
|
||||
"COPY <<EOF /app/x.py\nfrom os import path\nEOF\n" +
|
||||
"RUN --mount=type=bind,from=docker.io/vendor/two:1,target=/t cp /t/x /x\n" +
|
||||
"FROM scratch\n"
|
||||
bases, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true})
|
||||
if strings.Join(copies, "|") != "docker.io/vendor/one:latest|docker.io/vendor/two:1" {
|
||||
t.Fatalf("copies: %v", copies)
|
||||
}
|
||||
if len(bases) != 0 {
|
||||
t.Fatalf("a heredoc line or a continued stage was read as a base: %v", bases)
|
||||
}
|
||||
}
|
||||
|
||||
// What a build was handed as its bases is what it stood on — recorded, so a changed base knows what
|
||||
// to rebuild (novox/hq 04-ISSUES/131). A recipe reads the base from an argument, so nothing else
|
||||
// could know.
|
||||
func TestTheBasesABuildWasHandedAreWhatItStoodOn(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "gitea",
|
||||
Build: &catalogue.Build{
|
||||
On: []catalogue.BuildsOn{
|
||||
{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"},
|
||||
{Arg: "BUILD_BASE", Module: "mesh-tools", Artifact: "build"},
|
||||
},
|
||||
Artifacts: []catalogue.Artifact{{Name: "runtime", Kind: catalogue.ArtifactImage, From: "Dockerfile"}},
|
||||
},
|
||||
}
|
||||
held := map[string]string{
|
||||
"mesh-tools/runtime": "127.0.0.1:5000/mesh-tools/runtime@sha256:" + strings.Repeat("a", 64),
|
||||
"mesh-tools/build": "127.0.0.1:5000/mesh-tools/build@sha256:" + strings.Repeat("b", 64),
|
||||
}
|
||||
_, resolved, err := standingOn(context.Background(), manifest, held, noMirror)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := against(t.TempDir(), manifest, resolved)
|
||||
if len(got) != 2 || got[0] != held["mesh-tools/build"] || got[1] != held["mesh-tools/runtime"] {
|
||||
t.Fatalf("the bases the build was handed were not what it stood on: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// What a build read besides its module's own repository is the second repository its recipes name,
|
||||
// each once: a module that packages source living elsewhere is affected when that source moves.
|
||||
func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
|
||||
elsewhere := catalogue.ArtifactContext{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "builder",
|
||||
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{
|
||||
{Name: "server", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
|
||||
{Name: "tools", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
|
||||
{Name: "config", Kind: catalogue.ArtifactArchive, From: "etc"},
|
||||
}},
|
||||
}
|
||||
read := readBy(manifest)
|
||||
if len(read) != 1 || read[0] != elsewhere {
|
||||
t.Fatalf("the repositories this build read are %+v", read)
|
||||
}
|
||||
if readBy(catalogue.Manifest{Module: "gitea", Build: &catalogue.Build{}}) != nil {
|
||||
t.Fatal("a module whose recipes name no other repository read one")
|
||||
}
|
||||
}
|
||||
|
||||
// novox/hq 04-ISSUES/212: a toolchain stands on the SDK's published package, and is built with the
|
||||
// exact version the mesh published — an argument that changes when the SDK does, so a rebuild after
|
||||
// a release never reuses an install of the version before it.
|
||||
func TestAPackageTheMeshPublishedIsPassedByItsExactVersion(t *testing.T) {
|
||||
manifest := catalogue.Manifest{
|
||||
Module: "mesh-tools",
|
||||
Build: &catalogue.Build{
|
||||
On: []catalogue.BuildsOn{{Arg: "MESH_SDK", Module: "mesh-sdk", Artifact: "lib"}},
|
||||
},
|
||||
}
|
||||
held := map[string]string{"mesh-sdk/lib": "@novox/mesh-sdk@0.1.6"}
|
||||
args, resolved, err := standingOn(context.Background(), manifest, held, noMirror)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fmt.Sprint(args) != "[--build-arg MESH_SDK=@novox/mesh-sdk@0.1.6]" || fmt.Sprint(resolved) != "[@novox/mesh-sdk@0.1.6]" {
|
||||
t.Errorf("the package was passed as %v, recorded as %v", args, resolved)
|
||||
}
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A host built without knowing its system refuses every declaration before applying anything —
|
||||
// safely, totally, and with nothing reporting it. The mesh built one, delivered it, started it, and
|
||||
// it would have refused the first thing it was asked to do (novox/hq 04-ISSUES/161).
|
||||
|
||||
func TestTheGoToolchainStampsTheArtifactsSystem(t *testing.T) {
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if chain.SystemStamp != "main.builtFor" {
|
||||
t.Fatalf("the go toolchain fills %q", chain.SystemStamp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestALanguageWithNoPinnedSystemStampsNothing(t *testing.T) {
|
||||
// Interpreted output is not pinned to a system, and a manifest declaring one for it is already
|
||||
// refused. Nothing to fill.
|
||||
for _, language := range []string{"typescript", "python"} {
|
||||
chain, err := ToolchainFor(language)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if chain.SystemStamp != "" {
|
||||
t.Fatalf("%s fills %q, and its output is not pinned to a system",
|
||||
language, chain.SystemStamp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheStampIsTheOneThingTakenFromTheArtifact(t *testing.T) {
|
||||
// The toolchain accepts nothing else from the module — anything it could override it would be
|
||||
// writing a Dockerfile to override. The system is the stated exception, because a compiled
|
||||
// binary is per system and the artifact is what declares one (ADR 0142).
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
joined := strings.Join(chain.Compile, " ")
|
||||
if strings.Contains(joined, "${") || strings.Contains(joined, "%s") {
|
||||
t.Fatalf("the compile line takes something from the module: %q", joined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLinkerIsToldOnceNotTwice(t *testing.T) {
|
||||
// A repeated flag is not a merged one: the Go command takes the last -ldflags and drops the
|
||||
// first. Passing the toolchain's flags and then the stamp separately produced a binary that knew
|
||||
// its system and had lost -s -w — 12.2MB against 8.5MB, with its debug info (04-ISSUES/161).
|
||||
chain, err := ToolchainFor("go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, arg := range chain.Compile {
|
||||
if arg == "-ldflags" {
|
||||
t.Fatal("the compile line carries -ldflags, so composing one here makes two")
|
||||
}
|
||||
}
|
||||
if len(chain.LinkerFlags) == 0 {
|
||||
t.Fatal("the go toolchain passes no linker flags, so the binary keeps its debug info")
|
||||
}
|
||||
var stripped bool
|
||||
for _, f := range chain.LinkerFlags {
|
||||
if f == "-s" {
|
||||
stripped = true
|
||||
}
|
||||
}
|
||||
if !stripped {
|
||||
t.Fatalf("the go toolchain does not strip: %v", chain.LinkerFlags)
|
||||
}
|
||||
}
|
||||
@@ -1,95 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"path/filepath"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// withTemplateTexts is what was built with each template's text, for a module whose files name templates (novox/hq ADR
|
||||
// 0349 §2): its templates archive — the one built from its templates directory, files/ unless its manifest names
|
||||
// another (issue 731) — gains the text of every file in it, read back from the packed bytes, by its path inside the
|
||||
// archive. Resolve reads them into the files' content.
|
||||
//
|
||||
// **Read from the archive, proven by its digest.** The module's templates directory is packed again, as the build packs
|
||||
// it, and the bytes are read only when they hash to the digest the build made or took: a fresh build's own archive, or
|
||||
// a check's build a walk copied by digest without building (novox/hq ADR 0350 §2). Packing is deterministic, so one
|
||||
// source packs to one digest; a tree that packs to another is not the archive, and its text is refused.
|
||||
func withTemplateTexts(manifest catalogue.Manifest, within string, built []catalogue.Built) ([]catalogue.Built, error) {
|
||||
if !manifest.Templated() {
|
||||
return built, nil
|
||||
}
|
||||
name := catalogue.TemplatesArtifact(manifest)
|
||||
at := -1
|
||||
for i, b := range built {
|
||||
if b.Name == name {
|
||||
at = i
|
||||
}
|
||||
}
|
||||
if name == "" || at < 0 {
|
||||
return nil, fmt.Errorf("%s names templates and built no archive of %s/ to read them from",
|
||||
manifest.Module, manifest.TemplatesDir())
|
||||
}
|
||||
body, err := pack(filepath.Join(within, filepath.FromSlash(manifest.TemplatesDir())))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: packing %s/ to read its templates back failed: %w", manifest.Module,
|
||||
manifest.TemplatesDir(), err)
|
||||
}
|
||||
sum := sha256.Sum256(body)
|
||||
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||
if built[at].Digest != digest {
|
||||
return nil, fmt.Errorf("%s: %s/ packs to the digest %s, and the archive %q built or taken is %s: the "+
|
||||
"templates are read only from the archive the mesh records", manifest.Module, manifest.TemplatesDir(),
|
||||
digest, name, orNone(built[at].Digest))
|
||||
}
|
||||
texts, err := unpackTexts(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: reading the archive %q back failed: %w", manifest.Module, name, err)
|
||||
}
|
||||
out := append([]catalogue.Built(nil), built...)
|
||||
out[at].Texts = texts
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// unpackTexts is every file of a packed archive (pack), by its path inside it.
|
||||
func unpackTexts(body []byte) (map[string]string, error) {
|
||||
zipped, err := gzip.NewReader(bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer zipped.Close()
|
||||
reader := tar.NewReader(zipped)
|
||||
texts := map[string]string{}
|
||||
for {
|
||||
header, err := reader.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
return texts, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if header.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
text, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
texts[header.Name] = string(text)
|
||||
}
|
||||
}
|
||||
|
||||
func orNone(digest string) string {
|
||||
if digest == "" {
|
||||
return "no digest"
|
||||
}
|
||||
return digest
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
//go:build integration
|
||||
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
func TestABuildReadsEachTemplateBackFromItsArchive(t *testing.T) {
|
||||
r, workspace := aRepository(t, templatedModule, templatedFiles)
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
|
||||
t.Fatalf("the template's text is %q, not the file's", text)
|
||||
}
|
||||
if text := contentOf(t, got.Manifest, "deeper"); text != templatedFiles["files/a/deeper.conf"] {
|
||||
t.Fatalf("the template's text is %q, not the file's", text)
|
||||
}
|
||||
if r.archives["meshboard/files"] == "" {
|
||||
t.Fatalf("the templates archive was not published: %v", r.archives)
|
||||
}
|
||||
if problem := catalogue.HeldTemplateProblem(got.Manifest); problem != "" {
|
||||
t.Fatal(problem)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildWhoseTemplateIsMissingFailsNamingIt(t *testing.T) {
|
||||
files := map[string]string{}
|
||||
for k, v := range templatedFiles {
|
||||
files[k] = v
|
||||
}
|
||||
delete(files, "files/a/deeper.conf")
|
||||
r, workspace := aRepository(t, templatedModule, files)
|
||||
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "files/a/deeper.conf") {
|
||||
t.Fatalf("a build whose template is missing did not fail naming it: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWalkTakingAChecksBuildReadsTheTemplatesBackToo(t *testing.T) {
|
||||
withVersion(t, "builder-1")
|
||||
contents := map[string]string{"mod/" + ManifestName: templatedModule}
|
||||
for k, v := range templatedFiles {
|
||||
contents["mod/"+k] = v
|
||||
}
|
||||
scratch := newScratch()
|
||||
r, run, workspace := aModuleWith(t, "tree-one", contents)
|
||||
checked := aCheckBuiltBy(t, scratch, r, run, workspace)
|
||||
|
||||
r, run, workspace = aModuleWith(t, "tree-one", contents)
|
||||
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
|
||||
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Reused == nil || len(r.archives) > 0 {
|
||||
t.Fatalf("the walk built again: reused %+v, why %q, published %v", got.Reused, got.Why, r.archives)
|
||||
}
|
||||
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
|
||||
t.Fatalf("the template's text is %q, not the file's", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildReadsEachTemplateBackFromTheTemplatesDirectoryItsManifestNames(t *testing.T) {
|
||||
files := map[string]string{"files/bin/meshboard-hook": "#!/bin/sh\n", "templates/meshboard.conf": "listen = 8080\n"}
|
||||
r, workspace := aRepository(t, unpackingModule, files)
|
||||
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
|
||||
Npmrc{}, GitCredential{}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if text := contentOf(t, got.Manifest, "conf"); text != files["templates/meshboard.conf"] {
|
||||
t.Fatalf("the template's text is %q, not the file's", text)
|
||||
}
|
||||
if got.Manifest.Templates != "" {
|
||||
t.Fatalf("the built manifest still names its templates directory: %q", got.Manifest.Templates)
|
||||
}
|
||||
// files/ packs exactly as it would with no templates beside it.
|
||||
alone, err := pack(writeTree(t, t.TempDir(), map[string]string{"bin/meshboard-hook": "#!/bin/sh\n"}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum := sha256.Sum256(alone)
|
||||
packed := ""
|
||||
for _, b := range got.Built {
|
||||
if b.Name == "scripts" {
|
||||
packed = b.Digest
|
||||
}
|
||||
}
|
||||
if packed != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||
t.Fatalf("files/ packed to %q, not to what it packs to alone", packed)
|
||||
}
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/catalogue"
|
||||
)
|
||||
|
||||
// A module's files kept as template files in its repository (novox/hq ADR 0349 §2, design 57 package T1, issue
|
||||
// 654): the build packs files/ into the module's templates archive and reads each template's text back from the
|
||||
// packed bytes into the file's content, so the manifest the mesh holds is the one it would hold for the text inline.
|
||||
|
||||
const templatedModule = `{"module":"meshboard","version":"1",
|
||||
"build":{"artifacts":[
|
||||
{"name":"server","kind":"image","from":"Dockerfile"},
|
||||
{"name":"files","kind":"archive","from":"files"}]},
|
||||
"resources":[
|
||||
{"id":"svc","type":"container","name":"meshboard","artifact":"server"},
|
||||
{"id":"conf","type":"file","path":"/etc/meshboard/meshboard.conf","template":"files/meshboard.conf"},
|
||||
{"id":"deeper","type":"file","path":"/etc/meshboard/deeper.conf","template":"files/a/deeper.conf"}]}`
|
||||
|
||||
var templatedFiles = map[string]string{
|
||||
"Dockerfile": "FROM scratch", "files/meshboard.conf": "listen = 8080\n\"quoted\" é\n",
|
||||
"files/a/deeper.conf": "deeper\n", "files/unused.conf": "packed, named by nothing",
|
||||
}
|
||||
|
||||
func contentOf(t *testing.T, m catalogue.Manifest, id string) string {
|
||||
t.Helper()
|
||||
for _, r := range m.Resources {
|
||||
if r["id"] == id {
|
||||
if _, has := r[catalogue.TemplateField]; has {
|
||||
t.Fatalf("%s still names its template in the built manifest: %v", id, r)
|
||||
}
|
||||
text, ok := r["content"].(string)
|
||||
if !ok {
|
||||
t.Fatalf("%s carries no content: %v", id, r)
|
||||
}
|
||||
return text
|
||||
}
|
||||
}
|
||||
t.Fatalf("no resource %s in %v", id, m.Resources)
|
||||
return ""
|
||||
}
|
||||
|
||||
// A walk that takes its check's build copies the archive by digest and builds nothing: the templates are read back
|
||||
// from the tree packed again, which is the archive only when it packs to the digest taken.
|
||||
|
||||
// The tree packed again must be the archive taken; text read from anything else is not the archive's.
|
||||
func TestTemplateTextsAreRefusedWhenTheTreeDoesNotPackToTheDigestTaken(t *testing.T) {
|
||||
m, err := catalogue.ParseManifest([]byte(templatedModule))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
within := writeTree(t, t.TempDir(), templatedFiles)
|
||||
built := []catalogue.Built{{Name: "files", Kind: catalogue.ArtifactArchive, Digest: "sha256:" + strings.Repeat("0", 64)}}
|
||||
if _, err := withTemplateTexts(m, within, built); err == nil || !strings.Contains(err.Error(), "digest") {
|
||||
t.Fatalf("text was read from a tree that does not pack to the archive taken: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// writeTree writes files under dir and answers dir.
|
||||
func writeTree(t *testing.T, dir string, files map[string]string) string {
|
||||
t.Helper()
|
||||
for name, body := range files {
|
||||
full := filepath.Join(dir, filepath.FromSlash(name))
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return dir
|
||||
}
|
||||
|
||||
// A module whose files/ is an archive unpacked onto the machine keeps its templates in a directory it names (issue
|
||||
// 731): the build reads each template back from the archive packed from that directory, and files/ is packed as it
|
||||
// was, its templates nowhere in it.
|
||||
const unpackingModule = `{"module":"meshboard","version":"1",
|
||||
"templates":"templates",
|
||||
"build":{"artifacts":[
|
||||
{"name":"scripts","kind":"archive","from":"files"},
|
||||
{"name":"templates","kind":"archive","from":"templates"}]},
|
||||
"resources":[
|
||||
{"id":"scripts","type":"archive","path":"/usr/local/lib/meshboard","artifact":"scripts"},
|
||||
{"id":"conf","type":"file","path":"/etc/meshboard/meshboard.conf","template":"templates/meshboard.conf"}]}`
|
||||
-7
@@ -1,7 +0,0 @@
|
||||
Captured 2026-10-11 from real `go test` runs (go1.27.1, linux/amd64) of a throwaway module whose
|
||||
packages fail each way a repository's own check fails: a test failing early in a run its own logs
|
||||
then fill (early: 600 log lines after the failures), a panic, a data race under -race, a build error,
|
||||
and one -race run of every package. Only the module's directory was rewritten to /src/cap.
|
||||
|
||||
They are the output shapes of novox/hq issue 460: mesh-controller #218's check printed its
|
||||
`--- FAIL:` line before the store-backed package's logs, which pushed it out of the report's tail.
|
||||
@@ -1,5 +0,0 @@
|
||||
# example.com/cap/broken
|
||||
broken/broken.go:3:28: undefined: undefinedThing
|
||||
FAIL example.com/cap/broken [build failed]
|
||||
ok example.com/cap/fine (cached)
|
||||
FAIL
|
||||
@@ -1,31 +0,0 @@
|
||||
==================
|
||||
WARNING: DATA RACE
|
||||
Read at 0x00c000018398 by goroutine 8:
|
||||
example.com/cap/race.TestACounterIsShared.func1()
|
||||
/src/cap/race/race_test.go:13 +0x7b
|
||||
|
||||
Previous write at 0x00c000018398 by goroutine 9:
|
||||
example.com/cap/race.TestACounterIsShared.func1()
|
||||
/src/cap/race/race_test.go:13 +0x8d
|
||||
|
||||
Goroutine 8 (running) created at:
|
||||
example.com/cap/race.TestACounterIsShared()
|
||||
/src/cap/race/race_test.go:13 +0x78
|
||||
testing.tRunner()
|
||||
/usr/lib/go/src/testing/testing.go:2193 +0x21c
|
||||
testing.(*T).Run.gowrap1()
|
||||
/usr/lib/go/src/testing/testing.go:2258 +0x38
|
||||
|
||||
Goroutine 9 (finished) created at:
|
||||
example.com/cap/race.TestACounterIsShared()
|
||||
/src/cap/race/race_test.go:13 +0x78
|
||||
testing.tRunner()
|
||||
/usr/lib/go/src/testing/testing.go:2193 +0x21c
|
||||
testing.(*T).Run.gowrap1()
|
||||
/usr/lib/go/src/testing/testing.go:2258 +0x38
|
||||
==================
|
||||
--- FAIL: TestACounterIsShared (0.00s)
|
||||
testing.go:1865: race detected during execution of test
|
||||
FAIL
|
||||
FAIL example.com/cap/race 0.008s
|
||||
FAIL
|
||||
@@ -1,609 +0,0 @@
|
||||
--- FAIL: TestTheEnvelopeIsPinned (0.00s)
|
||||
early_test.go:9: the envelope's subject is "mesh.a", not "mesh.b"
|
||||
early_test.go:10: a second line of the same failure
|
||||
--- FAIL: TestSubtests (0.00s)
|
||||
--- FAIL: TestSubtests/the_hub (0.00s)
|
||||
early_test.go:14: the hub does not compose
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d0
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d1
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d2
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d3
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d4
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d5
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d6
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d7
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d8
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d9
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d10
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d11
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d12
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d13
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d14
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d15
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d16
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d17
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d18
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d19
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d20
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d21
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d22
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d23
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d24
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d25
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d26
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d27
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d28
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d29
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d30
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d31
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d32
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d33
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d34
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d35
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d36
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d37
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d38
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d39
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d40
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d41
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d42
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d43
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d44
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d45
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d46
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d47
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d48
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d49
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d50
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d51
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d52
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d53
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d54
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d55
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d56
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d57
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d58
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d59
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d60
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d61
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d62
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d63
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d64
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d65
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d66
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d67
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d68
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d69
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d70
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d71
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d72
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d73
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d74
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d75
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d76
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d77
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d78
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d79
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d80
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d81
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d82
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d83
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d84
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d85
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d86
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d87
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d88
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d89
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d90
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d91
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d92
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d93
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d94
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d95
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d96
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d97
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d98
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d99
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d100
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d101
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d102
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d103
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d104
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d105
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d106
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d107
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d108
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d109
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d110
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d111
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d112
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d113
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d114
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d115
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d116
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d117
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d118
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d119
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d120
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d121
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d122
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d123
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d124
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d125
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d126
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d127
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d128
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d129
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d130
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d131
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d132
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d133
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d134
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d135
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d136
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d137
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d138
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d139
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d140
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d141
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d142
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d143
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d144
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d145
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d146
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d147
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d148
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d149
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d150
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d151
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d152
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d153
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d154
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d155
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d156
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d157
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d158
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d159
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d160
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d161
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d162
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d163
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d164
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d165
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d166
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d167
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d168
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d169
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d170
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d171
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d172
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d173
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d174
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d175
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d176
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d177
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d178
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d179
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d180
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d181
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d182
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d183
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d184
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d185
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d186
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d187
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d188
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d189
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d190
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d191
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d192
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d193
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d194
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d195
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d196
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d197
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d198
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d199
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d200
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d201
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d202
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d203
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d204
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d205
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d206
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d207
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d208
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d209
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d210
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d211
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d212
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d213
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d214
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d215
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d216
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d217
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d218
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d219
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d220
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d221
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d222
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d223
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d224
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d225
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d226
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d227
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d228
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d229
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d230
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d231
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d232
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d233
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d234
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d235
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d236
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d237
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d238
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d239
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d240
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d241
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d242
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d243
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d244
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d245
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d246
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d247
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d248
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d249
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d250
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d251
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d252
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d253
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d254
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d255
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d256
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d257
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d258
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d259
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d260
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d261
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d262
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d263
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d264
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d265
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d266
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d267
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d268
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d269
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d270
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d271
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d272
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d273
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d274
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d275
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d276
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d277
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d278
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d279
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d280
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d281
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d282
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d283
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d284
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d285
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d286
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d287
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d288
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d289
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d290
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d291
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d292
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d293
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d294
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d295
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d296
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d297
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d298
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d299
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d300
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d301
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d302
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d303
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d304
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d305
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d306
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d307
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d308
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d309
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d310
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d311
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d312
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d313
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d314
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d315
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d316
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d317
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d318
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d319
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d320
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d321
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d322
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d323
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d324
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d325
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d326
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d327
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d328
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d329
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d330
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d331
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d332
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d333
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d334
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d335
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d336
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d337
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d338
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d339
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d340
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d341
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d342
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d343
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d344
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d345
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d346
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d347
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d348
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d349
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d350
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d351
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d352
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d353
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d354
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d355
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d356
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d357
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d358
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d359
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d360
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d361
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d362
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d363
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d364
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d365
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d366
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d367
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d368
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d369
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d370
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d371
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d372
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d373
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d374
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d375
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d376
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d377
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d378
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d379
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d380
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d381
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d382
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d383
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d384
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d385
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d386
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d387
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d388
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d389
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d390
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d391
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d392
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d393
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d394
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d395
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d396
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d397
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d398
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d399
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d400
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d401
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d402
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d403
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d404
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d405
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d406
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d407
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d408
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d409
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d410
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d411
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d412
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d413
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d414
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d415
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d416
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d417
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d418
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d419
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d420
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d421
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d422
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d423
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d424
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d425
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d426
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d427
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d428
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d429
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d430
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d431
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d432
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d433
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d434
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d435
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d436
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d437
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d438
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d439
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d440
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d441
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d442
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d443
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d444
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d445
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d446
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d447
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d448
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d449
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d450
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d451
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d452
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d453
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d454
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d455
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d456
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d457
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d458
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d459
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d460
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d461
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d462
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d463
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d464
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d465
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d466
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d467
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d468
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d469
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d470
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d471
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d472
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d473
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d474
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d475
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d476
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d477
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d478
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d479
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d480
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d481
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d482
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d483
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d484
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d485
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d486
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d487
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d488
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d489
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d490
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d491
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d492
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d493
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d494
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d495
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d496
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d497
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d498
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d499
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d500
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d501
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d502
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d503
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d504
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d505
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d506
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d507
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d508
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d509
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d510
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d511
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d512
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d513
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d514
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d515
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d516
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d517
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d518
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d519
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d520
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d521
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d522
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d523
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d524
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d525
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d526
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d527
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d528
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d529
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d530
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d531
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d532
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d533
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d534
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d535
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d536
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d537
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d538
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d539
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d540
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d541
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d542
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d543
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d544
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d545
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d546
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d547
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d548
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d549
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d550
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d551
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d552
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d553
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d554
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d555
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d556
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d557
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d558
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d559
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d560
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d561
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d562
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d563
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d564
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d565
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d566
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d567
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d568
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d569
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d570
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d571
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d572
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d573
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d574
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d575
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d576
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d577
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d578
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d579
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d580
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d581
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d582
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d583
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d584
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d585
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d586
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d587
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d588
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d589
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d590
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d591
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d592
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d593
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d594
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d595
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d596
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d597
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d598
|
||||
2026/10/11 04:22:00 kept what home-server says about the machine, and refused its account of declaration d599
|
||||
FAIL
|
||||
FAIL example.com/cap/early 0.003s
|
||||
FAIL
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user