diff --git a/modules/mesh-build-go/README.md b/modules/mesh-build-go/README.md new file mode 100644 index 00000000..45afecf2 --- /dev/null +++ b/modules/mesh-build-go/README.md @@ -0,0 +1,88 @@ +# mesh-build-go — the Go build module + +Design 47, release R4c-3 (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470). The module that builds and checks Go +code on a build machine, as an account of its own (`runs-as`, no login shell, home at mode 0700), with the +toolchain installed as the packages it declares (`go`, `git`, `openssh`, `postgresql`) and the capability +`user-namespaces`. It claims no seat: it is assigned to the build machines (ADR 0469 §6) and runs dark until the +builder uses it. + +## The recipe + +`cmd/mesh-build-go/recipe.go` names the one Go version the module builds with (`RecipeGo`), the postgres major +version the integration level tests with, and the recipe's own version. The recipe version is a hash of those and of +the nats-server library's version, read from the program's build information, so a change of any of them moves it. + +## The guard and the health + +At start, every ten minutes, before every step and at every health look, the module reads the installed Go +(`go version`) and writes it with the recipe version to `.go` of its `toolchains` state. A machine whose Go +is not the recipe's refuses every step, the step ending `refused` on its key in `steps` with both versions named, +and `build_go_health` (the module's `runtime-health`) answers unhealthy with the same text, so the controller raises +the machine's condition. Moving to a new Go is a change of `RecipeGo`. + +## Tools + +`build_go_status`, `build_go_health` and `build_go_fetch` are served now; the other step tools (`build_go_test`, +`build_go_digest`, `build_go_build`) are added by their own issues to `StepTools` and run behind the guard: a long +step answers at once with its id and ends on `.` of `steps` (ADR 0469 §2); the digest answers in its call. + +## Fetch + +`build_go_fetch {repository, commit, ref?}` fetches the commit over SSH as the module's account (`mesh-forge` is the +forge-account provision's ssh name, so the account's own ssh files and git rewrite are used) into a workspace of the +work directory, then runs `go mod download` and `go mod verify` in every directory with a `go.mod` that does not vendor, +with `GOTOOLCHAIN=local`, `GOFLAGS=-mod=mod` and the module's cache directory. Every command runs with an +environment built from scratch (`StepRunner`): `PATH`, `HOME` and what the step passes. Workspaces older than a day +are removed. + +## Test + +`build_go_test {workspace, level, packages?}` runs the tests of a fetched workspace by Go's convention, one level at a +time. The unit level is `go test -json` over every package without the `integration` build tag, in each directory +with a `go.mod`, judged by its one bound: its running time, that of the slowest package (compile not counted), within +five seconds. The step ends passed when the level ran, and its result says whether the level did: `passed`, the +packages with their times, the tests that failed with their first lines of output, the five slowest tests, and `why` +when it failed (a failed test, a build that failed, a bound kept or not). A step ends failed only when go test could not +run. A repository with no tests at the level has none, and the result says so: that is a pass. `testdata/captured` holds +the output of the real go for the fixture repositories in `testdata/repos`; the unit tests read it, and the +integration-tagged test runs the real go on the fixtures. + +## The integration level + +`build_go_test` with `level: integration` runs `go test -tags integration` against a store and a bus started for the +run alone (novox/hq ADR 0470). The store is a postgres of the installed package: `initdb` and `pg_ctl` into +`//run-/pg`, as the module's account, listening on a unix socket there and on no TCP address, with +durability off; the tests get `MESH_TEST_POSTGRES` (the socket directory percent-encoded in the URL, because the +controller's tests cut the URL at its last slash). The bus is a nats-server from the Go library at the version this +module's `go.mod` names, run by this same binary started as `mesh-build-go bus ` (`bus.go`: JetStream on, a +loopback port of its own, its address on the first line, stopped when its input closes); the tests get `MESH_TEST_NATS`. +Both run inside the step, are stopped, and the run's directory is removed whatever the tests did. + +Setup (compiling the tests, the store, the bus) is judged within 30 s and the tests' running time (the wall time of +`go test`) within 60 s, apart; a level over either fails and names the bound, with the slowest tests for the running +time. A machine whose installed postgres major is not the recipe's refuses this level, naming both; the unit level +still runs. The result names the postgres major and the nats-server version it ran against. The mesh's own store major +is not named: no state or grant gives it to this module (ADR 0470 §4 asks for it when it differs), so only the recipe's +is said until a source exists. + +## Digest + +`build_go_digest {workspace, level}` answers in its call, for each package of the workspace that has tests at the level, +one digest of everything those tests read (ADR 0445 §5): the files, by content and under their path in the tree, of the +package and every package its test binary reaches (`go list -test -deps`) that lies in the module's own tree; the version +and checksum of every module outside it; the package's own `testdata/` files; and the Go version, the level, the build tags +and, at the integration level, the postgres major and the nats-server version. Packages of the standard library are the Go +version's. A package with no tests at the level has no digest and is named in `without_tests`. A directory that vendors +is listed and tested from its vendor directory (`-mod=vendor`). + +## Build + +`build_go_build {workspace, module, artifact}` builds one Go bundle artifact of a module of the workspace with the build +agent's recipe (mesh-controller's `toolchain.go`, the Go entry, copied in `testdata/captured/toolchain-go-entry.txt`, and +`builder.go`): `CGO_ENABLED=0`, `GOFLAGS=-trimpath -buildvcs=false` (with `-mod=mod`, or `-mod=vendor` for a directory +that vendors), `go build -ldflags "-s -w -X main.builtFor=" -o .mesh-build// ./` in the +module's directory, by the installed Go (`GOTOOLCHAIN=local`) from the module cache the fetch filled and no network +(`GOPROXY=off`). The binary is named as the build agent names it. The output directory is packed with mesh-sdk's `Pack`, +so the digest is the build agent's for the same tree. The step ends with the artifact, the binary, the digest, the size and +the path of the archive, which stays in the workspace's `artifacts` directory: publishing it is the builder's, in a later +release. An artifact that is not a Go bundle fails the step naming it. diff --git a/modules/mesh-build-go/cmd/mesh-build-go/bus.go b/modules/mesh-build-go/cmd/mesh-build-go/bus.go new file mode 100644 index 00000000..8aa52548 --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/bus.go @@ -0,0 +1,116 @@ +package main + +// The integration level's throwaway bus (novox/hq ADR 0470 §2; issue 1486): the module's own program runs a +// nats-server from the Go library, at the one version this module's go.mod names, with JetStream on, its store in the +// run's directory and a loopback port of its own. The program is this binary started again as `mesh-build-go bus +// `, so the bus is built from the same go.mod as the version the recipe names (ADR 0470 §4) and no second program +// has to be built and shipped; the tools bundle never serves a bus, and a bus never serves tools. It prints its +// address on one line and stops when its standard input closes. + +import ( + "bufio" + "context" + "fmt" + "io" + "os" + "os/exec" + "os/signal" + "strings" + "syscall" + "time" + + "github.com/nats-io/nats-server/v2/server" +) + +// busStartBound is how long a bus is given to answer its address. +const busStartBound = 30 * time.Second + +// busArg is the first argument that makes this binary a bus. +const busArg = "bus" + +// runBus is the bus program: args are the directory of its store. It returns the exit status. +func runBus(args []string) int { + if len(args) != 1 { + fmt.Fprintln(os.Stderr, "usage: mesh-build-go bus ") + return 2 + } + s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: args[0], NoLog: true, NoSigs: true}) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + go s.Start() + if !s.ReadyForConnections(busStartBound) { + fmt.Fprintln(os.Stderr, "the bus was not ready within", busStartBound) + s.Shutdown() + return 1 + } + fmt.Println(s.ClientURL()) + done := make(chan struct{}) + go func() { _, _ = io.Copy(io.Discard, os.Stdin); close(done) }() + sig := make(chan os.Signal, 1) + signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT) + select { + case <-done: + case <-sig: + } + s.Shutdown() + s.WaitForShutdown() + return 0 +} + +// startBus starts the bus of a run as a process of this binary under the step's environment, and answers its address. +func (m *Module) startBus(ctx context.Context, dir string) (string, func(), error) { + if m.StartBus != nil { + return m.StartBus(ctx, dir) + } + exe, err := os.Executable() + if err != nil { + return "", nil, err + } + cmd := exec.Command(exe, busArg, dir) + cmd.Env = baseEnv(m.Home) + in, err := cmd.StdinPipe() + if err != nil { + return "", nil, err + } + out, err := cmd.StdoutPipe() + if err != nil { + return "", nil, err + } + var errb strings.Builder + cmd.Stderr = &errb + if err := cmd.Start(); err != nil { + return "", nil, err + } + stop := func() { + _ = in.Close() + exited := make(chan struct{}) + go func() { _ = cmd.Wait(); close(exited) }() + select { + case <-exited: + case <-time.After(10 * time.Second): + _ = cmd.Process.Kill() + <-exited + } + } + line := make(chan string, 1) + go func() { + l, _ := bufio.NewReader(out).ReadString('\n') + line <- strings.TrimSpace(l) + }() + select { + case url := <-line: + if !strings.HasPrefix(url, "nats://") { + stop() + return "", nil, fmt.Errorf("the bus did not answer an address: %s", firstLine(errb.String()+" "+url)) + } + return url, stop, nil + case <-time.After(busStartBound + 5*time.Second): + stop() + return "", nil, fmt.Errorf("the bus did not answer an address within %s", busStartBound) + case <-ctx.Done(): + stop() + return "", nil, ctx.Err() + } +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/keys.go b/modules/mesh-build-go/cmd/mesh-build-go/keys.go new file mode 100644 index 00000000..fb93a0de --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/keys.go @@ -0,0 +1,42 @@ +package main + +// The module's own forge key (novox/hq ADR 0466 §2, issue 1484): mesh-sdk's key keeper, kind builder, for the +// module's account in its home. It makes ~/.ssh/mesh_ed25519 when it is missing and writes one entry to this +// module's `keys` state at ., which gitea's provider reads to register the key on the forge user +// that reads and pushes nothing (ADR 0466 §3). + +import ( + "context" + "fmt" + "os" + "os/user" + "strings" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" + "git.novox.be/novox/mesh-sdk/go/forgekey" +) + +// keeperFor is this module's keeper: go/forgekey's, of kind builder, for account in home on node, whose entry lies +// at . of the keys state. +func keeperFor(home, node, account string, run forgekey.Runner, state forgekey.KeyPutter) *forgekey.KeyKeeper { + return &forgekey.KeyKeeper{Home: home, Node: node, Account: account, Kind: forgekey.KindBuilder, Run: run, State: state, Now: time.Now} +} + +// startKeeper runs the keeper when the runtime launched this process for a node: outside a launch — a build, a +// check — there is no state to put the entry in. +func startKeeper(ctx context.Context) { + node := strings.TrimSpace(os.Getenv("MESH_NODE")) + u, err := user.Current() + if node == "" || err != nil || u.HomeDir == "" { + return + } + k := keeperFor(u.HomeDir, node, u.Username, forgekey.ExecRunner, keysState{}) + k.Keep(ctx, func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-build-go] "+f+"\n", a...) }) +} + +type keysState struct{} + +func (keysState) Put(key string, value any) (uint64, error) { + return stdio.State("keys").Put(key, value) +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/main.go b/modules/mesh-build-go/cmd/mesh-build-go/main.go new file mode 100644 index 00000000..12317389 --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/main.go @@ -0,0 +1,57 @@ +// mesh-build-go: the Go build module (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470; design 47 R4c-3, issue +// 1483). A Go tools bundle the node's runtime launches as the module's own account (runs-as). It names the one Go +// toolchain version it builds with, writes the version it finds to .go of its `toolchains` state at start and +// before every step, and refuses every step on a node that has another: its health is then unhealthy, so the +// controller raises the node's condition. What it says, it says on stderr; stdout is the MCP channel. +package main + +import ( + "context" + "fmt" + "os" + "os/user" + "strings" + + stdio "git.novox.be/novox/mesh-sdk/go" + "git.novox.be/novox/mesh-sdk/go/delivery" +) + +// The directories the module's manifest places outside its home (ADR 0466 §1): the workspaces, and the Go module +// and build caches. +const ( + WorkDir = "/var/lib/mesh-build-go-work" + CacheDir = "/var/lib/mesh-build-go-cache" +) + +func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-build-go] "+format+"\n", a...) } + +func main() { + if len(os.Args) > 1 && os.Args[1] == busArg { + os.Exit(runBus(os.Args[2:])) + } + ctx := context.Background() + m := &Module{ + Node: strings.TrimSpace(os.Getenv("MESH_NODE")), + Recipe: Current(), + Probe: ProbeGo, + Toolchains: stdio.State(delivery.ToolchainsState).Put, + Steps: stdio.State(delivery.StepsState).Put, + Ctx: ctx, + Say: logf, + Work: WorkDir, + Cache: CacheDir, + } + if u, err := user.Current(); err == nil { + m.Home = u.HomeDir + } + m.Run = StepRunner(m.Home) + // Outside a launch by the runtime — a build, a check — there is no node and no state to write. + if m.Node != "" { + go m.Start(ctx) + } + go startKeeper(ctx) + if err := stdio.Serve("", m.Tools(m.StepTools())); err != nil { + logf("%v", err) + os.Exit(1) + } +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/module.go b/modules/mesh-build-go/cmd/mesh-build-go/module.go new file mode 100644 index 00000000..61f8ce2f --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/module.go @@ -0,0 +1,125 @@ +package main + +import ( + "context" + "fmt" + "time" + + "git.novox.be/novox/mesh-sdk/go/buildstep" + "git.novox.be/novox/mesh-sdk/go/delivery" + "git.novox.be/novox/mesh-sdk/go/forgekey" + "git.novox.be/novox/mesh-sdk/go/health" +) + +// CheckRound is how often the module reads the installed toolchain again, beside the check before every step and +// the one every health look makes. +const CheckRound = 10 * time.Minute + +// Module is the Go build module's one running state: the node it runs on, its recipe, the toolchain guard and the +// states it writes (toolchains, steps). +type Module struct { + Node string + Recipe Recipe + // Probe reads the installed Go version (ProbeGo). + Probe func(ctx context.Context) (string, error) + // Toolchains and Steps are the Put of the states of those names. + Toolchains, Steps buildstep.Put + // Ctx is the process's context, which a step runs under (the tool call's is cut at 30 s, the step's is not). + Ctx context.Context + Say func(format string, a ...any) + // Run runs the commands of a step (StepRunner); Work holds the workspaces and Cache the module and build caches, + // both outside the account's home (ADR 0466 §1); Home is the account's home. + Run forgekey.Runner + Home, Work, Cache string + // UnitBound is the unit level's bound; zero is UnitBound. A test lowers it to see a level over its bound. + UnitBound time.Duration + // IntegrationBounds are the integration level's bounds; zero values are IntegrationSetup and IntegrationRun. + IntegrationBounds Bounds + // Now is the clock the levels' times are read from; time.Now when nil. + Now func() time.Time + // StartBus starts the throwaway bus of a run in dir and answers its address and what stops it (the module's own + // program, StartBus in bus.go, when nil). + StartBus func(ctx context.Context, dir string) (url string, stop func(), err error) +} + +func (m *Module) guard() buildstep.Guard { + return buildstep.Guard{Language: "go", Want: m.Recipe.Go, Recipe: m.Recipe.Version(), Probe: m.Probe, Node: m.Node, Put: m.Toolchains} +} + +// Check reads the installed toolchain, writes .go of the toolchains state and answers nil, or the refusal +// (buildstep.Guard.Check). It runs at start, before every step, at every health look and every CheckRound. +func (m *Module) Check(ctx context.Context) error { + return m.guard().Check(ctx) +} + +// Start checks once, as the module starts, and then every CheckRound until ctx ends. +func (m *Module) Start(ctx context.Context) { + for { + if err := m.Check(ctx); err != nil && m.Say != nil { + m.Say("the toolchain: %v", err) + } + m.CleanWorkspaces(time.Now()) + select { + case <-ctx.Done(): + return + case <-time.After(CheckRound): + } + } +} + +// Health is the module's health (the runtime-health tool, ADR 0366): unhealthy while the toolchain installed is not +// the recipe's, saying both versions, so the controller raises the node's condition. It looks again each time it is +// asked. +func (m *Module) Health(ctx context.Context) health.Answer { + err := m.Check(ctx) + switch { + case err == nil: + return health.Healthy(fmt.Sprintf("%s installed, as the recipe %s builds with", m.Recipe.Go, m.Recipe.Version())) + default: + return health.Unhealthy(err.Error()) + } +} + +// Long is a tool that runs a long step (ADR 0469 §2): it answers at once with the step's id, and the step ends on +// its key in the steps state. The guard runs inside the step, before run, so a node that refuses ends every step +// refused with both versions named. +func (m *Module) Long(tool string, run func(ctx context.Context, args map[string]any) (any, error)) func(map[string]any) (any, error) { + return func(args map[string]any) (any, error) { + id, err := buildstep.Start(m.Ctx, m.Steps, m.Node, tool, func(ctx context.Context) (any, error) { + if err := m.Check(ctx); err != nil { + return nil, err + } + return run(ctx, args) + }) + if err != nil { + return nil, err + } + key, _ := delivery.StepKey(m.Node, id) + return map[string]any{"step": id, "key": key, "state": delivery.StepRunning}, nil + } +} + +// Short is a tool that answers in its own call; the guard runs before it, and a refusal is the call's error. +func (m *Module) Short(run func(ctx context.Context, args map[string]any) (any, error)) func(map[string]any) (any, error) { + return func(args map[string]any) (any, error) { + if err := m.Check(m.Ctx); err != nil { + return nil, err + } + return run(m.Ctx, args) + } +} + +// Status is what the module reports of itself: the node, the recipe, the toolchain found and why it refuses, if it +// does. +func (m *Module) Status(ctx context.Context) map[string]any { + err := m.Check(ctx) + installed, _ := m.Probe(ctx) + out := map[string]any{ + "node": m.Node, "recipe": m.Recipe.Version(), "go": m.Recipe.Go, "installed": installed, + "postgres-major": m.Recipe.PostgresMajor, "nats-server": m.Recipe.NatsServer, "refusing": err != nil, + } + if err != nil { + out["why"] = err.Error() + } + return out +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/probe.go b/modules/mesh-build-go/cmd/mesh-build-go/probe.go new file mode 100644 index 00000000..838ece4d --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/probe.go @@ -0,0 +1,36 @@ +package main + +import ( + "context" + "fmt" + "os" + "os/exec" + "strings" + "time" +) + +// ProbeTimeout bounds `go version`. +const ProbeTimeout = 15 * time.Second + +// ProbeGo reads the installed Go version: the third word `go version` prints ("go version go1.27.1 linux/amd64"), +// as the recipe's RecipeGo is written. GOTOOLCHAIN=local keeps an older Go from fetching another toolchain: what is +// installed is what is read. +func ProbeGo(ctx context.Context) (string, error) { + ctx, cancel := context.WithTimeout(ctx, ProbeTimeout) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "version") + cmd.Env = append(os.Environ(), "GOTOOLCHAIN=local") + out, err := cmd.Output() + if err != nil { + return "", err + } + return parseGoVersion(string(out)) +} + +func parseGoVersion(out string) (string, error) { + f := strings.Fields(out) + if len(f) < 3 || f[0] != "go" || f[1] != "version" || !strings.HasPrefix(f[2], "go") { + return "", fmt.Errorf("`go version` answered %q", strings.TrimSpace(out)) + } + return f[2], nil +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/recipe.go b/modules/mesh-build-go/cmd/mesh-build-go/recipe.go new file mode 100644 index 00000000..0303045e --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/recipe.go @@ -0,0 +1,87 @@ +package main + +// The recipe of the Go build module (novox/hq ADR 0405 §1 and §7, ADR 0469 §1, ADR 0470 §4; issue 1483): the one +// Go toolchain version it builds with, the one postgres major version its integration level tests with, and the +// recipe's own version. A node whose installed Go differs from the recipe's refuses every step (guard.go), so all +// builders run this version or none builds. + +import ( + "crypto/sha256" + "encoding/hex" + "runtime/debug" + "strconv" + "strings" + + "github.com/nats-io/nats-server/v2/server" +) + +// RecipeGo is the Go version the recipe builds with: the text `go version` prints for it, which is what the guard +// compares and what the indexer reads from .go of the toolchains state. +const RecipeGo = "go1.27.1" + +// PostgresMajor is the postgres major version the integration level tests with (ADR 0470 §4). +const PostgresMajor = 18 + +// RecipeSteps is the recipe's own version: changed whenever the steps of fetch, test or build change in a way +// that can change what they produce (ADR 0405 §7). It is one of the four values the recipe version hashes. +const RecipeSteps = "1" + +// natsModule is the library the integration level's bus is started from (ADR 0470 §2). +const natsModule = "github.com/nats-io/nats-server/v2" + +// Recipe is every value that makes up the recipe version. +type Recipe struct { + // Steps is RecipeSteps. + Steps string + // Go is the toolchain's version as `go version` prints it. + Go string + // PostgresMajor is the integration level's postgres major version. + PostgresMajor int + // NatsServer is the nats-server library's version, from the program's build information. + NatsServer string +} + +// Current is the recipe of this build of the module. +func Current() Recipe { + return Recipe{Steps: RecipeSteps, Go: RecipeGo, PostgresMajor: PostgresMajor, NatsServer: natsServerVersion()} +} + +// Version is the recipe version: a hash of the recipe's own version, the Go version, the postgres major version and +// the nats-server version, so a change of any of the four moves it and with it every fingerprint that holds it +// (ADR 0405 §7, ADR 0465 §2). The fields are named in the hashed text, so no two recipes share a text. +func (r Recipe) Version() string { + h := sha256.New() + for _, f := range [][2]string{ + {"steps", r.Steps}, {"go", r.Go}, {"postgres", strconv.Itoa(r.PostgresMajor)}, {"nats-server", r.NatsServer}, + } { + h.Write([]byte(f[0] + "=" + f[1] + "\n")) + } + return hex.EncodeToString(h.Sum(nil))[:16] +} + +// natsServerVersion is the version of the nats-server library this program was built with: its build information +// names it as the module's go.mod does, and has no second source (ADR 0470 §4). A binary built without build +// information (a stripped test binary) falls back to the library's own version constant, which is the same text. +func natsServerVersion() string { + if info, ok := debug.ReadBuildInfo(); ok { + if v := natsVersionOf(info); v != "" { + return v + } + } + return server.VERSION +} + +// natsVersionOf is the version build information gives the nats-server library, "v" stripped, or "" when it lists +// none. A replaced library is the replacement's version. +func natsVersionOf(info *debug.BuildInfo) string { + for _, d := range info.Deps { + if d.Path != natsModule { + continue + } + if d.Replace != nil { + d = d.Replace + } + return strings.TrimPrefix(d.Version, "v") + } + return "" +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/runner.go b/modules/mesh-build-go/cmd/mesh-build-go/runner.go new file mode 100644 index 00000000..691ef6f5 --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/runner.go @@ -0,0 +1,32 @@ +package main + +import ( + "context" + "os" + + "git.novox.be/novox/mesh-sdk/go/forgekey" +) + +// StepRunner runs the commands of a build step with an environment built from scratch (novox/hq ADR 0466 §4, issue +// 1485): PATH, HOME (the account's own home, where its ssh files and git configuration are) and a fixed locale, plus +// the words the caller passes, and nothing else of this process's environment — which holds the module's bus +// credentials and might hold a password or a token. It is mesh-sdk's clean runner (ExecRunnerClean, issue 1497) with +// that base given; a command is bound by its context alone (the bound 0), so a fetch may outlast a tool call's thirty +// seconds (the step has its own context). +func StepRunner(home string) forgekey.Runner { + run := forgekey.ExecRunnerClean(0) + return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran { + return run(ctx, append(baseEnv(home), env...), name, args...) + } +} + +// defaultPath is where a step looks for git, ssh and go when this process was given no PATH. +const defaultPath = "/usr/local/bin:/usr/bin:/bin" + +func baseEnv(home string) []string { + path := os.Getenv("PATH") + if path == "" { + path = defaultPath + } + return []string{"PATH=" + path, "HOME=" + home, "LC_ALL=C"} +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/store.go b/modules/mesh-build-go/cmd/mesh-build-go/store.go new file mode 100644 index 00000000..032d64d5 --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/store.go @@ -0,0 +1,79 @@ +package main + +// The integration level's throwaway store (novox/hq ADR 0470 §1; issue 1486): one postgres server of the installed +// package for each run, `initdb` into the run's own directory, listening on a unix socket in that directory and on no +// TCP address, as the module's account, with durability off as today's check sets it. + +import ( + "context" + "fmt" + "net/url" + "os" + "path/filepath" + "regexp" + "strconv" + "time" + + "git.novox.be/novox/mesh-sdk/go/buildstep" +) + +// storeStopBound is how long stopping the store is given: it runs after the level, whatever the level did. +const storeStopBound = 30 * time.Second + +var postgresVersion = regexp.MustCompile(`PostgreSQL\)\s+(\d+)`) + +// postgresMajor is the major version of the installed postgres: `postgres --version` ("postgres (PostgreSQL) 18.6"). +func (m *Module) postgresMajor(ctx context.Context) (int, error) { + r := m.Run(ctx, nil, "postgres", "--version") + if r.Status != 0 || r.Err != "" { + return 0, fmt.Errorf("`postgres --version`: %s", firstLine(r.Err+" "+r.Stderr)) + } + f := postgresVersion.FindStringSubmatch(r.Stdout) + if f == nil { + return 0, fmt.Errorf("`postgres --version` answered %q", firstLine(r.Stdout)) + } + return strconv.Atoi(f[1]) +} + +// checkPostgres refuses the integration level when the installed postgres major is not the recipe's, naming both +// (ADR 0470 §4); the unit level is not refused. It answers the major it found. +func (m *Module) checkPostgres(ctx context.Context) (int, error) { + major, err := m.postgresMajor(ctx) + if err != nil { + return 0, &buildstep.Refusal{Text: fmt.Sprintf("the installed postgres version could not be read: %v", err), Want: strconv.Itoa(m.Recipe.PostgresMajor)} + } + if major != m.Recipe.PostgresMajor { + return major, &buildstep.Refusal{Text: fmt.Sprintf("postgres %d installed, the integration level is built for postgres %d", major, m.Recipe.PostgresMajor), + Want: strconv.Itoa(m.Recipe.PostgresMajor), Have: strconv.Itoa(major)} + } + return major, nil +} + +// storeSettings are the server's own settings: no TCP address, a socket directory, and no durability. +func storeOptions(socket string) string { + return "-k " + socket + " -c listen_addresses='' -c fsync=off -c synchronous_commit=off -c full_page_writes=off" +} + +// startStore makes and starts the run's postgres in run/pg and answers the address tests are given as +// MESH_TEST_POSTGRES, and what stops it. The address names the socket directory percent-encoded, as the controller's +// tests cut the URL at its last slash to name their own databases. +func (m *Module) startStore(ctx context.Context, run string) (string, func(), error) { + data := filepath.Join(run, "pg", "data") + socket := filepath.Join(run, "pg", "socket") + if err := os.MkdirAll(socket, 0o700); err != nil { + return "", nil, err + } + stop := func() { + sctx, cancel := context.WithTimeout(context.Background(), storeStopBound) + defer cancel() + m.Run(sctx, nil, "pg_ctl", "-D", data, "-m", "immediate", "-w", "stop") + } + if r := m.Run(ctx, nil, "initdb", "-D", data, "-U", "postgres", "--auth=trust", "-E", "UTF8", "--no-locale"); r.Status != 0 || r.Err != "" { + return "", nil, fmt.Errorf("initdb: %s", firstLine(r.Err+" "+r.Stderr+" "+r.Stdout)) + } + if r := m.Run(ctx, nil, "pg_ctl", "-D", data, "-l", filepath.Join(run, "pg", "log"), "-w", "-t", "30", "-o", storeOptions(socket), "start"); r.Status != 0 || r.Err != "" { + stop() // a server that was half started is stopped + return "", nil, fmt.Errorf("pg_ctl start: %s", firstLine(r.Err+" "+r.Stderr+" "+r.Stdout)) + } + return "postgres://postgres@/postgres?host=" + url.QueryEscape(socket) + "&sslmode=disable", stop, nil +} diff --git a/modules/mesh-build-go/cmd/mesh-build-go/tools.go b/modules/mesh-build-go/cmd/mesh-build-go/tools.go new file mode 100644 index 00000000..91c2fd1a --- /dev/null +++ b/modules/mesh-build-go/cmd/mesh-build-go/tools.go @@ -0,0 +1,53 @@ +package main + +import ( + "context" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// The tools this module serves. A tool is named as the module is: build_go_ and its verb (a tool name has no dot). +const ( + StatusTool = "build_go_status" + HealthTool = "build_go_health" +) + +// StepTool is one step tool of the module: fetch, test, digest or build. Long is a step that answers at once and +// ends on its key (ADR 0469 §2); the others answer in their call. Both run behind the toolchain guard. +type StepTool struct { + Name string + Description string + Input map[string]any + Long bool + Run func(ctx context.Context, args map[string]any) (any, error) +} + +// StepTools are the module's step tools. Each step's issue adds its own. +func (m *Module) StepTools() []StepTool { + return []StepTool{m.fetchTool(), m.testTool(), m.digestTool(), m.buildTool()} +} + +// Tools is every tool the module serves: its step tools behind the guard, its status and its health. +func (m *Module) Tools(steps []StepTool) []stdio.Tool { + tools := []stdio.Tool{} + for _, s := range steps { + t := stdio.Tool{Name: s.Name, Description: s.Description, Input: s.Input} + if s.Long { + t.Run = m.Long(s.Name, s.Run) + } else { + t.Run = m.Short(s.Run) + } + tools = append(tools, t) + } + return append(tools, + stdio.Tool{Name: StatusTool, + Description: "What the Go build module is on this machine: the Go version its recipe builds with and the one installed, " + + "the recipe version, the postgres major version and the nats-server version the integration level uses, and " + + "why it refuses to build if it does. Read-only.", + Run: func(map[string]any) (any, error) { return m.Status(m.Ctx), nil }}, + stdio.Tool{Name: HealthTool, + Description: "How the Go build module is: unhealthy while the Go installed on this machine is not the version its " + + "recipe builds with, saying both versions; healthy otherwise. Read-only.", + Run: func(map[string]any) (any, error) { return m.Health(m.Ctx), nil }}, + ) +}