Add the Go build module's main, recipe, guard, key keeper, runner, store and bus, and its README (issues 1483, 1484, 1485, 1486), part 7 of 20
This commit is contained in:
@@ -0,0 +1,88 @@
|
||||
# mesh-build-go — the Go build module
|
||||
|
||||
Design 47, release R4c-3 (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470). The module that builds and checks Go
|
||||
code on a build machine, as an account of its own (`runs-as`, no login shell, home at mode 0700), with the
|
||||
toolchain installed as the packages it declares (`go`, `git`, `openssh`, `postgresql`) and the capability
|
||||
`user-namespaces`. It claims no seat: it is assigned to the build machines (ADR 0469 §6) and runs dark until the
|
||||
builder uses it.
|
||||
|
||||
## The recipe
|
||||
|
||||
`cmd/mesh-build-go/recipe.go` names the one Go version the module builds with (`RecipeGo`), the postgres major
|
||||
version the integration level tests with, and the recipe's own version. The recipe version is a hash of those and of
|
||||
the nats-server library's version, read from the program's build information, so a change of any of them moves it.
|
||||
|
||||
## The guard and the health
|
||||
|
||||
At start, every ten minutes, before every step and at every health look, the module reads the installed Go
|
||||
(`go version`) and writes it with the recipe version to `<node>.go` of its `toolchains` state. A machine whose Go
|
||||
is not the recipe's refuses every step, the step ending `refused` on its key in `steps` with both versions named,
|
||||
and `build_go_health` (the module's `runtime-health`) answers unhealthy with the same text, so the controller raises
|
||||
the machine's condition. Moving to a new Go is a change of `RecipeGo`.
|
||||
|
||||
## Tools
|
||||
|
||||
`build_go_status`, `build_go_health` and `build_go_fetch` are served now; the other step tools (`build_go_test`,
|
||||
`build_go_digest`, `build_go_build`) are added by their own issues to `StepTools` and run behind the guard: a long
|
||||
step answers at once with its id and ends on `<node>.<id>` of `steps` (ADR 0469 §2); the digest answers in its call.
|
||||
|
||||
## Fetch
|
||||
|
||||
`build_go_fetch {repository, commit, ref?}` fetches the commit over SSH as the module's account (`mesh-forge` is the
|
||||
forge-account provision's ssh name, so the account's own ssh files and git rewrite are used) into a workspace of the
|
||||
work directory, then runs `go mod download` and `go mod verify` in every directory with a `go.mod` that does not vendor,
|
||||
with `GOTOOLCHAIN=local`, `GOFLAGS=-mod=mod` and the module's cache directory. Every command runs with an
|
||||
environment built from scratch (`StepRunner`): `PATH`, `HOME` and what the step passes. Workspaces older than a day
|
||||
are removed.
|
||||
|
||||
## Test
|
||||
|
||||
`build_go_test {workspace, level, packages?}` runs the tests of a fetched workspace by Go's convention, one level at a
|
||||
time. The unit level is `go test -json` over every package without the `integration` build tag, in each directory
|
||||
with a `go.mod`, judged by its one bound: its running time, that of the slowest package (compile not counted), within
|
||||
five seconds. The step ends passed when the level ran, and its result says whether the level did: `passed`, the
|
||||
packages with their times, the tests that failed with their first lines of output, the five slowest tests, and `why`
|
||||
when it failed (a failed test, a build that failed, a bound kept or not). A step ends failed only when go test could not
|
||||
run. A repository with no tests at the level has none, and the result says so: that is a pass. `testdata/captured` holds
|
||||
the output of the real go for the fixture repositories in `testdata/repos`; the unit tests read it, and the
|
||||
integration-tagged test runs the real go on the fixtures.
|
||||
|
||||
## The integration level
|
||||
|
||||
`build_go_test` with `level: integration` runs `go test -tags integration` against a store and a bus started for the
|
||||
run alone (novox/hq ADR 0470). The store is a postgres of the installed package: `initdb` and `pg_ctl` into
|
||||
`<work>/<workspace>/run-<id>/pg`, as the module's account, listening on a unix socket there and on no TCP address, with
|
||||
durability off; the tests get `MESH_TEST_POSTGRES` (the socket directory percent-encoded in the URL, because the
|
||||
controller's tests cut the URL at its last slash). The bus is a nats-server from the Go library at the version this
|
||||
module's `go.mod` names, run by this same binary started as `mesh-build-go bus <dir>` (`bus.go`: JetStream on, a
|
||||
loopback port of its own, its address on the first line, stopped when its input closes); the tests get `MESH_TEST_NATS`.
|
||||
Both run inside the step, are stopped, and the run's directory is removed whatever the tests did.
|
||||
|
||||
Setup (compiling the tests, the store, the bus) is judged within 30 s and the tests' running time (the wall time of
|
||||
`go test`) within 60 s, apart; a level over either fails and names the bound, with the slowest tests for the running
|
||||
time. A machine whose installed postgres major is not the recipe's refuses this level, naming both; the unit level
|
||||
still runs. The result names the postgres major and the nats-server version it ran against. The mesh's own store major
|
||||
is not named: no state or grant gives it to this module (ADR 0470 §4 asks for it when it differs), so only the recipe's
|
||||
is said until a source exists.
|
||||
|
||||
## Digest
|
||||
|
||||
`build_go_digest {workspace, level}` answers in its call, for each package of the workspace that has tests at the level,
|
||||
one digest of everything those tests read (ADR 0445 §5): the files, by content and under their path in the tree, of the
|
||||
package and every package its test binary reaches (`go list -test -deps`) that lies in the module's own tree; the version
|
||||
and checksum of every module outside it; the package's own `testdata/` files; and the Go version, the level, the build tags
|
||||
and, at the integration level, the postgres major and the nats-server version. Packages of the standard library are the Go
|
||||
version's. A package with no tests at the level has no digest and is named in `without_tests`. A directory that vendors
|
||||
is listed and tested from its vendor directory (`-mod=vendor`).
|
||||
|
||||
## Build
|
||||
|
||||
`build_go_build {workspace, module, artifact}` builds one Go bundle artifact of a module of the workspace with the build
|
||||
agent's recipe (mesh-controller's `toolchain.go`, the Go entry, copied in `testdata/captured/toolchain-go-entry.txt`, and
|
||||
`builder.go`): `CGO_ENABLED=0`, `GOFLAGS=-trimpath -buildvcs=false` (with `-mod=mod`, or `-mod=vendor` for a directory
|
||||
that vendors), `go build -ldflags "-s -w -X main.builtFor=<system>" -o .mesh-build/<artifact>/<binary> ./<from>` in the
|
||||
module's directory, by the installed Go (`GOTOOLCHAIN=local`) from the module cache the fetch filled and no network
|
||||
(`GOPROXY=off`). The binary is named as the build agent names it. The output directory is packed with mesh-sdk's `Pack`,
|
||||
so the digest is the build agent's for the same tree. The step ends with the artifact, the binary, the digest, the size and
|
||||
the path of the archive, which stays in the workspace's `artifacts` directory: publishing it is the builder's, in a later
|
||||
release. An artifact that is not a Go bundle fails the step naming it.
|
||||
@@ -0,0 +1,116 @@
|
||||
package main
|
||||
|
||||
// The integration level's throwaway bus (novox/hq ADR 0470 §2; issue 1486): the module's own program runs a
|
||||
// nats-server from the Go library, at the one version this module's go.mod names, with JetStream on, its store in the
|
||||
// run's directory and a loopback port of its own. The program is this binary started again as `mesh-build-go bus
|
||||
// <dir>`, so the bus is built from the same go.mod as the version the recipe names (ADR 0470 §4) and no second program
|
||||
// has to be built and shipped; the tools bundle never serves a bus, and a bus never serves tools. It prints its
|
||||
// address on one line and stops when its standard input closes.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
)
|
||||
|
||||
// busStartBound is how long a bus is given to answer its address.
|
||||
const busStartBound = 30 * time.Second
|
||||
|
||||
// busArg is the first argument that makes this binary a bus.
|
||||
const busArg = "bus"
|
||||
|
||||
// runBus is the bus program: args are the directory of its store. It returns the exit status.
|
||||
func runBus(args []string) int {
|
||||
if len(args) != 1 {
|
||||
fmt.Fprintln(os.Stderr, "usage: mesh-build-go bus <store directory>")
|
||||
return 2
|
||||
}
|
||||
s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: args[0], NoLog: true, NoSigs: true})
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
go s.Start()
|
||||
if !s.ReadyForConnections(busStartBound) {
|
||||
fmt.Fprintln(os.Stderr, "the bus was not ready within", busStartBound)
|
||||
s.Shutdown()
|
||||
return 1
|
||||
}
|
||||
fmt.Println(s.ClientURL())
|
||||
done := make(chan struct{})
|
||||
go func() { _, _ = io.Copy(io.Discard, os.Stdin); close(done) }()
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
|
||||
select {
|
||||
case <-done:
|
||||
case <-sig:
|
||||
}
|
||||
s.Shutdown()
|
||||
s.WaitForShutdown()
|
||||
return 0
|
||||
}
|
||||
|
||||
// startBus starts the bus of a run as a process of this binary under the step's environment, and answers its address.
|
||||
func (m *Module) startBus(ctx context.Context, dir string) (string, func(), error) {
|
||||
if m.StartBus != nil {
|
||||
return m.StartBus(ctx, dir)
|
||||
}
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
cmd := exec.Command(exe, busArg, dir)
|
||||
cmd.Env = baseEnv(m.Home)
|
||||
in, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
var errb strings.Builder
|
||||
cmd.Stderr = &errb
|
||||
if err := cmd.Start(); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
stop := func() {
|
||||
_ = in.Close()
|
||||
exited := make(chan struct{})
|
||||
go func() { _ = cmd.Wait(); close(exited) }()
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(10 * time.Second):
|
||||
_ = cmd.Process.Kill()
|
||||
<-exited
|
||||
}
|
||||
}
|
||||
line := make(chan string, 1)
|
||||
go func() {
|
||||
l, _ := bufio.NewReader(out).ReadString('\n')
|
||||
line <- strings.TrimSpace(l)
|
||||
}()
|
||||
select {
|
||||
case url := <-line:
|
||||
if !strings.HasPrefix(url, "nats://") {
|
||||
stop()
|
||||
return "", nil, fmt.Errorf("the bus did not answer an address: %s", firstLine(errb.String()+" "+url))
|
||||
}
|
||||
return url, stop, nil
|
||||
case <-time.After(busStartBound + 5*time.Second):
|
||||
stop()
|
||||
return "", nil, fmt.Errorf("the bus did not answer an address within %s", busStartBound)
|
||||
case <-ctx.Done():
|
||||
stop()
|
||||
return "", nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package main
|
||||
|
||||
// The module's own forge key (novox/hq ADR 0466 §2, issue 1484): mesh-sdk's key keeper, kind builder, for the
|
||||
// module's account in its home. It makes ~/.ssh/mesh_ed25519 when it is missing and writes one entry to this
|
||||
// module's `keys` state at <node>.<account>, which gitea's provider reads to register the key on the forge user
|
||||
// that reads and pushes nothing (ADR 0466 §3).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/forgekey"
|
||||
)
|
||||
|
||||
// keeperFor is this module's keeper: go/forgekey's, of kind builder, for account in home on node, whose entry lies
|
||||
// at <node>.<account> of the keys state.
|
||||
func keeperFor(home, node, account string, run forgekey.Runner, state forgekey.KeyPutter) *forgekey.KeyKeeper {
|
||||
return &forgekey.KeyKeeper{Home: home, Node: node, Account: account, Kind: forgekey.KindBuilder, Run: run, State: state, Now: time.Now}
|
||||
}
|
||||
|
||||
// startKeeper runs the keeper when the runtime launched this process for a node: outside a launch — a build, a
|
||||
// check — there is no state to put the entry in.
|
||||
func startKeeper(ctx context.Context) {
|
||||
node := strings.TrimSpace(os.Getenv("MESH_NODE"))
|
||||
u, err := user.Current()
|
||||
if node == "" || err != nil || u.HomeDir == "" {
|
||||
return
|
||||
}
|
||||
k := keeperFor(u.HomeDir, node, u.Username, forgekey.ExecRunner, keysState{})
|
||||
k.Keep(ctx, func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-build-go] "+f+"\n", a...) })
|
||||
}
|
||||
|
||||
type keysState struct{}
|
||||
|
||||
func (keysState) Put(key string, value any) (uint64, error) {
|
||||
return stdio.State("keys").Put(key, value)
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// mesh-build-go: the Go build module (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470; design 47 R4c-3, issue
|
||||
// 1483). A Go tools bundle the node's runtime launches as the module's own account (runs-as). It names the one Go
|
||||
// toolchain version it builds with, writes the version it finds to <node>.go of its `toolchains` state at start and
|
||||
// before every step, and refuses every step on a node that has another: its health is then unhealthy, so the
|
||||
// controller raises the node's condition. What it says, it says on stderr; stdout is the MCP channel.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"strings"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
)
|
||||
|
||||
// The directories the module's manifest places outside its home (ADR 0466 §1): the workspaces, and the Go module
|
||||
// and build caches.
|
||||
const (
|
||||
WorkDir = "/var/lib/mesh-build-go-work"
|
||||
CacheDir = "/var/lib/mesh-build-go-cache"
|
||||
)
|
||||
|
||||
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-build-go] "+format+"\n", a...) }
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == busArg {
|
||||
os.Exit(runBus(os.Args[2:]))
|
||||
}
|
||||
ctx := context.Background()
|
||||
m := &Module{
|
||||
Node: strings.TrimSpace(os.Getenv("MESH_NODE")),
|
||||
Recipe: Current(),
|
||||
Probe: ProbeGo,
|
||||
Toolchains: stdio.State(delivery.ToolchainsState).Put,
|
||||
Steps: stdio.State(delivery.StepsState).Put,
|
||||
Ctx: ctx,
|
||||
Say: logf,
|
||||
Work: WorkDir,
|
||||
Cache: CacheDir,
|
||||
}
|
||||
if u, err := user.Current(); err == nil {
|
||||
m.Home = u.HomeDir
|
||||
}
|
||||
m.Run = StepRunner(m.Home)
|
||||
// Outside a launch by the runtime — a build, a check — there is no node and no state to write.
|
||||
if m.Node != "" {
|
||||
go m.Start(ctx)
|
||||
}
|
||||
go startKeeper(ctx)
|
||||
if err := stdio.Serve("", m.Tools(m.StepTools())); err != nil {
|
||||
logf("%v", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buildstep"
|
||||
"git.novox.be/novox/mesh-sdk/go/delivery"
|
||||
"git.novox.be/novox/mesh-sdk/go/forgekey"
|
||||
"git.novox.be/novox/mesh-sdk/go/health"
|
||||
)
|
||||
|
||||
// CheckRound is how often the module reads the installed toolchain again, beside the check before every step and
|
||||
// the one every health look makes.
|
||||
const CheckRound = 10 * time.Minute
|
||||
|
||||
// Module is the Go build module's one running state: the node it runs on, its recipe, the toolchain guard and the
|
||||
// states it writes (toolchains, steps).
|
||||
type Module struct {
|
||||
Node string
|
||||
Recipe Recipe
|
||||
// Probe reads the installed Go version (ProbeGo).
|
||||
Probe func(ctx context.Context) (string, error)
|
||||
// Toolchains and Steps are the Put of the states of those names.
|
||||
Toolchains, Steps buildstep.Put
|
||||
// Ctx is the process's context, which a step runs under (the tool call's is cut at 30 s, the step's is not).
|
||||
Ctx context.Context
|
||||
Say func(format string, a ...any)
|
||||
// Run runs the commands of a step (StepRunner); Work holds the workspaces and Cache the module and build caches,
|
||||
// both outside the account's home (ADR 0466 §1); Home is the account's home.
|
||||
Run forgekey.Runner
|
||||
Home, Work, Cache string
|
||||
// UnitBound is the unit level's bound; zero is UnitBound. A test lowers it to see a level over its bound.
|
||||
UnitBound time.Duration
|
||||
// IntegrationBounds are the integration level's bounds; zero values are IntegrationSetup and IntegrationRun.
|
||||
IntegrationBounds Bounds
|
||||
// Now is the clock the levels' times are read from; time.Now when nil.
|
||||
Now func() time.Time
|
||||
// StartBus starts the throwaway bus of a run in dir and answers its address and what stops it (the module's own
|
||||
// program, StartBus in bus.go, when nil).
|
||||
StartBus func(ctx context.Context, dir string) (url string, stop func(), err error)
|
||||
}
|
||||
|
||||
func (m *Module) guard() buildstep.Guard {
|
||||
return buildstep.Guard{Language: "go", Want: m.Recipe.Go, Recipe: m.Recipe.Version(), Probe: m.Probe, Node: m.Node, Put: m.Toolchains}
|
||||
}
|
||||
|
||||
// Check reads the installed toolchain, writes <node>.go of the toolchains state and answers nil, or the refusal
|
||||
// (buildstep.Guard.Check). It runs at start, before every step, at every health look and every CheckRound.
|
||||
func (m *Module) Check(ctx context.Context) error {
|
||||
return m.guard().Check(ctx)
|
||||
}
|
||||
|
||||
// Start checks once, as the module starts, and then every CheckRound until ctx ends.
|
||||
func (m *Module) Start(ctx context.Context) {
|
||||
for {
|
||||
if err := m.Check(ctx); err != nil && m.Say != nil {
|
||||
m.Say("the toolchain: %v", err)
|
||||
}
|
||||
m.CleanWorkspaces(time.Now())
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(CheckRound):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Health is the module's health (the runtime-health tool, ADR 0366): unhealthy while the toolchain installed is not
|
||||
// the recipe's, saying both versions, so the controller raises the node's condition. It looks again each time it is
|
||||
// asked.
|
||||
func (m *Module) Health(ctx context.Context) health.Answer {
|
||||
err := m.Check(ctx)
|
||||
switch {
|
||||
case err == nil:
|
||||
return health.Healthy(fmt.Sprintf("%s installed, as the recipe %s builds with", m.Recipe.Go, m.Recipe.Version()))
|
||||
default:
|
||||
return health.Unhealthy(err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// Long is a tool that runs a long step (ADR 0469 §2): it answers at once with the step's id, and the step ends on
|
||||
// its key in the steps state. The guard runs inside the step, before run, so a node that refuses ends every step
|
||||
// refused with both versions named.
|
||||
func (m *Module) Long(tool string, run func(ctx context.Context, args map[string]any) (any, error)) func(map[string]any) (any, error) {
|
||||
return func(args map[string]any) (any, error) {
|
||||
id, err := buildstep.Start(m.Ctx, m.Steps, m.Node, tool, func(ctx context.Context) (any, error) {
|
||||
if err := m.Check(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return run(ctx, args)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
key, _ := delivery.StepKey(m.Node, id)
|
||||
return map[string]any{"step": id, "key": key, "state": delivery.StepRunning}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Short is a tool that answers in its own call; the guard runs before it, and a refusal is the call's error.
|
||||
func (m *Module) Short(run func(ctx context.Context, args map[string]any) (any, error)) func(map[string]any) (any, error) {
|
||||
return func(args map[string]any) (any, error) {
|
||||
if err := m.Check(m.Ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return run(m.Ctx, args)
|
||||
}
|
||||
}
|
||||
|
||||
// Status is what the module reports of itself: the node, the recipe, the toolchain found and why it refuses, if it
|
||||
// does.
|
||||
func (m *Module) Status(ctx context.Context) map[string]any {
|
||||
err := m.Check(ctx)
|
||||
installed, _ := m.Probe(ctx)
|
||||
out := map[string]any{
|
||||
"node": m.Node, "recipe": m.Recipe.Version(), "go": m.Recipe.Go, "installed": installed,
|
||||
"postgres-major": m.Recipe.PostgresMajor, "nats-server": m.Recipe.NatsServer, "refusing": err != nil,
|
||||
}
|
||||
if err != nil {
|
||||
out["why"] = err.Error()
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ProbeTimeout bounds `go version`.
|
||||
const ProbeTimeout = 15 * time.Second
|
||||
|
||||
// ProbeGo reads the installed Go version: the third word `go version` prints ("go version go1.27.1 linux/amd64"),
|
||||
// as the recipe's RecipeGo is written. GOTOOLCHAIN=local keeps an older Go from fetching another toolchain: what is
|
||||
// installed is what is read.
|
||||
func ProbeGo(ctx context.Context) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, ProbeTimeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "go", "version")
|
||||
cmd.Env = append(os.Environ(), "GOTOOLCHAIN=local")
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return parseGoVersion(string(out))
|
||||
}
|
||||
|
||||
func parseGoVersion(out string) (string, error) {
|
||||
f := strings.Fields(out)
|
||||
if len(f) < 3 || f[0] != "go" || f[1] != "version" || !strings.HasPrefix(f[2], "go") {
|
||||
return "", fmt.Errorf("`go version` answered %q", strings.TrimSpace(out))
|
||||
}
|
||||
return f[2], nil
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package main
|
||||
|
||||
// The recipe of the Go build module (novox/hq ADR 0405 §1 and §7, ADR 0469 §1, ADR 0470 §4; issue 1483): the one
|
||||
// Go toolchain version it builds with, the one postgres major version its integration level tests with, and the
|
||||
// recipe's own version. A node whose installed Go differs from the recipe's refuses every step (guard.go), so all
|
||||
// builders run this version or none builds.
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"runtime/debug"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
)
|
||||
|
||||
// RecipeGo is the Go version the recipe builds with: the text `go version` prints for it, which is what the guard
|
||||
// compares and what the indexer reads from <node>.go of the toolchains state.
|
||||
const RecipeGo = "go1.27.1"
|
||||
|
||||
// PostgresMajor is the postgres major version the integration level tests with (ADR 0470 §4).
|
||||
const PostgresMajor = 18
|
||||
|
||||
// RecipeSteps is the recipe's own version: changed whenever the steps of fetch, test or build change in a way
|
||||
// that can change what they produce (ADR 0405 §7). It is one of the four values the recipe version hashes.
|
||||
const RecipeSteps = "1"
|
||||
|
||||
// natsModule is the library the integration level's bus is started from (ADR 0470 §2).
|
||||
const natsModule = "github.com/nats-io/nats-server/v2"
|
||||
|
||||
// Recipe is every value that makes up the recipe version.
|
||||
type Recipe struct {
|
||||
// Steps is RecipeSteps.
|
||||
Steps string
|
||||
// Go is the toolchain's version as `go version` prints it.
|
||||
Go string
|
||||
// PostgresMajor is the integration level's postgres major version.
|
||||
PostgresMajor int
|
||||
// NatsServer is the nats-server library's version, from the program's build information.
|
||||
NatsServer string
|
||||
}
|
||||
|
||||
// Current is the recipe of this build of the module.
|
||||
func Current() Recipe {
|
||||
return Recipe{Steps: RecipeSteps, Go: RecipeGo, PostgresMajor: PostgresMajor, NatsServer: natsServerVersion()}
|
||||
}
|
||||
|
||||
// Version is the recipe version: a hash of the recipe's own version, the Go version, the postgres major version and
|
||||
// the nats-server version, so a change of any of the four moves it and with it every fingerprint that holds it
|
||||
// (ADR 0405 §7, ADR 0465 §2). The fields are named in the hashed text, so no two recipes share a text.
|
||||
func (r Recipe) Version() string {
|
||||
h := sha256.New()
|
||||
for _, f := range [][2]string{
|
||||
{"steps", r.Steps}, {"go", r.Go}, {"postgres", strconv.Itoa(r.PostgresMajor)}, {"nats-server", r.NatsServer},
|
||||
} {
|
||||
h.Write([]byte(f[0] + "=" + f[1] + "\n"))
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))[:16]
|
||||
}
|
||||
|
||||
// natsServerVersion is the version of the nats-server library this program was built with: its build information
|
||||
// names it as the module's go.mod does, and has no second source (ADR 0470 §4). A binary built without build
|
||||
// information (a stripped test binary) falls back to the library's own version constant, which is the same text.
|
||||
func natsServerVersion() string {
|
||||
if info, ok := debug.ReadBuildInfo(); ok {
|
||||
if v := natsVersionOf(info); v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return server.VERSION
|
||||
}
|
||||
|
||||
// natsVersionOf is the version build information gives the nats-server library, "v" stripped, or "" when it lists
|
||||
// none. A replaced library is the replacement's version.
|
||||
func natsVersionOf(info *debug.BuildInfo) string {
|
||||
for _, d := range info.Deps {
|
||||
if d.Path != natsModule {
|
||||
continue
|
||||
}
|
||||
if d.Replace != nil {
|
||||
d = d.Replace
|
||||
}
|
||||
return strings.TrimPrefix(d.Version, "v")
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/forgekey"
|
||||
)
|
||||
|
||||
// StepRunner runs the commands of a build step with an environment built from scratch (novox/hq ADR 0466 §4, issue
|
||||
// 1485): PATH, HOME (the account's own home, where its ssh files and git configuration are) and a fixed locale, plus
|
||||
// the words the caller passes, and nothing else of this process's environment — which holds the module's bus
|
||||
// credentials and might hold a password or a token. It is mesh-sdk's clean runner (ExecRunnerClean, issue 1497) with
|
||||
// that base given; a command is bound by its context alone (the bound 0), so a fetch may outlast a tool call's thirty
|
||||
// seconds (the step has its own context).
|
||||
func StepRunner(home string) forgekey.Runner {
|
||||
run := forgekey.ExecRunnerClean(0)
|
||||
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
|
||||
return run(ctx, append(baseEnv(home), env...), name, args...)
|
||||
}
|
||||
}
|
||||
|
||||
// defaultPath is where a step looks for git, ssh and go when this process was given no PATH.
|
||||
const defaultPath = "/usr/local/bin:/usr/bin:/bin"
|
||||
|
||||
func baseEnv(home string) []string {
|
||||
path := os.Getenv("PATH")
|
||||
if path == "" {
|
||||
path = defaultPath
|
||||
}
|
||||
return []string{"PATH=" + path, "HOME=" + home, "LC_ALL=C"}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
// The integration level's throwaway store (novox/hq ADR 0470 §1; issue 1486): one postgres server of the installed
|
||||
// package for each run, `initdb` into the run's own directory, listening on a unix socket in that directory and on no
|
||||
// TCP address, as the module's account, with durability off as today's check sets it.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/buildstep"
|
||||
)
|
||||
|
||||
// storeStopBound is how long stopping the store is given: it runs after the level, whatever the level did.
|
||||
const storeStopBound = 30 * time.Second
|
||||
|
||||
var postgresVersion = regexp.MustCompile(`PostgreSQL\)\s+(\d+)`)
|
||||
|
||||
// postgresMajor is the major version of the installed postgres: `postgres --version` ("postgres (PostgreSQL) 18.6").
|
||||
func (m *Module) postgresMajor(ctx context.Context) (int, error) {
|
||||
r := m.Run(ctx, nil, "postgres", "--version")
|
||||
if r.Status != 0 || r.Err != "" {
|
||||
return 0, fmt.Errorf("`postgres --version`: %s", firstLine(r.Err+" "+r.Stderr))
|
||||
}
|
||||
f := postgresVersion.FindStringSubmatch(r.Stdout)
|
||||
if f == nil {
|
||||
return 0, fmt.Errorf("`postgres --version` answered %q", firstLine(r.Stdout))
|
||||
}
|
||||
return strconv.Atoi(f[1])
|
||||
}
|
||||
|
||||
// checkPostgres refuses the integration level when the installed postgres major is not the recipe's, naming both
|
||||
// (ADR 0470 §4); the unit level is not refused. It answers the major it found.
|
||||
func (m *Module) checkPostgres(ctx context.Context) (int, error) {
|
||||
major, err := m.postgresMajor(ctx)
|
||||
if err != nil {
|
||||
return 0, &buildstep.Refusal{Text: fmt.Sprintf("the installed postgres version could not be read: %v", err), Want: strconv.Itoa(m.Recipe.PostgresMajor)}
|
||||
}
|
||||
if major != m.Recipe.PostgresMajor {
|
||||
return major, &buildstep.Refusal{Text: fmt.Sprintf("postgres %d installed, the integration level is built for postgres %d", major, m.Recipe.PostgresMajor),
|
||||
Want: strconv.Itoa(m.Recipe.PostgresMajor), Have: strconv.Itoa(major)}
|
||||
}
|
||||
return major, nil
|
||||
}
|
||||
|
||||
// storeSettings are the server's own settings: no TCP address, a socket directory, and no durability.
|
||||
func storeOptions(socket string) string {
|
||||
return "-k " + socket + " -c listen_addresses='' -c fsync=off -c synchronous_commit=off -c full_page_writes=off"
|
||||
}
|
||||
|
||||
// startStore makes and starts the run's postgres in run/pg and answers the address tests are given as
|
||||
// MESH_TEST_POSTGRES, and what stops it. The address names the socket directory percent-encoded, as the controller's
|
||||
// tests cut the URL at its last slash to name their own databases.
|
||||
func (m *Module) startStore(ctx context.Context, run string) (string, func(), error) {
|
||||
data := filepath.Join(run, "pg", "data")
|
||||
socket := filepath.Join(run, "pg", "socket")
|
||||
if err := os.MkdirAll(socket, 0o700); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
stop := func() {
|
||||
sctx, cancel := context.WithTimeout(context.Background(), storeStopBound)
|
||||
defer cancel()
|
||||
m.Run(sctx, nil, "pg_ctl", "-D", data, "-m", "immediate", "-w", "stop")
|
||||
}
|
||||
if r := m.Run(ctx, nil, "initdb", "-D", data, "-U", "postgres", "--auth=trust", "-E", "UTF8", "--no-locale"); r.Status != 0 || r.Err != "" {
|
||||
return "", nil, fmt.Errorf("initdb: %s", firstLine(r.Err+" "+r.Stderr+" "+r.Stdout))
|
||||
}
|
||||
if r := m.Run(ctx, nil, "pg_ctl", "-D", data, "-l", filepath.Join(run, "pg", "log"), "-w", "-t", "30", "-o", storeOptions(socket), "start"); r.Status != 0 || r.Err != "" {
|
||||
stop() // a server that was half started is stopped
|
||||
return "", nil, fmt.Errorf("pg_ctl start: %s", firstLine(r.Err+" "+r.Stderr+" "+r.Stdout))
|
||||
}
|
||||
return "postgres://postgres@/postgres?host=" + url.QueryEscape(socket) + "&sslmode=disable", stop, nil
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// The tools this module serves. A tool is named as the module is: build_go_ and its verb (a tool name has no dot).
|
||||
const (
|
||||
StatusTool = "build_go_status"
|
||||
HealthTool = "build_go_health"
|
||||
)
|
||||
|
||||
// StepTool is one step tool of the module: fetch, test, digest or build. Long is a step that answers at once and
|
||||
// ends on its key (ADR 0469 §2); the others answer in their call. Both run behind the toolchain guard.
|
||||
type StepTool struct {
|
||||
Name string
|
||||
Description string
|
||||
Input map[string]any
|
||||
Long bool
|
||||
Run func(ctx context.Context, args map[string]any) (any, error)
|
||||
}
|
||||
|
||||
// StepTools are the module's step tools. Each step's issue adds its own.
|
||||
func (m *Module) StepTools() []StepTool {
|
||||
return []StepTool{m.fetchTool(), m.testTool(), m.digestTool(), m.buildTool()}
|
||||
}
|
||||
|
||||
// Tools is every tool the module serves: its step tools behind the guard, its status and its health.
|
||||
func (m *Module) Tools(steps []StepTool) []stdio.Tool {
|
||||
tools := []stdio.Tool{}
|
||||
for _, s := range steps {
|
||||
t := stdio.Tool{Name: s.Name, Description: s.Description, Input: s.Input}
|
||||
if s.Long {
|
||||
t.Run = m.Long(s.Name, s.Run)
|
||||
} else {
|
||||
t.Run = m.Short(s.Run)
|
||||
}
|
||||
tools = append(tools, t)
|
||||
}
|
||||
return append(tools,
|
||||
stdio.Tool{Name: StatusTool,
|
||||
Description: "What the Go build module is on this machine: the Go version its recipe builds with and the one installed, " +
|
||||
"the recipe version, the postgres major version and the nats-server version the integration level uses, and " +
|
||||
"why it refuses to build if it does. Read-only.",
|
||||
Run: func(map[string]any) (any, error) { return m.Status(m.Ctx), nil }},
|
||||
stdio.Tool{Name: HealthTool,
|
||||
Description: "How the Go build module is: unhealthy while the Go installed on this machine is not the version its " +
|
||||
"recipe builds with, saying both versions; healthy otherwise. Read-only.",
|
||||
Run: func(map[string]any) (any, error) { return m.Health(m.Ctx), nil }},
|
||||
)
|
||||
}
|
||||
Reference in New Issue
Block a user