mesh-tools
d699708fb3
mesh-build-go — the Go build module
Design 47, release R4c-3 (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470). The module that builds and checks Go
code on a build machine, as an account of its own (runs-as, no login shell, home at mode 0700), with the
toolchain installed as the packages it declares (go, git, openssh, postgresql) and the capability
user-namespaces. It claims no seat: it is assigned to the build machines (ADR 0469 §6) and runs dark until the
builder uses it.
The recipe
The module builds with the go its node has installed (GOTOOLCHAIN=local; ADR 0477 §1, issue 1572) and names no Go
version. The versions of the build nodes are agreed by the indexer, which builds a language's modules only while the
nodes that report agree (ADR 0465), and kept equal by the weekly upgrade of every node's packages (ADR 0477 §5).
cmd/mesh-build-go/recipe.go holds the steps' own version (RecipeSteps), the postgres major version the integration
level tests with, and the recipe version: a hash of those and of the nats-server library's version, read from the
program's build information, never of a Go version (ADR 0477 §3).
The guard and the health
At start, every ten minutes, before every step and at every health look, the module reads the installed Go (go version
with GOTOOLCHAIN=local; nothing is downloaded) and writes it with the recipe version to <node>.go of its
toolchains state. Whatever the version, the module is healthy and build_go_health (the module's runtime-health)
names it. Only a go that cannot be run refuses every step, the step ending refused on its key in steps with the
cause named, and the health is then unhealthy with the same text, so the controller raises the machine's condition.
Tools
build_go_status, build_go_health and build_go_fetch are served now; the other step tools (build_go_test,
build_go_digest, build_go_build) are added by their own issues to StepTools and run behind the guard: a long
step answers at once with its id and ends on <node>.<id> of steps (ADR 0469 §2); the digest answers in its call.
Fetch
build_go_fetch {repository, commit, ref?} fetches the commit over SSH as the module's account (mesh-forge is the
forge-account provision's ssh name, so the account's own ssh files and git rewrite are used) into a workspace of the
work directory, then runs go mod download and go mod verify in every directory with a go.mod that does not vendor,
with GOTOOLCHAIN=local, GOFLAGS=-mod=mod and the module's cache directory. Every command runs with an
environment built from scratch (StepRunner): PATH, HOME and what the step passes. Workspaces older than a day
are removed.
The account's ssh files (files/forge, files/known_hosts, files/gitconfig) are filled on the node by the
provisioner's client from the module's entry of forge-account (the manifest says moved, ADR 0466 §4, ADR 0400 §5).
The forge writes that entry only once the module has run and reported its key (ADR 0400 §2), so until then the client
says it waits and the files are not written: a fetch cannot reach the forge before them (issue 1537).
Test
build_go_test {workspace, level, packages?} runs the tests of a fetched workspace by Go's convention, one level at a
time. The unit level is go test -json over every package without the integration build tag, in each directory
with a go.mod, judged by its one bound: its running time, that of the slowest package (compile not counted), within
five seconds. The step ends passed when the level ran, and its result says whether the level did: passed, the
packages with their times, the tests that failed with their first lines of output, the five slowest tests, and why
when it failed (a failed test, a build that failed) or ran over a bound. passed is the tests' own verdict: a level over its bound still passes and says over_bound (and over_which, setup or run) with the slowest tests, and the checker judges the bounds (ADR 0453 §9). A step ends failed only when go test could not
run. A repository with no tests at the level has none, and the result says so: that is a pass. testdata/captured holds
the output of the real go for the fixture repositories in testdata/repos; the unit tests read it, and the
integration-tagged test runs the real go on the fixtures.
The integration level
build_go_test with level: integration runs go test -tags integration against a store and a bus started for the
run alone (novox/hq ADR 0470). The store is a postgres of the installed package: initdb and pg_ctl into
<work>/<workspace>/run-<id>/pg, as the module's account, listening on a unix socket there and on no TCP address, with
durability off; the tests get MESH_TEST_POSTGRES (the socket directory percent-encoded in the URL, because the
controller's tests cut the URL at its last slash). The bus is a nats-server from the Go library at the version this
module's go.mod names, run by this same binary started as mesh-build-go bus <dir> (bus.go: JetStream on, a
loopback port of its own, its address on the first line, stopped when its input closes); the tests get MESH_TEST_NATS.
Both run inside the step, are stopped, and the run's directory is removed whatever the tests did.
Setup (compiling the tests, the store, the bus) is judged within 30 s and the tests' running time (the wall time of
go test) within 60 s, apart; a level over either says over_bound and names the bound, with the slowest tests for the running
time. A machine whose installed postgres major is not the recipe's refuses this level, naming both; the unit level
still runs. The result names the postgres major and the nats-server version it ran against. The mesh's own store major
is not named: no state or grant gives it to this module (ADR 0470 §4 asks for it when it differs), so only the recipe's
is said until a source exists.
Digest
build_go_digest {workspace, level} answers in its call, for each package of the workspace that has tests at the level,
one digest of everything those tests read (ADR 0445 §5): the files, by content and under their path in the tree, of the
package and every package its test binary reaches (go list -test -deps) that lies in the module's own tree; the version
and checksum of every module outside it; the package's own testdata/ files; and the level, the build tags and, at the
integration level, the postgres major and the nats-server version. The Go version is not in it: the toolchain enters a
fingerprint once, as the version the build nodes agree on (ADR 0477 §3), and the standard library's packages are that
version's. A package with no tests at the level has no digest and is named in without_tests. A directory that vendors
is listed and tested from its vendor directory (-mod=vendor).
Build
build_go_build {workspace, module, artifact} builds one Go bundle artifact of a module of the workspace with the build
agent's recipe (mesh-controller's toolchain.go, the Go entry, copied in testdata/captured/toolchain-go-entry.txt, and
builder.go): CGO_ENABLED=0, GOFLAGS=-trimpath -buildvcs=false (with -mod=mod, or -mod=vendor for a directory
that vendors), go build -ldflags "-s -w -X main.builtFor=<system>" -o .mesh-build/<artifact>/<binary> ./<from> in the
module's directory, by the installed go (GOTOOLCHAIN=local) from the module cache the fetch filled and no network
(GOPROXY=off). The binary is named as the build agent names it. The output directory is packed with mesh-sdk's Pack,
as the build agent packs; two build nodes on the same toolchain give the same digest for the same tree (ADR 0477 §6). The step ends with the artifact, the binary, the digest, the size and
the path of the archive, which stays in the workspace's artifacts directory: publishing it is the builder's, in a later
release. An artifact that is not a Go bundle fails the step naming it.