mesh-tools
c7e892ca63
mesh/merge-gate head c7e892ca63b3 merged into trunk 82649bc770d1 — pass: builds mesh-build-go, mesh-build-node, sent nowhere; no bus step; every machine co…
mesh/repo-check head c7e892ca63b3 merged into trunk 82649bc770d1 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
94 lines
7.3 KiB
Markdown
94 lines
7.3 KiB
Markdown
# mesh-build-go — the Go build module
|
|
|
|
Design 47, release R4c-3 (novox/hq ADR 0405, ADR 0466 §1, ADR 0469, ADR 0470). The module that builds and checks Go
|
|
code on a build machine, as an account of its own (`runs-as`, no login shell, home at mode 0700), with the
|
|
toolchain installed as the packages it declares (`go`, `git`, `openssh`, `postgresql`) and the capability
|
|
`user-namespaces`. It claims no seat: it is assigned to the build machines (ADR 0469 §6) and runs dark until the
|
|
builder uses it.
|
|
|
|
## The recipe
|
|
|
|
`cmd/mesh-build-go/recipe.go` names the one Go version the module builds with (`RecipeGo`), the postgres major
|
|
version the integration level tests with, and the recipe's own version. The recipe version is a hash of those and of
|
|
the nats-server library's version, read from the program's build information, so a change of any of them moves it.
|
|
|
|
## The guard and the health
|
|
|
|
At start, every ten minutes, before every step and at every health look, the module reads the installed Go
|
|
(`go version`) and writes it with the recipe version to `<node>.go` of its `toolchains` state. A machine whose Go
|
|
is not the recipe's refuses every step, the step ending `refused` on its key in `steps` with both versions named,
|
|
and `build_go_health` (the module's `runtime-health`) answers unhealthy with the same text, so the controller raises
|
|
the machine's condition. Moving to a new Go is a change of `RecipeGo`.
|
|
|
|
## Tools
|
|
|
|
`build_go_status`, `build_go_health` and `build_go_fetch` are served now; the other step tools (`build_go_test`,
|
|
`build_go_digest`, `build_go_build`) are added by their own issues to `StepTools` and run behind the guard: a long
|
|
step answers at once with its id and ends on `<node>.<id>` of `steps` (ADR 0469 §2); the digest answers in its call.
|
|
|
|
## Fetch
|
|
|
|
`build_go_fetch {repository, commit, ref?}` fetches the commit over SSH as the module's account (`mesh-forge` is the
|
|
forge-account provision's ssh name, so the account's own ssh files and git rewrite are used) into a workspace of the
|
|
work directory, then runs `go mod download` and `go mod verify` in every directory with a `go.mod` that does not vendor,
|
|
with `GOTOOLCHAIN=local`, `GOFLAGS=-mod=mod` and the module's cache directory. Every command runs with an
|
|
environment built from scratch (`StepRunner`): `PATH`, `HOME` and what the step passes. Workspaces older than a day
|
|
are removed.
|
|
|
|
The account's ssh files (`files/forge`, `files/known_hosts`, `files/gitconfig`) are filled on the node by the
|
|
provisioner's client from the module's entry of `forge-account` (the manifest says `moved`, ADR 0466 §4, ADR 0400 §5).
|
|
The forge writes that entry only once the module has run and reported its key (ADR 0400 §2), so until then the client
|
|
says it waits and the files are not written: a fetch cannot reach the forge before them (issue 1537).
|
|
|
|
## Test
|
|
|
|
`build_go_test {workspace, level, packages?}` runs the tests of a fetched workspace by Go's convention, one level at a
|
|
time. The unit level is `go test -json` over every package without the `integration` build tag, in each directory
|
|
with a `go.mod`, judged by its one bound: its running time, that of the slowest package (compile not counted), within
|
|
five seconds. The step ends passed when the level ran, and its result says whether the level did: `passed`, the
|
|
packages with their times, the tests that failed with their first lines of output, the five slowest tests, and `why`
|
|
when it failed (a failed test, a build that failed) or ran over a bound. `passed` is the tests' own verdict: a level over its bound still passes and says `over_bound` (and `over_which`, setup or run) with the slowest tests, and the checker judges the bounds (ADR 0453 §9). A step ends failed only when go test could not
|
|
run. A repository with no tests at the level has none, and the result says so: that is a pass. `testdata/captured` holds
|
|
the output of the real go for the fixture repositories in `testdata/repos`; the unit tests read it, and the
|
|
integration-tagged test runs the real go on the fixtures.
|
|
|
|
## The integration level
|
|
|
|
`build_go_test` with `level: integration` runs `go test -tags integration` against a store and a bus started for the
|
|
run alone (novox/hq ADR 0470). The store is a postgres of the installed package: `initdb` and `pg_ctl` into
|
|
`<work>/<workspace>/run-<id>/pg`, as the module's account, listening on a unix socket there and on no TCP address, with
|
|
durability off; the tests get `MESH_TEST_POSTGRES` (the socket directory percent-encoded in the URL, because the
|
|
controller's tests cut the URL at its last slash). The bus is a nats-server from the Go library at the version this
|
|
module's `go.mod` names, run by this same binary started as `mesh-build-go bus <dir>` (`bus.go`: JetStream on, a
|
|
loopback port of its own, its address on the first line, stopped when its input closes); the tests get `MESH_TEST_NATS`.
|
|
Both run inside the step, are stopped, and the run's directory is removed whatever the tests did.
|
|
|
|
Setup (compiling the tests, the store, the bus) is judged within 30 s and the tests' running time (the wall time of
|
|
`go test`) within 60 s, apart; a level over either says `over_bound` and names the bound, with the slowest tests for the running
|
|
time. A machine whose installed postgres major is not the recipe's refuses this level, naming both; the unit level
|
|
still runs. The result names the postgres major and the nats-server version it ran against. The mesh's own store major
|
|
is not named: no state or grant gives it to this module (ADR 0470 §4 asks for it when it differs), so only the recipe's
|
|
is said until a source exists.
|
|
|
|
## Digest
|
|
|
|
`build_go_digest {workspace, level}` answers in its call, for each package of the workspace that has tests at the level,
|
|
one digest of everything those tests read (ADR 0445 §5): the files, by content and under their path in the tree, of the
|
|
package and every package its test binary reaches (`go list -test -deps`) that lies in the module's own tree; the version
|
|
and checksum of every module outside it; the package's own `testdata/` files; and the Go version, the level, the build tags
|
|
and, at the integration level, the postgres major and the nats-server version. Packages of the standard library are the Go
|
|
version's. A package with no tests at the level has no digest and is named in `without_tests`. A directory that vendors
|
|
is listed and tested from its vendor directory (`-mod=vendor`).
|
|
|
|
## Build
|
|
|
|
`build_go_build {workspace, module, artifact}` builds one Go bundle artifact of a module of the workspace with the build
|
|
agent's recipe (mesh-controller's `toolchain.go`, the Go entry, copied in `testdata/captured/toolchain-go-entry.txt`, and
|
|
`builder.go`): `CGO_ENABLED=0`, `GOFLAGS=-trimpath -buildvcs=false` (with `-mod=mod`, or `-mod=vendor` for a directory
|
|
that vendors), `go build -ldflags "-s -w -X main.builtFor=<system>" -o .mesh-build/<artifact>/<binary> ./<from>` in the
|
|
module's directory, by the installed Go (`GOTOOLCHAIN=local`) from the module cache the fetch filled and no network
|
|
(`GOPROXY=off`). The binary is named as the build agent names it. The output directory is packed with mesh-sdk's `Pack`,
|
|
so the digest is the build agent's for the same tree. The step ends with the artifact, the binary, the digest, the size and
|
|
the path of the archive, which stays in the workspace's `artifacts` directory: publishing it is the builder's, in a later
|
|
release. An artifact that is not a Go bundle fails the step naming it.
|