jschoubben
b9cdca78b0
build_node_fetch takes buildstep.FetchRequest (dir and paths narrow the workspace) and an
optional package {repository, commit, ref}: the group's package member, built and packed
shut off and published nowhere. Each member's lock decides its SDK: the group's packed
file when the version and integrity are the lock's, else the registry's (integrity
checked); a version the registry lacks fails waits for: @novox/mesh-sdk@<version>. A
package artifact is built by its own script and packed with npm pack, shut off; only a
trunk build publishes it, from this process with the write credential (never in a
workspace): absent is published, the same file is done, another is version taken:. The
registry configuration file now lies outside every workspace.
93 lines
9.0 KiB
Markdown
93 lines
9.0 KiB
Markdown
# mesh-build-node — the Node and TypeScript build module
|
|
|
|
Design 47, release R4c-3 (novox/hq ADR 0405, ADR 0466 §1, ADR 0469 §1 and §4). The module that builds and checks
|
|
TypeScript code on a build machine, as an account of its own (`runs-as`, no login shell, home at mode 0700), with the
|
|
packages it declares (`git`, `openssh`; `nodejs` and `npm` come from the `nodejs` module, which this module requires as `node-toolchain`) and the capability `user-namespaces`. The bundle is Go
|
|
(`cmd/mesh-build-node`). It claims no seat: it is assigned to the build machines (ADR 0469 clause 6) and runs dark until the
|
|
builder uses it. Its forge key and ssh files are the same as mesh-build-go's (issue 1484), filled on the node by the
|
|
provisioner's client (issue 1537).
|
|
|
|
## The recipe
|
|
|
|
The module builds with the `nodejs` and `npm` its node has installed and names no Node version (ADR 0477 §1, issue
|
|
1572); the versions of the build nodes are agreed by the indexer (ADR 0465) and kept equal by the weekly upgrade of
|
|
every node's packages (ADR 0477 §5). `recipe.go` holds the recipe's own version (`RecipeSteps`). The recipe version is a
|
|
hash of that and of the bytes of the lock file `toolchain/package-lock.json`, never of a Node version (ADR 0477 §3), so
|
|
one changed byte of the lock or another recipe moves it, and typescript and esbuild change only when the lock does.
|
|
|
|
## The toolchain directory
|
|
|
|
`toolchain/package.json` and its committed `package-lock.json` name typescript 5.9.3, esbuild 0.25.12, @types/node 22.20.1 (the types tsc checks node imports against),
|
|
@novox/mesh-sdk 0.1.14 and nats 2.29.3. The mesh unpacks both, as the archive artifact `toolchain`, into `<home>/toolchain/` (resource `toolchain-dir`; a
|
|
manifest names no host, and the lock does, so it is not a template); the program reads the lock from there. When `<cache dir>/toolchain/<recipe version>` is missing
|
|
the module runs `npm ci --ignore-scripts` there once (a directory without its `.installed` mark counts as missing).
|
|
|
|
**How the registry is reached** (the choice of issue 1488): the lock holds @novox/mesh-sdk at the forge's public
|
|
address, and npm 12 installs a locked tarball only from the registry it is configured with. The module therefore
|
|
requires `npm-package-registry` (read access, provided by gitea) and, like the lab module, takes its binding and sealed
|
|
secret as files in its home. The install gets a private npm config file with the `@novox` scope at the binding's
|
|
`public-root` (else scheme, address, port and npm path) and the bound account's credential, and removes it afterwards;
|
|
every other package resolves from the public registry. This is the way today's build agent names it.
|
|
|
|
The runtime tree for bundles with externals (the pruned sdk and nats today in `/app/runtime`) is not installed
|
|
separately: sdk and nats are in this directory's `node_modules`, and the build step (issue 1490) takes them from there.
|
|
|
|
## The guard and the health
|
|
|
|
At start, every ten minutes, before every step and at every health look, the module reads the lock again and the
|
|
installed Node (`node --version`) and writes it with the recipe version to `<node>.typescript` of its `toolchains`
|
|
state. Whatever the version, the module is healthy and `build_node_health` (the module's `runtime-health`) names it. Only a
|
|
`node` that cannot be run, or a lock that cannot be read, refuses every step, the step ending `refused` on its key in
|
|
`steps` with the cause named; the health is then unhealthy with the same text, so the controller raises the machine's
|
|
condition. A step runs only after the toolchain directory is installed.
|
|
|
|
Three read tools answer what the module stored in its own states, without a grant (issue 1562): `build_node_step {id}` is a step's stored value (state, reason, result), `build_node_steps {limit}` this machine's latest steps without their results, and `build_node_toolchains` every `<node>.<language>` key of `toolchains`.
|
|
|
|
## Tools
|
|
|
|
`build_node_status` and `build_node_health` answer in their call. The step tools run behind the guard and the
|
|
toolchain install: a long step answers at once with its id and ends on `<node>.<id>` of `steps` (ADR 0469 §2).
|
|
|
|
- `build_node_fetch` {repository, commit, ref} (issue 1489): the commit over ssh as the module's account
|
|
(`buildstep.FetchCommit`, remote `ssh://mesh-forge/<owner>/<name>.git`) into `<work>/<id>/tree`, then for each
|
|
TypeScript module (a module.json with a typescript artifact) the packages its package.json names beyond the SDK,
|
|
`typescript` and `@types/*` (the toolchain carries those): none installs nothing; some need a package-lock.json, and
|
|
are installed as the build agent does (scratch copy without the SDK,
|
|
devDependencies and scripts; `npm ci --omit=dev --omit=peer --ignore-scripts --no-bin-links`; copies of the SDK
|
|
removed). The mesh's registry reaches `npm ci` through a private config file (`--userconfig`), so
|
|
other `@novox/` packages install. A module with packages and no lock does not fail the fetch: its result lists it
|
|
under `refused` with the reason, and a build of that bundle fails naming it (issue 1459). Every command runs through `forgekey.ExecRunnerClean(0)` with PATH, HOME, the locale and its own
|
|
words, nothing of this process's environment. Workspaces older than a day are removed at each fetch.
|
|
The arguments are mesh-sdk's `buildstep.FetchRequest` (ADR 0499 §4): with `dir` and `paths` (from the builder, never
|
|
from a manifest) the workspace is narrowed to those paths, the manifest and the lock files before anything else. A
|
|
member is built against the SDK version its own lock names (the lead's decision L10 of R4e, ADR 0500 §5-§6): the
|
|
fetch places that version in the module's `node_modules/@novox/mesh-sdk`, from the group's package when the call
|
|
names one (`package: {repository, commit, ref}`: fetched, its dependencies installed from its own lock, built and
|
|
packed shut off, published nowhere) and its version and integrity are the lock's, else from the registry (integrity
|
|
checked); a version the registry lacks fails `waits for: @novox/mesh-sdk@<version>` (the builder queues the key), or,
|
|
with a group package of another version, fails naming both. A module whose lock names no SDK keeps the toolchain's.
|
|
A module whose artifact is a `package` gets its dependencies, devDependencies included, from its own lock instead.
|
|
- `build_node_test` {workspace, module, level} (issue 1491): the unit level is the type-check (the toolchain's tsc, the
|
|
package's `tsconfig.json` or the build agent's flags over its .ts files) and the `test` script; the integration level
|
|
is the `test:integration` script, the type-check being its setup. Bounds (the Go module's; ADR 0445 names none for
|
|
TypeScript): unit 5 s, integration 30 s setup and 60 s running. The test script finds `tsc` and `esbuild` on its PATH through `<work>/<id>/bin` (shell scripts calling the toolchain's). A level over its bound says `over_bound` and passes;
|
|
a failed one finishes the step with `passed` false and the output (the step fails only when the tests could not be run). Neither script nor .ts files: no tests. The toolchain makes no bin
|
|
links, so a script runs `node` and files, not `tsc` by name.
|
|
- `build_node_build` {workspace, module, artifact} (issue 1490): the build agent's tsc and esbuild lines
|
|
(mesh-controller internal/builder, cited in `build.go`), the launchers, `package.json`, `buildstep.Pack`, and the
|
|
digest. No network. The build runs in a staging copy of the module at `<work>/<id>/stage-<artifact>/modules/module`, with the
|
|
toolchain's packages (not typescript or esbuild) copied to `stage-<artifact>/node_modules`: the build agent's depth
|
|
(`/app/modules/module`, `/app/node_modules`), so esbuild's `../../node_modules/...` path comments, and the bundle's
|
|
digest, equal its. Copies, not links, because the mesh makes no symlink. A bundle that keeps an external gets them in its own `node_modules`.
|
|
Once the build ended, this process pushes the packed archive to the artifact store (novox/hq ADR 0498 §1 §2, mesh-sdk's
|
|
`buildstep.PushBlob`, the address in `${dir:state}/registry`) under `<module>/<artifact>` by its digest, a candidate's as a
|
|
trunk build's, and answers `reference: artifact-store://<module>/<artifact>/blobs/<digest>`; a push that fails fails the
|
|
step `not stored: <artifact>: <the store's words>`. The packed bytes never stay on the node. tsc and esbuild, like the
|
|
type-check and the test script, run shut off in bubblewrap (ADR 0448): no network, none of the account, the toolchain
|
|
read-only, the npm cache beneath a throwaway layer unless the step is told `publish` (a trunk build).
|
|
A `package` artifact (ADR 0498 §5) is built by its own `build` script and packed with `npm pack --ignore-scripts`, both
|
|
shut off; its digest is the packed file's. Only with `publish` is it published, by this process with the registry's
|
|
write credential (never in a workspace or a step's mounts): a version the registry lacks is published, the same
|
|
packed file already there is done without a publish, another file under that version fails `version taken:
|
|
<name>@<version>`; the reference `<name>@<version>` is answered only then. A candidate answers its digest alone.
|