jschoubben
5d1f7762c2
The images a bed stocks are almost entirely the same bytes: the base they share is 227 MB and a module's own code is a few. Exported one at a time that base is written, pushed and loaded once per image — for this bed, the same 227 MB crossed thirty-odd times, and the whole set measured 9.7 GB. Measured on eight of them: 2.24 GB as separate archives, 0.29 GB as one. 87 per cent less, and it improves with the count. This is the slowest thing a raise does, and the four-machine bed has been exceeding its own ninety-minute limit while still copying — so it was failing on the clock rather than on anything it was testing. Also stops shipping a compiler in every module image. The image runs compiled code and never compiles any; tsc runs on the workstation. Worth 26 MB an image, which is small beside the above but was pure waste.
109 lines
6.9 KiB
Bash
Executable File
109 lines
6.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build a per-module runtime image (novox/hq ADR 0052): the tool runtime carrying ONE module's
|
|
# compiled code, which serves that module's tools and runs its events/provisioner under the module's
|
|
# own scoped broker account. Generalises build-runtime-image.sh from the audit-logger to any module.
|
|
#
|
|
# build-module-runtime.sh <module> <output.tar>
|
|
# -> tags mesh-runtime-<module>:development and saves it to <output.tar>
|
|
set -euo pipefail
|
|
|
|
MODULE="${1:?usage: build-module-runtime.sh <module> <output.tar>}"
|
|
OUT="${2:?usage: build-module-runtime.sh <module> <output.tar>}"
|
|
HERE="$(cd "$(dirname "$0")/.." && pwd)"; ROOT="$(cd "$HERE/.." && pwd)"
|
|
MESH_TOOLS="${MESH_TOOLS:-$ROOT/mesh-tools}"
|
|
MESH_SDK="${MESH_SDK:-$ROOT/mesh-sdk}"
|
|
MESH_CATALOG="${MESH_CATALOG:-$ROOT/mesh-catalog}"
|
|
MOD="$MESH_CATALOG/modules/$MODULE"
|
|
TAG="${RUNTIME_TAG:-mesh-runtime-$MODULE:development}"
|
|
BASE="${RUNTIME_BASE:-node:22-bookworm-slim}"
|
|
[ -d "$MOD" ] || { echo "no module $MODULE at $MOD" >&2; exit 1; }
|
|
|
|
( cd "$MESH_SDK" && npm run build >/dev/null )
|
|
( cd "$MESH_TOOLS" && npm run build >/dev/null )
|
|
# Compile whichever of the module's entrypoints exist. Besides the serve-time entrypoints (tools,
|
|
# events, provisioner) and the run-once bootstrap, a module may carry scheduled/one-shot entrypoints
|
|
# it names in a `schedule`/`run-once` container's args (novox/hq ADR 0052/0053) — refresh/apply/usage
|
|
# for the anthropic model-access modules, migrate for model-usage's run-once schema step. tsc pulls
|
|
# in their imports, so leaf files they use are compiled with them. A module may also carry an ambient
|
|
# `.d.ts` typing a third-party dep it default-imports (model-usage's pg.d.ts) — listed here so the
|
|
# ambient declaration is in the program even though the dep is only installed into the image below.
|
|
SRCS=(); for f in \
|
|
client.ts index.ts tools/index.ts provisioner/index.ts bootstrap/index.ts \
|
|
adopt/index.ts refresh/index.ts apply/index.ts usage/index.ts migrate/index.ts \
|
|
pg.d.ts; do
|
|
[ -f "$MOD/$f" ] && SRCS+=("$f")
|
|
done
|
|
TSC="$MESH_SDK/node_modules/.bin/tsc"; ( cd "$MOD" && "$TSC" "${SRCS[@]}" --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist >/dev/null )
|
|
|
|
STAGE="$(mktemp -d)"; trap 'rm -rf "$STAGE"' EXIT
|
|
cp -r "$MESH_TOOLS/dist" "$STAGE/dist"
|
|
cp -rL "$MESH_TOOLS/node_modules" "$STAGE/node_modules"
|
|
# And the sdk, from the sibling this script just built, whatever form the installed tree holds it
|
|
# in. It used to be relied on being a symlink into that sibling, which `-L` above materialised —
|
|
# true only on a workstation where somebody had linked them, and false the moment the runtime's
|
|
# dependencies are installed the ordinary way, which now fetches the sdk as sources with nothing
|
|
# compiled in it. The image built then looked fine and every entry point inside it pointed at
|
|
# nothing.
|
|
rm -rf "$STAGE/node_modules/@novox/mesh-sdk"
|
|
mkdir -p "$STAGE/node_modules/@novox"
|
|
cp -rL "$MESH_SDK" "$STAGE/node_modules/@novox/mesh-sdk"
|
|
rm -rf "$STAGE/node_modules/@novox/mesh-sdk/node_modules"
|
|
# **The image runs compiled code and never compiles any**, so it does not need a compiler. The
|
|
# tree copied above is the runtime's full install, development dependencies and all — and the
|
|
# compiler alone is 23 of its 28 MB. Every module image carried one, on every machine, for nothing:
|
|
# tsc runs on the workstation a few lines above, not in here.
|
|
#
|
|
# Removed by name rather than by `npm prune --omit=dev`, which would re-resolve dependencies — one
|
|
# of them a git URL with no registry behind it — and could drop something the image needs.
|
|
rm -rf "$STAGE/node_modules/typescript" "$STAGE/node_modules/@types"
|
|
mkdir -p "$STAGE/modules/$MODULE"; cp -r "$MOD/dist" "$STAGE/modules/$MODULE/dist"
|
|
cp "$MESH_TOOLS/package.json" "$STAGE/package.json"
|
|
|
|
# A module may declare its own third-party runtime deps (the anthropic-manager seals with
|
|
# tweetnacl-sealedbox-js). The shared node_modules copied above carries the common packages and
|
|
# @novox/* — but not a module's private deps. Install those under the module itself, so Node
|
|
# resolves them from /app/modules/<module>/node_modules and still falls back to the shared tree
|
|
# at /app/node_modules for @novox/* and everything common. Modules with no non-@novox deps are a
|
|
# no-op. (@novox/* are workspace deps with no registry to fetch from, so they are excluded here.)
|
|
MOD_DEPS="$(node -e 'const d=(require("'"$MOD"'/package.json").dependencies)||{};process.stdout.write(Object.keys(d).filter(k=>!k.startsWith("@novox/")).map(k=>k+"@"+d[k]).join(" "))')"
|
|
if [ -n "$MOD_DEPS" ]; then
|
|
# shellcheck disable=SC2086
|
|
npm install --prefix "$STAGE/modules/$MODULE" --omit=dev --no-save --no-package-lock --ignore-scripts $MOD_DEPS >/dev/null
|
|
fi
|
|
|
|
# The entrypoints the runtime loads: tools, events and (a provider's) provisioner, whichever exist.
|
|
ENTRIES=""; for e in tools/index.js index.js provisioner/index.js; do
|
|
[ -f "$STAGE/modules/$MODULE/dist/$e" ] && ENTRIES="${ENTRIES:+$ENTRIES,}/app/modules/$MODULE/dist/$e"
|
|
done
|
|
|
|
# A module whose code drives a CLI needs that CLI in the image — postgres shells out to `psql`, minio
|
|
# to `mc`. Everything else speaks a wire protocol or HTTP and needs nothing added.
|
|
EXTRA=""
|
|
case "$MODULE" in
|
|
postgres) EXTRA='RUN apt-get update && apt-get install -y --no-install-recommends postgresql-client && rm -rf /var/lib/apt/lists/*' ;;
|
|
minio) EXTRA='COPY --from=minio/mc:latest /usr/bin/mc /usr/bin/mc' ;;
|
|
# mosquitto drives its dynsec admin — and its run-once bootstrap seeds the store — through
|
|
# `mosquitto_ctrl`. It is not in `mosquitto-clients` on bookworm; the `mosquitto` package carries
|
|
# it (with its shared libraries), and installing from apt keeps them together — copying the binary
|
|
# out of the (musl) eclipse-mosquitto image into this (glibc) base would not load.
|
|
mosquitto) EXTRA='RUN apt-get update && apt-get install -y --no-install-recommends mosquitto && rm -rf /var/lib/apt/lists/*' ;;
|
|
# mongodb's client shells out to `mongosh`. Install it from MongoDB's own apt repo so its shared
|
|
# libraries come with it — copying just the binary out of the mongo image leaves it unable to load.
|
|
mongodb) EXTRA='RUN apt-get update && apt-get install -y --no-install-recommends gnupg curl ca-certificates && curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg --dearmor -o /usr/share/keyrings/mongodb.gpg && echo "deb [signed-by=/usr/share/keyrings/mongodb.gpg] https://repo.mongodb.org/apt/debian bookworm/mongodb-org/7.0 main" > /etc/apt/sources.list.d/mongodb.list && apt-get update && apt-get install -y --no-install-recommends mongodb-mongosh && rm -rf /var/lib/apt/lists/*' ;;
|
|
esac
|
|
|
|
cat > "$STAGE/Dockerfile" <<DOCKER
|
|
FROM $BASE
|
|
WORKDIR /app
|
|
$EXTRA
|
|
COPY package.json ./
|
|
COPY node_modules ./node_modules
|
|
COPY dist ./dist
|
|
COPY modules ./modules
|
|
ENV MESH_TOOL_MODULES=$ENTRIES
|
|
ENTRYPOINT ["node", "dist/main.js"]
|
|
DOCKER
|
|
docker build -t "$TAG" "$STAGE"
|
|
docker save -o "$OUT" "$TAG"
|
|
echo "built $TAG (entrypoints: $ENTRIES) -> $OUT"
|