Compare commits

...
Author SHA1 Message Date
mesh-admin 27315d35cf Merge pull request 'The store does not collect until every controller composes the window (hq ADR 0189)' (#254) from fix/the-store-collects-once-the-window-is-understood into main 2026-10-04 02:26:57 +00:00
jschoubben 525c639041 The store does not collect until every controller composes the window (hq ADR 0189)
mesh-controller#259 fixes while-stopped to name the container as the machine
knows it — `distribution.store`, not `store`. Until that controller is the
one composing, novox refuses its whole declaration and takes nothing at all.

The step comes out; deletion stays on, already applied and harmless on its
own. A collect step without its window would be worse than none: garbage
collection against a live registry can sweep a blob a build is pushing.

Put back once the fixed controller is deployed and stays.
2026-10-04 04:26:38 +02:00
mesh-admin c0159ca0a1 Merge pull request 'Group 8: minio declares the bucket it derives (hq ADR 0201), and the store collects nightly (hq ADR 0189)' (#229) from feat/the-store-keeps-what-the-records-name into main 2026-10-04 01:47:48 +00:00
jschoubben 159ed53103 Rebased onto main: ADR 0188 renumbered to 0201, and minio takes the sdk at 0.1.7
The bundles refactor took ADR 0188 on main, so minio's comments cite 0201.
The sdk is 0.1.7 after the same rebase, and minio needs the `derived` field
it carries.
2026-10-04 02:45:13 +02:00
jschoubben 723e676b75 The store enables deletion and collects nightly (hq ADR 0189)
REGISTRY_STORAGE_DELETE_ENABLED on the server — the door already accepts a
push — and a scheduled step running the registry's own collector over the
volume at 03:30 with the server held still. Plain garbage-collect: what the
mesh keeps is still a manifest, so --delete-untagged is not needed and would
delete images machines are running.
2026-10-04 02:34:06 +02:00
jschoubben 661114370f minio declares the bucket it derives; its consumers stop transcribing it (hq ADR 0188)
serves.s3-bucket.bucket is ${consumer:as:dns}; the provisioner uses what it
is given. nextcloud, invoicing and photos ask for ${bound:s3-bucket:bucket}
instead of naming mesh-novox-* literals, which also named this node.
bucketFor and the long-dead accessKeyFor are gone.
2026-10-04 02:34:06 +02:00
mesh-admin 9e8146192c Merge pull request 'mssql: give TLS a host name when the server is an address' (#252) from fix/mssql-tls-names-the-host into main 2026-10-03 23:31:40 +00:00
jochen 6171d747db mssql: give TLS a host name when the server is an address
Node 25 refuses an IP address as the TLS server name, and the module reaches its server on
loopback, so every connection failed on the live machines. The certificate is trusted either way.
2026-10-04 01:31:31 +02:00
mesh-admin 84609c0373 Merge pull request 'The last three: mesh-catalog, mongodb and mssql code moves into bundles the node's runtime serves (hq ADR 0198, to-be 38 WP4c)' (#250) from feat/0198-the-last-three-module-code-moves into main 2026-10-03 23:28:58 +00:00
mesh-admin 28d5e7f939 Merge pull request 'audit-logger: the test subscribes as the module does' (#251) from fix/audit-logger-test-hears-every-event into main 2026-10-03 23:21:52 +00:00
jochen 4128380a3d audit-logger: its test subscribes as the module does and expects local event names
The test subscribed '**', which its in-memory broker never matched, while the module subscribes
'#'; and it still expected the module-qualified type from before event names became local.
2026-10-04 01:21:40 +02:00
jochen cf57d3fd8f mssql: its handlers, tools and provisioner run in the node's runtime, through the driver in its bundle (hq ADR 0198)
The mesh-mssql container goes with its Dockerfile (and the sqlcmd it fetched), build bases and bus credential. The client speaks TDS through the mssql driver its package.json names, inlined into the bundle by the builder (ADR 0198 §4): one session per call as one sqlcmd invocation was, FOR JSON rendering rows exactly as before, the consumer's password checked as a bound parameter. A caller's statement still runs only as the reader login (issue 193); the one-line rule and -x guarded against sqlcmd's own commands and variable substitution, which no longer stand between the caller and the server. The server is reached on loopback at the port the machine published (${port:1433}). The reader test drives a fake session in place of a fake sqlcmd.
2026-10-04 01:17:41 +02:00
jochen da8a46cfe8 mongodb: its handlers, tools and provisioner run in the node's runtime, through the driver in its bundle (hq ADR 0198)
The mesh-mongodb container goes with its Dockerfile, build bases and bus credential. Its client shelled out to mongosh, which no machine's system carries, so it now speaks to the server through the official mongodb driver its package.json names, inlined into the bundle by the builder (ADR 0198 §4); the tools answer exactly as before (relaxed Extended JSON). The server is reached on loopback at the port the machine published (${port:27017}). The root secret was owned by the mongo image's user (secrets-owner 999:999), which the runtime's account cannot read; the module's own copy is now the runtime's, and the server is given its own 999-owned copy rendered from the same secret.
2026-10-04 01:17:41 +02:00
jochen ed50130a6a mesh-catalog: its consumer and tools run in the node's runtime, and its preparation is a run-once process (hq ADR 0198)
The mesh-catalog container goes with its Dockerfile, build bases, bus credential and mesh-state directory. Its words are the database URL file where the mesh writes it. `pg` is a dependency in its package.json, which the builder now installs and inlines into the bundle (mesh-controller: a TypeScript bundle installs its module's own packages). `prepares: true` needs a container running the module's own artifact, so it becomes what ADR 0198 §3 says it is: prepare/index.js run by node as a run-once process, with the same words and no bus, before the runtime is started with the version that needs it, and again when the database URL changes.
2026-10-04 01:17:41 +02:00
mesh-admin bd2123166f Merge pull request 'Waves 2-3: nine modules' code moves into bundles the node's runtime serves (hq ADR 0198, to-be 38 WP4c)' (#248) from feat/0198-waves-2-3-module-code-moves into main 2026-10-03 23:01:18 +00:00
mesh-admin d9db931bd0 Merge pull request 'mosquitto: run mosquitto_ctrl inside the broker's container' (#249) from fix/mosquitto-ctrl-from-its-container into main 2026-10-03 23:01:03 +00:00
jochen 69f2efb591 mosquitto: run mosquitto_ctrl inside the broker's container
The module's code moved out of its container and took mosquitto_ctrl from a host package. A
machine whose package index is stale cannot install it (hq issue 205), so the tools failed. The
broker's own image carries the tool at the broker's version: the tools exec into the running
broker, and the bootstrap seeds from a throwaway container of the same image.
2026-10-04 01:00:51 +02:00
jochen 568674fef7 anthropic-consumer: its usage runs in the node's runtime, and its apply is a scheduled process (hq ADR 0198)
Both containers go with the Dockerfile, build bases, bus credential and state directory. apply needs no bus and runs every five minutes as a process on the machine at the host paths the container mounted. usage emitted by spawning the runtime image's own emit command with the module's credential, which exists nowhere now, so it is loaded by the node's runtime instead: it emits through the SDK as this module and reads on the cadence the schedule gave it, once at start and every five minutes. That is the one code change.
2026-10-04 00:52:31 +02:00
jochen 3c6b70845c openai-consumer: its apply is a scheduled process (hq ADR 0198)
The mesh-openai-consumer-apply container goes with its Dockerfile and build bases. The same entrypoint runs every five minutes as a process on the machine, reading the binding and writing the credentials at the host paths the container used to mount.
2026-10-04 00:52:31 +02:00
jochen 35ef72081f route-adapter: its step is a run-once process (hq ADR 0198)
The mesh-route-adapter container goes with its Dockerfile and build bases. The step runs node on the bundle as a run-once process, reading what the mesh contributed and its config where the mesh writes them and writing the proxy's dynamic directory at the path the container used to mount; it still runs again when a route or its config changes.
2026-10-04 00:52:31 +02:00
jochen 59c42b2086 lab: its tools run in the node's runtime (hq ADR 0198)
The mesh-lab container goes with its Dockerfile, build bases, bus credential and state directory. What the image installed — git, make, python, file, iproute2, sudo, npm, go and the incus client — are packages of the machine, and docker and incus are reached through their sockets as the runtime's account. The forge is an operator's setting, which reaches a file and never a bundle's words, so the tools read it from the env-file the mesh already fills, at each call; that is the one code change.
2026-10-04 00:52:31 +02:00
jochen 3b8164f1c0 mailu: its handlers, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-mailu container goes with its Dockerfile, the mesh-tools build bases and its bus credential; automx keeps its own image. The code reached the admin API by its name on the mailu network, which a process on the machine cannot, so the admin container publishes 8080 to this machine only and the bundle reaches it on loopback at that port. Mail is still read through docker exec into mailu-imap, so the runtime's account needs the docker socket as nextcloud's does.
2026-10-04 00:52:31 +02:00
jochen 8877f893e5 records: its consumer and tools run in the node's runtime (hq ADR 0198)
The records container goes with its Dockerfile, build bases and bus credential. The checkout, the config file and the origin file are read where the mesh writes them, and git comes from the machine's git package instead of the image's apt layer.
2026-10-04 00:52:31 +02:00
jochen 043ae17fbf mesh-vault: its handlers, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-vault container goes with its Dockerfile, build bases, bus credential and state directory; its env was already host paths, so it becomes the bundle's words unchanged.
2026-10-04 00:52:31 +02:00
jochen 944f086ec7 gitea: its watcher, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-gitea container goes with its Dockerfile, build bases and bus credential; its env becomes the bundle's words with mount targets folded back to host paths: the config file, the admin password and the kept-token state directory are read where the mesh writes them.
2026-10-04 00:52:31 +02:00
jochen dd93cfd613 audit-logger: its handler runs in the node's runtime (hq ADR 0198)
The mesh-audit-logger container goes with its Dockerfile, build bases and bus credential: its one entrypoint is a load of one bundle, which subscribes to every event through the runtime and writes the trail at the host path the container used to mount.
2026-10-04 00:52:31 +02:00
mesh-admin 64cc292d7e Merge pull request 'Wave 1: thirteen modules' code moves into bundles the node's runtime serves (hq ADR 0198, to-be 38 WP4c)' (#245) from feat/0198-wave-1-module-code-moves into main 2026-10-03 22:52:08 +00:00
mesh-admin 7e889adf71 Merge pull request 'netcheck: one module, a Go tools bundle and a TypeScript one (hq ADR 0188, 0193)' (#246) from feat/netcheck-a-module-in-two-languages into main 2026-10-03 22:35:32 +00:00
jochen 7e9ef899c1 netcheck: one module, a Go tools bundle and a TypeScript one (hq ADR 0188, 0193)
ADR 0193 says the node's runtime launches every served bundle over MCP stdio and knows no
language, and ADR 0188 says one module may carry several bundles in any language. Nothing in
the catalogue shows both at once: every tools bundle is TypeScript, and the only Go bundle is
the runtime itself. netcheck is the smallest real module that does — read-only checks from a
machine, worth having on their own:

- tools-go (Go SDK go/v0.1.6): netcheck_tcp (one connect, nothing sent) and netcheck_dns
  (A/AAAA/CNAME/TXT/MX through the machine's resolver).
- tools-typescript (@novox/mesh-sdk): netcheck_http (HEAD or GET, body neither sent nor read,
  redirects reported not followed, anything but http(s) refused).

Both say loads; the module lists its tools. No container, no image, no env: nothing to be
given, so the runtime's own words suffice.
2026-10-04 00:34:31 +02:00
jochen f79199777d minio: its tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-minio container goes with its Dockerfile, build bases, bus credential and state directory. The client reaches minio on the published port, runs the minio-client package's mcli instead of the image's mc, and keeps mc's config, which holds the root alias, in the module's own state directory rather than a shared /tmp.
2026-10-03 23:33:51 +02:00
jochen b9d0884335 nextcloud: its handlers and tools run in the node's runtime (hq ADR 0198)
The mesh-nextcloud container goes with its Dockerfile, build bases and bus credential. occ still runs through docker exec, now with the host's own docker CLI and socket.
2026-10-03 23:33:51 +02:00
jochen 6a6d5747a3 nodered: the runtime serves its tools, and its mqtt step is a run-once process (hq ADR 0198)
The mesh-nodered container goes with its Dockerfile, build bases and bus credential. The mqtt step runs node on the bundle and reads the binding and settings files where the mesh writes them, from an env-file the mesh fills because a process's env is not given ${port:…}.
2026-10-03 23:33:51 +02:00
jochen b19c4a2593 home-assistant: the runtime serves its code, and its provisions step is a run-once process (hq ADR 0198)
The mesh-home-assistant container goes with its Dockerfile, build bases and bus credential. The provisions step runs node on the bundle and reads the binding files where the mesh writes them, from an env-file the mesh fills because a process's env is not given ${port:…}. It still runs again when a binding it reads changes.
2026-10-03 23:33:51 +02:00
jochen a934e2a69f icecast: its handlers and tools run in the node's runtime (hq ADR 0198)
The mesh-icecast container goes with its Dockerfile, build bases and bus credential. The bundle reaches icecast on the port this machine published rather than the container network's name.
2026-10-03 23:33:51 +02:00
jochen af346f6066 grafana: its handlers and tools run in the node's runtime (hq ADR 0198)
The mesh-grafana container goes with its Dockerfile, build bases and bus credential; its env becomes the bundle's words with mount targets folded back to host paths.
2026-10-03 23:33:51 +02:00
jochen 7b0cfceb68 cloudflare-dns: its tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-cloudflare-dns container goes with its Dockerfile, build bases, bus credential and state directory. MESH_RECEIVES now names the grants directory itself; the container's value pointed at a path nothing was mounted on.
2026-10-03 23:33:51 +02:00
jochen 26021865c1 umami: its tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-umami container goes with its Dockerfile, build bases, bus credential and state directory. The provisioner's env-file only told it umami's container-network address, so it becomes a word on the published port and the file goes.
2026-10-03 23:33:51 +02:00
jochen 7440b8d009 keycloak: its handlers, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-keycloak container goes with its Dockerfile, build bases and bus credential; its env becomes the bundle's words with mount targets folded back to host paths.
2026-10-03 23:33:51 +02:00
jochen 0cb67e856f influxdb: its tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-influxdb container goes with its Dockerfile, build bases and bus credential; its env becomes the bundle's words with mount targets folded back to host paths.
2026-10-03 23:33:51 +02:00
jochen 038a0a25ce mosquitto: the runtime serves its code, and its bootstrap is a run-once process (hq ADR 0198)
The mesh-mosquitto container goes with its Dockerfile, build bases and bus credential. mosquitto_ctrl comes from the mosquitto package, and the bootstrap step runs node on the bundle, reading the broker's published port from an env-file the mesh fills, because a process's env is not given ${port:…}.
2026-10-03 23:33:50 +02:00
jochen 1b27ce319a redis: its handlers, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-redis container goes with its Dockerfile, build bases, bus credential and the state directory only that credential lived in. The bundle reaches redis on the port this machine published rather than the container network's name.
2026-10-03 23:33:50 +02:00
jochen 7563569c8a postgres: its handlers, tools and provisioner run in the node's runtime (hq ADR 0198)
The mesh-postgres container goes with its Dockerfile, build bases and bus credential: its three entrypoints are loads of one bundle, given their words as host paths, and psql comes from postgresql-libs instead of the image's apt layer. The seat word is dropped, since a bundle's words cannot carry one and the client treats it as optional.
2026-10-03 23:33:50 +02:00
85 changed files with 1552 additions and 2155 deletions
-22
View File
@@ -1,22 +0,0 @@
# anthropic-consumer's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/anthropic-consumer
COPY . .
RUN node /app/node_modules/typescript/bin/tsc apply/index.ts usage/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/anthropic-consumer/dist /app/modules/anthropic-consumer/dist
# No serve-time entrypoints: every container of this module names its command (`run` on a
# schedule), so nothing here serves — deliberately no MESH_TOOL_MODULES.
+26 -62
View File
@@ -14,19 +14,10 @@
"secrets": { "secrets": {
"model-access": "${dir:state}/access-token" "model-access": "${dir:state}/access-token"
}, },
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"emits": [ "emits": [
"usage.session" "usage.session"
], ],
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -46,66 +37,39 @@
}, },
{ {
"id": "apply", "id": "apply",
"type": "container", "type": "process",
"name": "mesh-anthropic-consumer-apply", "name": "anthropic-consumer-apply",
"network": "host", "artifact": "code",
"run": [
"node",
"apply/index.js"
],
"schedule": "*/5 * * * *", "schedule": "*/5 * * * *",
"args": [
"run",
"/app/modules/anthropic-consumer/dist/apply/index.js"
],
"volumes": [
"${dir:state}:/run/state"
],
"env": { "env": {
"MESH_MODEL_ACCESS_SECRET_FILE": "/run/state/access-token", "MESH_MODEL_ACCESS_SECRET_FILE": "${dir:state}/access-token",
"MESH_MODEL_ACCESS_BIND_FILE": "/run/state/model.json", "MESH_MODEL_ACCESS_BIND_FILE": "${dir:state}/model.json",
"MESH_CLAUDE_CREDENTIALS_FILE": "/run/state/claude/.credentials.json", "MESH_CLAUDE_CREDENTIALS_FILE": "${dir:state}/claude/.credentials.json",
"MESH_CLAUDE_IDENTITY_FILE": "/run/state/claude/.claude.json" "MESH_CLAUDE_IDENTITY_FILE": "${dir:state}/claude/.claude.json"
}, }
"artifact": "runtime"
},
{
"id": "usage",
"type": "container",
"name": "mesh-anthropic-consumer-usage",
"network": "host",
"schedule": "*/5 * * * *",
"args": [
"run",
"/app/modules/anthropic-consumer/dist/usage/index.js"
],
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:state}:/run/state"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_CLAUDE_PROJECTS_DIR": "/run/state/claude/projects",
"MESH_ANTHROPIC_USAGE_OUT": "/run/state/out/session-usage.json",
"MESH_TOOLS_MAIN": "/app/dist/main.js"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"apply/index.js",
"usage/index.js"
],
"loads": [
"usage/index.js"
],
"env": {
"MESH_CLAUDE_PROJECTS_DIR": "${dir:state}/claude/projects",
"MESH_ANTHROPIC_USAGE_OUT": "${dir:state}/out/session-usage.json"
}
} }
] ]
} }
+18 -17
View File
@@ -3,12 +3,15 @@
// per session. The consumer IS the (node,module) session's fixed binding, so no per-message account // per session. The consumer IS the (node,module) session's fixed binding, so no per-message account
// attribution is done — just the totals (port map "don't-map" #3). // attribution is done — just the totals (port map "don't-map" #3).
// //
// Runs as `mesh-tools run` (no broker), so events are emitted best-effort via the sibling mesh-tools // Runs in the node's runtime (novox/hq ADR 0198), every five minutes, so events are emitted through
// `emit` primitive; the totals are also written to a file so the reading is observable without one. // the runtime as this module; the totals are also written to a file so the reading is observable
// without one.
import { readdirSync, statSync, readFileSync, writeFileSync, renameSync, mkdirSync } from "node:fs"; import { readdirSync, statSync, readFileSync, writeFileSync, renameSync, mkdirSync } from "node:fs";
import { join, dirname } from "node:path"; import { join, dirname } from "node:path";
import { emit } from "@novox/mesh-sdk/events";
import { readSessionFile, type SessionUsage } from "../transcript.js"; import { readSessionFile, type SessionUsage } from "../transcript.js";
/** The vendor-neutral usage row ADR 0054 fixes — the shape the model-usage store upserts. Kept local /** The vendor-neutral usage row ADR 0054 fixes — the shape the model-usage store upserts. Kept local
@@ -116,22 +119,20 @@ function atomicWrite(path: string, content: string): void {
renameSync(tmp, path); renameSync(tmp, path);
} }
/** Emit best-effort via the sibling mesh-tools `emit`, which wires a broker a run step has none. */ /** Emit best-effort through the runtime: a reading that could not be announced is still in the file. */
async function emitUsage(body: Record<string, unknown>): Promise<void> { async function emitUsage(body: Record<string, unknown>): Promise<void> {
const main = process.env.MESH_TOOLS_MAIN ?? "/app/dist/main.js"; try {
const { spawn } = await import("node:child_process"); await emit("usage.session", body);
await new Promise<void>((resolve) => { } catch (err) {
const child = spawn(
process.execPath,
[main, "emit", "usage.session", JSON.stringify(body)],
{ stdio: "inherit" },
);
child.on("exit", () => resolve());
child.on("error", (err) => {
console.error(`[anthropic-consumer] could not emit usage: ${err}`); console.error(`[anthropic-consumer] could not emit usage: ${err}`);
resolve(); }
});
});
} }
await main(); // The cadence the scheduled container had: once at start, then every five minutes. Not awaited, so the
// runtime's handshake is answered while a long first reading is still under way.
const EVERY_MS = 5 * 60 * 1000;
const tick = (): void => {
void main().catch((err) => console.error(`[anthropic-consumer] usage reading failed: ${err}`));
};
tick();
setInterval(tick, EVERY_MS);
-33
View File
@@ -1,33 +0,0 @@
# audit-logger's runtime: the shared runtime image, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The toolkit is in the base image, so
# nothing is copied out of a neighbouring checkout — which is what lets the mesh build this from a
# repository and a path (novox/hq ADR 0069) rather than only on a workstation that happens to have
# the siblings laid out beside it.
# Two bases, named rather than pinned: the image this is COMPILED in, and the image it RUNS in.
# They are different images on purpose — the first carries a compiler and the second must not, or
# every running container would carry one it never invokes. The mesh answers both with the copies it
# holds, because a fingerprint written here would name one particular copy and no other mesh has it
# (novox/hq issue 044). Declared in module.json's `build.on`; deliberately no defaults, so a build
# nobody told stops here and says which module to build first.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the toolkit it will run against.
WORKDIR /app/modules/audit-logger
COPY . .
# The compiler is invoked by its real path rather than through node_modules/.bin, whose entries are
# symlinks to a launcher that requires its library relatively — resolved away when the base image
# was assembled.
RUN node /app/node_modules/typescript/bin/tsc audit.ts index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/audit-logger/dist /app/modules/audit-logger/dist
# **Served, not run.** This subscribes on import, and the serve mode binds the broker before it
# imports anything — `run` exists for a step that works offline and exits, and would leave this
# with nothing to subscribe to.
ENV MESH_TOOL_MODULES=/app/modules/audit-logger/dist/index.js
+12 -33
View File
@@ -5,27 +5,21 @@
"consumes": [ "consumes": [
"**" "**"
], ],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js"
],
"loads": [
"index.js"
],
"env": {
"AUDIT_LOG": "${dir:trail}/audit.log"
}
} }
] ]
}, },
@@ -40,21 +34,6 @@
"id": "trail", "id": "trail",
"type": "directory", "type": "directory",
"mode": "0700" "mode": "0700"
},
{
"id": "run",
"type": "container",
"name": "mesh-audit-logger",
"network": "host",
"volumes": [
"${dir:state}/broker:/run/secrets/broker:ro",
"${dir:trail}:/trail"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"AUDIT_LOG": "/trail/audit.log"
},
"artifact": "runtime"
} }
], ],
"capabilities": [ "capabilities": [
+4 -2
View File
@@ -15,7 +15,7 @@ test("audit-logger records every event to the trail as one line each", async ()
const path = join(dir, "audit.log"); const path = join(dir, "audit.log");
// The audit-logger's whole behaviour: consume everything, record it. // The audit-logger's whole behaviour: consume everything, record it.
await on("**", async (event) => record(event, path)); await on("#", async (event) => record(event, path)); // the pattern index.ts subscribes
process.env.MESH_MODULE = "umami"; process.env.MESH_MODULE = "umami";
process.env.MESH_NODE = "anchor"; process.env.MESH_NODE = "anchor";
@@ -24,7 +24,9 @@ test("audit-logger records every event to the trail as one line each", async ()
const lines = (await readFile(path, "utf8")).trim().split("\n").map((l) => JSON.parse(l)); const lines = (await readFile(path, "utf8")).trim().split("\n").map((l) => JSON.parse(l));
assert.equal(lines.length, 2); assert.equal(lines.length, 2);
assert.deepEqual(lines.map((l) => l.type), ["umami.site.created", "node.anchor.joined"]); // A module names its events locally (design 29); the module is the `source`, which together with
// the type says whose event it was. This broker does no namespacing, so the type is as emitted.
assert.deepEqual(lines.map((l) => l.type), ["site.created", "node.anchor.joined"]);
assert.equal(lines[0].source, "umami"); assert.equal(lines[0].source, "umami");
assert.equal(lines[0].node, "anchor"); assert.equal(lines[0].node, "anchor");
assert.equal(lines[0].body.domain, "my-app"); assert.equal(lines[0].body.domain, "my-app");
-24
View File
@@ -1,24 +0,0 @@
# cloudflare-dns's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/cloudflare-dns
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/cloudflare-dns/dist /app/modules/cloudflare-dns/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/cloudflare-dns/dist/tools/index.js,/app/modules/cloudflare-dns/dist/provisioner/index.js
+17 -42
View File
@@ -18,20 +18,13 @@
"public-dns": "${dir:grants}/mesh.json" "public-dns": "${dir:grants}/mesh.json"
}, },
"own-secrets": { "own-secrets": {
"token": "${dir:state}/token", "token": "${dir:state}/token"
"broker": "${dir:mesh-state}/broker"
}, },
"emits": [ "emits": [
"record.created", "record.created",
"record.removed" "record.removed"
], ],
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -50,48 +43,30 @@
"merge": "json", "merge": "json",
"content": "{}", "content": "{}",
"mode": "0600" "mode": "0600"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-cloudflare-dns",
"network": "host",
"volumes": [
"${dir:state}/config.json:/run/config/config.json:ro",
"${dir:grants}:/grants",
"${dir:state}/token:/run/secrets/token:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro"
],
"env": {
"MESH_CLOUDFLARE_TOKEN_FILE": "/run/secrets/token",
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_CLOUDFLARE_CONFIG_FILE": "/run/config/config.json",
"MESH_RECEIVES": "/var/lib/cloudflare-dns/grants/mesh.json"
},
"artifact": "runtime"
} }
], ],
"capabilities": [ "capabilities": [
"container-runtime" "container-runtime"
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_CLOUDFLARE_TOKEN_FILE": "${dir:state}/token",
"MESH_CLOUDFLARE_CONFIG_FILE": "${dir:state}/config.json",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
} }
] ]
} }
+4 -1
View File
@@ -59,7 +59,10 @@
], ],
"volumes": [ "volumes": [
"/var/lib/mesh-registry:/var/lib/registry" "/var/lib/mesh-registry:/var/lib/registry"
] ],
"env": {
"REGISTRY_STORAGE_DELETE_ENABLED": "true"
}
} }
] ]
} }
-37
View File
@@ -1,37 +0,0 @@
# gitea's runtime: the tool runtime, carrying this module's compiled provisioner, tools and event
# consumer.
#
# **Built from this module's own directory and nothing else.** The sdk is in the base image, so
# nothing is copied out of a neighbouring checkout — which is what lets the mesh build this from a
# repository and a path (novox/hq ADR 0069) rather than only on a workstation that happens to have
# the siblings.
#
# Two bases, named rather than pinned: the image this is COMPILED in, and the image it RUNS in.
# They are different images on purpose — the first carries a compiler and the second must not, or
# every running container would carry one it never invokes. The mesh answers both with the copies it
# holds, because a fingerprint written here would name one particular copy and no other mesh has it
# (novox/hq issue 044). Declared in module.json's `build.on`; deliberately no defaults, so a build
# nobody told stops here and says which module to build first.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against.
WORKDIR /app/modules/gitea
COPY . .
# The compiler is invoked by its real path rather than through node_modules/.bin, whose entries are
# symlinks to a launcher that requires its library relatively — resolved away when the base image
# was assembled.
RUN node /app/node_modules/typescript/bin/tsc client.ts token.ts index.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
# **No apt packages.** gitea's provisioner talks to the forge over HTTP (the gitea REST API), not
# through a CLI the way postgres drives psql — so the runtime base holds everything this needs.
COPY --from=build /app/modules/gitea/dist /app/modules/gitea/dist
# What a tool host should load from this module: its event consumer and its tools, which are
# separate entrypoints because they are loaded by different things. The provisioner is the third,
# and is not listed here — the declaration names it in the container's `args`, because it is what
# this module's own container runs. One image, because they are one module and share a client.
ENV MESH_TOOL_MODULES=/app/modules/gitea/dist/index.js,/app/modules/gitea/dist/tools/index.js,/app/modules/gitea/dist/provisioner/index.js
+21 -44
View File
@@ -85,9 +85,6 @@
"scope": "mesh" "scope": "mesh"
} }
], ],
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"resources": [ "resources": [
{ {
"id": "mesh-state", "id": "mesh-state",
@@ -180,32 +177,6 @@
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-gitea",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"${dir:runtime-state}:/run/state"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GITEA_CONFIG_FILE": "/run/config/config.json",
"MESH_GITEA_ADMIN_USER": "mesh-admin",
"MESH_GITEA_ADMIN_PASSWORD_FILE": "/run/secrets/admin",
"MESH_GITEA_STATE_DIR": "/run/state",
"MESH_RECEIVES": "${dir:grants}/npm.json"
},
"artifact": "runtime",
"restart-on": [
"runtime-config"
]
} }
], ],
"provides": [ "provides": [
@@ -219,23 +190,29 @@
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GITEA_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_GITEA_ADMIN_USER": "mesh-admin",
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_GITEA_STATE_DIR": "${dir:runtime-state}",
"MESH_RECEIVES": "${dir:grants}/npm.json"
}
} }
] ]
}, },
-24
View File
@@ -1,24 +0,0 @@
# grafana's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/grafana
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/grafana/dist /app/modules/grafana/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/grafana/dist/index.js,/app/modules/grafana/dist/tools/index.js
+16 -36
View File
@@ -5,8 +5,7 @@
"alert.firing" "alert.firing"
], ],
"own-secrets": { "own-secrets": {
"admin": "${dir:mesh-state}/admin", "admin": "${dir:mesh-state}/admin"
"broker": "${dir:mesh-state}/broker"
}, },
"capabilities": [ "capabilities": [
"container-runtime" "container-runtime"
@@ -112,25 +111,6 @@
"mode": "0600", "mode": "0600",
"content": "{\n \"user\": \"admin\",\n \"password\": \"${secret:admin}\"\n}\n", "content": "{\n \"user\": \"admin\",\n \"password\": \"${secret:admin}\"\n}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-grafana",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_GRAFANA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GRAFANA_CONFIG_FILE": "/run/config/config.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
} }
], ],
"requires": [ "requires": [
@@ -162,23 +142,23 @@
"influxdb-api": "${dir:mesh-state}/influxdb-api" "influxdb-api": "${dir:mesh-state}/influxdb-api"
}, },
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"MESH_GRAFANA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GRAFANA_CONFIG_FILE": "${dir:mesh-state}/config.json"
}
} }
] ]
} }
-27
View File
@@ -1,27 +0,0 @@
# home-assistant's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/home-assistant
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts provisions/hass.ts provisions/probe.ts provisions/connections.ts provisions/mesh.ts provisions/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/home-assistant/dist /app/modules/home-assistant/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/home-assistant/dist/index.js,/app/modules/home-assistant/dist/tools/index.js
# NOT dist/provisions/index.js: that is a step the host runs to completion, named by the
# `provisions` container's args as `mesh-tools run …` (novox/hq ADR 0052). Listed here it would run
# inside the serving sidecar too, and exit it.
+33 -61
View File
@@ -9,7 +9,6 @@
"state.changed" "state.changed"
], ],
"own-secrets": { "own-secrets": {
"broker": "${dir:mesh-state}/broker",
"token": "${dir:mesh-state}/token" "token": "${dir:mesh-state}/token"
}, },
"listens": [ "listens": [
@@ -80,55 +79,27 @@
"merge": "json" "merge": "json"
}, },
{ {
"id": "runtime", "id": "provisions-env",
"type": "container", "type": "file",
"name": "mesh-home-assistant", "path": "${dir:state}/provisions.env",
"network": "host", "mode": "0600",
"volumes": [ "content": "MESH_HOMEASSISTANT_URL=http://127.0.0.1:${port:8123}\nMESH_HOMEASSISTANT_TOKEN_FILE=${dir:mesh-state}/token\nMESH_PROVISIONS_DIR=${dir:state}\nMESH_WRITTEN_DIR=${dir:written}\n"
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/token:/run/secrets/token:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_HOMEASSISTANT_URL": "http://127.0.0.1:${port:8123}",
"MESH_HOMEASSISTANT_TOKEN_FILE": "/run/secrets/token",
"MESH_HOMEASSISTANT_CONFIG_FILE": "/run/config/config.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
}, },
{ {
"id": "provisions", "id": "provisions",
"type": "container", "type": "process",
"name": "mesh-home-assistant-provisions", "name": "home-assistant-provisions",
"network": "host", "artifact": "code",
"run-once": true, "run": [
"volumes": [ "node",
"${dir:mesh-state}/token:/run/secrets/token:ro", "provisions/index.js"
"${dir:written}:/var/lib/home-assistant-provisions",
"${dir:state}/mqtt-topic.json:/run/provisions/mqtt-topic.json:ro",
"${dir:state}/mqtt-topic.secret:/run/provisions/mqtt-topic.secret:ro",
"${dir:state}/sonarr-api.json:/run/provisions/sonarr-api.json:ro",
"${dir:state}/sonarr-api.secret:/run/provisions/sonarr-api.secret:ro",
"${dir:state}/radarr-api.json:/run/provisions/radarr-api.json:ro",
"${dir:state}/radarr-api.secret:/run/provisions/radarr-api.secret:ro",
"${dir:state}/lidarr-api.json:/run/provisions/lidarr-api.json:ro",
"${dir:state}/lidarr-api.secret:/run/provisions/lidarr-api.secret:ro"
], ],
"env": { "run-once": true,
"MESH_HOMEASSISTANT_URL": "http://127.0.0.1:${port:8123}", "env-file": [
"MESH_HOMEASSISTANT_TOKEN_FILE": "/run/secrets/token", "${dir:state}/provisions.env"
"MESH_PROVISIONS_DIR": "/run/provisions",
"MESH_WRITTEN_DIR": "/var/lib/home-assistant-provisions"
},
"args": [
"run",
"/app/modules/home-assistant/dist/provisions/index.js"
], ],
"restart-on": [ "restart-on": [
"provisions-env",
"bound-mqtt-topic", "bound-mqtt-topic",
"secret-mqtt-topic", "secret-mqtt-topic",
"bound-sonarr-api", "bound-sonarr-api",
@@ -137,8 +108,7 @@
"secret-radarr-api", "secret-radarr-api",
"bound-lidarr-api", "bound-lidarr-api",
"secret-lidarr-api" "secret-lidarr-api"
], ]
"artifact": "runtime"
} }
], ],
"requires": [ "requires": [
@@ -173,23 +143,25 @@
"lidarr-api": "${dir:state}/lidarr-api.secret" "lidarr-api": "${dir:state}/lidarr-api.secret"
}, },
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisions/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"MESH_HOMEASSISTANT_URL": "http://127.0.0.1:${port:8123}",
"MESH_HOMEASSISTANT_TOKEN_FILE": "${dir:mesh-state}/token",
"MESH_HOMEASSISTANT_CONFIG_FILE": "${dir:mesh-state}/config.json"
}
} }
] ]
} }
-24
View File
@@ -1,24 +0,0 @@
# icecast's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/icecast
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/icecast/dist /app/modules/icecast/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/icecast/dist/index.js,/app/modules/icecast/dist/tools/index.js
+15 -37
View File
@@ -28,9 +28,6 @@
"stream.started", "stream.started",
"stream.stopped" "stream.stopped"
], ],
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [ "listens": [
{ {
"name": "stream", "name": "stream",
@@ -95,45 +92,26 @@
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-icecast",
"network": "icecast",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_ICECAST_URL": "http://icecast:8000",
"MESH_ICECAST_CONFIG_FILE": "/run/config/config.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"MESH_ICECAST_URL": "http://127.0.0.1:${port:8000}",
"MESH_ICECAST_CONFIG_FILE": "${dir:mesh-state}/config.json"
}
} }
] ]
} }
-24
View File
@@ -1,24 +0,0 @@
# influxdb's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/influxdb
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts grants.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/influxdb/dist /app/modules/influxdb/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/influxdb/dist/tools/index.js,/app/modules/influxdb/dist/provisioner/index.js
+17 -39
View File
@@ -11,7 +11,6 @@
"container-runtime" "container-runtime"
], ],
"own-secrets": { "own-secrets": {
"broker": "${dir:mesh-state}/broker",
"admin": "${dir:state}/admin.secret", "admin": "${dir:state}/admin.secret",
"admin-token": "${dir:state}/admin-token.secret" "admin-token": "${dir:state}/admin-token.secret"
}, },
@@ -100,29 +99,6 @@
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-influxdb",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"${dir:state}/admin-token.secret:/run/secrets/admin-token:ro",
"${dir:grants}:${dir:grants}:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_INFLUXDB_URL": "http://127.0.0.1:${port:8086}",
"MESH_INFLUXDB_CONFIG_FILE": "/run/config/config.json",
"MESH_INFLUXDB_TOKEN_FILE": "/run/secrets/admin-token",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
} }
], ],
"requires": [ "requires": [
@@ -135,23 +111,25 @@
} }
}, },
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_INFLUXDB_URL": "http://127.0.0.1:${port:8086}",
"MESH_INFLUXDB_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_INFLUXDB_TOKEN_FILE": "${dir:state}/admin-token.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
} }
] ]
} }
+1 -1
View File
@@ -68,7 +68,7 @@
"type": "file", "type": "file",
"path": "${dir:state}/api.env", "path": "${dir:state}/api.env",
"mode": "0600", "mode": "0600",
"content": "NODE_ENV=production\nPORT=9000\nMONGO_URL=mongodb://${bound:mongodb-database:as}:${secret:mongodb-database}@${bound:mongodb-database:at}:${bound:mongodb-database:port}/${bound:mongodb-database:as}?authSource=${bound:mongodb-database:as}\nMONGO_DB=${bound:mongodb-database:as}\nMINIO_BUCKET=mesh-novox-invoice\nMINIO_ENDPOINT=${bound:s3-bucket:at}\nMINIO_PORT=${bound:s3-bucket:port}\nMINIO_ACCESSKEY=${bound:s3-bucket:as}\nMINIO_SECRET=${secret:s3-bucket}\n" "content": "NODE_ENV=production\nPORT=9000\nMONGO_URL=mongodb://${bound:mongodb-database:as}:${secret:mongodb-database}@${bound:mongodb-database:at}:${bound:mongodb-database:port}/${bound:mongodb-database:as}?authSource=${bound:mongodb-database:as}\nMONGO_DB=${bound:mongodb-database:as}\nMINIO_BUCKET=${bound:s3-bucket:bucket}\nMINIO_ENDPOINT=${bound:s3-bucket:at}\nMINIO_PORT=${bound:s3-bucket:port}\nMINIO_ACCESSKEY=${bound:s3-bucket:as}\nMINIO_SECRET=${secret:s3-bucket}\n"
}, },
{ {
"id": "net", "id": "net",
-30
View File
@@ -1,30 +0,0 @@
# keycloak's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/keycloak
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts oidc.ts index.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/keycloak/dist /app/modules/keycloak/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/keycloak/dist/index.js,/app/modules/keycloak/dist/tools/index.js,/app/modules/keycloak/dist/provisioner/index.js
+20 -40
View File
@@ -62,8 +62,7 @@
"oidc-client": "${dir:grants}" "oidc-client": "${dir:grants}"
}, },
"own-secrets": { "own-secrets": {
"admin": "${dir:state}/admin.secret", "admin": "${dir:state}/admin.secret"
"broker": "${dir:mesh-state}/broker"
}, },
"resources": [ "resources": [
{ {
@@ -144,49 +143,30 @@
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-keycloak",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"${dir:grants}:${dir:grants}:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_KEYCLOAK_URL": "http://127.0.0.1:${port:8080}",
"MESH_KEYCLOAK_CONFIG_FILE": "/run/config/config.json",
"MESH_KEYCLOAK_PASSWORD_FILE": "/run/secrets/admin",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_KEYCLOAK_URL": "http://127.0.0.1:${port:8080}",
"MESH_KEYCLOAK_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_KEYCLOAK_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
} }
] ]
} }
-31
View File
@@ -1,31 +0,0 @@
# lab's runtime: the tool runtime, carrying this module's code, and the toolchain the lab's suite
# builds the mesh with (novox/hq ADR 0172). It reaches the machine's virtualisation and container
# runtime through their sockets, so what it raises is what a hand run on this machine raises.
#
# Every download is pinned by its checksum: an image that builds the mesh is the last place to take
# whatever an upstream serves today.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/lab
COPY . .
RUN node /app/node_modules/typescript/bin/tsc tools/index.ts tools/runs.ts --rootDir . \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
RUN apt-get update \
&& apt-get install -y --no-install-recommends git make ca-certificates curl python3 file iproute2 sudo \
&& rm -rf /var/lib/apt/lists/*
RUN curl -fsSL -o /tmp/go.tgz https://go.dev/dl/go1.26.8.linux-amd64.tar.gz \
&& echo "d0f743b33e8d8945e6b1f432edd15785c70507121d6e2a723b21285eddf8b57b /tmp/go.tgz" | sha256sum -c - \
&& tar -C /usr/local -xzf /tmp/go.tgz && rm /tmp/go.tgz
RUN curl -fsSL -o /usr/local/bin/incus https://github.com/lxc/incus/releases/download/v7.5.1/bin.linux.incus.x86_64 \
&& echo "7bd6223b369f4d693fcde695bd8549a73b5b3d403735329212483702aa22c179 /usr/local/bin/incus" | sha256sum -c - \
&& chmod 0755 /usr/local/bin/incus
RUN curl -fsSL -o /tmp/docker.tgz https://download.docker.com/linux/static/stable/x86_64/docker-28.5.2.tgz \
&& echo "ea90cfd12e1eeb12aa1c971741adb8bd4ed88e2a574eaac13f5029a1dbc6300d /tmp/docker.tgz" | sha256sum -c - \
&& tar -C /tmp -xzf /tmp/docker.tgz docker/docker && mv /tmp/docker/docker /usr/local/bin/docker && rm -rf /tmp/docker /tmp/docker.tgz
ENV PATH=/usr/local/go/bin:$PATH
COPY --from=build /app/modules/lab/dist /app/modules/lab/dist
ENV MESH_TOOL_MODULES=/app/modules/lab/dist/tools/index.js
+55 -44
View File
@@ -5,16 +5,7 @@
"container-runtime", "container-runtime",
"virtualisation" "virtualisation"
], ],
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -35,47 +26,67 @@
"content": "MESH_LAB_FORGE=${setting:forge}\n" "content": "MESH_LAB_FORGE=${setting:forge}\n"
}, },
{ {
"id": "runtime", "id": "git",
"type": "container", "type": "package",
"name": "mesh-lab", "package": "git"
"network": "host",
"env-file": [
"${dir:state}/lab.env"
],
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:work}:${dir:work}",
"/var/run/docker.sock:/var/run/docker.sock",
"/var/lib/incus/unix.socket:/var/lib/incus/unix.socket"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_LAB_WORK": "${dir:work}"
}, },
"restart-on": [ {
"runtime-env" "id": "make",
], "type": "package",
"artifact": "runtime" "package": "make"
},
{
"id": "python",
"type": "package",
"package": "python"
},
{
"id": "file",
"type": "package",
"package": "file"
},
{
"id": "iproute2",
"type": "package",
"package": "iproute2"
},
{
"id": "sudo",
"type": "package",
"package": "sudo"
},
{
"id": "npm",
"type": "package",
"package": "npm"
},
{
"id": "go",
"type": "package",
"package": "go"
},
{
"id": "incus",
"type": "package",
"package": "incus"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js"
],
"loads": [
"tools/index.js"
],
"env": {
"MESH_LAB_WORK": "${dir:work}",
"MESH_LAB_ENV_FILE": "${dir:state}/lab.env"
}
} }
] ]
} }
+23 -1
View File
@@ -2,18 +2,23 @@
// lab is assigned to, and only there: a bed raises virtual machines on that machine's virtualisation. // lab is assigned to, and only there: a bed raises virtual machines on that machine's virtualisation.
import { spawnSync } from "node:child_process"; import { spawnSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools"; import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { listRuns, readStatus, REPOSITORIES, running, start, stop, tail } from "./runs.js"; import { listRuns, readStatus, REPOSITORIES, running, start, stop, tail } from "./runs.js";
export function getLabTools(env: NodeJS.ProcessEnv): ToolDefinition[] { export function getLabTools(env: NodeJS.ProcessEnv): ToolDefinition[] {
const work = env.MESH_LAB_WORK ?? "/var/lib/mesh-lab-runs"; const work = env.MESH_LAB_WORK ?? "/var/lib/mesh-lab-runs";
const forge = (env.MESH_LAB_FORGE ?? "").replace(/\/+$/, ""); // The forge is an operator's setting, which reaches a file and never a bundle's words (novox/hq
// ADR 0192): read from the env-file the mesh fills, at each call, so a changed setting is used
// without restarting the runtime. MESH_LAB_FORGE itself still wins, for a hand-run instance.
const forgeOf = (): string => (env.MESH_LAB_FORGE ?? wordIn(env.MESH_LAB_ENV_FILE, "MESH_LAB_FORGE")).replace(/\/+$/, "");
return [ return [
{ {
name: "lab_check", name: "lab_check",
description: "Whether this machine can run the lab's beds: the lab's own check, against the forge's main branch.", description: "Whether this machine can run the lab's beds: the lab's own check, against the forge's main branch.",
input: {}, input: {},
run: async () => { run: async () => {
const forge = forgeOf();
if (!forge) return { ok: false, output: "the lab's forge is not set: settings for lab, {\"forge\": \"<url>\"}" }; if (!forge) return { ok: false, output: "the lab's forge is not set: settings for lab, {\"forge\": \"<url>\"}" };
const dir = `${work}/check`; const dir = `${work}/check`;
spawnSync("rm", ["-rf", dir]); spawnSync("rm", ["-rf", dir]);
@@ -38,6 +43,7 @@ export function getLabTools(env: NodeJS.ProcessEnv): ToolDefinition[] {
}, },
}, },
run: async (args) => { run: async (args) => {
const forge = forgeOf();
if (!forge) return { started: false, reason: "the lab's forge is not set: settings for lab, {\"forge\": \"<url>\"}" }; if (!forge) return { started: false, reason: "the lab's forge is not set: settings for lab, {\"forge\": \"<url>\"}" };
const tests = String(args.tests ?? "").split(",").map((s) => s.trim()).filter(Boolean); const tests = String(args.tests ?? "").split(",").map((s) => s.trim()).filter(Boolean);
if (tests.length === 0) return { started: false, reason: "name at least one bed test file" }; if (tests.length === 0) return { started: false, reason: "name at least one bed test file" };
@@ -83,4 +89,20 @@ export function getLabTools(env: NodeJS.ProcessEnv): ToolDefinition[] {
]; ];
} }
/** One word from an env-file (`KEY=value` lines), or "" when the file or the word is absent. */
export function wordIn(file: string | undefined, word: string): string {
if (!file) return "";
let text: string;
try {
text = readFileSync(file, "utf8");
} catch {
return "";
}
for (const line of text.split("\n")) {
const at = line.indexOf("=");
if (at > 0 && line.slice(0, at).trim() === word) return line.slice(at + 1).trim();
}
return "";
}
registerModuleTools("lab", (env) => getLabTools(env)); registerModuleTools("lab", (env) => getLabTools(env));
-30
View File
@@ -1,30 +0,0 @@
# mailu's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/mailu
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/mailu/dist /app/modules/mailu/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/mailu/dist/index.js,/app/modules/mailu/dist/tools/index.js,/app/modules/mailu/dist/provisioner/index.js
+30 -41
View File
@@ -135,11 +135,15 @@
"protocol": "tcp", "protocol": "tcp",
"from": "mesh", "from": "mesh",
"why": "automx: mail client autoconfiguration; the autoconfig, autodiscover and automx names are route grants reaching it here" "why": "automx: mail client autoconfiguration; the autoconfig, autodiscover and automx names are route grants reaching it here"
},
{
"name": "admin-api",
"port": 8080,
"protocol": "tcp",
"from": "machine",
"why": "the admin API, which this module's own code reaches on loopback from the node's runtime now that it runs outside the mailu network"
} }
], ],
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"resources": [ "resources": [
{ {
"id": "mesh-state", "id": "mesh-state",
@@ -305,6 +309,9 @@
"name": "mailu-admin", "name": "mailu-admin",
"image": "ghcr.io/mailu/admin@sha256:6dbfdadc4a9590dcb7652357b505200115b689b74008653bbf369e4599a3be5a", "image": "ghcr.io/mailu/admin@sha256:6dbfdadc4a9590dcb7652357b505200115b689b74008653bbf369e4599a3be5a",
"network": "mailu", "network": "mailu",
"ports": [
"8080"
],
"env-file": [ "env-file": [
"${dir:state}/mailu.env", "${dir:state}/mailu.env",
"${dir:state}/secret.env", "${dir:state}/secret.env",
@@ -473,31 +480,6 @@
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
}, },
{
"id": "runtime",
"type": "container",
"name": "mesh-mailu",
"network": "mailu",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:state}/api-token.secret:/run/secrets/api-token:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"/var/run/docker.sock:/var/run/docker.sock"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_MAILU_URL": "http://mailu-admin:8080/api/v1",
"MESH_MAILU_API_KEY_FILE": "/run/secrets/api-token",
"MESH_MAILU_IMAP_CONTAINER": "mailu-imap",
"MESH_MAILU_CONFIG_FILE": "/run/config/config.json",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
},
{ {
"id": "automx", "id": "automx",
"type": "container", "type": "container",
@@ -517,16 +499,6 @@
], ],
"build": { "build": {
"on": [ "on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
},
{ {
"arg": "PYTHON_BASE", "arg": "PYTHON_BASE",
"image": "python@sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228" "image": "python@sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"
@@ -534,9 +506,26 @@
], ],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_MAILU_URL": "http://127.0.0.1:${port:8080}/api/v1",
"MESH_MAILU_API_KEY_FILE": "${dir:state}/api-token.secret",
"MESH_MAILU_IMAP_CONTAINER": "mailu-imap",
"MESH_MAILU_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
}, },
{ {
"name": "automx", "name": "automx",
-55
View File
@@ -1,55 +0,0 @@
# mesh-catalog's runtime: the tool runtime, carrying the catalogue's compiled graph, its consumer
# of what the builder announces, and its tools.
#
# **Built from this module's own directory and nothing else.** The sdk is in the base image, so
# nothing is copied out of a neighbouring checkout — which is what lets the mesh build this from a
# repository and a path (novox/hq ADR 0069) rather than only on a workstation with the siblings.
#
# Two bases, named rather than pinned: the image this is COMPILED in, and the image it RUNS in.
# They are different images on purpose — the first carries a compiler and the second must not, or
# every running container would carry one it never invokes. The mesh answers both with the copies it
# holds, because a fingerprint written here would name one particular copy and no other mesh has it
# (novox/hq issue 044). Declared in module.json's `build.on`; deliberately no defaults, so a build
# nobody told stops here and says which module to build first.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against.
WORKDIR /app/modules/mesh-catalog
COPY . .
# The compiler is invoked by its real path rather than through node_modules/.bin, whose entries are
# symlinks to a launcher that requires its library relatively — resolved away when the base image
# was assembled.
RUN node /app/node_modules/typescript/bin/tsc pg.d.ts store.ts index.ts tools/index.ts prepare/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
# **A module may need something the base image does not carry.** The base holds what every module
# needs — the sdk, the broker client — and a postgres driver is not that: the one other module that
# reaches a database shells out to psql instead. So the catalogue brings its own.
#
# Installed into an empty directory rather than into the module's, because the module's package.json
# also names `@novox/mesh-sdk`, which is not on any registry — it is in the base image. Asking npm to
# resolve this module's dependencies would therefore fail on the one it already has.
RUN mkdir -p /deps && cd /deps && \
npm install --omit=dev --no-audit --no-fund --no-package-lock pg@8
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/mesh-catalog/dist /app/modules/mesh-catalog/dist
# Beside the compiled code, so `pg` resolves from it while `@novox/mesh-sdk` keeps walking up to the
# base image's own node_modules — the module gets its extra dependency without shadowing the sdk it
# was compiled against.
COPY --from=build /deps/node_modules /app/modules/mesh-catalog/node_modules
# Both entrypoints, loaded in serve mode.
#
# **A consumer cannot be started with `run`.** That mode imports an entrypoint without binding a
# broker — it is for a step that does its work offline and exits — and the catalogue's whole job is
# to listen for what the builder announces. Serve binds the broker first, then imports these, so
# `on()` has something to subscribe to.
ENV MESH_TOOL_MODULES=/app/modules/mesh-catalog/dist/index.js,/app/modules/mesh-catalog/dist/tools/index.js
# And what prepares this module's state, for the runtime's `prepare` mode (novox/hq ADR 0135). Named
# here, beside the entrypoints above, because the module knows which of its files prepares its state
# and nothing else could: the mesh asks one word and this says what answers it.
ENV MESH_PREPARE=/app/modules/mesh-catalog/dist/prepare/index.js
+24 -36
View File
@@ -25,9 +25,6 @@
"secrets": { "secrets": {
"postgres-database": "${dir:state}/database.secret" "postgres-database": "${dir:state}/database.secret"
}, },
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"consumes": [ "consumes": [
"mesh-build-machine.built", "mesh-build-machine.built",
"mesh-controller.built-before" "mesh-controller.built-before"
@@ -38,14 +35,7 @@
"rebuild-needed", "rebuild-needed",
"catching-up" "catching-up"
], ],
"prepares": true,
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -60,43 +50,41 @@
"content": "postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}\n" "content": "postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}\n"
}, },
{ {
"id": "runtime", "id": "prepare",
"type": "container", "type": "process",
"name": "mesh-catalog", "name": "mesh-catalog-prepare",
"network": "host", "artifact": "code",
"volumes": [ "run": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro", "node",
"${dir:state}:/run/state", "prepare/index.js"
"${dir:state}/database.url:/run/secrets/database-url:ro"
], ],
"run-once": true,
"env": { "env": {
"MESH_BROKER_FILE": "/run/secrets/broker", "DATABASE_URL_FILE": "${dir:state}/database.url"
"DATABASE_URL_FILE": "/run/secrets/database-url"
}, },
"artifact": "runtime",
"restart-on": [ "restart-on": [
"database-url" "database-url"
] ]
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"prepare/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"DATABASE_URL_FILE": "${dir:state}/database.url"
}
} }
] ]
} }
+3 -3
View File
@@ -1,9 +1,9 @@
// Ambient types for `pg` (node-postgres), which ships its types only via the separate `@types/pg` // Ambient types for `pg` (node-postgres), which ships its types only via the separate `@types/pg`
// package. Rather than pull that in at tsc time, this declares the exact slice model-usage uses — // package. Rather than pull that in at tsc time, this declares the exact slice model-usage uses —
// the same precedent anthropic-manager sets for `tweetnacl-sealedbox-js` (a local ambient .d.ts, // the same precedent anthropic-manager sets for `tweetnacl-sealedbox-js` (a local ambient .d.ts,
// listed in tsconfig `include`, default-imported). The real `pg` is installed into the module's // listed in tsconfig `include`, default-imported). The real `pg` is the package.json dependency the
// runtime image (package.json `dependencies`; novox/hq ADR 0052), so this types the code without // builder installs and inlines into the module's bundle (novox/hq ADR 0198 §4), so this types the
// deciding what runs. // code without deciding what runs.
declare module "pg" { declare module "pg" {
/** One checked-out connection. Needed because registering a module-version and its edges is one /** One checked-out connection. Needed because registering a module-version and its edges is one
* act: a half-written registration is a graph that lies about what something was built against. */ * act: a half-written registration is a graph that lies about what something was built against. */
+3 -2
View File
@@ -7,8 +7,9 @@
// nothing anywhere said so. // nothing anywhere said so.
// //
// Nothing here connects to the broker. Preparation runs before the version that would use it, so // Nothing here connects to the broker. Preparation runs before the version that would use it, so
// there is nothing yet to talk to; the runtime's `prepare` mode imports this and awaits it, and this // there is nothing yet to talk to: the host runs this file as a run-once process, with the module's
// process exiting non-zero is how the host knows not to start the runtime. // words and no bus (novox/hq ADR 0198 §3), before the node's runtime is started with the version
// that needs it, and this process exiting non-zero is how the host knows the step did not complete.
import { Graph } from "../store.js"; import { Graph } from "../store.js";
const graph = Graph.fromEnv(); const graph = Graph.fromEnv();
-22
View File
@@ -1,22 +0,0 @@
# mesh-vault's runtime: the tool runtime, carrying this module's compiled provisioner, tools and event
# consumer. The same shape as postgres's, minus the client the database needs: mesh-vault reaches no
# server, because what it provides is a value the mesh already delivered to its node.
#
# **Built from this module's own directory and nothing else.** The sdk is in the base image, so
# nothing is copied out of a neighbouring checkout (novox/hq ADR 0069). Two bases, named rather than
# pinned — the image this is COMPILED in and the image it RUNS in — answered by the mesh from
# `build.on` in module.json (novox/hq issue 044).
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/vault
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/vault/dist /app/modules/vault/dist
# The entrypoints a tool host loads from this module: its event consumer, its tools and its
# provisioner — one image, one process, one broker account (novox/hq ADR 0052).
ENV MESH_TOOL_MODULES=/app/modules/vault/dist/index.js,/app/modules/vault/dist/tools/index.js,/app/modules/vault/dist/provisioner/index.js
+18 -43
View File
@@ -27,16 +27,7 @@
"secret": "${dir:grants}" "secret": "${dir:grants}"
}, },
"keeps": "/var/lib/mesh-vault/root", "keeps": "/var/lib/mesh-vault/root",
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -57,45 +48,29 @@
"id": "root", "id": "root",
"type": "directory", "type": "directory",
"mode": "0700" "mode": "0700"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-vault",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:ledger}:${dir:ledger}",
"${dir:root}:${dir:root}:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_VAULT_LEDGER": "${dir:ledger}",
"MESH_VAULT_ROOT": "${dir:root}"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_VAULT_LEDGER": "${dir:ledger}",
"MESH_VAULT_ROOT": "${dir:root}"
}
} }
] ]
}, },
-40
View File
@@ -1,40 +0,0 @@
# minio's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
ARG MC_CLI
# Named so the final stage's COPY --from can reference a stage, not an ARG — the legacy builder
# this host still runs doesn't expand ARGs inside COPY --from, only inside FROM.
FROM ${MC_CLI} AS mccli
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/minio
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/minio/dist /app/modules/minio/dist
# The provisioner shells out to mc to actually create buckets and service accounts on the running
# minio server — mc itself was never in this runtime image, only in minio's own. Silently retried
# "spawn mc ENOENT" forever: a requirement was granted at the control-plane level without ever
# materializing the credential on minio. /usr/bin/mc there is a symlink to the real binary, mcli —
# both copied so the symlink resolves.
COPY --from=mccli /usr/bin/mcli /usr/bin/mcli
COPY --from=mccli /usr/bin/mc /usr/bin/mc
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/minio/dist/tools/index.js,/app/modules/minio/dist/provisioner/index.js
+5 -15
View File
@@ -303,21 +303,11 @@ export class MinioClient {
// --- module-scoped helpers ------------------------------------------------- // --- module-scoped helpers -------------------------------------------------
/** A deterministic 20-char access key id from a consumer name, so removal needs no stored state: // **Neither the access key nor the bucket is derived here any more.** `accessKeyFor` minted an id
* the provisioner recomputes the same id at teardown that it minted at creation. */ // of its own until the mesh took that over (ADR 0048: the login is the mesh's, handed to both
export function accessKeyFor(consumer: string): string { // ends), and `bucketFor` derived the bucket until the mesh took that over too (ADR 0201: the rule
const chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; // is a line of this module's manifest, filled per consumer and delivered to both ends). Both
const digest = createHash("sha256").update(consumer).digest(); // survived with no callers, which is the state a rule comes back from; they are gone.
let out = "";
for (let i = 0; i < 20; i++) out += chars[digest[i] % chars.length];
return out;
}
/** A DNS-safe bucket name derived from a consumer — the removable identity of its storage. */
export function bucketFor(consumer: string): string {
const name = consumer.toLowerCase().replace(/[^a-z0-9-]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 63);
return name.length >= 3 ? name : `mesh-${name}`;
}
function bucketPolicy(bucket: string): string { function bucketPolicy(bucket: string): string {
return JSON.stringify({ return JSON.stringify({
+26 -46
View File
@@ -49,7 +49,8 @@
"s3-bucket": { "s3-bucket": {
"scheme": "http", "scheme": "http",
"region": "eu-west", "region": "eu-west",
"port": 9000 "port": 9000,
"bucket": "${consumer:as:dns}"
} }
}, },
"receives": { "receives": {
@@ -59,16 +60,9 @@
"s3-bucket": "${dir:grants}" "s3-bucket": "${dir:grants}"
}, },
"own-secrets": { "own-secrets": {
"root": "${dir:state}/root.secret", "root": "${dir:state}/root.secret"
"broker": "${dir:mesh-state}/broker"
}, },
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -127,48 +121,34 @@
} }
}, },
{ {
"id": "runtime", "id": "client",
"type": "container", "type": "package",
"name": "mesh-minio", "package": "minio-client"
"network": "minio-net",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:state}/root.secret:/run/secrets/root:ro"
],
"env": {
"MESH_MINIO_ENDPOINT": "http://minio:9000",
"MESH_MINIO_ROOT_USER": "meshroot",
"MESH_MINIO_ROOT_PASSWORD_FILE": "/run/secrets/root",
"MESH_MINIO_REGION": "eu-west",
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
},
{
"arg": "MC_CLI",
"image": "docker.io/pgsty/minio@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_MINIO_ENDPOINT": "http://127.0.0.1:${port:9000}",
"MESH_MINIO_ROOT_USER": "meshroot",
"MESH_MINIO_ROOT_PASSWORD_FILE": "${dir:state}/root.secret",
"MESH_MINIO_REGION": "eu-west",
"MESH_MINIO_MC_BIN": "mcli",
"MESH_MINIO_MC_CONFIG": "${dir:state}/mc",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
} }
] ]
} }
+1 -1
View File
@@ -5,7 +5,7 @@
"type": "module", "type": "module",
"private": true, "private": true,
"dependencies": { "dependencies": {
"@novox/mesh-sdk": "^0.1.1" "@novox/mesh-sdk": "^0.1.7"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
+31 -7
View File
@@ -10,18 +10,24 @@
// **The access key and its secret are the mesh's, not the provisioner's (ADR 0048).** The mesh // **The access key and its secret are the mesh's, not the provisioner's (ADR 0048).** The mesh
// derives the login (the access-key id) and hands it to both ends, and mints the secret key. minio // derives the login (the access-key id) and hands it to both ends, and mints the secret key. minio
// creates the service account under exactly that access key with exactly that secret — a credential // creates the service account under exactly that access key with exactly that secret — a credential
// the provisioner invented is one the consumer could never present. The bucket is derived from the // the provisioner invented is one the consumer could never present.
// login, so teardown recomputes it with nothing to persist. //
// **The bucket name is the mesh's too (ADR 0201).** It used to be computed here, from the login,
// and every consumer transcribed the same rule into its own definition by hand — two copies of
// one rule with nothing comparing them, and one of three was wrong for months. Now the rule is a
// line of this module's manifest (`serves.s3-bucket.bucket: ${consumer:as:dns}`), the mesh fills
// it per consumer, and the same filled value reaches this provisioner and the consumer's own
// configuration. There is no second computation to disagree with.
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner"; import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
import { emit } from "@novox/mesh-sdk/events"; import { emit } from "@novox/mesh-sdk/events";
import { MinioClient, bucketFor } from "../client.js"; import { MinioClient } from "../client.js";
const minio = MinioClient.fromEnv(); const minio = MinioClient.fromEnv();
runProvisioner("s3-bucket", { runProvisioner("s3-bucket", {
async create(p: Provision): Promise<void> { async create(p: Provision): Promise<void> {
const bucket = bucketFor(p.as); const bucket = bucketNamed(p.derived);
const accessKeyId = p.as; const accessKeyId = p.as;
if (!(await minio.bucketExists(bucket))) await minio.createBucket(bucket); if (!(await minio.bucketExists(bucket))) await minio.createBucket(bucket);
@@ -38,8 +44,8 @@ runProvisioner("s3-bucket", {
}); });
}, },
async remove(p: { as: string }): Promise<void> { async remove(p: { as: string; derived: Readonly<Record<string, unknown>> }): Promise<void> {
const bucket = bucketFor(p.as); const bucket = bucketNamed(p.derived);
// Revoking the key is what cuts the consumer's access. The bucket is emptied-then-dropped only if // Revoking the key is what cuts the consumer's access. The bucket is emptied-then-dropped only if
// empty; a bucket that still holds objects is left for an operator rather than erroring on every // empty; a bucket that still holds objects is left for an operator rather than erroring on every
@@ -57,10 +63,28 @@ runProvisioner("s3-bucket", {
// Asked every minute by the harness: whether the backend still holds this consumer exactly as // Asked every minute by the harness: whether the backend still holds this consumer exactly as
// the mesh gave it, so a login lost behind the provisioner's back is made again (novox/hq issue 120). // the mesh gave it, so a login lost behind the provisioner's back is made again (novox/hq issue 120).
async holds(p: Provision): Promise<boolean> { async holds(p: Provision): Promise<boolean> {
return minio.canReachAs(bucketFor(p.as), p.as, p.password); return minio.canReachAs(bucketNamed(p.derived), p.as, p.password);
}, },
}); });
/** The bucket the mesh derived for this consumer.
*
* Absent means this module is running against a control plane that does not fill `${consumer:…}`
* yet, or a manifest whose `serves` block lost the line. Both are the same mistake from here —
* nobody said which bucket — and both are said rather than guessed: a provisioner that fell back
* to deriving one would restore the second rule and hide the fault behind a bucket that happens
* to be right. */
function bucketNamed(derived: Readonly<Record<string, unknown>>): string {
const bucket = derived.bucket;
if (typeof bucket !== "string" || bucket === "") {
throw new Error(
"the mesh did not say which bucket this consumer gets: minio's manifest must serve " +
"`bucket` under s3-bucket (novox/hq ADR 0201)",
);
}
return bucket;
}
/** Emit best-effort: a broker hiccup is logged and dropped, never allowed to throw back and fail a /** Emit best-effort: a broker hiccup is logged and dropped, never allowed to throw back and fail a
* bucket that was made. */ * bucket that was made. */
async function announce(type: string, body: unknown): Promise<void> { async function announce(type: string, body: unknown): Promise<void> {
-37
View File
@@ -1,37 +0,0 @@
# mongodb's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/mongodb
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
# mongodb's client shells out to `mongosh`, installed from MongoDB's own apt repo so its shared
# libraries come with it — copying the bare binary out of the mongo image leaves it unable to load.
RUN apt-get update && apt-get install -y --no-install-recommends gnupg curl ca-certificates \
&& curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg --dearmor -o /usr/share/keyrings/mongodb.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/mongodb.gpg] https://repo.mongodb.org/apt/debian bookworm/mongodb-org/7.0 main" > /etc/apt/sources.list.d/mongodb.list \
&& apt-get update && apt-get install -y --no-install-recommends mongodb-mongosh \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app/modules/mongodb/dist /app/modules/mongodb/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/mongodb/dist/index.js,/app/modules/mongodb/dist/tools/index.js,/app/modules/mongodb/dist/provisioner/index.js
+69 -78
View File
@@ -1,19 +1,16 @@
// mongodb's admin client — mongodb's own code, living in the module (novox/hq ADR 0039). Both this // mongodb's admin client — mongodb's own code, living in the module (novox/hq ADR 0039). Both this
// module's tools and its provisioner import it, and nothing outside mongodb does. // module's tools and its provisioner import it, and nothing outside mongodb does.
// //
// Commands run through `mongosh`, not a wire-protocol driver: the module may take NO npm dependency // **The backend's own driver, inside the bundle** (novox/hq ADR 0198 §4). This used to shell out to
// beyond @novox/mesh-sdk, and hand-rolling the MongoDB wire protocol + SCRAM auth is more surface // `mongosh`, which the module's container installed from MongoDB's apt repository; the module's code
// than this should carry — so it shells out to the shell the mongodb image ships, the same way // now runs in the node's runtime, on machines whose system carries no mongosh, so it speaks to the
// postgres drives itself through `psql`, minio through `mc` and mailu through doveadm. One boundary, // server through the official `mongodb` driver its package.json names — installed and inlined into
// `evalJs()`, and every method is built on it: a snippet of JavaScript is evaluated server-side and // the bundle by the builder. One connection per call, as one mongosh invocation was: the module is
// its result comes back as EJSON on stdout. // called rarely, and a pool held open across calls would hold a credential the mesh may rotate.
import { randomBytes } from "node:crypto"; import { randomBytes } from "node:crypto";
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import { execFile } from "node:child_process"; import { MongoClient as Driver, MongoServerError, BSON, type Document } from "mongodb";
import { promisify } from "node:util";
const run = promisify(execFile);
export interface DatabaseInfo { export interface DatabaseInfo {
readonly name: string; readonly name: string;
@@ -59,32 +56,26 @@ export class MongoClient {
return this.conn.port; return this.conn.port;
} }
/** The admin connection URI mongosh authenticates with, credentials percent-encoded. */ /** The admin connection URI, credentials percent-encoded. */
private uri(): string { private uri(): string {
const u = encodeURIComponent(this.conn.user); const u = encodeURIComponent(this.conn.user);
const p = encodeURIComponent(this.conn.password); const p = encodeURIComponent(this.conn.password);
const a = encodeURIComponent(this.conn.authSource); const a = encodeURIComponent(this.conn.authSource);
return `mongodb://${u}:${p}@${this.conn.host}:${this.conn.port}/?authSource=${a}`; return `mongodb://${u}:${p}@${this.conn.host}:${this.conn.port}/?authSource=${a}&directConnection=true`;
} }
/** /**
* Evaluate a JavaScript snippet server-side through `mongosh` and parse the JSON it prints (see * The one execution boundary: connect as the administrator, do `work`, and close — a failure to
* header). The snippet MUST `print()` exactly one JSON document as its only stdout — every method * connect or to authenticate rejects here rather than returning a partial success.
* below ends in `print(EJSON.stringify(...))`. `--quiet` suppresses the shell banner so stdout is
* the JSON alone; a non-zero exit (auth failure, bad command) rejects here rather than returning
* a partial success.
*/ */
async evalJs<T>(js: string): Promise<T> { private async admin<T>(work: (client: Driver) => Promise<T>): Promise<T> {
const { stdout } = await run( const client = new Driver(this.uri(), { serverSelectionTimeoutMS: 10_000 });
"mongosh", try {
[this.uri(), "--quiet", "--eval", js], await client.connect();
{ maxBuffer: 16 << 20 }, return await work(client);
); } finally {
const text = stdout.trim(); await client.close();
if (text.length === 0) {
throw new Error("mongosh returned no output — the eval printed nothing");
} }
return JSON.parse(text) as T;
} }
/** /**
@@ -94,19 +85,16 @@ export class MongoClient {
* password and roles, so a rotated credential converges. * password and roles, so a rotated credential converges.
*/ */
async createDatabaseAndUser(database: string, user: string, password: string): Promise<void> { async createDatabaseAndUser(database: string, user: string, password: string): Promise<void> {
const js = ` await this.admin(async (client) => {
const target = db.getSiblingDB(${lit(database)}); const target = client.db(database);
let existing = null; const roles = [{ role: "dbOwner", db: database }];
try { existing = target.getUser(${lit(user)}); } catch (e) { existing = null; } const found = await target.command({ usersInfo: user });
const roles = [{ role: "dbOwner", db: ${lit(database)} }]; if (Array.isArray(found.users) && found.users.length > 0) {
if (existing) { await target.command({ updateUser: user, pwd: password, roles });
target.updateUser(${lit(user)}, { pwd: ${lit(password)}, roles: roles });
} else { } else {
target.createUser({ user: ${lit(user)}, pwd: ${lit(password)}, roles: roles }); await target.command({ createUser: user, pwd: password, roles });
} }
print(EJSON.stringify({ ok: 1 })); });
`;
await this.evalJs<{ ok: number }>(js);
} }
/** /**
@@ -115,45 +103,43 @@ print(EJSON.stringify({ ok: 1 }));
* authentication failure or a missing role; an unreachable server rejects (novox/hq issue 120). * authentication failure or a missing role; an unreachable server rejects (novox/hq issue 120).
*/ */
async canAuthenticateAs(database: string, user: string, password: string): Promise<boolean> { async canAuthenticateAs(database: string, user: string, password: string): Promise<boolean> {
// Connected without credentials, then authenticated inside the eval from the environment, so // Credentials as options, never in a URI, so the consumer's password is in no message a failed
// the consumer's password is neither on argv nor in the message of a failed command. // connection prints.
const uri = `mongodb://${this.conn.host}:${this.conn.port}/?serverSelectionTimeoutMS=10000`; const client = new Driver(`mongodb://${this.conn.host}:${this.conn.port}/?directConnection=true`, {
const js = auth: { username: user, password },
"const t = db.getSiblingDB(process.env.MESH_HOLDS_DB);" + authSource: database,
"t.auth(process.env.MESH_HOLDS_USER, process.env.MESH_HOLDS_PW);" + serverSelectionTimeoutMS: 10_000,
"print(EJSON.stringify(t.runCommand({ connectionStatus: 1 }).authInfo.authenticatedUserRoles))"; });
let stdout: string;
try { try {
({ stdout } = await run("mongosh", [uri, "--quiet", "--eval", js], { await client.connect();
env: { ...process.env, MESH_HOLDS_DB: database, MESH_HOLDS_USER: user, MESH_HOLDS_PW: password }, const status = await client.db(database).command({ connectionStatus: 1 });
timeout: 30_000, const roles = (status.authInfo?.authenticatedUserRoles ?? []) as { role: string; db: string }[];
}));
} catch (err) {
const text = `${(err as { stderr?: string }).stderr ?? ""}${(err as { stdout?: string }).stdout ?? ""}`;
if (/Authentication failed|AuthenticationFailed/i.test(text)) return false;
throw new Error(`mongosh could not check ${user}: ${text.trim().slice(0, 500) || String((err as Error).message).split("\n")[0]}`);
}
const roles = JSON.parse(stdout.trim()) as { role: string; db: string }[];
return roles.some((r) => r.role === "dbOwner" && r.db === database); return roles.some((r) => r.role === "dbOwner" && r.db === database);
} catch (err) {
if (isAuthFailure(err)) return false;
throw new Error(`mongodb could not check ${user}: ${String((err as Error).message).split("\n")[0]}`);
} finally {
await client.close();
}
} }
/** Drop a database and its owning user, idempotently. Dropping the database evicts its data; the /** Drop a database and its owning user, idempotently. Dropping the database evicts its data; the
* user is removed first so a re-grant of the same login starts clean. */ * user is removed first so a re-grant of the same login starts clean. */
async dropDatabaseAndUser(database: string, user: string): Promise<void> { async dropDatabaseAndUser(database: string, user: string): Promise<void> {
const js = ` await this.admin(async (client) => {
const target = db.getSiblingDB(${lit(database)}); const target = client.db(database);
try { target.dropUser(${lit(user)}); } catch (e) {} try {
target.dropDatabase(); await target.command({ dropUser: user });
print(EJSON.stringify({ ok: 1 })); } catch (err) {
`; if (!(err instanceof MongoServerError && err.code === 11)) throw err; // 11: UserNotFound
await this.evalJs<{ ok: number }>(js); }
await target.dropDatabase();
});
} }
/** List the databases on the server, with on-disk size, for the mongodb_list_databases tool. */ /** List the databases on the server, with on-disk size, for the mongodb_list_databases tool. */
async listDatabases(): Promise<DatabaseInfo[]> { async listDatabases(): Promise<DatabaseInfo[]> {
const res = await this.evalJs<{ databases: { name: string; sizeOnDisk?: number }[] }>( const res = await this.admin((client) => client.db("admin").admin().listDatabases());
`print(EJSON.stringify(db.adminCommand({ listDatabases: 1 })));`,
);
return (res.databases ?? []) return (res.databases ?? [])
.map((d) => ({ name: String(d.name), sizeBytes: Number(d.sizeOnDisk ?? 0) })) .map((d) => ({ name: String(d.name), sizeBytes: Number(d.sizeOnDisk ?? 0) }))
.sort((a, b) => a.name.localeCompare(b.name)); .sort((a, b) => a.name.localeCompare(b.name));
@@ -162,6 +148,8 @@ print(EJSON.stringify({ ok: 1 }));
/** /**
* Run a read-only `find` against a collection in a named database, for the mongodb_query tool. * Run a read-only `find` against a collection in a named database, for the mongodb_query tool.
* `find` mutates nothing; the limit is capped so a tool call cannot stream an unbounded result. * `find` mutates nothing; the limit is capped so a tool call cannot stream an unbounded result.
* Documents come back as relaxed Extended JSON — an ObjectId as `{"$oid": …}` — exactly as the
* shell's `EJSON.stringify` rendered them before.
*/ */
async find( async find(
database: string, database: string,
@@ -170,26 +158,29 @@ print(EJSON.stringify({ ok: 1 }));
limit: number, limit: number,
): Promise<Record<string, unknown>[]> { ): Promise<Record<string, unknown>[]> {
const capped = Math.max(1, Math.min(limit, 1000)); const capped = Math.max(1, Math.min(limit, 1000));
const js = const docs = await this.admin((client) =>
`print(EJSON.stringify(` + client
`db.getSiblingDB(${lit(database)}).getCollection(${lit(collection)})` + .db(database)
`.find(${JSON.stringify(filter)}).limit(${capped}).toArray()` + .collection(collection)
`));`; .find(BSON.EJSON.deserialize(filter as Document, { relaxed: true }) as Document)
return this.evalJs<Record<string, unknown>[]>(js); .limit(capped)
.toArray(),
);
return BSON.EJSON.serialize(docs, { relaxed: true }) as Record<string, unknown>[];
} }
} }
/** An authentication failure, as the server or the driver reports it. */
function isAuthFailure(err: unknown): boolean {
if (err instanceof MongoServerError && err.code === 18) return true; // 18: AuthenticationFailed
return /Authentication failed|AuthenticationFailed/i.test(String((err as Error)?.message ?? ""));
}
/** Generate a URL-safe password. */ /** Generate a URL-safe password. */
export function generatePassword(): string { export function generatePassword(): string {
return randomBytes(24).toString("base64url"); return randomBytes(24).toString("base64url");
} }
/** Embed a value as a JavaScript literal inside a mongosh snippet — JSON.stringify escapes quotes,
* backslashes and control characters, so a string cannot break out of the snippet. */
function lit(val: unknown): string {
return JSON.stringify(val);
}
function readSecretFile(path: string | undefined): string | undefined { function readSecretFile(path: string | undefined): string | undefined {
if (!path) return undefined; if (!path) return undefined;
try { try {
+3 -3
View File
@@ -1,9 +1,9 @@
// mongodb's events entrypoint, loaded by the per-node tool host (the provisioner container runs // mongodb's events entrypoint, launched by the node's runtime beside its tools and provisioner
// ./provisioner separately). The database lifecycle events are EMITTED from the provisioner, where // (novox/hq ADR 0198). The database lifecycle events are EMITTED from the provisioner, where
// the lifecycle actually happens (novox/hq ADR 0041/0042): // the lifecycle actually happens (novox/hq ADR 0041/0042):
// module.mongodb.database.provisioned — a consumer's database + owning user was created // module.mongodb.database.provisioned — a consumer's database + owning user was created
// module.mongodb.database.deprovisioned — that database was removed // module.mongodb.database.deprovisioned — that database was removed
// Here in the tool host we react to them, keeping a lightweight audit trail of who was granted a // Here in the runtime we react to them, keeping a lightweight audit trail of who was granted a
// database and who lost one — observability the provider itself is best placed to log. // database and who lost one — observability the provider itself is best placed to log.
import { on } from "@novox/mesh-sdk/events"; import { on } from "@novox/mesh-sdk/events";
+28 -43
View File
@@ -39,17 +39,9 @@
"mongodb-database": "${dir:grants}" "mongodb-database": "${dir:grants}"
}, },
"own-secrets": { "own-secrets": {
"root": "${dir:state}/root.secret", "root": "${dir:state}/root.secret"
"broker": "${dir:mesh-state}/broker"
}, },
"secrets-owner": "999:999",
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -71,6 +63,14 @@
"type": "network", "type": "network",
"name": "mongodb" "name": "mongodb"
}, },
{
"id": "server-root",
"type": "file",
"path": "${dir:state}/server-root.secret",
"mode": "0400",
"owner": "999:999",
"content": "${secret:root}"
},
{ {
"id": "server", "id": "server",
"type": "container", "type": "container",
@@ -86,46 +86,31 @@
], ],
"volumes": [ "volumes": [
"${dir:data}:/data/db", "${dir:data}:/data/db",
"${dir:state}/root.secret:/run/secrets/root:ro" "${dir:state}/server-root.secret:/run/secrets/root:ro"
] ]
},
{
"id": "runtime",
"type": "container",
"name": "mesh-mongodb",
"network": "mongodb",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:state}/root.secret:/run/secrets/root:ro"
],
"env": {
"MESH_PROVISION_MONGODB": "mongodb://root@mongodb-server:27017/admin?authSource=admin",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/root",
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_PROVISION_MONGODB": "mongodb://root@127.0.0.1:${port:27017}/admin?authSource=admin",
"MESH_PROVISION_PASSWORD_FILE": "${dir:state}/root.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
}
} }
] ]
} }
+2 -1
View File
@@ -5,7 +5,8 @@
"type": "module", "type": "module",
"private": true, "private": true,
"dependencies": { "dependencies": {
"@novox/mesh-sdk": "^0.1.1" "@novox/mesh-sdk": "^0.1.1",
"mongodb": "^6.21.0"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
+1 -2
View File
@@ -11,8 +11,7 @@
// same-named database under exactly that login — a name the consumer cannot learn is a database it // same-named database under exactly that login — a name the consumer cannot learn is a database it
// cannot reach. // cannot reach.
// //
// The commands run through MongoClient.evalJs(), which is the module's one execution boundary (see // The commands run through MongoClient, the official driver inside this bundle (see client.ts).
// client.ts).
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner"; import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
import { emit } from "@novox/mesh-sdk/events"; import { emit } from "@novox/mesh-sdk/events";
+2 -2
View File
@@ -1,6 +1,6 @@
// mongodb's tools — mongodb's own code (novox/hq ADR 0039), importing mongodb's own client. They // mongodb's tools — mongodb's own code (novox/hq ADR 0039), importing mongodb's own client. They
// return structured data; the mesh serves them through the sdk's tool harness. Both call through // return structured data; the mesh serves them through the sdk's tool harness. Both call the server
// MongoClient.evalJs(), the module's one execution boundary (see client.ts). // through MongoClient, the driver inside this bundle (see client.ts).
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools"; import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { MongoClient } from "../client.js"; import { MongoClient } from "../client.js";
-28
View File
@@ -1,28 +0,0 @@
# mosquitto's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/mosquitto
COPY . .
RUN node /app/node_modules/typescript/bin/tsc topics.ts client.ts index.ts tools/index.ts provisioner/index.ts bootstrap/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
# mosquitto's client and bootstrap drive `mosquitto_ctrl`; the apt package carries it with its
# shared libraries — the musl binary from the eclipse image would not load on this glibc base.
RUN apt-get update && apt-get install -y --no-install-recommends mosquitto \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app/modules/mosquitto/dist /app/modules/mosquitto/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/mosquitto/dist/index.js,/app/modules/mosquitto/dist/tools/index.js,/app/modules/mosquitto/dist/provisioner/index.js
+36 -4
View File
@@ -18,6 +18,7 @@ import { randomBytes } from "node:crypto";
import { connect as tcpConnect } from "node:net"; import { connect as tcpConnect } from "node:net";
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import { execFile } from "node:child_process"; import { execFile } from "node:child_process";
import { basename, dirname } from "node:path";
import { promisify } from "node:util"; import { promisify } from "node:util";
import { missingAcls, parseRoleAcls, staleAcls, wantedAcls } from "./topics.js"; import { missingAcls, parseRoleAcls, staleAcls, wantedAcls } from "./topics.js";
@@ -30,6 +31,14 @@ export interface MqttConn {
/** The Dynamic Security admin client the runtime authenticates as. */ /** The Dynamic Security admin client the runtime authenticates as. */
readonly adminUser: string; readonly adminUser: string;
readonly adminPassword: string; readonly adminPassword: string;
/**
* The broker's own container, when `mosquitto_ctrl` is run inside it rather than from this
* machine's packages. The broker's image carries the tool at the broker's version, and inside it
* the broker listens on 127.0.0.1:1883 whatever port the machine publishes.
*/
readonly container?: string;
/** The broker's image, to seed the security file before the broker has ever started. */
readonly image?: string;
} }
export class MosquittoClient { export class MosquittoClient {
@@ -53,7 +62,11 @@ export class MosquittoClient {
"mosquitto host or admin password is not set — mosquitto's own code cannot reach the broker", "mosquitto host or admin password is not set — mosquitto's own code cannot reach the broker",
); );
} }
return new MosquittoClient({ host, port, adminUser, adminPassword: adminPassword ?? "" }); return new MosquittoClient({
host, port, adminUser, adminPassword: adminPassword ?? "",
container: env.MESH_MQTT_CTRL_CONTAINER || undefined,
image: env.MESH_MQTT_CTRL_IMAGE || undefined,
});
} }
get host(): string { get host(): string {
@@ -84,16 +97,18 @@ export class MosquittoClient {
* to this single-purpose runtime container; see the module README. * to this single-purpose runtime container; see the module README.
*/ */
async ctl(...args: string[]): Promise<string> { async ctl(...args: string[]): Promise<string> {
const inside = this.conn.container !== undefined;
const base = [ const base = [
"-h", this.conn.host, "-h", inside ? "127.0.0.1" : this.conn.host,
"-p", String(this.conn.port), "-p", inside ? "1883" : String(this.conn.port),
"-u", this.conn.adminUser, "-u", this.conn.adminUser,
"-P", this.conn.adminPassword, "-P", this.conn.adminPassword,
]; ];
let stdout: string; let stdout: string;
let stderr: string; let stderr: string;
try { try {
({ stdout, stderr } = await run("mosquitto_ctrl", [...base, "dynsec", ...args], { const [command, argv] = this.ctrl([...base, "dynsec", ...args]);
({ stdout, stderr } = await run(command, argv, {
maxBuffer: 16 << 20, maxBuffer: 16 << 20,
timeout: 30_000, timeout: 30_000,
})); }));
@@ -240,10 +255,27 @@ export class MosquittoClient {
async initBootstrapFile(configFile: string): Promise<void> { async initBootstrapFile(configFile: string): Promise<void> {
// `dynsec init <file> <admin-username> [admin-password]` is an offline file operation — it does // `dynsec init <file> <admin-username> [admin-password]` is an offline file operation — it does
// not connect to the broker. The password is a positional argument (omitting it prompts). // not connect to the broker. The password is a positional argument (omitting it prompts).
if (this.conn.image) {
// Before the broker has ever started there is no container to enter: a throwaway one from the
// broker's own image writes the file into the directory the broker will mount.
await run("docker", [
"run", "--rm", "--entrypoint", "mosquitto_ctrl",
"-v", `${dirname(configFile)}:/mosquitto/data`,
this.conn.image,
"dynsec", "init", `/mosquitto/data/${basename(configFile)}`, this.conn.adminUser, this.conn.adminPassword,
], { maxBuffer: 16 << 20 });
return;
}
await run("mosquitto_ctrl", ["dynsec", "init", configFile, this.conn.adminUser, this.conn.adminPassword], { await run("mosquitto_ctrl", ["dynsec", "init", configFile, this.conn.adminUser, this.conn.adminPassword], {
maxBuffer: 16 << 20, maxBuffer: 16 << 20,
}); });
} }
/** How `mosquitto_ctrl` is run here: inside the broker's container when one is named. */
ctrl(argv: string[]): [string, string[]] {
if (this.conn.container) return ["docker", ["exec", this.conn.container, "mosquitto_ctrl", ...argv]];
return ["mosquitto_ctrl", argv];
}
} }
/** Generate a URL-safe password with no argv- or MQTT-hostile characters. */ /** Generate a URL-safe password with no argv- or MQTT-hostile characters. */
+42 -53
View File
@@ -32,8 +32,7 @@
"mqtt-topic": "${dir:grants}" "mqtt-topic": "${dir:grants}"
}, },
"own-secrets": { "own-secrets": {
"admin": "${dir:mesh-state}/admin", "admin": "${dir:mesh-state}/admin"
"broker": "${dir:mesh-state}/broker"
}, },
"listens": [ "listens": [
{ {
@@ -89,25 +88,28 @@
"name": "mosquitto" "name": "mosquitto"
}, },
{ {
"id": "bootstrap", "id": "bootstrap-env",
"type": "container", "type": "file",
"name": "mosquitto-bootstrap", "path": "${dir:state}/bootstrap.env",
"run-once": true, "mode": "0600",
"volumes": [ "content": "MESH_PROVISION_MQTT=127.0.0.1:${port:1883}\nMESH_PROVISION_ADMIN_USER=mesh-admin\nMESH_PROVISION_PASSWORD_FILE=${dir:mesh-state}/admin\nMESH_DYNSEC_FILE=${dir:data}/dynamic-security.json\nMESH_MQTT_CTRL_IMAGE=eclipse-mosquitto@sha256:38c0da4f2ef84284d47b3b3eeea1cb3bdeabe81ee10caf0cd5c5ff61ee3ea408\n"
"${dir:data}:/mosquitto/data",
"${dir:mesh-state}/admin:/run/secrets/admin:ro"
],
"env": {
"MESH_PROVISION_MQTT": "mosquitto:1883",
"MESH_PROVISION_ADMIN_USER": "mesh-admin",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/admin",
"MESH_DYNSEC_FILE": "/mosquitto/data/dynamic-security.json"
}, },
"args": [ {
"run", "id": "bootstrap",
"/app/modules/mosquitto/dist/bootstrap/index.js" "type": "process",
"name": "mosquitto-bootstrap",
"artifact": "code",
"run": [
"node",
"bootstrap/index.js"
], ],
"artifact": "runtime" "run-once": true,
"env-file": [
"${dir:state}/bootstrap.env"
],
"restart-on": [
"bootstrap-env"
]
}, },
{ {
"id": "server", "id": "server",
@@ -123,45 +125,32 @@
"${dir:data}:/mosquitto/data", "${dir:data}:/mosquitto/data",
"${dir:state}/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro" "${dir:state}/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro"
] ]
},
{
"id": "runtime",
"type": "container",
"name": "mesh-mosquitto",
"network": "mosquitto",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:mesh-state}/admin:/run/secrets/admin:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_PROVISION_MQTT": "mosquitto:1883",
"MESH_PROVISION_ADMIN_USER": "mesh-admin",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/admin"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js",
"bootstrap/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_PROVISION_MQTT": "127.0.0.1:${port:1883}",
"MESH_PROVISION_ADMIN_USER": "mesh-admin",
"MESH_PROVISION_PASSWORD_FILE": "${dir:mesh-state}/admin",
"MESH_MQTT_CTRL_CONTAINER": "mosquitto"
}
} }
] ]
} }
+19
View File
@@ -0,0 +1,19 @@
// Run after `npm run build`.
// mosquitto_ctrl runs inside the broker's own container when the manifest names it, so a machine
// needs no mosquitto package (whose index may be too stale to install from) and the tool always
// matches the broker's version.
import assert from "node:assert/strict";
import { test } from "node:test";
import { MosquittoClient } from "../dist/client.js"; // compiled: client.ts uses parameter properties, which type stripping cannot run
const env = { MESH_PROVISION_MQTT: "127.0.0.1:21883", MESH_MQTT_PASSWORD: "pw" };
test("named, the broker's container runs mosquitto_ctrl", () => {
const c = MosquittoClient.fromEnv({ ...env, MESH_MQTT_CTRL_CONTAINER: "mosquitto" });
assert.deepEqual(c.ctrl(["dynsec", "listClients"]), ["docker", ["exec", "mosquitto", "mosquitto_ctrl", "dynsec", "listClients"]]);
});
test("unnamed, this machine's mosquitto_ctrl runs", () => {
const c = MosquittoClient.fromEnv(env);
assert.deepEqual(c.ctrl(["dynsec", "listClients"]), ["mosquitto_ctrl", ["dynsec", "listClients"]]);
});
-43
View File
@@ -1,43 +0,0 @@
# mssql's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/mssql
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
# **sqlcmd, which this module's client drives, has to be here** — it never was, so every tool failed
# with `spawn sqlcmd ENOENT`. go-sqlcmd is one static binary; fetched at a pinned release and checked
# against its digest, so a build that receives anything else stops here.
FROM ${BUILD_BASE} AS sqlcmd
ARG SQLCMD_VERSION=v1.10.0
ARG SQLCMD_SHA256=92516d98c63d99b0994de5b61350c91f6915f9b76f139a59039fbcb225c2e987
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates bzip2 \
&& curl -fsSL -o /tmp/sqlcmd.tar.bz2 \
"https://github.com/microsoft/go-sqlcmd/releases/download/${SQLCMD_VERSION}/sqlcmd-linux-amd64.tar.bz2" \
&& echo "${SQLCMD_SHA256} /tmp/sqlcmd.tar.bz2" | sha256sum -c - \
&& tar -xjf /tmp/sqlcmd.tar.bz2 -C /usr/local/bin sqlcmd
FROM ${RUNTIME_BASE}
COPY --from=sqlcmd /usr/local/bin/sqlcmd /usr/local/bin/sqlcmd
COPY --from=build /app/modules/mssql/dist /app/modules/mssql/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/mssql/dist/index.js,/app/modules/mssql/dist/tools/index.js,/app/modules/mssql/dist/provisioner/index.js
+111 -98
View File
@@ -1,22 +1,74 @@
// mssql's admin client — mssql's own code, living in the module (novox/hq ADR 0039). Both this // mssql's admin client — mssql's own code, living in the module (novox/hq ADR 0039). Both this
// module's tools and its provisioner import it, and nothing outside mssql does. // module's tools and its provisioner import it, and nothing outside mssql does.
// //
// SQL is executed through `sqlcmd`, not a wire-protocol driver: the module may take NO npm // **The backend's own driver, inside the bundle** (novox/hq ADR 0198 §4). This used to shell out to
// dependency beyond @novox/mesh-sdk, and hand-rolling the TDS handshake, pre-login and query // `sqlcmd`, a binary the module's container fetched; the module's code now runs in the node's
// protocol is more surface than this should carry — so it shells out to the client the mssql // runtime, on machines whose system carries no SQL Server client, so it speaks TDS through the
// tools ship, the same way postgres drives itself through `psql`, minio through `mc`, and mailu // `mssql` driver its package.json names — installed and inlined into the bundle by the builder. One
// through doveadm. One boundary, `run()`, and every method is built on it. // boundary, `session()`, and every method is built on it: a connection as one login to one database,
// opened for one call and closed after, as one sqlcmd invocation was.
// //
// Structured rows come back as JSON: SQL Server itself renders the result with `FOR JSON`, and // Structured rows still come back as JSON rendered by SQL Server itself (`FOR JSON`), so a tool's
// this parses the single JSON document sqlcmd prints — far more robust than parsing sqlcmd's // answer is shaped exactly as it was: SQL Server owns the quoting and typing.
// column-aligned text, since SQL Server owns the quoting and typing.
import { isIP } from "node:net";
import { randomBytes } from "node:crypto"; import { randomBytes } from "node:crypto";
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import { execFile } from "node:child_process"; import sql from "mssql";
import { promisify } from "node:util";
const run = promisify(execFile); /** Where a session connects, and as whom. */
export interface Target {
readonly host: string;
readonly port: number;
readonly user: string;
readonly password: string;
readonly database: string;
}
/** One login's connection to one database: run a batch, answer the rows of its last result set. */
export interface Session {
/** `params` are bound as NVARCHAR parameters (`@name`), never written into the text. */
run(text: string, params?: Record<string, string>): Promise<Record<string, unknown>[]>;
close(): Promise<void>;
}
/** How a session is opened — the driver, or a test's fake. */
export type Connect = (to: Target) => Promise<Session>;
/**
* The driver's session: TLS, trusting the self-signed certificate the mssql image ships with (what
* sqlcmd's `-C` did), one connection, closed with the session.
*/
export const connectWithDriver: Connect = async (to) => {
const pool = new sql.ConnectionPool({
server: to.host,
port: to.port,
user: to.user,
password: to.password,
database: to.database,
// TLS names a host, never an address: Node refuses an IP as the server name (DEP0123, an error
// since Node 25), and the module reaches its server on loopback. The certificate is trusted
// either way, so the name only has to be one TLS accepts.
options: { encrypt: true, trustServerCertificate: true, ...(isIP(to.host) ? { serverName: "localhost" } : {}) },
pool: { min: 0, max: 1 },
connectionTimeout: 15_000,
requestTimeout: 60_000,
});
await pool.connect();
return {
async run(text, params = {}) {
const request = pool.request();
const names = Object.keys(params);
for (const name of names) request.input(name, sql.NVarChar, params[name]);
// A batch when nothing is bound — CREATE DATABASE must stand alone in its batch, which a
// parameterised query (sp_executesql) is not.
const result = names.length > 0 ? await request.query(text) : await request.batch(text);
const sets = (result.recordsets ?? []) as Record<string, unknown>[][];
return sets.length > 0 ? sets[sets.length - 1] : [];
},
close: () => pool.close(),
};
};
export interface QueryResult { export interface QueryResult {
/** The leading keyword of the statement, e.g. "SELECT", "CREATE". */ /** The leading keyword of the statement, e.g. "SELECT", "CREATE". */
@@ -45,20 +97,17 @@ export interface MssqlConn {
*/ */
export const READER = "mesh_mssql_reader"; export const READER = "mesh_mssql_reader";
/** Who a sqlcmd invocation logs in as, and whether the text is a caller's rather than the module's. */ /** Who a session logs in as. */
interface Invocation { interface Invocation {
readonly user: string; readonly user: string;
readonly password: string; readonly password: string;
/**
* A caller's text: sqlcmd substitutes no `$(NAME)` in it, which would read this process's
* environment — the administrator's password among it. (Its own commands are kept out by the
* caller's text never beginning a line; see readOnlyQuery.)
*/
readonly caller: boolean;
} }
export class MssqlClient { export class MssqlClient {
constructor(private readonly conn: MssqlConn) {} constructor(
private readonly conn: MssqlConn,
private readonly connect: Connect = connectWithDriver,
) {}
/** The reader is made once per process: idempotent, and repeating it re-sets a rotated password. */ /** The reader is made once per process: idempotent, and repeating it re-sets a rotated password. */
private readerReady?: Promise<void>; private readerReady?: Promise<void>;
@@ -90,63 +139,41 @@ export class MssqlClient {
return this.conn.port; return this.conn.port;
} }
/** /** Execute a batch that returns no rows (DDL and the like). A failed statement rejects. */
* Execute a batch that returns no rows (DDL and the like), through `sqlcmd`. The password is async exec(text: string, database = "master"): Promise<void> {
* passed by SQLCMDPASSWORD, never on argv, the way postgres passes PGPASSWORD; `-b` makes a await this.session(text, database);
* failed statement an error here rather than a success with a warning, and `-C` trusts the
* server's self-signed certificate the mssql image ships with.
*/
async exec(sql: string, database = "master"): Promise<void> {
await this.sqlcmd(sql, database);
} }
/** /**
* Run a SELECT and return its rows as objects. The caller's SQL must be a single SELECT; it is * Run a SELECT and return its rows as objects. The caller's SQL must be a single SELECT; it is
* wrapped so SQL Server renders the result with `FOR JSON PATH`, and the JSON document sqlcmd * wrapped so SQL Server renders the result with `FOR JSON PATH`, and the JSON document it answers
* prints (split across output lines for a large result, and reassembled here) is parsed. An * (split across rows for a large result, and reassembled here) is parsed. An empty result yields
* empty result yields no output at all — an empty array. * no rows — an empty array. `params` are bound as `@name`, never written into the text.
*/ */
async query( async query(
select: string, select: string,
database = "master", database = "master",
variables: Record<string, string> = {}, params: Record<string, string> = {},
): Promise<Record<string, unknown>[]> { ): Promise<Record<string, unknown>[]> {
const wrapped = `SET NOCOUNT ON;\n${stripTrailingSemis(select)}\nFOR JSON PATH, INCLUDE_NULL_VALUES;`; const wrapped = `SET NOCOUNT ON;\n${stripTrailingSemis(select)}\nFOR JSON PATH, INCLUDE_NULL_VALUES;`;
const stdout = await this.sqlcmd(wrapped, database, variables); return parseJsonRows(await this.session(wrapped, database, params));
return parseJsonRows(stdout);
} }
/** The one execution boundary: invoke `sqlcmd` and return its concatenated stdout. */ /** The one execution boundary: open a session as `as`, run `text`, close it. */
private async sqlcmd( private async session(
sql: string, text: string,
database: string, database: string,
variables: Record<string, string> = {}, params: Record<string, string> = {},
as: Invocation = { user: this.conn.user, password: this.conn.password, caller: false }, as: Invocation = { user: this.conn.user, password: this.conn.password },
): Promise<string> { ): Promise<Record<string, unknown>[]> {
// `-h -1` drops the column-header rule; `-y 0`/`-Y 0` lift the display-width cap so a long const session = await this.connect({
// JSON document is not truncated; `-W` trims trailing whitespace so the JSON chunks rejoin host: this.conn.host, port: this.conn.port, user: as.user, password: as.password, database,
// cleanly. sqlcmd from the mssql-tools ships in the runtime container, the way `psql` ships });
// with postgres's — the module owns its own code (ADR 0039) and shells out to it. try {
const { stdout } = await run( return await session.run(text, params);
"sqlcmd", } finally {
[ await session.close();
"-S", `${this.conn.host},${this.conn.port}`, }
"-U", as.user,
"-d", database,
...(as.caller ? ["-x"] : []),
"-C",
"-b",
"-h", "-1",
"-y", "0",
"-Y", "0",
"-W",
"-Q", sql,
],
// `variables` reach sqlcmd as environment variables, which it substitutes as `$(NAME)` scripting
// variables: a value that must not appear on argv, or in the message of a failed command.
{ env: { ...process.env, ...variables, SQLCMDPASSWORD: as.password }, maxBuffer: 16 << 20 },
);
return stdout;
} }
/** /**
@@ -173,7 +200,7 @@ export class MssqlClient {
`SELECT 1 AS ok FROM sys.databases WHERE name = ${literal(database)}`, `SELECT 1 AS ok FROM sys.databases WHERE name = ${literal(database)}`,
); );
if (dbs.length === 0) { if (dbs.length === 0) {
// CREATE DATABASE must stand alone in its batch; it runs as its own sqlcmd invocation. // CREATE DATABASE must stand alone in its batch; it runs as its own session.
await this.exec(`CREATE DATABASE ${ident(database)}`); await this.exec(`CREATE DATABASE ${ident(database)}`);
} }
@@ -206,15 +233,14 @@ export class MssqlClient {
* nothing logs in and no failed-login is recorded (novox/hq issue 120). * nothing logs in and no failed-login is recorded (novox/hq issue 120).
*/ */
async holdsLogin(database: string, login: string, password: string): Promise<boolean> { async holdsLogin(database: string, login: string, password: string): Promise<boolean> {
// The password reaches sqlcmd as a scripting variable from the environment, never inside the // The password is a bound parameter, never inside the query text, so it is in no message of a
// query text, so it is neither on argv nor in the message of a failed command. It is the mesh's // failed statement.
// minted value, which carries no quote.
const server = await this.query( const server = await this.query(
`SELECT CAST(CASE WHEN EXISTS (SELECT 1 FROM sys.sql_logins WHERE name = ${literal(login)} ` + `SELECT CAST(CASE WHEN EXISTS (SELECT 1 FROM sys.sql_logins WHERE name = ${literal(login)} ` +
`AND is_disabled = 0 AND PWDCOMPARE(N'$(MESHHOLDSPW)', password_hash) = 1) ` + `AND is_disabled = 0 AND PWDCOMPARE(@meshholdspw, password_hash) = 1) ` +
`AND DB_ID(${literal(database)}) IS NOT NULL THEN 1 ELSE 0 END AS int) AS ok`, `AND DB_ID(${literal(database)}) IS NOT NULL THEN 1 ELSE 0 END AS int) AS ok`,
"master", "master",
{ MESHHOLDSPW: password }, { meshholdspw: password },
); );
if (Number(server[0]?.ok) !== 1) return false; if (Number(server[0]?.ok) !== 1) return false;
// The user must be this login's, by SID, and a db_owner. A user orphaned by a restore has the // The user must be this login's, by SID, and a db_owner. A user orphaned by a restore has the
@@ -294,35 +320,27 @@ export class MssqlClient {
* (novox/hq issue 193). Read-only by the login, not by a transaction wrapped around the text; the * (novox/hq issue 193). Read-only by the login, not by a transaction wrapped around the text; the
* rows are rendered by FOR JSON. Never as the administrator: without the reader's password the call * rows are rendered by FOR JSON. Never as the administrator: without the reader's password the call
* is refused. * is refused.
*
* The text goes to the server as it is, over the driver: there is no client between that reads a
* line of its own (sqlcmd's `:!!`, which could start a program) or substitutes `$(NAME)` from this
* process's environment, so neither the one-line rule nor `-x` has anything left to guard.
*/ */
async readOnlyQuery(database: string, sql: string): Promise<QueryResult> { async readOnlyQuery(database: string, text: string): Promise<QueryResult> {
const password = this.conn.readerPassword; const password = this.conn.readerPassword;
if (!password) throw readerMissing(); if (!password) throw readerMissing();
// **One line, refused otherwise.** sqlcmd reads a line that BEGINS with `:` or `!!` as its own
// command rather than SQL, and `:!!` starts a program in this container, which holds the
// administrator's password. Its switch for refusing those (-X) makes it ignore -Q in the
// version shipped here, so instead no line of a caller's text can begin one: the text follows
// this module's own on the first line, and a line break in it is refused. Proven on a throwaway
// server: the same text at the start of a line ran a program; mid-line it is a syntax error.
if (/[\r\n]/.test(sql)) {
throw new Error(
"mssql_query: the statement must be one line — sqlcmd takes a line beginning with ':' or " +
"'!!' as a command of its own, which can start a program (novox/hq issue 193)",
);
}
this.readerReady ??= this.ensureReader().catch((err) => { this.readerReady ??= this.ensureReader().catch((err) => {
this.readerReady = undefined; // asked again next call, not failed for the process's life this.readerReady = undefined; // asked again next call, not failed for the process's life
throw err; throw err;
}); });
await this.readerReady; await this.readerReady;
const stdout = await this.sqlcmd( const rows = await this.session(
`SET NOCOUNT ON; ${stripTrailingSemis(sql)}\nFOR JSON PATH, INCLUDE_NULL_VALUES;`, `SET NOCOUNT ON; ${stripTrailingSemis(text)}\nFOR JSON PATH, INCLUDE_NULL_VALUES;`,
database, database,
{}, {},
{ user: READER, password, caller: true }, { user: READER, password },
); );
const command = /^\s*([A-Za-z]+)/.exec(sql)?.[1]?.toUpperCase() ?? ""; const command = /^\s*([A-Za-z]+)/.exec(text)?.[1]?.toUpperCase() ?? "";
return { command, rows: parseJsonRows(stdout) }; return { command, rows: parseJsonRows(rows) };
} }
} }
@@ -372,18 +390,13 @@ function safeUrl(raw: string): URL | undefined {
} }
/** /**
* Parse the JSON a FOR JSON query prints through sqlcmd. SQL Server splits a large FOR JSON result * Parse the JSON a FOR JSON query answers. SQL Server splits a large FOR JSON result into
* into ~2033-character chunks, one per output row; with `-h -1 -W` each lands on its own line, so * ~2033-character chunks, one per row of a single column, so the document is reassembled by
* the document is reassembled by concatenating the non-empty lines. No output (an empty result, or * concatenating that column in order. No rows (an empty result, or a pure DDL batch) means none.
* a pure DDL batch) means no rows.
*/ */
function parseJsonRows(stdout: string): Record<string, unknown>[] { function parseJsonRows(rows: Record<string, unknown>[]): Record<string, unknown>[] {
const joined = stdout const joined = rows.map((row) => String(Object.values(row)[0] ?? "")).join("");
.split(/\r?\n/) if (joined.trim().length === 0) return [];
.map((l) => l.trimEnd())
.filter((l) => l.length > 0)
.join("");
if (joined.length === 0) return [];
const parsed = JSON.parse(joined); const parsed = JSON.parse(joined);
return Array.isArray(parsed) ? (parsed as Record<string, unknown>[]) : [parsed as Record<string, unknown>]; return Array.isArray(parsed) ? (parsed as Record<string, unknown>[]) : [parsed as Record<string, unknown>];
} }
+3 -3
View File
@@ -1,9 +1,9 @@
// mssql's events entrypoint, loaded by the per-node tool host (the provisioner container runs // mssql's events entrypoint, launched by the node's runtime beside its tools and provisioner
// ./provisioner separately). The database lifecycle events are EMITTED from the provisioner, where // (novox/hq ADR 0198). The database lifecycle events are EMITTED from the provisioner, where
// the lifecycle actually happens (novox/hq ADR 0041/0042): // the lifecycle actually happens (novox/hq ADR 0041/0042):
// module.mssql.database.provisioned — a consumer's database + login/user was created // module.mssql.database.provisioned — a consumer's database + login/user was created
// module.mssql.database.deprovisioned — that database was removed // module.mssql.database.deprovisioned — that database was removed
// Here in the tool host we react to them, keeping a lightweight audit trail of who was granted a // Here in the runtime we react to them, keeping a lightweight audit trail of who was granted a
// database and who lost one — observability the provider itself is best placed to log. // database and who lost one — observability the provider itself is best placed to log.
import { on } from "@novox/mesh-sdk/events"; import { on } from "@novox/mesh-sdk/events";
+19 -42
View File
@@ -38,16 +38,9 @@
}, },
"own-secrets": { "own-secrets": {
"sa": "${dir:state}/sa.secret", "sa": "${dir:state}/sa.secret",
"broker": "${dir:mesh-state}/broker",
"reader": "${dir:state}/reader.secret" "reader": "${dir:state}/reader.secret"
}, },
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -93,46 +86,30 @@
"${dir:data}:/var/opt/mssql" "${dir:data}:/var/opt/mssql"
], ],
"secrets-in-environment": "the image documents only MSSQL_SA_PASSWORD, no _FILE and no configuration field; not convertible without a wrapper entrypoint" "secrets-in-environment": "the image documents only MSSQL_SA_PASSWORD, no _FILE and no configuration field; not convertible without a wrapper entrypoint"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-mssql",
"network": "mssql",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:/var/lib/mssql/grants:ro",
"${dir:state}/sa.secret:/run/secrets/sa:ro",
"${dir:state}/reader.secret:/run/secrets/reader:ro"
],
"env": {
"MESH_PROVISION_MSSQL": "mssql://sa@mssql:1433/master",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/sa",
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "/var/lib/mssql/grants/mesh.json",
"MESH_MSSQL_READER_PASSWORD_FILE": "/run/secrets/reader"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_PROVISION_MSSQL": "mssql://sa@127.0.0.1:${port:1433}/master",
"MESH_PROVISION_PASSWORD_FILE": "${dir:state}/sa.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_MSSQL_READER_PASSWORD_FILE": "${dir:state}/reader.secret"
}
} }
] ]
} }
+32
View File
@@ -0,0 +1,32 @@
// Ambient types for `mssql`, which ships its types only in the separate `@types/mssql` package. This
// declares the slice client.ts uses — the precedent mesh-catalog's pg.d.ts sets — so the module
// type-checks without deciding what runs: the real `mssql` is the package.json dependency the
// builder installs and inlines into the bundle (novox/hq ADR 0198 §4).
declare module "mssql" {
interface Result {
recordsets: unknown;
}
interface Request {
input(name: string, type: unknown, value: unknown): Request;
query(text: string): Promise<Result>;
batch(text: string): Promise<Result>;
}
class ConnectionPool {
constructor(config: {
server: string;
port?: number;
user?: string;
password?: string;
database?: string;
options?: { encrypt?: boolean; trustServerCertificate?: boolean; serverName?: string };
pool?: { min?: number; max?: number };
connectionTimeout?: number;
requestTimeout?: number;
});
connect(): Promise<ConnectionPool>;
request(): Request;
close(): Promise<void>;
}
const sql: { ConnectionPool: typeof ConnectionPool; NVarChar: unknown };
export default sql;
}
+3 -2
View File
@@ -5,11 +5,12 @@
"type": "module", "type": "module",
"private": true, "private": true,
"scripts": { "scripts": {
"build": "tsc client.ts index.ts tools/index.ts provisioner/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist", "build": "tsc mssql.d.ts client.ts index.ts tools/index.ts provisioner/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
"test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'" "test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'"
}, },
"dependencies": { "dependencies": {
"@novox/mesh-sdk": "^0.1.1" "@novox/mesh-sdk": "^0.1.1",
"mssql": "^11.0.2"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^22.0.0", "@types/node": "^22.0.0",
+50 -63
View File
@@ -1,96 +1,83 @@
// What holds mssql_query to being read-only (novox/hq issue 193): a caller's statement runs as the // What holds mssql_query to being read-only (novox/hq issue 193): a caller's statement runs as the
// reader login and never as the administrator, with sqlcmd's variable substitution off, on one line // reader login and never as the administrator, with no transaction wrapped around it as text, and
// that follows the module's own — a line break is refused before sqlcmd starts — and with no // without the reader's password the statement is refused.
// transaction wrapped around it as text. Without the reader's password the statement is refused.
// //
// sqlcmd is a fake on PATH that records each call's login, flags and text. That the reader cannot // The driver is a fake session that records each call's login, database, text and bound
// write is the server's to enforce and was proven against a real server; this holds the module to // parameters. That the reader cannot write is the server's to enforce and was proven against a real
// asking for it. Run against the compiled module (npm test builds first), the way the runtime loads it. // server; this holds the module to asking for it. Run against the compiled module (npm test builds
// first), the way the runtime loads it.
import { test, before, after } from "node:test"; import { test } from "node:test";
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { chmod, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { MssqlClient, READER } from "../dist/client.js"; import { MssqlClient, READER, type Connect, type Target } from "../dist/client.js";
let dir: string; interface Call extends Target {
let log: string; text: string;
const originalPath = process.env.PATH; params: Record<string, string>;
}
before(async () => { function recording(): { connect: Connect; calls: Call[] } {
dir = await mkdtemp(join(tmpdir(), "mssql-reader-")); const calls: Call[] = [];
log = join(dir, "calls.jsonl"); const connect: Connect = async (to) => ({
await writeFile(join(dir, "sqlcmd"), `#!/usr/bin/env node async run(text, params = {}) {
const fs = require("node:fs"); calls.push({ ...to, text, params });
const args = process.argv.slice(2); if (/FROM sys.server_principals/.test(text)) return [];
const at = (flag) => args[args.indexOf(flag) + 1]; // FOR JSON answers its document split across rows of one column.
fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify({ if (/FOR JSON/.test(text)) return [{ JSON_F52E: '[{"name":"al' }, { JSON_F52E: 'pha","n":1}]' }];
user: at("-U"), database: at("-d"), sql: at("-Q"), noVariables: args.includes("-x"), return [];
password: process.env.SQLCMDPASSWORD, },
}) + "\\n"); async close() {},
const sql = at("-Q");
if (/FROM sys.server_principals/.test(sql)) process.stdout.write("");
else if (/FOR JSON/.test(sql)) process.stdout.write('[{"name":"alpha","n":1}]\\n');
`);
await chmod(join(dir, "sqlcmd"), 0o755);
process.env.PATH = `${dir}:${originalPath}`;
}); });
return { connect, calls };
after(async () => {
process.env.PATH = originalPath;
await rm(dir, { recursive: true, force: true });
});
async function calls(): Promise<Record<string, unknown>[]> {
const text = await readFile(log, "utf8").catch(() => "");
await writeFile(log, "");
return text.split("\n").filter(Boolean).map((line) => JSON.parse(line));
} }
const conn = { host: "127.0.0.1", port: 1433, user: "sa", password: "admin-secret" }; const conn = { host: "127.0.0.1", port: 1433, user: "sa", password: "admin-secret" };
test("a caller's statement runs as the reader, without variables, on the module's first line", async () => { test("a caller's statement runs as the reader, as it was written, on the database it names", async () => {
const client = new MssqlClient({ ...conn, readerPassword: "reader-secret" }); const { connect, calls } = recording();
const client = new MssqlClient({ ...conn, readerPassword: "reader-secret" }, connect);
const result = await client.readOnlyQuery("inventory", "SELECT '$(SQLCMDPASSWORD)' AS p"); const result = await client.readOnlyQuery("inventory", "SELECT '$(SQLCMDPASSWORD)' AS p");
const asked = (await calls()).at(-1)!; const asked = calls.at(-1)!;
assert.equal(asked.user, READER, "the statement never runs as the administrator"); assert.equal(asked.user, READER, "the statement never runs as the administrator");
assert.equal(asked.password, "reader-secret"); assert.equal(asked.password, "reader-secret");
assert.equal(asked.noVariables, true, "no $(NAME) is substituted in a caller's text"); assert.equal(asked.database, "inventory");
const [first] = String(asked.sql).split("\n"); assert.ok(asked.text.startsWith("SET NOCOUNT ON; SELECT '$(SQLCMDPASSWORD)' AS p\nFOR JSON PATH"),
assert.ok(first.startsWith("SET NOCOUNT ON; SELECT '$(SQLCMDPASSWORD)'"), "the caller's text never begins a line"); "the caller's text reaches the server unaltered");
assert.doesNotMatch(String(asked.sql), /BEGIN TRANSACTION|ROLLBACK/, "no transaction wrapped around it as text"); assert.doesNotMatch(asked.text, /BEGIN TRANSACTION|ROLLBACK/, "no transaction wrapped around it as text");
assert.deepEqual(result.rows, [{ name: "alpha", n: 1 }]); assert.deepEqual(result.rows, [{ name: "alpha", n: 1 }], "a FOR JSON document split across rows is reassembled");
assert.equal(result.command, "SELECT"); assert.equal(result.command, "SELECT");
}); });
test("a line break in a caller's statement is refused before sqlcmd starts", async () => {
const client = new MssqlClient({ ...conn, readerPassword: "reader-secret" });
for (const sql of ["SELECT 1\n:!! id", "SELECT 1\r\n:!! id", "SELECT 1\r:!! id"]) {
await assert.rejects(client.readOnlyQuery("inventory", sql), /must be one line/);
}
assert.deepEqual(await calls(), []);
});
test("the reader is made as the administrator, kept out of sysadmin, and granted only reading", async () => { test("the reader is made as the administrator, kept out of sysadmin, and granted only reading", async () => {
const client = new MssqlClient({ ...conn, readerPassword: "reader-secret" }); const { connect, calls } = recording();
const client = new MssqlClient({ ...conn, readerPassword: "reader-secret" }, connect);
await client.readOnlyQuery("inventory", "SELECT 1 AS x"); await client.readOnlyQuery("inventory", "SELECT 1 AS x");
await client.readOnlyQuery("inventory", "SELECT 2 AS x"); await client.readOnlyQuery("inventory", "SELECT 2 AS x");
const made = await calls(); const asAdmin = calls.filter((c) => c.user === "sa").map((c) => c.text);
const asAdmin = made.filter((c) => c.user === "sa").map((c) => String(c.sql));
assert.ok(asAdmin.some((s) => s.startsWith(`CREATE LOGIN [${READER}]`))); assert.ok(asAdmin.some((s) => s.startsWith(`CREATE LOGIN [${READER}]`)));
assert.ok(asAdmin.some((s) => /ALTER SERVER ROLE sysadmin DROP MEMBER/.test(s))); assert.ok(asAdmin.some((s) => /ALTER SERVER ROLE sysadmin DROP MEMBER/.test(s)));
assert.ok(asAdmin.includes(`GRANT CONNECT ANY DATABASE TO [${READER}]`)); assert.ok(asAdmin.includes(`GRANT CONNECT ANY DATABASE TO [${READER}]`));
assert.ok(asAdmin.includes(`GRANT SELECT ALL USER SECURABLES TO [${READER}]`)); assert.ok(asAdmin.includes(`GRANT SELECT ALL USER SECURABLES TO [${READER}]`));
assert.equal(asAdmin.filter((s) => s.startsWith("CREATE LOGIN")).length, 1, "made once, not per call"); assert.equal(asAdmin.filter((s) => s.startsWith("CREATE LOGIN")).length, 1, "made once, not per call");
assert.equal(made.filter((c) => c.user === READER).length, 2); assert.equal(calls.filter((c) => c.user === READER).length, 2);
}); });
test("without the reader's password the statement is refused, and nothing runs as the administrator", async () => { test("without the reader's password the statement is refused, and nothing runs as the administrator", async () => {
const client = new MssqlClient(conn); const { connect, calls } = recording();
const client = new MssqlClient(conn, connect);
await assert.rejects(client.readOnlyQuery("inventory", "SELECT 1"), /refused rather than run as the administrator/); await assert.rejects(client.readOnlyQuery("inventory", "SELECT 1"), /refused rather than run as the administrator/);
assert.deepEqual(await calls(), []); assert.deepEqual(calls, []);
});
test("a consumer's password is checked as a bound parameter, never in the text", async () => {
const { connect, calls } = recording();
const client = new MssqlClient(conn, connect);
await client.holdsLogin("shop", "shop_login", "minted-secret");
const asked = calls[0];
assert.equal(asked.params.meshholdspw, "minted-secret");
assert.doesNotMatch(asked.text, /minted-secret/);
assert.match(asked.text, /PWDCOMPARE\(@meshholdspw, password_hash\)/);
}); });
+1 -1
View File
@@ -8,5 +8,5 @@
"skipLibCheck": true, "skipLibCheck": true,
"noEmit": true "noEmit": true
}, },
"include": ["client.ts", "index.ts", "provisioner/index.ts", "tools/index.ts"] "include": ["mssql.d.ts", "client.ts", "index.ts", "provisioner/index.ts", "tools/index.ts"]
} }
+173
View File
@@ -0,0 +1,173 @@
package main
import (
"context"
"errors"
"fmt"
"math"
"net"
"sort"
"strconv"
"strings"
"time"
)
// Bounds on what a caller may ask: a check is a probe, never a wait anyone can make long.
const (
DefaultTimeout = 3 * time.Second
MostTimeout = 30 * time.Second
)
// TCPResult is what netcheck_tcp answers.
type TCPResult struct {
Host string `json:"host"`
Port int `json:"port"`
Address string `json:"address,omitempty"`
Reachable bool `json:"reachable"`
ElapsedMS int64 `json:"elapsed_ms"`
Error string `json:"error,omitempty"`
}
// CheckTCP opens one TCP connection and closes it, sending nothing. A port that refuses or a host
// that does not answer is a result, not a failure of the tool; only a malformed question is.
func CheckTCP(host string, port int, timeout time.Duration) (TCPResult, error) {
if port < 1 || port > 65535 {
return TCPResult{}, fmt.Errorf("port %d is not a TCP port (1-65535)", port)
}
out := TCPResult{Host: host, Port: port}
start := time.Now()
conn, err := net.DialTimeout("tcp", net.JoinHostPort(host, strconv.Itoa(port)), timeout)
out.ElapsedMS = time.Since(start).Milliseconds()
if err != nil {
out.Error = err.Error()
return out, nil
}
out.Address = conn.RemoteAddr().String()
out.Reachable = true
_ = conn.Close()
return out, nil
}
// DNSResult is what netcheck_dns answers.
type DNSResult struct {
Name string `json:"name"`
Type string `json:"type"`
Answers []string `json:"answers"`
ElapsedMS int64 `json:"elapsed_ms"`
Error string `json:"error,omitempty"`
}
// DNSTypes are the record types netcheck_dns looks up.
var DNSTypes = []string{"A", "AAAA", "CNAME", "TXT", "MX"}
// CheckDNS looks a name up with the machine's resolver. Built without cgo, Go's own resolver reads
// the machine's /etc/resolv.conf and /etc/hosts, which is the resolver this machine's programs use.
// A name that does not resolve is a result with its error; an unknown type is refused.
func CheckDNS(name, kind string, timeout time.Duration) (DNSResult, error) {
kind = strings.ToUpper(strings.TrimSpace(kind))
if kind == "" {
kind = "A"
}
known := false
for _, t := range DNSTypes {
known = known || t == kind
}
if !known {
return DNSResult{}, fmt.Errorf("type %q is not one netcheck_dns looks up (%s)", kind, strings.Join(DNSTypes, ", "))
}
out := DNSResult{Name: name, Type: kind, Answers: []string{}}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
r := net.DefaultResolver
start := time.Now()
var err error
switch kind {
case "A", "AAAA":
network := "ip4"
if kind == "AAAA" {
network = "ip6"
}
var ips []net.IP
if ips, err = r.LookupIP(ctx, network, name); err == nil {
for _, ip := range ips {
out.Answers = append(out.Answers, ip.String())
}
}
case "CNAME":
var cname string
if cname, err = r.LookupCNAME(ctx, name); err == nil {
out.Answers = append(out.Answers, cname)
}
case "TXT":
var txts []string
if txts, err = r.LookupTXT(ctx, name); err == nil {
out.Answers = append(out.Answers, txts...)
}
case "MX":
var mxs []*net.MX
if mxs, err = r.LookupMX(ctx, name); err == nil {
for _, mx := range mxs {
out.Answers = append(out.Answers, fmt.Sprintf("%d %s", mx.Pref, mx.Host))
}
}
}
out.ElapsedMS = time.Since(start).Milliseconds()
if err != nil {
out.Error = err.Error()
}
if kind != "MX" {
sort.Strings(out.Answers)
}
return out, nil
}
// text is a required string argument.
func text(args map[string]any, key string) (string, error) {
s, _ := args[key].(string)
s = strings.TrimSpace(s)
if s == "" {
return "", fmt.Errorf("%s is required", key)
}
return s, nil
}
// whole is an integer argument, given as a JSON number or a numeric string; fallback when absent.
func whole(args map[string]any, key string, fallback int) (int, error) {
v, given := args[key]
if !given || v == nil {
if fallback == 0 {
return 0, fmt.Errorf("%s is required", key)
}
return fallback, nil
}
switch n := v.(type) {
case float64:
if n != math.Trunc(n) {
return 0, fmt.Errorf("%s must be a whole number, not %v", key, n)
}
return int(n), nil
case string:
i, err := strconv.Atoi(strings.TrimSpace(n))
if err != nil {
return 0, fmt.Errorf("%s must be a whole number, not %q", key, n)
}
return i, nil
}
return 0, errors.New(key + " must be a whole number")
}
// timeoutOf is timeout_ms, defaulted and bounded.
func timeoutOf(args map[string]any) (time.Duration, error) {
ms, err := whole(args, "timeout_ms", int(DefaultTimeout/time.Millisecond))
if err != nil {
return 0, err
}
if ms < 1 {
return 0, fmt.Errorf("timeout_ms must be at least 1, not %d", ms)
}
d := time.Duration(ms) * time.Millisecond
if d > MostTimeout {
d = MostTimeout
}
return d, nil
}
@@ -0,0 +1,68 @@
package main
import (
"net"
"testing"
"time"
)
func TestATCPPortThatListensIsReachableAndOneThatDoesNotIsNot(t *testing.T) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
port := l.Addr().(*net.TCPAddr).Port
got, err := CheckTCP("127.0.0.1", port, time.Second)
if err != nil || !got.Reachable || got.Error != "" {
t.Fatalf("a listening port: %+v, %v", got, err)
}
l.Close()
got, err = CheckTCP("127.0.0.1", port, time.Second)
if err != nil || got.Reachable || got.Error == "" {
t.Fatalf("a closed port is reported as a result with its error, not a failure: %+v, %v", got, err)
}
}
func TestAPortOutsideTheRangeIsRefused(t *testing.T) {
for _, p := range []int{0, -1, 65536} {
if _, err := CheckTCP("127.0.0.1", p, time.Second); err == nil {
t.Errorf("port %d was accepted", p)
}
}
}
func TestDNSAnswersFromTheMachinesResolverAndRefusesAnUnknownType(t *testing.T) {
got, err := CheckDNS("localhost", "a", time.Second)
if err != nil || got.Type != "A" || len(got.Answers) == 0 {
t.Fatalf("localhost A: %+v, %v", got, err)
}
if _, err := CheckDNS("localhost", "SRV", time.Second); err == nil {
t.Fatal("an unknown record type was accepted")
}
got, err = CheckDNS("no-such-name.invalid", "A", time.Second)
if err != nil || got.Error == "" || len(got.Answers) != 0 {
t.Fatalf("a name that does not resolve is a result with its error: %+v, %v", got, err)
}
}
func TestTimeoutIsDefaultedAndBounded(t *testing.T) {
if d, _ := timeoutOf(map[string]any{}); d != DefaultTimeout {
t.Errorf("default: %v", d)
}
if d, _ := timeoutOf(map[string]any{"timeout_ms": float64(10 * 60 * 1000)}); d != MostTimeout {
t.Errorf("bounded: %v", d)
}
if _, err := timeoutOf(map[string]any{"timeout_ms": float64(0)}); err == nil {
t.Error("a zero timeout was accepted")
}
}
func TestBothToolsAreListedUnprefixed(t *testing.T) {
names := map[string]bool{}
for _, tool := range tools() {
names[tool.Name] = true
}
if !names["netcheck_tcp"] || !names["netcheck_dns"] || len(names) != 2 {
t.Fatalf("tools: %v", names)
}
}
+72
View File
@@ -0,0 +1,72 @@
// netcheck's Go tools bundle (novox/hq ADR 0188, ADR 0193): a process the node's runtime launches
// and speaks MCP over stdio to, through the Go SDK. It serves the two checks that are the machine's
// own sockets and resolver — a TCP connect and a DNS lookup — and nothing that changes anything.
// The module's HTTP check is its TypeScript bundle; the runtime serves both under one module.
package main
import (
"fmt"
"os"
stdio "git.novox.be/novox/mesh-sdk/go"
)
func main() {
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): netcheck.
if err := stdio.Serve("", tools()); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func tools() []stdio.Tool {
return []stdio.Tool{
{
Name: "netcheck_tcp",
Description: "Check whether a TCP port is reachable from this machine: opens one connection " +
"and closes it at once, sending nothing. Answers reachable, elapsed_ms and the error when not.",
Input: map[string]any{
"host": map[string]any{"type": "string", "description": "host name or IP address"},
"port": map[string]any{"type": "integer", "description": "TCP port, 1-65535"},
"timeout_ms": map[string]any{"type": "integer", "description": "give up after this long (default 3000, at most 30000)"},
},
Run: func(args map[string]any) (any, error) {
host, err := text(args, "host")
if err != nil {
return nil, err
}
port, err := whole(args, "port", 0)
if err != nil {
return nil, err
}
timeout, err := timeoutOf(args)
if err != nil {
return nil, err
}
return CheckTCP(host, port, timeout)
},
},
{
Name: "netcheck_dns",
Description: "Look a name up with this machine's resolver (its /etc/resolv.conf and /etc/hosts). " +
"type is A, AAAA, CNAME, TXT or MX; answers the records found, or the error.",
Input: map[string]any{
"name": map[string]any{"type": "string", "description": "the name to look up"},
"type": map[string]any{"type": "string", "enum": []string{"A", "AAAA", "CNAME", "TXT", "MX"}, "description": "record type (default A)"},
"timeout_ms": map[string]any{"type": "integer", "description": "give up after this long (default 3000, at most 30000)"},
},
Run: func(args map[string]any) (any, error) {
name, err := text(args, "name")
if err != nil {
return nil, err
}
kind, _ := args["type"].(string)
timeout, err := timeoutOf(args)
if err != nil {
return nil, err
}
return CheckDNS(name, kind, timeout)
},
},
}
}
+5
View File
@@ -0,0 +1,5 @@
module netcheck
go 1.22
require git.novox.be/novox/mesh-sdk/go v0.1.6
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.6 h1:9qzdYONYbJdWcu6sxQcq9v1LI0JxcfkiKYkMUzJSkVQ=
git.novox.be/novox/mesh-sdk/go v0.1.6/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+84
View File
@@ -0,0 +1,84 @@
// netcheck's HTTP check — the module's own code, in TypeScript (novox/hq ADR 0039, ADR 0188). One
// request, HEAD or GET, never a body sent and never a body read: the status, how long it took and
// a few headers that say what answered. Redirects are reported, not followed, so a check reaches
// exactly the address it was given.
export const METHODS = ["HEAD", "GET"] as const;
export type Method = (typeof METHODS)[number];
/** The headers worth reporting: what answered and what it says it is, nothing it set for a client. */
export const REPORTED_HEADERS = [
"content-type", "content-length", "server", "location", "date",
"cache-control", "last-modified", "etag",
] as const;
export const DEFAULT_TIMEOUT_MS = 5000;
export const MOST_TIMEOUT_MS = 30000;
export interface HttpResult {
url: string;
method: Method;
status?: number;
statusText?: string;
elapsed_ms: number;
headers: Record<string, string>;
error?: string;
}
/** Only http and https are checked; anything else — file:, data:, ftp: — is refused by name. */
export function checkedUrl(raw: unknown): URL {
const text = typeof raw === "string" ? raw.trim() : "";
if (!text) throw new Error("url is required");
let url: URL;
try {
url = new URL(text);
} catch {
throw new Error(`${JSON.stringify(text)} is not a URL`);
}
if (url.protocol !== "http:" && url.protocol !== "https:") {
throw new Error(`netcheck_http checks http and https URLs only, not ${url.protocol}`);
}
return url;
}
export function checkedMethod(raw: unknown): Method {
const m = (typeof raw === "string" && raw.trim() ? raw.trim() : "HEAD").toUpperCase();
if (!(METHODS as readonly string[]).includes(m)) {
throw new Error(`method ${m} is not one netcheck_http uses (${METHODS.join(", ")}): a check never changes anything`);
}
return m as Method;
}
export function checkedTimeout(raw: unknown): number {
if (raw === undefined || raw === null || raw === "") return DEFAULT_TIMEOUT_MS;
const n = Number(raw);
if (!Number.isInteger(n) || n < 1) throw new Error(`timeout_ms must be a whole number of at least 1, not ${String(raw)}`);
return Math.min(n, MOST_TIMEOUT_MS);
}
/** Make one request and report how it went. A refused connection or a timeout is a result with its
* error; only a malformed question throws. */
export async function checkHttp(args: Readonly<Record<string, unknown>>, fetcher: typeof fetch = fetch): Promise<HttpResult> {
const url = checkedUrl(args.url);
const method = checkedMethod(args.method);
const timeout = checkedTimeout(args.timeout_ms);
const started = performance.now();
const out: HttpResult = { url: url.toString(), method, elapsed_ms: 0, headers: {} };
try {
const res = await fetcher(url, { method, redirect: "manual", signal: AbortSignal.timeout(timeout) });
out.elapsed_ms = Math.round(performance.now() - started);
out.status = res.status;
out.statusText = res.statusText;
for (const h of REPORTED_HEADERS) {
const v = res.headers.get(h);
if (v !== null) out.headers[h] = v;
}
// The body is not read: a check asks whether something answers, not what it says.
await res.body?.cancel().catch(() => {});
} catch (err) {
out.elapsed_ms = Math.round(performance.now() - started);
const e = err as Error & { cause?: { message?: string; code?: string } };
out.error = e.name === "TimeoutError" ? `no answer within ${timeout} ms` : (e.cause?.code ?? e.cause?.message ?? e.message);
}
return out;
}
+35
View File
@@ -0,0 +1,35 @@
{
"module": "netcheck",
"version": "1",
"tools": [
"netcheck_tcp",
"netcheck_dns",
"netcheck_http"
],
"build": {
"artifacts": [
{
"name": "tools-go",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/netcheck",
"binary": "netcheck",
"loads": [
"netcheck"
]
},
{
"name": "tools-typescript",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"tools/index.js"
],
"loads": [
"tools/index.js"
]
}
]
}
}
+17
View File
@@ -0,0 +1,17 @@
{
"name": "@novox/module-netcheck",
"version": "0.1.0",
"description": "netcheck — read-only network checks from a machine, as one module carrying a Go tools bundle (TCP, DNS) and a TypeScript one (HTTP) (novox/hq ADR 0188, ADR 0193).",
"type": "module",
"private": true,
"scripts": {
"test": "node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.6"
},
"devDependencies": {
"@types/node": "^22.0.0",
"typescript": "^5.6.0"
}
}
+52
View File
@@ -0,0 +1,52 @@
// The HTTP check refuses what is not http(s) and what would change something, and reports a status,
// a refusal and a timeout as results (novox/hq ADR 0188: a tools bundle is read-only and harmless).
import { test } from "node:test";
import assert from "node:assert/strict";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { checkHttp, checkedMethod, checkedUrl } from "../http.ts";
test("only http and https URLs are checked", () => {
for (const bad of ["file:///etc/passwd", "ftp://example.org/", "data:text/plain,hi", "javascript:1", "", "not a url"]) {
assert.throws(() => checkedUrl(bad), `${bad} was accepted`);
}
assert.equal(checkedUrl("https://example.org/x").protocol, "https:");
});
test("only HEAD and GET are used", () => {
assert.equal(checkedMethod(undefined), "HEAD");
assert.equal(checkedMethod("get"), "GET");
for (const bad of ["POST", "PUT", "DELETE", "PATCH"]) assert.throws(() => checkedMethod(bad));
});
test("a status, its headers and a redirect not followed", async () => {
const server = createServer((req, res) => {
if (req.url === "/moved") { res.writeHead(302, { location: "/elsewhere" }); res.end(); return; }
res.writeHead(200, { "content-type": "text/plain", "x-secret": "not reported" });
res.end(req.method === "GET" ? "body" : undefined);
});
await new Promise<void>((ok) => server.listen(0, "127.0.0.1", ok));
const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
try {
const head = await checkHttp({ url: base + "/" });
assert.equal(head.status, 200);
assert.equal(head.method, "HEAD");
assert.equal(head.headers["content-type"], "text/plain");
assert.equal(head.headers["x-secret"], undefined);
const moved = await checkHttp({ url: base + "/moved", method: "GET" });
assert.equal(moved.status, 302);
assert.equal(moved.headers.location, "/elsewhere");
} finally {
server.close();
}
});
test("a refused connection is a result with its error", async () => {
const server = createServer();
await new Promise<void>((ok) => server.listen(0, "127.0.0.1", ok));
const port = (server.address() as AddressInfo).port;
await new Promise<void>((ok) => server.close(() => ok()));
const got = await checkHttp({ url: `http://127.0.0.1:${port}/`, timeout_ms: 2000 });
assert.equal(got.status, undefined);
assert.ok(got.error, "no error reported");
});
+28
View File
@@ -0,0 +1,28 @@
// netcheck's TypeScript tools bundle (novox/hq ADR 0188, ADR 0193): what the builder's launcher
// imports and serves over MCP on stdio. Its Go bundle serves the TCP and DNS checks; this one the
// HTTP check — one module, two languages, one runtime that knows neither.
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { checkHttp, DEFAULT_TIMEOUT_MS, METHODS, MOST_TIMEOUT_MS } from "../http.js";
export function getNetcheckHttpTools(): ToolDefinition[] {
return [
{
name: "netcheck_http",
description:
"Check whether an http(s) URL answers from this machine: one HEAD or GET, no body sent or read, " +
"redirects reported and not followed. Answers status, elapsed_ms and a few headers.",
input: {
url: { type: "string", description: "an http:// or https:// URL" },
method: { type: "string", enum: [...METHODS], description: "HEAD (default) or GET" },
timeout_ms: {
type: "integer",
description: `give up after this long (default ${DEFAULT_TIMEOUT_MS}, at most ${MOST_TIMEOUT_MS})`,
},
},
run: async (args) => checkHttp(args),
},
];
}
registerModuleTools("netcheck", () => getNetcheckHttpTools());
+12
View File
@@ -0,0 +1,12 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"noEmit": true
},
"include": ["http.ts", "tools/index.ts"]
}
-40
View File
@@ -1,40 +0,0 @@
# nextcloud's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
ARG DOCKER_CLI
# Named so the final stage's COPY --from can reference a stage, not an ARG — the legacy builder
# this host still runs doesn't expand ARGs inside COPY --from, only inside FROM.
FROM ${DOCKER_CLI} AS dockercli
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/nextcloud
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/nextcloud/dist /app/modules/nextcloud/dist
# occ runs inside nextcloud's own container, reached over the mounted docker socket — which needs
# the docker CLI itself present here, not only the socket. Copied from Docker's own official client
# image rather than apt-installed, so this stays the one binary and nothing else (no daemon, no
# systemd unit, no package manager tree pulled in for it).
COPY --from=dockercli /usr/local/bin/docker /usr/local/bin/docker
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/nextcloud/dist/index.js,/app/modules/nextcloud/dist/tools/index.js
+19 -45
View File
@@ -30,8 +30,7 @@
"share.created" "share.created"
], ],
"own-secrets": { "own-secrets": {
"admin": "${dir:state}/admin.secret", "admin": "${dir:state}/admin.secret"
"broker": "${dir:mesh-state}/broker"
}, },
"capabilities": [ "capabilities": [
"container-runtime" "container-runtime"
@@ -63,7 +62,7 @@
"type": "file", "type": "file",
"path": "${dir:state}/server.env", "path": "${dir:state}/server.env",
"mode": "0600", "mode": "0600",
"content": "POSTGRES_HOST=${bound:postgres-database:at}:${bound:postgres-database:port}\nPOSTGRES_DB=${bound:postgres-database:as}\nPOSTGRES_USER=${bound:postgres-database:as}\nPOSTGRES_PASSWORD=${secret:postgres-database}\nNEXTCLOUD_ADMIN_USER=mesh-admin\nNEXTCLOUD_ADMIN_PASSWORD=${secret:admin}\nOBJECTSTORE_S3_HOST=${bound:s3-bucket:at}\nOBJECTSTORE_S3_PORT=${bound:s3-bucket:port}\nOBJECTSTORE_S3_BUCKET=mesh-novox-ncloud\nOBJECTSTORE_S3_KEY=${bound:s3-bucket:as}\nOBJECTSTORE_S3_SECRET=${secret:s3-bucket}\nOBJECTSTORE_S3_SSL=false\nOBJECTSTORE_S3_USEPATH_STYLE=true\nOBJECTSTORE_S3_REGION=${bound:s3-bucket:region}\n" "content": "POSTGRES_HOST=${bound:postgres-database:at}:${bound:postgres-database:port}\nPOSTGRES_DB=${bound:postgres-database:as}\nPOSTGRES_USER=${bound:postgres-database:as}\nPOSTGRES_PASSWORD=${secret:postgres-database}\nNEXTCLOUD_ADMIN_USER=mesh-admin\nNEXTCLOUD_ADMIN_PASSWORD=${secret:admin}\nOBJECTSTORE_S3_HOST=${bound:s3-bucket:at}\nOBJECTSTORE_S3_PORT=${bound:s3-bucket:port}\nOBJECTSTORE_S3_BUCKET=${bound:s3-bucket:bucket}\nOBJECTSTORE_S3_KEY=${bound:s3-bucket:as}\nOBJECTSTORE_S3_SECRET=${secret:s3-bucket}\nOBJECTSTORE_S3_SSL=false\nOBJECTSTORE_S3_USEPATH_STYLE=true\nOBJECTSTORE_S3_REGION=${bound:s3-bucket:region}\n"
}, },
{ {
"id": "html", "id": "html",
@@ -94,53 +93,28 @@
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n",
"merge": "json" "merge": "json"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-nextcloud",
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"/var/run/docker.sock:/var/run/docker.sock"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_NEXTCLOUD_URL": "http://127.0.0.1:${port:80}",
"MESH_NEXTCLOUD_CONFIG_FILE": "/run/config/config.json",
"MESH_NEXTCLOUD_ADMIN_USER": "mesh-admin",
"MESH_NEXTCLOUD_ADMIN_PASSWORD_FILE": "/run/secrets/admin"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
},
{
"arg": "DOCKER_CLI",
"image": "docker@sha256:018edbc908e08fcc9dbf029c812c34251e9b4719e6f71ca0e5eae2a987d014ca"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"MESH_NEXTCLOUD_URL": "http://127.0.0.1:${port:80}",
"MESH_NEXTCLOUD_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_NEXTCLOUD_ADMIN_USER": "mesh-admin",
"MESH_NEXTCLOUD_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret"
}
} }
] ]
} }
-27
View File
@@ -1,27 +0,0 @@
# nodered's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/nodered
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts tools/index.ts mqtt/probe.ts mqtt/connection.ts mqtt/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/nodered/dist /app/modules/nodered/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/nodered/dist/tools/index.js
# NOT dist/mqtt/index.js: that is a step the host runs to completion, named by the `mqtt`
# container's args as `mesh-tools run …` (novox/hq ADR 0052). Listed here it would run inside the
# serving sidecar too, and exit it.
+31 -55
View File
@@ -12,8 +12,7 @@
"api-token": { "api-token": {
"path": "${dir:mesh-state}/api-token", "path": "${dir:mesh-state}/api-token",
"taken": "at-start" "taken": "at-start"
}, }
"broker": "${dir:mesh-state}/broker"
}, },
"capabilities": [ "capabilities": [
"container-runtime" "container-runtime"
@@ -101,53 +100,31 @@
"content": "{\n \"token\": \"${secret:api-token}\"\n}\n" "content": "{\n \"token\": \"${secret:api-token}\"\n}\n"
}, },
{ {
"id": "runtime", "id": "mqtt-env",
"type": "container", "type": "file",
"name": "mesh-nodered", "path": "${dir:state}/mqtt.env",
"network": "host", "mode": "0600",
"volumes": [ "content": "MESH_NODERED_URL=http://127.0.0.1:${port:1880}\nMESH_NODERED_CONFIG_FILE=${dir:mesh-state}/config.json\nMESH_PROVISIONS_DIR=${dir:state}\nMESH_WRITTEN_DIR=${dir:written}\n"
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_NODERED_URL": "http://127.0.0.1:${port:1880}",
"MESH_NODERED_CONFIG_FILE": "/run/config/config.json"
},
"restart-on": [
"runtime-config"
],
"artifact": "runtime"
}, },
{ {
"id": "mqtt", "id": "mqtt",
"type": "container", "type": "process",
"name": "mesh-nodered-mqtt", "name": "nodered-mqtt",
"network": "host", "artifact": "code",
"run-once": true, "run": [
"volumes": [ "node",
"${dir:mesh-state}/config.json:/run/config/config.json:ro", "mqtt/index.js"
"${dir:written}:/var/lib/nodered-provisions",
"${dir:state}/mqtt-topic.json:/run/provisions/mqtt-topic.json:ro",
"${dir:state}/mqtt-topic.secret:/run/provisions/mqtt-topic.secret:ro",
"${dir:state}/settings.json:/run/provisions/settings.json:ro"
], ],
"env": { "run-once": true,
"MESH_NODERED_URL": "http://127.0.0.1:${port:1880}", "env-file": [
"MESH_NODERED_CONFIG_FILE": "/run/config/config.json", "${dir:state}/mqtt.env"
"MESH_PROVISIONS_DIR": "/run/provisions",
"MESH_WRITTEN_DIR": "/var/lib/nodered-provisions"
},
"args": [
"run",
"/app/modules/nodered/dist/mqtt/index.js"
], ],
"restart-on": [ "restart-on": [
"mqtt-env",
"bound-mqtt-topic", "bound-mqtt-topic",
"secret-mqtt-topic", "secret-mqtt-topic",
"settings" "settings"
], ]
"artifact": "runtime"
} }
], ],
"requires": [ "requires": [
@@ -173,23 +150,22 @@
"mqtt-topic": "${dir:state}/mqtt-topic.secret" "mqtt-topic": "${dir:state}/mqtt-topic.secret"
}, },
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js",
"mqtt/index.js"
],
"loads": [
"tools/index.js"
],
"env": {
"MESH_NODERED_URL": "http://127.0.0.1:${port:1880}",
"MESH_NODERED_CONFIG_FILE": "${dir:mesh-state}/config.json"
}
} }
] ]
} }
-22
View File
@@ -1,22 +0,0 @@
# openai-consumer's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/openai-consumer
COPY . .
RUN node /app/node_modules/typescript/bin/tsc apply/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/openai-consumer/dist /app/modules/openai-consumer/dist
# No serve-time entrypoints: every container of this module names its command (`run` on a
# schedule), so nothing here serves — deliberately no MESH_TOOL_MODULES.
+18 -31
View File
@@ -28,44 +28,31 @@
}, },
{ {
"id": "apply", "id": "apply",
"type": "container", "type": "process",
"name": "mesh-openai-consumer-apply", "name": "openai-consumer-apply",
"network": "host", "artifact": "code",
"run": [
"node",
"apply/index.js"
],
"schedule": "*/5 * * * *", "schedule": "*/5 * * * *",
"args": [
"run",
"/app/modules/openai-consumer/dist/apply/index.js"
],
"volumes": [
"${dir:state}:/run/state"
],
"env": { "env": {
"MESH_MODEL_ACCESS_SECRET_FILE": "/run/state/api-key", "MESH_MODEL_ACCESS_SECRET_FILE": "${dir:state}/api-key",
"MESH_MODEL_ACCESS_BIND_FILE": "/run/state/model.json", "MESH_MODEL_ACCESS_BIND_FILE": "${dir:state}/model.json",
"MESH_OPENAI_ENV_FILE": "/run/state/config/openai.env", "MESH_OPENAI_ENV_FILE": "${dir:state}/config/openai.env",
"MESH_OPENAI_CREDENTIALS_FILE": "/run/state/config/auth.json" "MESH_OPENAI_CREDENTIALS_FILE": "${dir:state}/config/auth.json"
}, }
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"apply/index.js"
]
} }
] ]
} }
+1 -1
View File
@@ -58,7 +58,7 @@
"type": "file", "type": "file",
"path": "${dir:state}/server.env", "path": "${dir:state}/server.env",
"mode": "0600", "mode": "0600",
"content": "NODE_ENV=production\nPORT=9000\nMONGO_URL=mongodb://${bound:mongodb-database:as}:${secret:mongodb-database}@${bound:mongodb-database:at}:${bound:mongodb-database:port}/${bound:mongodb-database:as}?authSource=admin\nMONGO_DB=${bound:mongodb-database:as}\nMINIO_ENDPOINT=${bound:s3-bucket:at}\nMINIO_PORT=${bound:s3-bucket:port}\nMINIO_BUCKET=mesh-novox-photos\nMINIO_ACCESSKEY=${bound:s3-bucket:as}\nMINIO_SECRET=${secret:s3-bucket}\nMINIO_USE_SSL=false\n" "content": "NODE_ENV=production\nPORT=9000\nMONGO_URL=mongodb://${bound:mongodb-database:as}:${secret:mongodb-database}@${bound:mongodb-database:at}:${bound:mongodb-database:port}/${bound:mongodb-database:as}?authSource=admin\nMONGO_DB=${bound:mongodb-database:as}\nMINIO_ENDPOINT=${bound:s3-bucket:at}\nMINIO_PORT=${bound:s3-bucket:port}\nMINIO_BUCKET=${bound:s3-bucket:bucket}\nMINIO_ACCESSKEY=${bound:s3-bucket:as}\nMINIO_SECRET=${secret:s3-bucket}\nMINIO_USE_SSL=false\n"
}, },
{ {
"id": "net", "id": "net",
-41
View File
@@ -1,41 +0,0 @@
# postgres's runtime: the tool runtime, carrying this module's compiled provisioner, tools and
# event consumer.
#
# **Built from this module's own directory and nothing else.** The sdk is in the base image, so
# nothing is copied out of a neighbouring checkout — which is what lets the mesh build this from a
# repository and a path (novox/hq ADR 0069) rather than only on a workstation that happens to have
# the siblings.
#
# Two bases, named rather than pinned: the image this is COMPILED in, and the image it RUNS in.
# They are different images on purpose — the first carries a compiler and the second must not, or
# every running container would carry one it never invokes. The mesh answers both with the copies it
# holds, because a fingerprint written here would name one particular copy and no other mesh has it
# (novox/hq issue 044). Declared in module.json's `build.on`; deliberately no defaults, so a build
# nobody told stops here and says which module to build first.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against.
WORKDIR /app/modules/postgres
COPY . .
# The compiler is invoked by its real path rather than through node_modules/.bin, whose entries are
# symlinks to a launcher that requires its library relatively — resolved away when the base image
# was assembled.
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
# **This module talks to its database through psql, so psql has to be here.** The client is how
# postgres's provisioner runs DDL — it does not carry a driver — and the runtime base holds only
# what every module needs.
RUN apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app/modules/postgres/dist /app/modules/postgres/dist
# What a tool host should load from this module: its event consumer and its tools, which are
# separate entrypoints because they are loaded by different things. The provisioner is the third,
# and is not listed here — the declaration names it in the container's `args`, because it is what
# this module's own container runs. One image, because they are one module and share a client.
ENV MESH_TOOL_MODULES=/app/modules/postgres/dist/index.js,/app/modules/postgres/dist/tools/index.js,/app/modules/postgres/dist/provisioner/index.js
+22 -41
View File
@@ -53,16 +53,9 @@
}, },
"own-secrets": { "own-secrets": {
"superuser": "${dir:state}/superuser.secret", "superuser": "${dir:state}/superuser.secret",
"broker": "${dir:mesh-state}/broker",
"reader": "${dir:state}/reader.secret" "reader": "${dir:state}/reader.secret"
}, },
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -99,45 +92,33 @@
] ]
}, },
{ {
"id": "runtime", "id": "client",
"type": "container", "type": "package",
"name": "mesh-postgres", "package": "postgresql-libs"
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}:ro",
"${dir:state}/superuser.secret:/run/secrets/superuser:ro",
"${dir:state}/reader.secret:/run/secrets/reader:ro"
],
"env": {
"MESH_PROVISION_POSTGRES": "postgres://postgres@127.0.0.1:${port:5432}/postgres?sslmode=disable",
"MESH_PROVISION_POSTGRES_PORT": "${seat:mesh-store:5432}",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/superuser",
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_POSTGRES_READER_PASSWORD_FILE": "/run/secrets/reader"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_PROVISION_POSTGRES": "postgres://postgres@127.0.0.1:${port:5432}/postgres?sslmode=disable",
"MESH_PROVISION_PASSWORD_FILE": "${dir:state}/superuser.secret",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_POSTGRES_READER_PASSWORD_FILE": "${dir:state}/reader.secret"
}
} }
] ]
} }
-26
View File
@@ -1,26 +0,0 @@
# records' runtime: the tool runtime, carrying this module's compiled reader and its tools.
#
# **Built from this module's own directory and nothing else.** The sdk is in the base image, so
# nothing is copied out of a neighbouring checkout (novox/hq ADR 0069).
#
# Two bases, named rather than pinned: the image this is COMPILED in, and the image it RUNS in
# (novox/hq issue 044). Declared in module.json's `build.on`; deliberately no defaults.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/records
COPY . .
RUN node /app/node_modules/typescript/bin/tsc records.ts index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
# **A module may need something the base image does not carry.** The reader keeps a checkout of the
# repository it reads (novox/hq ADR 0153) — a git working copy, kept current, not a derived copy — and
# the base image has no git. Certificates too, because the origin may be reached over TLS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app/modules/records/dist /app/modules/records/dist
# Both entrypoints, loaded in serve mode: the consumer that pulls on a merge, and the tools.
ENV MESH_TOOL_MODULES=/app/modules/records/dist/index.js,/app/modules/records/dist/tools/index.js
+19 -39
View File
@@ -11,9 +11,6 @@
"binds": { "binds": {
"git": "${dir:mesh-state}/git.json" "git": "${dir:mesh-state}/git.json"
}, },
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"consumes": [ "consumes": [
"gitea.pull.merged" "gitea.pull.merged"
], ],
@@ -53,47 +50,30 @@
"content": "${bound:git:scheme}://${bound:git:at}:${bound:git:port}\n" "content": "${bound:git:scheme}://${bound:git:at}:${bound:git:port}\n"
}, },
{ {
"id": "runtime", "id": "git",
"type": "container", "type": "package",
"name": "records", "package": "git"
"network": "host",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/config.json:/run/config/config.json:ro",
"${dir:mesh-state}/origin:/run/config/origin:ro",
"${dir:checkout}:${dir:checkout}"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECORDS_CONFIG_FILE": "/run/config/config.json",
"MESH_RECORDS_ORIGIN_FILE": "/run/config/origin",
"MESH_RECORDS_DIR": "${dir:checkout}"
},
"artifact": "runtime",
"restart-on": [
"config",
"origin"
]
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js"
],
"loads": [
"index.js",
"tools/index.js"
],
"env": {
"MESH_RECORDS_CONFIG_FILE": "${dir:mesh-state}/config.json",
"MESH_RECORDS_ORIGIN_FILE": "${dir:mesh-state}/origin",
"MESH_RECORDS_DIR": "${dir:checkout}"
}
} }
] ]
} }
-30
View File
@@ -1,30 +0,0 @@
# redis's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/redis
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/redis/dist /app/modules/redis/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/redis/dist/index.js,/app/modules/redis/dist/tools/index.js,/app/modules/redis/dist/provisioner/index.js
+18 -42
View File
@@ -35,9 +35,6 @@
"secrets": { "secrets": {
"secret": "${dir:state}/default.secret" "secret": "${dir:state}/default.secret"
}, },
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [ "listens": [
{ {
"name": "cache", "name": "cache",
@@ -48,12 +45,6 @@
} }
], ],
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -103,44 +94,29 @@
"restart-on": [ "restart-on": [
"server-conf" "server-conf"
] ]
},
{
"id": "runtime",
"type": "container",
"name": "mesh-redis",
"network": "redis",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:/var/lib/redis-module/grants:ro",
"${dir:state}/default.secret:/run/secrets/default:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "/var/lib/redis-module/grants/mesh.json",
"MESH_PROVISION_REDIS": "redis:6379",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/default"
},
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"index.js",
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_PROVISION_REDIS": "127.0.0.1:${port:6379}",
"MESH_PROVISION_PASSWORD_FILE": "${dir:state}/default.secret"
}
} }
] ]
} }
-24
View File
@@ -1,24 +0,0 @@
# route-adapter's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/route-adapter
COPY . .
RUN node /app/node_modules/typescript/bin/tsc adapter.ts index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/route-adapter/dist /app/modules/route-adapter/dist
# **No MESH_TOOL_MODULES, deliberately.** This module serves no tool and consumes no event: it is a
# step the host runs to completion, named by the container's `args` as `mesh-tools run …`. Setting a
# serve-time entrypoint here would give the image a second way to be started — one that connects to
# the broker and never exits.
+14 -28
View File
@@ -47,23 +47,18 @@
}, },
{ {
"id": "adapt", "id": "adapt",
"type": "container", "type": "process",
"name": "mesh-route-adapter", "name": "route-adapter",
"artifact": "runtime", "artifact": "code",
"run-once": true, "run": [
"volumes": [ "node",
"${dir:routes-dir}/mesh.json:${dir:routes-dir}/mesh.json:ro", "index.js"
"${dir:state}/config.json:/run/config/config.json:ro",
"${access:dynamic}:/services/traefik/dynamic"
], ],
"run-once": true,
"env": { "env": {
"MESH_RECEIVES": "${dir:routes-dir}/mesh.json", "MESH_RECEIVES": "${dir:routes-dir}/mesh.json",
"MESH_ROUTE_ADAPTER_CONFIG": "/run/config/config.json" "MESH_ROUTE_ADAPTER_CONFIG": "${dir:state}/config.json"
}, },
"args": [
"run",
"/app/modules/route-adapter/dist/index.js"
],
"restart-on": [ "restart-on": [
"received-route", "received-route",
"config" "config"
@@ -71,23 +66,14 @@
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"index.js"
]
} }
] ]
} }
-30
View File
@@ -1,30 +0,0 @@
# umami's runtime: the tool runtime, carrying this module's compiled code.
#
# **Built from this module's own directory and nothing else.** The sdk and the tool runtime are in
# the base images, published like any other artifact — which is what makes this buildable by the
# mesh from a repository and a path (novox/hq ADR 0069) rather than only on a workstation that
# happens to have the siblings.
#
# Two bases, named rather than pinned (novox/hq issue 044): the image this is COMPILED in and the
# image it RUNS in — the second must not carry a compiler. Declared in module.json's `build.on`.
ARG BUILD_BASE
ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
# Compiled under /app/modules so `@novox/mesh-sdk` resolves upward into the base's own
# node_modules — the module is compiled against exactly the sdk it will run against. The compiler
# is invoked by its real path: node_modules/.bin entries are launcher symlinks the base image
# resolved away.
WORKDIR /app/modules/umami
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts tools/index.ts provisioner/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
COPY --from=build /app/modules/umami/dist /app/modules/umami/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/umami/dist/tools/index.js,/app/modules/umami/dist/provisioner/index.js
+16 -51
View File
@@ -44,9 +44,6 @@
"grants": { "grants": {
"analytics": "${dir:grants}" "analytics": "${dir:grants}"
}, },
"own-secrets": {
"broker": "${dir:mesh-state}/broker"
},
"listens": [ "listens": [
{ {
"name": "web", "name": "web",
@@ -57,12 +54,6 @@
} }
], ],
"resources": [ "resources": [
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
},
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
@@ -81,13 +72,6 @@
"mode": "0600", "mode": "0600",
"content": "DATABASE_URL=postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}\nDATABASE_TYPE=postgresql\nAPP_SECRET=${secret:app-secret}\n" "content": "DATABASE_URL=postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}\nDATABASE_TYPE=postgresql\nAPP_SECRET=${secret:app-secret}\n"
}, },
{
"id": "provisioner-env",
"type": "file",
"path": "${dir:state}/provisioner.env",
"mode": "0600",
"content": "MESH_PROVISION_UMAMI_URL=http://umami:3000\nGRANTS=${dir:grants}\n"
},
{ {
"id": "net", "id": "net",
"type": "network", "type": "network",
@@ -106,46 +90,27 @@
"3000" "3000"
], ],
"secrets-in-environment": "a Next.js/Prisma application: DATABASE_URL and APP_SECRET are read from the environment only; not convertible" "secrets-in-environment": "a Next.js/Prisma application: DATABASE_URL and APP_SECRET are read from the environment only; not convertible"
},
{
"id": "runtime",
"type": "container",
"name": "mesh-umami",
"network": "umami",
"volumes": [
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:grants}:${dir:grants}",
"${dir:state}/admin.secret:/run/secrets/admin:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_UMAMI_ADMIN_PASSWORD_FILE": "/run/secrets/admin"
},
"env-file": [
"${dir:state}/provisioner.env"
],
"artifact": "runtime"
} }
], ],
"build": { "build": {
"on": [
{
"arg": "BUILD_BASE",
"module": "mesh-tools",
"artifact": "build"
},
{
"arg": "RUNTIME_BASE",
"module": "mesh-tools",
"artifact": "runtime"
}
],
"artifacts": [ "artifacts": [
{ {
"name": "runtime", "name": "code",
"kind": "image", "kind": "bundle",
"from": "Dockerfile" "language": "typescript",
"entrypoints": [
"tools/index.js",
"provisioner/index.js"
],
"loads": [
"tools/index.js",
"provisioner/index.js"
],
"env": {
"MESH_RECEIVES": "${dir:grants}/mesh.json",
"MESH_UMAMI_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_PROVISION_UMAMI_URL": "http://127.0.0.1:${port:3000}"
}
} }
] ]
} }