The breadth install (catalogue-broad) surfaced two latent resolution bugs that
single-module compile checks never caught (compile != resolve):
1. postgres/redis/mongodb/minio served no "port", yet seven consumers
(baserow, letta, invoicing, gitea, umami, keycloak, nextcloud) reference
${bound:<provision>:port}. A binding auto-carries at/from/as; the port is the
provider's half of the answer and must be declared in `serves` (manifest.go:
"Serves is what a consumer needs to know ... a port, a path, a realm"). Added
port to each: postgres 5432, redis 6379, mongodb 27017, minio 9000. Without it
no DB consumer could resolve, let alone deploy.
2. baserow declared an empty contribution `contributes: {"redis-cache": {}}`.
redis's serves spec for the cache is empty (a cache takes no per-consumer
payload), so a consumer only `requires` it; an empty contribution is refused.
Dropped it (requires/binds/secrets unchanged).
Found by mesh-lab catalogue-broad; the same fixes are mirrored in that bed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Port the HAL confluence and jira integrations to the tools-only,
outbound-only external-SaaS pattern proven by the merged gitlab module:
runtime-only containers (container-runtime capability), own-secret token +
broker, a settings-managed config.json for public config, and a
per-module runtime image.
Each module carries its own Atlassian API client and tools (ADR 0039),
translated from HAL's @hal/sdk zod-schema/MCP-content shape into
mesh-sdk's input/run-returns-data shape. Public config (ATLASSIAN_URL,
ATLASSIAN_EMAIL) lives in config.json; the API token is the one
own-secret. Clients are built lazily and never throw at registration, so
each runtime serves its full tool surface with no credentials (the
Servarr lesson) — confluence serves 3 tools, jira serves 8.
jira's periodic ticket-poller (update-tickets.service/.timer) is NOT
ported: the mesh has no scheduled-task primitive yet (a pending
decision). Only jira's tools are ported; a top-of-file note records the
deferral.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Port the HAL gitlab module (whose tools lived in @hal/sdk) into a
self-contained mesh-catalog module modelled on cloudflare-dns: the GitLab
API client and all its tools live in the module (ADR 0039), served through
mesh-sdk's registerModuleTools harness.
Tools-only, outbound-only external-SaaS shape: a runtime-only container on
network:host, no service, no listener, no provisioner. The token is an
own-secret; GITLAB_URL is a public setting in a merge:json config file.
The client is built lazily and never throws at registration, so the runtime
comes up and serves all 23 tools even with no valid token (the Servarr
lesson) — it only fails when a tool is actually invoked unconfigured.
Ported 23 tools: projects (list, get), merge requests (list, get, create,
approve, add note), pipelines (list, get, retry, cancel, list jobs, job log),
and project + group CI/CD variables (list, get, create, update, delete each).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The conversion left the image host as a placeholder (registry-api.example).
Point both containers at the actual registry the built app/api images live in
(novox/invoicing-{app,api}, tags 51-55 + latest present). Tracks :latest to
match the app's continuous-deploy model; pin by digest later if reproducibility
of a specific build is wanted.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Mirror the postgres-consumer shape (keycloak/nextcloud): requires the
backing service(s), binds/secrets for the delivered credential, and a
server container that reads it from an interpolated env file.
- baserow: consumes postgres-database + redis-cache; tooled (dormant
until an account is configured, like gitea's token).
- letta: consumes postgres-database; tooled, live via a mesh-minted
server-password injected into both server and runtime.
- invoicing: consumes mongodb-database + s3-bucket; plain two-container
service (app + api), no tools.
Digests pinned for baserow and letta; invoicing keeps private-registry
tags (DIGEST-UNRESOLVED). redis-cache and mongodb-database consumer
shapes are inferred (no prior consumer in the catalog).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
mosquitto_ctrl's dynsec subcommands exit 0 even when they fail — a
"Client not found", an "already exists", a "Connection error: Not
authorized", an "Unable to connect" all return status 0 and report the
failure only as a line of text (verified live against 2.0.11). ctl()
trusted the exit code, so clientExists()'s getClient probe never threw
and always returned true; createScopedClient therefore took the
setClientPassword branch, never ran createClient, and the consumer's
client never landed in the dynsec store — while the provisioner logged
it as provisioned. That is the assigned-catalogue-mqtt failure.
ctl() now scans the combined stdout/stderr for the tool's error markers
and raises a match as the failure it is. Surfacing those errors exposed
two calls that only "worked" by being swallowed: addRoleACL re-run
reports "already exists" (now ignored like createRole), and addClientRole
re-run reports a bare "Internal error" that cannot be told from a real
fault — so the binding is checked with clientHasRole and only added when
absent. Fresh provision, idempotent re-provision, password rotation, bad
admin auth and unreachable broker all verified against a live broker.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The Dynamic Security plugin refuses to start the broker unless
dynamic-security.json already holds an admin client, and no reconcile loop
seeds it (novox/hq ADR 0052). Add a run-once init container, declared before
the server container, that runs mosquitto's own bootstrap entrypoint in the
runtime image: it seeds the store offline via mosquitto_ctrl and exits, and
the host gates the broker on its completion.
The bootstrap hands the seeded file to the broker's user (uid 1883, chown +
0600): the broker must read the seed at startup AND persist to it as clients
come and go, but the init container runs as root and would otherwise leave a
file the broker can neither read nor rewrite. This is the ownership question
ADR 0052 left for the lab to settle. It seeds only when the file is absent, so
what the running plugin grows is never clobbered (issue 035).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The radarr/sonarr/lidarr/bookshelf events entrypoints built their client
with XClient.fromEnv() at import time, which throws when no API key/URL is
available yet — crash-looping the runtime container. The tools already
guard this; the events entrypoint did not.
Mirror the tools' guard: build the client in a try/catch, and only start
the poll loop when it succeeds. When it fails, log one line and stay idle
until a key is available. Behaviour when configured is unchanged.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
04-ISSUES/036: eight media modules each declared the shared library and
download directories under /services/media/* as their own `directory`
resources. Six of them owning one path is the collision the resolver
refuses — so the stack's only sensible assignment, all on one machine
sharing one filesystem, would be refused the first time two landed
together.
The media library is the operator's, owned by no module (novox/hq
ADR 0051). Move every /services/media/* path from an owned `directory`
resource to an `accesses` entry: the mesh mounts it and owns nothing —
does not create, chown, reconcile or remove it — and several modules may
access one path with no conflict. Each module's own config and mesh-state
directories stay owned resources.
Modes are least-privilege: plex reads the libraries it streams; the
managers and download clients get read-write on what they import and
write; bazarr writes subtitles into the libraries (read-write) and only
reads the download spool. The container volume mounts are unchanged.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
- bookshelf: Servarr v1 fork on the radarr template (4 tools).
- unifi: portainer-shaped tooled app (7 tools, 9 ports), settings-merged config.
- fail2ban: host-level security module mirroring firewall (service + restart-on,
no container); ban actions preserved as source ufw/iptables and FLAGGED to be
rewritten nftables-native before it actually bans.
- marrytts: manifest-only plain container (no tools), like resolv-conf.
All typecheck against the built @novox/mesh-sdk; service images digest-pinned.
Held from merge pending the hq initialization reconciliation.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Mirrors the proven catalog patterns field-for-field:
- lidarr -> the Servarr twin of radarr/sonarr (API v1, artist content); no
provisioner (it is a consumer app).
- mongodb -> postgres shape: mongodb-database provider, provisioner mints a
per-consumer db+user (ADR 0053), client shells to mongosh (no npm driver,
the psql convention).
- mssql -> postgres shape: mssql-database provider, sqlcmd client.
- mosquitto -> redis shape: mqtt-topic provider via the Dynamic Security
plugin, deliberately avoiding hal's password_file (that file is nox issue
011 exactly); provisioner mints a per-consumer MQTT client+role.
All four typecheck (strict, NodeNext) against the built @novox/mesh-sdk, and
their service images are digest-pinned to resolved registry digests. The
mesh-runtime-<mod> images keep the all-zeros placeholder the pipeline pins,
as postgres/redis do, and must bundle each module's CLI (mongosh/sqlcmd/
mosquitto_ctrl) as mesh-runtime-postgres bundles psql.
Not yet lab-verified: each module lists in-code what an integration test must
prove (auth model, provisioner reconcile, mosquitto dynsec bootstrap ordering).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Each provider's separate provisioner container becomes a runtime container that
serves the module's tools and runs its provisioner under the module's scoped broker
account: mesh-runtime-<module>, on the backend's own network (reaching the backend
by name and the broker by NAT), with MESH_BROKER_FILE + MESH_RECEIVES replacing
GRANTS. umami gains the broker own-secret it lacked. cloudflare-dns's adapter is
re-pointed at the ADR 0053 contract (a data provision, like umami — its record
return is the scoped-out concern).
Proven: provider-on-backend-network green — redis's runtime, on the private redis
network, binds the broker and provisions a consumer with the mesh's credential.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
redis, postgres and minio adapters drop generatePassword + the returned credential:
each creates the resource under the login the mesh derived (`as`) with the password
the mesh minted (`p.password`). minio's client gains a secret-key argument so it sets
the mesh's secret rather than generating one. umami (analytics) is re-pointed at the
new contract too; its siteId return is a data-provision concern ADR 0053 scopes out.
Proven: mesh-lab provider-uses-mesh-credential green — redis creates the consumer's
login with the mesh's password, the consumer authenticates (PONG), no seal key set.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Give each settings-config runtime a mergeable config file with its own id
(runtime-config) — the previous "config" collided with modules that already own a
config directory, so ten runtimes mounted a config file no resource declared. Point
each runtime's restart-on at it, so a settings change recreates the runtime and it
re-reads the new value (needs the mesh-host container restart-on fix on
issue/009-container-restart-on). Proven: runtime-restart-on-config e2e green.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Nineteen modules gain a broker-bound runtime container that serves the module's
tools under its own scoped account: bazarr, gitea, grafana, home-assistant,
icecast, influxdb, jackett, keycloak, mailu, nextcloud, nodered, nzbget, ombi,
photos, portainer, qbittorrent, searxng, tautulli, verdaccio.
Config is the assignment's, not the manifest's (ADR 0051): each client's fromEnv
overlays a settings-merged config file (MESH_<M>_CONFIG_FILE) over its env
fallbacks, so URL and credentials come from `settings set`, with the URL defaulting
to the server on the node. nextcloud and mailu also mount the docker socket for
their exec-based tools.
Proven in the mesh-lab: assigned-grafana green — settings deliver the URL and token,
the runtime reads the merged config and serves grafana's tools under the scoped
account, with nothing in the manifest. Two gaps this surfaced are filed as hq
issues 008 (a provider runtime's seal key) and 009 (a settings change does not
restart a container runtime).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Mirrors plex: a broker-bound runtime that serves the module's tools, discovering
the app's API key from its own config.xml under a read-only config-dir mount, URL
defaulting to the server on the node. Proven in the mesh-lab: assigned-sonarr green
(key detected, tools served under the scoped account, no live Sonarr needed).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
plex gains a broker-bound tools/events runtime container (mesh-runtime-plex)
alongside its server, and a never-throwing plex_reachable health probe. redis and
postgres subscribe to their own lifecycle events in index.ts but declared no
consumes — so the substrate never made the queue the runtime binds and it crashed
on start (404). Declare the consume, as ADR 0046 requires. Proven end-to-end in the
mesh-lab: assigned-plex and assigned-redis both green.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The review found broker own-secret paths drifting: mostly /var/lib/<module>/
broker, but grafana/redis/icecast/nextcloud carried a '-module' suffix to dodge
a collision with the service's own /var/lib/<name> data, and photos sat under
/etc. Normalized to one collision-free namespace a service never owns:
/var/lib/mesh/<module>/broker, with a mesh-state directory resource for the
parent, across 24 modules. audit-logger is grandfathered (lab-proven, referenced
by the assigned test, and it has no service to collide with). All 33 manifests
parse; no client hardcoded a path, so nothing in code moved.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The review found minio's provisioner emitting with a bare await, which throws
when no broker is bound (a provisioner is not yet a runtime — ADR 0052) and so
fails create/remove. Wrapped like the other providers: the event is logged and
dropped, the bucket still made. The real fix — the provisioner carrying a broker
credential — is ADR 0052.
Which zone, domain and ingress are a mesh's facts, not the module's — so they
are settings merged into a config file the mesh manages, read by fromEnv, rather
than the empty env placeholders I wrongly baked in. The token stays the one
own-secret. The module now describes a Cloudflare registrar; which zone is a
setting, so the same description serves every mesh. Typechecks; manifest parses.
The first registrar behind the neutral public-dns interface. Provider shape
like minio: provides public-dns, a provisioner that registers a consumer's
public name at Cloudflare pointing it at the mesh's ingress, and removes it on
withdrawal. The name is derived from the consumer identity under the mesh's
domain (so stateless teardown recomputes it); the returned {fqdn,target,ttl}
is public, the Cloudflare token the only secret and it never leaves. Emits
record.created/.removed (best-effort). A cloudflare_dns_records diagnostic tool.
Config (zone, domain, ingress) is left to settings, so it fails closed until a
mesh provides them. Typechecks; manifest parses.
The missing applier. mesh-control already derives a node's whole nftables rule
set from the union of its modules' listens and writes it to /etc/nftables.conf;
this module declares filtering:{into} to receive it and loads it — the nftables
service, reloaded on 'filtering' whenever the rules change. A firewall_rules
tool reads the live table so a declared scope can be checked against what is
really enforced. Closes the loop from listens.from to a packet actually dropped.
Manifest parses; tool typechecks.
The mesh's resolver is more than config after all. Tools: dnsmasq_names (what
this node answers, from the generated wildcard file) and dnsmasq_resolve (resolve
a name through this node's own resolver — the check that wildcard-resolution
actually answers). Event: it watches the wildcard file and emits
module.dnsmasq.name.added/.removed as machines' names become resolvable here —
DNS having propagated to this node, distinct from the mesh's node.* events.
Typechecks; manifest parses.
searxng: a search tool — tools-only, a stateless search has nothing to observe.
icecast: status tool, emits stream.started/.stopped by diffing live mountpoints.
photos: status/albums/recent tools, emits item.added (immich-shaped API, coded
defensively since the stub ships a placeholder image — flagged in the code).
Typecheck; manifests parse.
registry (docker v2): catalog/tags/delete tools, emits image.pushed (a build's
image is now pullable). verdaccio (npm): list/info tools, emits
package.published. portainer: endpoints/stacks/containers tools — tools-only,
since its only events are the underlying containers' lifecycle, which the host
owns. Typecheck; manifests parse.
home-assistant: states/call-service/config tools, emits state.changed bounded
to actuator/contact domains (not attribute ticks), overridable via a watch
allowlist. influxdb: health/buckets/flux-query tools — tools-only, since a
time-series DB has no lifecycle event to emit here. Typecheck; manifests parse.
nzbget (usenet, JSON-RPC) and qbittorrent (torrents, WebUI API with manual SID
session). Tools: status, queue/torrents, add, pause/resume, delete. Both poll
and emit module.<app>.download.added and module.<app>.download.completed — the
key plex consumes to rescan; completion is keyed off real success (nzbget
history SUCCESS, qbittorrent progress reaching 1), not mere queue disappearance,
so a failed or deleted item is not reported as done. Typecheck; manifests parse.
Ported their real HTTP APIs (hal carried no client for these). jackett: an
indexer proxy — tools only (list indexers, search), no events, no broker
account, because it answers queries and has no timeline to observe. bazarr:
subtitle tools + emits module.bazarr.subtitle.downloaded (poll history, diff).
ombi: request tools + emits request.created/.approved. All pure emitters
(their decisions originate here). Typecheck; manifests parse.
redis provides redis-cache: a real admin client speaking RESP over a raw socket
(node:net, no deps); provisioner makes a keyspace-scoped ACL user per grant.
postgres provides postgres-database: admin client executing through psql (the
consistent shell-out port, like minio's mc), full DDL for create/drop database+
role, a CSV row parser, read-only query tool. Both emit
module.<x>.<thing>.provisioned/.deprovisioned from the provisioner. Both
typecheck; manifests parse.
Object store, an s3-bucket provider. Client ported with no npm deps: S3 data
plane over fetch + SigV4 (node:crypto), scoped access keys via the mc CLI (the
admin API needs an Argon2 payload node built-ins can't make — the honest port
hal also used). Tools: list buckets/objects, bucket info, presigned url. The
provisioner makes a bucket + scoped key per grant and emits
module.minio.bucket.created/removed (the secret stays off the bus). Typechecks;
manifest parses.
(Trimmed the generated self-consuming ledger: a provider need not subscribe to
its own emits.)
The Servarr apps (TV, movies), each self-contained from hal's shared arr
client. Tools: status, library, search, queue, calendar. Events by polling the
download queue: a new item emits module.<app>.<thing>.grabbed, an item that
leaves as completed emits module.<app>.download.completed — the exact key plex
consumes to rescan. A grab that left as failed/warning is not reported as a
completion. Both typecheck; manifests parse.
Git hosting. 15 tools (repos, issues, PRs, labels, api passthrough) moved out
of the shared sdk. Emits repo.created (from a light poll, catching repos born
of git push or the web UI), issue.opened and pull.merged (from the tools at the
moment of the action) — the poll owns repo.created alone so it is never
announced twice. Typechecks; manifest parses.
Identity provider. 22 admin tools (realms, users, clients + secrets, groups,
roles) over the admin API, moved out of the shared sdk. Emits user created/
deleted, password reset, client/group/role created — from the write tools
themselves, since Keycloak's value is the changes it makes, not pollable
state. Consumes nothing: it is upstream of everything that authenticates
against it. Typechecks; manifest parses.
Email server. Users/aliases/domains over the admin REST API, mail read via
doveadm in the imap container; ten tools. Emits user/alias created/deleted by
polling and diffing the admin API, so a change in the web admin is announced
as readily as one via a tool. Consumes nothing — deliberately, since mutating
mail accounts off another module's event could silently lose mail. Typechecks
against the sdk; manifest parses.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Moves plex's API client and tools out of the shared hal sdk into the module,
so a Plex API change rebuilds only plex. Tools: status, search, sessions,
recently-added, refresh. And a real event design: it emits playback
started/stopped and item.added by watching the server, and consumes
module.*.download.completed to rescan so a downloader's fetch becomes a
visible item. Typechecks against the sdk; manifest parses with its emits/
consumes and broker own-secret.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A container resource takes a digest-pinned image, not a build artifact — the
host refuses 'artifact' on a container. Verified: the mesh assigns it and the
host runs it in the lab.
Now a real assigned module, not just a handler: consumes '#', declares its
broker own-secret, and runs the runtime image as a container that mounts the
sealed credential and its trail. own-secrets:{broker} is the file the mesh
seals it (module issue); the container reads MESH_BROKER_FILE from the mount
and takes its node/module identity from the credential. Parses against the
catalogue schema.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The trail's id is now the event's x-event-id — the handle a reader dedups
the at-least-once stream on — not the type@time placeholder the first cut
used. Carries causation/schema through when present.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
jschoubben