keycloak, minio and n8n are told their names from their route bindings; mailu takes its domain, site
name, website and proxy address as settings and its front's name from its route, and the provisioner
reads the domain from the merged config; builder and route-proxy package the controller from the git
seat; the applications built outside the mesh say so per container; matrix says which of the world's
servers it means; the site module is named website, and the why prose no longer names a name (novox/hq
ADR 0155, issues 122 and 134). module check passes over all 77.
The image's /healthcheck.js requires /data/settings.js, so with the mesh's
settings mounted at /config the container ran fine but reported unhealthy
forever. Mount the same file at /data/settings.js and point --settings there.
The record is prose wrapped at a hundred columns; matched line by line, the first live search for a
sentence of ADR 0025 found nothing. A line is matched together with the next, emphasis marks are
ignored, and a hit still names the line it starts on.
The manifest declared its secrets as secrets.secret.<name> and required a
"secret" provision, so the controller saw no own secrets and every accept
was refused (the influxdb defect of #180). The directories functions and
storage shared their ids with the containers of the same name, which the
host refuses as two resources with one identity. The containers are now
edge-functions and storage-api; the placed directories keep their ids and
paths.
A module keeping a checkout of a repository of decisions, designs and issues from the git seat,
current on every announced merge and on a timer, answering records_search / records_read /
records_list / records_status / records_sync at the commit it read (novox/hq ADR 0025, ADR 0153).
The repository is a setting; it names no mesh.
The received grants file names each pair secret by its HOST path. A sidecar
that mounts the placed grants directory at another path inside its
container sees mesh.json and not the secrets beside it — mosquitto's
provisioner reported ENOENT for a file that was there, and nodered's mqtt
step failed against a login that was never created. Mounted at its own
path now, as postgres does.
The tool runtime's own client, mesh serve, started by the mesh on the credential it sealed to the
machine (novox/hq ADR 0152, design 34): invokes every tool, listens from the machine only, holds no
state. Checked with module check before it was ever registered (hq issue 148).
They came from `requires: secret`, minted by the vault — right for a fresh
setup (the image's INIT_* variables read them once), wrong for an instance
that already exists: setup is skipped, the minted values match nothing,
and the provisioner holds a token the server never issued (issue 100).
InfluxDB will not take a chosen token value, so the operator token must be
accepted from the instance (`secret accept ace influxdb admin-token`); the
password can be either. As own secrets both are minted for a fresh install
exactly as before, and accepted where the data already knows them.
Found migrating ace's influxdb.
A program that checks its resolver validates — Mailu's admin does, at
start — could not use the mesh's resolver, and the one it ships instead
knows no mesh name (novox/hq issue 171). proxy-dnssec copies the AD bit
from 1.1.1.1 and 8.8.8.8, both of which validate.
admin is the one container here that reaches something by a mesh name:
the database, bound in database.env. A mesh name is answered by the
machine's resolver, which the runtime hands every container that does
not name its own (novox/hq ADR 0148); Mailu's unbound knows no mesh
name, and since the mesh stopped copying names into containers admin
could not find its database. The others keep unbound — rspamd needs a
validating resolver for DNSBL lookups and asks for no mesh name.
dnsmasq admits a query by the interface it arrives on when told interface=,
and by the address it is sent to when told listen-address=. A container's
query is sent to the machine's private address but arrives on docker0, so
interface=mesh0 dropped it silently on every machine (novox/hq issue 110).
Name the address, not the interface.
novox/hq 04-ISSUES/110. The module writes the runtime's `dns` key and
deliberately ordered no restart, because a restart stops every container.
It also ordered no reload, and the key holds only for containers created
after the runtime next starts. On two of four machines the runtime
predated the file — one since August — so every container there was
handed a public resolver and no mesh name resolved, while everything read
as fine. The third machine works only because its runtime happened to
restart later.
The file now also sets live-restore, which the runtime reads on a reload,
and the module declares the runtime reloaded when the file changes (ADR
0102: reload, don't restart). A reload still does not make `dns` take
effect; what it does is make the one restart that key needs keep every
container running. That restart stays the operator's, once per machine,
and is harmless from the second time on.
The mesh restarts nothing here. Undeclared, the runtime's unit goes back
to the state it was found in (ADR 0118).
MESH_PROVISION_POSTGRES named 127.0.0.1:5432 literally; the seat twin
(${seat:mesh-store:5432}) corrected it only on the machine holding the
mesh-store seat. On any other machine — ace, where the module provides
postgres-database without the seat — the twin is empty and the sidecar
would have dialled whatever else holds 5432 (HAL's postgres). ${port:5432}
is the machine port on every node; on novox the twin still says the same
6852, so nothing moves there.
sonarr, radarr, lidarr and bookshelf reached jackett as http://jackett:9117 (a HAL
container name) or https://indexers.zurag.be (its public route), typed into each app by
hand. The mesh has neither: an app now requires jackett-api and its downloads step writes
the bound address into the app.
Serves scheme, port and url-base; `at` and the machine port come from the binding. Mesh
scope, like sonarr-api: an indexer proxy shares no files with its consumers.
The pair credential is jackett's one API key. The mesh cannot mint it, so the operator
accepts it per consumer pair (ADR 0092), as #156 does for sonarr-api; a consumer's step
refuses a minted value and names the accept.
Node-RED's one broker node pointed at zurag.be:1884, where nothing listens. nodered now requires
mqtt-topic (asking for every topic: flows follow the devices' own) and a run-once `mqtt` step —
declared last, restarted when the binding, credential or settings change — points the mesh's broker
nodes at the bound broker through Node-RED's admin API with the module's api-token: the node the
step makes itself when none is named, or the ones an assignment names in `mqtt.brokers`. Only host,
port, TLS and the login change; the broker is asked first whether it takes the login; the deploy is
against the revision read ("nodes", so only that node restarts) and a digest makes a rerun a no-op.
A broker node nobody named is never touched. settings.js keeps `mqtt` and `topics` out of Node-RED.
mqtt-topic served nothing: with two listens the mesh could not say which port a consumer dials, so a
consumer had to type 1883 into its config. It now serves the MQTT listener's port (the machine's,
once assigned) and the scheme, so `${bound:mqtt-topic:port}` fills.
The provisioner confined every consumer to `<as>/#`, which leaves nothing for the consumers the
broker exists for: Home Assistant discovers under homeassistant/# and tasmota/discovery/#, and
Node-RED's flows follow the devices' own topics. A consumer now contributes `topics` (MQTT topic
filters) to its mqtt-topic requirement and is granted exactly those; with none, its own subtree as
before. Settings merge into contributions, so an operator narrows a grant per assignment. The role
is brought to exactly the wanted ACLs (stale ones removed), `holds` checks the ACLs too, and an
invalid list is refused, never quietly narrowed. Only the role named for the consumer is touched:
a client carried from the predecessor's password file keeps its own.
grafana's data source and Node-RED's influxdb nodes reached ace's
InfluxDB by a LAN IP or a public name nobody routes, with a credential
somebody made by hand. Now a consumer requires influxdb-api and is told
where it is, which org and default bucket it serves, and signs in with
the password the mesh minted for the pair.
The credential is a v1-compatibility authorization, made per grant by
the new provisioner: InfluxDB 2.x generates API tokens itself and
ignores one the caller sends, so a v2 token could only be accepted by
hand per pair; a v1 authorization takes a caller-chosen password (8-72
characters, the mesh mints 40) and reads/writes every bucket as a
database of its name over InfluxQL and line protocol. A consumer
contributes `access` (read, write, read-write) and, for writing, the
buckets; a missing bucket is made and never deleted. Only
authorizations named mesh_* and marked [mesh] are ever changed or
removed; anything else of that name is refused and left alone.
The org and default bucket are served facts the assignment's settings
set, reaching both the consumers and the provisioner's config.json.
The module named /var/lib/letta, a layout no definition may carry (ADR
0112); state is now a placed directory holding the bindings and secrets.
letta had no route, while the server it replaces is reached by its public
name (a workflow calls it there). It now contributes one for its `web`
endpoint; reach is the assignment's.
The server needs an OpenAI key for agents on OpenAI models, and nothing
gave it one: `openai-api-key` is an own-secret, accepted from the
operator (a key someone chose, not one the mesh can mint). The
server-password is accepted the same way where a server already has
clients. Both, and the database password inside LETTA_PG_URI, stay in the
server's environment: letta 0.6.x reads settings from the environment
only, and its startup.sh starts an embedded PostgreSQL unless
LETTA_PG_URI is set - the declared reason now says so.
The runtime's tools never authenticated: the client sent only a Bearer
token, and 0.6.x's --secure mode checks X-BARE-PASSWORD ("password <it>")
and answers 401 otherwise. The client now sends both. Its password comes
from the runtime config file (the key client.ts reads first) instead of an
env-file, so the runtime container no longer carries a secret in its
environment.
Image: the same 0.6.8 image, now pinned by the index digest ace runs
rather than its amd64 manifest.
Verified: catalogue tests with MESH_CATALOGUE set; tsc -p tsconfig.json in
the mesh-tools build image. Throwaway containers: a fresh 0.6.8 with its
embedded PG and two blocks made through the API; stopped, copied, dumped
from the copy; restored (schema letta + vector pre-made by the superuser,
--no-owner --role, search_path set on the database as the original had
it) into a grant-shaped database on the postgres module's pgvector image
(PG17); started in this shape: alembic finds nothing to do, both blocks
are there, a wrong password is refused with 401, and the patched client
lists agents. Test containers and data removed.
The module named /var/lib/baserow and /services/baserow/data, a layout no
definition may carry (ADR 0112). State and data are now placed directories;
bindings and the grant's secret live in the placed state.
The all-in-one image's entrypoint honours DATABASE_PASSWORD_FILE (file_env
in /baserow.sh), so the grant's password is mounted rather than put in an
env-file, and "secrets-in-environment" is gone (ADR 0086). SECRET_KEY is no
longer minted: the image keeps it, and its JWT signing key, in the data
directory (.secret, .jwt_signing_key) and imports them on start, so a moved
data directory carries the keys its sessions and tokens were made with.
DISABLE_EMBEDDED_PSQL makes a missing grant fail loudly instead of starting
an empty embedded database.
BASEROW_PUBLIC_URL was http://localhost. Baserow answers only the host of
that URL - any other Host is looked up as a published builder site and gets
404, /api/_health/ included - so it is now https://${bound:route:name}
(depends on mesh-controller #149).
The runtime's tools could never have worked: its config was "{}", and the
client's Host override was silently dropped by Node's fetch, so calls by
container name would 404 even with credentials. The client now uses
node:http (which sends the Host it is given, with a Content-Length -
Baserow reads a chunked body as empty) and re-authenticates once when a
cached JWT is refused (access tokens last minutes, the runtime weeks). The
password is the accepted `admin` secret; the email is an assignment
setting merged into the same file, the host is the route's name.
Image pinned to the develop-latest build ace runs today (Baserow 2.3.4,
built 2026-09-18). The old pin (built 2026-09-04) is older than ace's data.
Verified: catalogue tests with MESH_CATALOGUE set; tsc -p tsconfig.json in
the mesh-tools build image. In throwaway containers of the pinned image: a
fresh embedded-PG instance with a user, workspace and 5-row table; stopped,
copied, dumped from the copy (start-only-db); restored with --no-owner
--role into a grant-shaped database on the pgvector image the postgres
module pins (PG17); started with this shape (root 0600 password file,
embedded PSQL disabled, copied data dir without postgres/): health 200,
the user logs in, the 5 rows are there, SECRET_KEY and the JWT key are
imported from the data dir. The patched client lists applications and rows
through the container name with the public Host, and recovers from a
refused token. Test containers and data removed.
ace runs Supabase under HAL as upstream's 13-container compose: a 2.2 GB
database (1.8 GB of it the dormant `novox` schema, 5.8 M rows in its largest
table), Kong at supabase.zurag.be, the pooler on 5433/6543. This is that stack
as a catalogue module, same images (the digests ace runs), same container
names so an assignment holds the running ones and a take replaces them.
The database stays inside the module. Supabase is a Postgres distribution: its
own image with pgsodium, pg_graphql, pg_net, vault and timescale preloaded, a
superuser (supabase_admin), a dozen reserved roles and a second database
(_supabase). A postgres-database grant - one database, one unprivileged role -
cannot hold it, so ace's data directory moves as a copy, not a dump/restore.
What HAL did by shell and environment the mesh now renders as files:
- kong.yml carries the anon/service keys and the dashboard login (owned by
kong's uid 100), instead of an entrypoint that eval'd the environment;
- GoTrue reads a dotenv file (auth -c), PostgREST a config file, Vector its yml
with the Logflare key in it, the database POSTGRES_PASSWORD_FILE and a
jwt.sql rendered with the secret (owned by postgres, uid 105). None of these
five containers has a secret in its environment.
- realtime, storage, meta, functions, analytics, studio and supavisor read
their credentials from the environment only; each declares
secrets-in-environment with the reason (ADR 0086).
- Upstreams are container names (supabase-db, supabase-kong, ...) instead of
compose service names, which the mesh does not have.
- SITE_URL / API_EXTERNAL_URL / SUPABASE_PUBLIC_URL are
https://${bound:route:name} (mesh-controller #149); on ace HAL rendered them
as "https://supabase." - broken today.
- Vector reads the docker socket, as upstream does, but now includes only this
module's containers instead of every container's logs on the machine.
Three things compose did that a declaration cannot, done as steps: a run-once
seed copies the image's /etc/postgresql-custom into the placed config
directory with cp -n (a named volume did that implicitly; never overwrites the
pgsodium root key), and two run-once gates wait for the database and for
Logflare, which compose expressed as depends_on: service_healthy.
The pooler bootstrap (pooler.exs) takes the tenant id and pool sizes from
settings.json, the module's one merge:json file, so ace keeps its tenant
"zurag"; and it repoints an existing tenant whose database host is not
supabase-db - HAL created ace's with host "db", which no longer resolves.
Secrets (vault, requires "secret"): postgres, jwt, anon-key,
service-role-key, dashboard-user, dashboard, logflare, pooler-vault,
key-base-a + key-base-b (concatenated: Phoenix wants 64+ bytes, a minted
secret is 40), openai. Three cannot be minted on any machine: anon-key and
service-role-key are JWTs signed with jwt, and pooler-vault must be exactly
32 bytes (AES-256-GCM, found in the bed). They are accepted. On ace every
secret the data already knows is accepted (all but key-base-a/b).
Not carried: Kong's 8443 and Logflare's 4000 on all interfaces (nothing
outside the module uses them); realtime's DB_ENC_KEY stays upstream's constant
(realtime deletes and re-seeds that tenant from its environment every start,
and the key must be exactly 16 bytes).
Verified: catalogue tests with MESH_CATALOGUE pointed here on mesh-controller
main and #149 (on main the render is refused for "name", never written
empty). The #149 resolution with stub providers, turned into a throwaway
stack of all 13 pinned digests with dummy secrets and the rendered files at
their owners and modes: the database initialised through the rendered scripts
(jwt setting applied, _analytics/_supavisor created, roles' password from
POSTGRES_PASSWORD_FILE); through Kong: REST 200 with the anon key and 401
without, auth health and settings 200, storage buckets 200, GraphQL 200, pg-meta
200, an edge function 200, Studio 401 without and 200 with the dashboard login,
realtime tenant health 200; the pooler in session and transaction mode as
postgres.zurag; a tenant set to host "db" was repointed to supabase-db by the
bootstrap and connections worked.
ace runs a Conduit homeserver (matrix.zurag.be, 5.4 GB of RocksDB, federating)
and Element Web under HAL, configured by environment with the domain
templated in, and Element's config.json carrying matrix.zurag.be literally.
A homeserver's server_name is its permanent identity - every user id, room id
and signature in the database carries it - and it is the name the module is
served under. So it comes from ${bound:route:name-homeserver} (mesh-controller
#149), rendered into a conduit.toml the container reads through CONDUIT_CONFIG,
and into Element's config.json (base_url, default_server_name, the room
directory). Without #149 the render is refused ("name-homeserver"), never
written empty. Two routes, one per endpoint: homeserver (label matrix, 6167)
and element (label element, 80).
Federation needs no 8448: Conduit answers /.well-known/matrix/server with
<name>:443, so peers federate through the route. HAL published 8448 on all
interfaces, but the router never forwarded it; checked from outside, the
well-known, federation version and client versions all answer on 443.
Registration defaults to off. HAL ran with CONDUIT_ALLOW_REGISTRATION=true,
which on ace means anyone on the internet can create an account with the
dummy flow (seen: /register offers m.login.dummy) - on 0.10.13, whose
successor 0.10.14 fixes an account-takeover by any local user. Existing
accounts are unaffected; the operator decides whether to reopen it.
Element's config.json is the one merge:json file (it tolerates `endpoints`),
so a machine can add keys. HAL's map_style_url is not carried: it embedded
a map-tile API key, which belongs in an assignment if wanted.
Images are the digests ace runs (Conduit 0.10.13, Element 1.12.28).
Verified: catalogue tests with MESH_CATALOGUE pointed here on mesh-controller
main and #149; a resolution on #149 renders both names (matrix.zurag.be,
element.zurag.be) into both files and both contributions; throwaway
containers of both pinned digests with the rendered files (root 0644, :ro):
client versions 200, well-known says matrix.zurag.be:443, register refused
M_FORBIDDEN, Element 200 serving the rendered config.json.
The manifest stated /services/redis/data and /var/lib/redis-module - novox's
old layout, paths no definition may carry (ADR 0112). State is now the
assignment's root, grants and data are placed, and the config file, the
secret file, receives and grants all name them as ${dir:...}. Paths inside
the sidecar are its own view and are unchanged.
The data directory and config are owned 999:1000: the image's redis user is
uid 999 in gid 1000 (checked in both builds), which is who owns ace's data
today; 999:999 named a group the image does not use.
Image pinned to the 7.4.11-alpine build ace runs (2026-09-17); the old pin was
the same version, built in August. Older-than-running is never the pin.
Nothing is assigned it anywhere today, so no machine changes.
Verified: catalogue tests pass with MESH_CATALOGUE on this tree; the
declaration composes for ace with every path under /var/lib/redis. The pinned
image ran as a throwaway with a 0600 999:1000 config and a 0700 data dir:
unauthenticated PING is refused (NOAUTH), authenticated SET/GET works,
appendonly is on, the server runs as redis.
The manifest stated /var/lib/mssql, its grants and its SA file by path, and
declared it listens on 4848 - the port one machine's predecessor published,
which is an assignment's fact (ADR 0112, 0138). ace is moving its own
SQL Server (80 GB of work databases) onto the mesh, so the module has to be
the same on every machine.
- state is the assignment's root (place "."), grants is placed, and every
reference (sa.env, the env-file, the SA and grants mounts, receives,
grants, own-secrets) names them as ${dir:...}.
- the database endpoint listens on 1433, the port SQL Server uses; a machine
that must keep an older number pins it in its assignment.
- the image pin is unchanged: it is the digest ace runs today (CU27,
16.0.4295), the same as novox.
novox is untouched: rendered with novox's own setting ({"ports":{"1433":4848}})
through the controller's Declaration and Rules, every resource - paths,
container names, volumes, env-file, the 4848:1433 mapping, owners, modes - is
byte-identical to what main renders; the only difference is the firewall
rule's comment text (still port 4848, from the mesh).
Verified: catalogue tests pass with MESH_CATALOGUE on this tree. The pinned
image ran as a throwaway on a 0700 10001:0 data dir with a root-owned 0600
env-file (dummy SA), answered sqlcmd as sa; a scratch database stopped,
copied with cp -a, checksummed and started on the copy kept its rows and
CHECKSUM_AGG.
The sidecar runs on the host network and dialled 127.0.0.1:1880, the
software's port; the mesh publishes nodered on a machine port it assigns,
so the tools reached whatever else holds 1880, or nothing (hq 088).
jschoubben
mesh-admin