fail2ban expands <HOST> to a named group, so two in one pattern is a duplicate group name and
the daemon refuses to start at all -- every jail on the machine, not just this one. Two patterns,
one <HOST> each: the certificate refused for an unserved name, and the request refused for one.
Caught live on the control node (hq ADR 0179).
The pattern ended at the line's end, which only the certificate refusal does; a request for
an unserved name carries trailing text and never matched. Caught against the live lines
before the jail counted anything (hq ADR 0179).
The module gains a runtime carrying only the fail2ban client with the daemon's socket shared in,
serving status/banned/ban/unban and its own fail2ban_settings. It declares jailing, so the
controller's composition lands in jail.d/mesh.conf and filter.d; mailu, route-proxy and gitea log to
the journal and declare a jail reading it by container name. The base is strict: three in a day for
a day, twice banned in two weeks for four; the mesh's range stays never banned.
Twenty-eight modules' data directories are placed: the root as place ".", a sub-directory named by
its id, and every host-side reference — binds, secrets, own secrets, grants, receives, file paths,
mounts, env-files — as ${dir:<id>}. Resolved on the default root every path is the one the manifest
named before, which the controller's TestPlacedDirectoriesKeepTheirPaths proves over both checkouts;
so no data moves and no machine sees a change. Five directories whose id is not their last segment
keep their path as a placement (novox/hq issue 119, ADR 0112, design 27).
keycloak, minio and n8n are told their names from their route bindings; mailu takes its domain, site
name, website and proxy address as settings and its front's name from its route, and the provisioner
reads the domain from the merged config; builder and route-proxy package the controller from the git
seat; the applications built outside the mesh say so per container; matrix says which of the world's
servers it means; the site module is named website, and the why prose no longer names a name (novox/hq
ADR 0155, issues 122 and 134). module check passes over all 77.
75 endpoints across 50 modules, named from what each one is for rather than by a
rule: mail's seven protocol ports are smtp, imaps, submission and the rest; unifi's
nine are inform, stun, discovery, the two portal ports and syslog; minio's two are
s3 and console; the resolver's two are dns-udp and dns-tcp.
And 35 route contributions name the endpoint they serve instead of repeating its
port. A route and a listen both carried a port and nothing said they were the same
thing; now one of them does. gitea's path-level deny rule names neither, because it
is a rule about a name rather than an endpoint.
novox/hq ADR 0138. The words shipped a release ahead in mesh-controller #138 and
#139, and the control plane running today is built from that merge — checked before
this was written, because an unknown manifest key is refused and a catalogue using
one against an older control plane would stop resolving.
builder and route-proxy both build from the mesh-controller repository's context,
so its go.mod is theirs, and `nats.go v1.54.0` puts that at `go >= 1.26`. Pinned at
1.25.14 they cannot compile it: the build machine's own build failed with "go.mod
requires go >= 1.26.0 (running go 1.25.14)".
Each moves to the 1.26.8 digest of the flavour it already used — alpine for
builder, debian for route-proxy — so nothing changes but the compiler version.
Two name spaces, two authorities (08-connectivity §2): a public name is
certified by a public CA, an internal one by the mesh's own. step-ca now
offers that second seat as internal-acme-ca beside its existing acme-ca,
and route-proxy requires both — the server dispatches by which authority
may certify the name at all, so an .internal alias stops being plain-HTTP
only without ever asking a public CA for a name it cannot validate.
Composed ACME_ROOTS unconditionally from ${bound:acme-ca:roots} even when
that field is empty — public-acme's own case, where an empty roots means
'the system trust store', not 'fetch from the bare authority host'. The
run-once step wget'd https://acme-v02.api.letsencrypt.org:443 (host, no
path) for two minutes every apply and failed, blocking every resource
after it — found live tonight, assigning route-proxy for the first time.
Carries the raw, uncomposed roots value alongside the composed URL
(ACME_ROOTS_PATH) so the step can tell 'nothing to fetch' apart from 'the
authority didn't answer' — a distinction the composed URL alone cannot
make. Empty copies the image's own system CA bundle to /ca/root.crt
instead of fetching one, so ACME_CA_BUNDLE stays the one path it has
always been rather than needing to become conditional itself.
Was still pinned to the scaffold's placeholder digest (mesh-route-
proxy@sha256:0000...0000) even after the build+context work landed —
never caught because nothing had assigned route-proxy before tonight.
artifact: server, matching trust's own reference a few lines up and
every other built module in the catalogue.
The Dockerfile's own comment already said it — 'the build context is the
mesh-controller repository root' — but nothing in the manifest actually
said so to the mesh, so every build attempt used mesh-catalog's own
directory instead and failed with 'stat go.mod: file does not exist'.
Never caught before because route-proxy has never been assigned anywhere.
Uses the context mechanism just added (mesh-controller#62), proven
working tonight on builder's own self-build.
step-ca's root certificate, its key and that key's password were own secrets — random
bytes the mesh minted, which no certificate is (novox/hq 04-ISSUES/076). The mesh
mints only the CA password now; step-ca makes its root at first start and serves it
at /roots.pem, which the manifest now names beside the ACME directory. The route
proxy fetches that root over the mesh network in a run-once step before it starts,
instead of being handed a served fact that could only be written before anything ran
(ADR 0098).
gitea, umami, influxdb, icecast and mailu require a secret and keep each of theirs
under a local name (novox/hq ADR 0094); the broker account stays their own. The
route proxy's recipe starts FROM the bases its manifest declares (ADR 0097).
An identity is `mesh_<node>_<slug-or-name>` and a backend keeps 20 characters
(an S3 access key). Overflow makes a module unresolvable, and this catalogue
was finding it one module at a time, on a raise: route-proxy on novox is 22,
home-assistant on ace is 23. Two found by hand where a sweep would have found
eighteen.
So the whole catalogue was swept instead, against the longest node name the
mesh actually has (`shanks`, six characters) rather than against the node each
module happens to sit on today — a module is assigned somewhere, and where is
not a property of the manifest. That leaves eight characters for the identity
source, and eighteen modules were over it.
Slugs added, chosen to stay greppable in a provider's user list:
anthropic-consumer claude openai-consumer openai
anthropic-manager anthmgr portainer portain
audit-logger audit public-acme pubacme
bookshelf books qbittorrent qbt
cloudflare-dns cfdns resolv-conf resolv
confluence confl resolved-split-dns splitdns
home-assistant hass route-proxy rproxy
invoicing invoice verdaccio verdacc
mosquitto mosq
nextcloud ncloud
A slug changes the login the mesh mints, so a module already provisioned under
its full name is re-minted under the slug and its old login withdrawn — which
is the provisioner's ordinary business, but it is a change, not a no-op.
Checked with the real parser: every one of the 66 manifests through
`catalogue.ParseManifest`, and every module's `CheckIdentity` against all four
node names. 0 problems, where the same check over the parent commit reports 44.
Piece A + B of ADR 0056, completing the internal-CA work in ff01ada.
Selectable issuer. acme-ca is now a role two providers can satisfy: step-ca
(internal CA) or the new public-acme (a fact-only module, no container/listen)
that serves Let's Encrypt production. A mesh assigns one or the other to satisfy
route-proxy's `requires: acme-ca`.
One directory shape for both. A provider serves the ACME directory's parts the
way the mesh already models any reachable service -- an address (`at`), a `port`
and a `path` -- and route-proxy composes `https://<at>:<port><path>`. step-ca
lets the mesh fill `at` (its node) and `port` (its single listen) and serves only
`path`; public-acme, not being a mesh service, serves all three (overriding `at`
with the public host). Same composition either way.
Empty root means the system trust store. Both providers serve `root`: step-ca
the operator root PEM (settled per mesh), public-acme an empty string. route-proxy
writes it to the CA bundle file unconditionally; the binary now reads an empty
bundle as "the root is already trusted by the OS" and falls back to system roots
(examples/route-proxy/main.go, committed on the mesh-control ADR-0056 branch).
step-ca inits from the operator's root. The operator's root cert, root key and
root-key password are mounted at the smallstep entrypoint's default init paths
(/run/secrets/root_ca.crt, root_ca_key, root_ca_key_password) with the matching
DOCKER_STEPCA_INIT_*_FILE vars, so `step ca init` adopts the operator's root
instead of self-generating one -- the CA that signs is the CA route-proxy trusts.
Route names are labels, not FQDNs. Every routed module now contributes a `label`
(the leftmost subdomain) instead of a full public hostname; the node's public
domain composes the name. Apex (novox.be) is left as a full name -- composeName
has no empty-label/apex convention yet (mesh-control follow-up).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Piece C of ADR 0056: an internal authority certifies routed names by the same
path a public one would, with the proxy pointed at whichever issuer the mesh
names and trusting that issuer's root.
step-ca (new): provides acme-ca (mesh scope), listens 9000 from mesh, persists
its CA home under uid 1000. The root CA cert reaches consumers as a served
value settled from a per-mesh operator setting (no baked root); the init
password and root key are sealed secrets, not literals. No route-name -> IP
hosts map (that is Piece B). Upstream smallstep/step-ca pinned by Docker Hub
digest.
route-proxy: requires + binds acme-ca. ACME_DIRECTORY is composed on the
consumer side from ${bound:acme-ca:at}:${bound:acme-ca:port}, and ACME_CA_BUNDLE
is a file whose content is ${bound:acme-ca:root} -- both from the binding,
replacing the hardcoded directory and the baked root PEM.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
route-proxy is the shipping form of the reference reverse proxy (novox/hq
ADR 0007, 08-connectivity section 3): it provides route, is given every
consumer as the file at receives.route, and forwards by the Host header. It
ships the Go proxy from mesh-control/examples/route-proxy via a multi-stage
Dockerfile; no broker, own-secret or provisioner, since it only reads the file
the mesh writes.
ACME_DIRECTORY defaults to Let's Encrypt staging and is overridable per node to
production, so there is no hardcoded production default -- resolving novox/hq
04-ISSUES/004. hello-web is a minimal consumer that requires route and
contributes name+port, to exercise the grant.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
jschoubben