Compare commits

...
Author SHA1 Message Date
jschoubben d2f03736fa grafana: its InfluxDB data source comes from the influxdb-api provision
ace's grafana reads InfluxDB through a data source somebody typed into
its database: a LAN address, a database InfluxDB 2 does not have, and a
password for a v1 user of an earlier instance. Nothing in the mesh knew
it existed, so migrating influxdb could only break it further.

grafana now requires influxdb-api, contributes read access, and the mesh
renders a provisioning file grafana reads at start: the address, port,
org's default bucket (as the InfluxQL database) and its own login from
the binding, the password by $__file from the pair credential the mesh
delivers, 0400 for grafana's uid 472. It is a data source of its own
name and uid, read-only in the UI and not the default, so the data
source a person made is never overwritten; a changed binding or a
rotated password restarts grafana, which re-reads the file.

Includes #152 (merged into this branch): influxdb provides influxdb-api.
2026-09-30 13:02:11 +02:00
jschoubben c5e273e232 Merge feat/influxdb-for-ace (#152) into feat/oidc-client-provision
grafana's data source requires influxdb-api, which influxdb provides only
on #152's branch; merged so this branch's catalogue has the provider of
everything grafana requires. #152 should merge first.
2026-09-30 12:55:43 +02:00
jschoubben 323ef9ec7e influxdb: provide influxdb-api, one mesh-made v1 credential per consumer
grafana's data source and Node-RED's influxdb nodes reached ace's
InfluxDB by a LAN IP or a public name nobody routes, with a credential
somebody made by hand. Now a consumer requires influxdb-api and is told
where it is, which org and default bucket it serves, and signs in with
the password the mesh minted for the pair.

The credential is a v1-compatibility authorization, made per grant by
the new provisioner: InfluxDB 2.x generates API tokens itself and
ignores one the caller sends, so a v2 token could only be accepted by
hand per pair; a v1 authorization takes a caller-chosen password (8-72
characters, the mesh mints 40) and reads/writes every bucket as a
database of its name over InfluxQL and line protocol. A consumer
contributes `access` (read, write, read-write) and, for writing, the
buckets; a missing bucket is made and never deleted. Only
authorizations named mesh_* and marked [mesh] are ever changed or
removed; anything else of that name is refused and left alone.

The org and default bucket are served facts the assignment's settings
set, reaching both the consumers and the provisioner's config.json.
2026-09-30 12:55:38 +02:00
jschoubben 5a906b757d keycloak, grafana: their public names come from the mesh, not the manifest
GF_SERVER_ROOT_URL=https://grafana.zurag.be, KC_HOSTNAME=https://keycloak.novox.be
and the served issuer's novox default were domains in definitions — wrong on
every other machine (ADR 0112). The names now come from ${bound:route:name}
(mesh-controller #149, hq 122): grafana's in oidc.env, keycloak's in a
hostname.env its server reads. The issuer includes the realm and stays the
assignment's, with no default: unset, a consumer asking for it is refused
and the provisioner says so, rather than both quietly using novox's URL.

Rendered through mesh-controller #149 from these manifests on a zurag.be
node: KC_HOSTNAME=https://keycloak.zurag.be, GF_SERVER_ROOT_URL=
https://grafana.zurag.be, OIDC URLs from the issuer setting. Needs #149
merged and rolled out first.
2026-09-30 00:49:08 +02:00
jschoubben d8ee88e487 grafana: log in through keycloak's oidc-client provision
HAL's grafana logged in through a hand-made Keycloak client whose secret sat
in its .env. Requiring oidc-client gives it a client the mesh makes and keeps:
the id and URLs come from the binding, the secret arrives as a file grafana
reads itself (__FILE), and the callback it contributes is what keycloak
registers as its redirect.

GF_SERVER_ROOT_URL is still a literal: a module cannot yet learn the public
name the mesh composes for its own endpoint (hq issue 122), and without it
grafana sends a redirect Keycloak refuses.
2026-09-30 00:39:16 +02:00
jschoubben 54557b77bf keycloak: provide oidc-client, one mesh-made client per consumer
A module that logs people in through Keycloak had to be given a client by
hand, with its secret copied into the consumer's environment. As a provision
the mesh derives the client id (the consumer's identity, mesh_<node>_<module>)
and mints its secret, and delivers both ends: keycloak creates exactly that
confidential client, the consumer names it through ${bound:oidc-client:as}.

The consumer says where its browser comes back to (`callback`) and which
endpoint it is reached on (`label`/`endpoint`), so the redirect is built from
the same names the mesh composes for its route. keycloak serves the issuer and
the endpoint paths under it; the issuer is the one value an assignment sets,
and the realm is read out of it, so consumer and client cannot disagree.

Only what the mesh made is touched: its clients carry mesh.provisioned=true;
a client of the same id without the mark is refused, never adopted, updated
or deleted. The runtime now gets the admin password as a file, which its
tools also needed and never had.
2026-09-30 00:39:16 +02:00
jschoubben a5e21cb438 grafana: its directories are placed, its admin password is a file, and it runs the build in use
The module stated /var/lib/grafana-module and /services/grafana/data, a
layout no definition may carry (ADR 0112). State and data are now placed
directories; the admin secret lives beside the broker account under the
mesh's own state.

The admin password reached grafana through an env-file. Grafana honours
GF_SECURITY_ADMIN_PASSWORD__FILE, so it is now a 0400 file owned by the
image's user (472) and mounted, and "secrets-in-environment" is gone
(ADR 0086).

The runtime sidecar was given no credential at all - its config file was
"{}", so GrafanaClient.fromEnv threw and the tools and the alert watcher
did nothing. It now carries user/password from the same secret, and it
calls grafana on the machine port the mesh assigned (${port:3000}) rather
than a literal 3000.

Image pinned to the 13.2.2 build ace's predecessor runs; the old pin was
13.2.1, older than the data it would open.

Verified: catalogue tests with MESH_CATALOGUE pointing here; a throwaway
container of the pinned image with the file-mounted secret answers
/api/health and authenticates admin with the file's value (default
admin/admin refused); restarted over the same data with a different file
value, the original password still holds - so a migrated instance's
password must be accepted, not minted; data owned by another uid fails to
start, so a moved data directory must be chowned to 472.
2026-09-29 23:43:00 +02:00
jschoubben fe0ed3b74e influxdb: place its directories, hand secrets over as files, name its UI
The manifest named /services/influxdb and /var/lib/influxdb-module — one
machine's paths — and passed the admin password and token through the
environment. ace is moving its 2022 instance onto the mesh, so the module
has to be what it is on any machine.

- data, config and state are placed directories; the data keeps 1000:1000,
  the image's influxdb user, which is who owns ace's data today.
- the init secrets reach the image through its own
  DOCKER_INFLUXDB_INIT_{PASSWORD,ADMIN_TOKEN}_FILE; the vault's files are
  mounted read-only. secrets-in-environment is gone.
- the sidecar reads its token from the same file (MESH_INFLUXDB_TOKEN_FILE,
  added to client.ts) and reaches the server at its assigned machine port
  (${port:8086}) instead of assuming 8086. The unused config-dir mount,
  which held the CLI's copy of the admin token, is dropped.
- the api endpoint contributes a route: the web UI is how people use it,
  and reach is the assignment's to say.

Verified: catalogue tests pass with MESH_CATALOGUE pointed at this tree.
The pinned 2.9.1 image, run on a scratch copy of ace's 2.4.0 data, opens
it, runs its metadata migrations (backing up the pre-upgrade bolt/sqlite)
and hashes the two stored tokens; /health passes. A fresh setup through
the _FILE variables, with dummy secrets as root-owned 0600 files, accepts
the token (200 on /api/v2/buckets) and the password (204 on /signin).
client.ts typechecks strict and reads the token file, tolerating the
endpoints key in its config.
2026-09-29 23:42:18 +02:00
mesh-admin 8064e5da8f Merge pull request 'searxng: its settings are a file the mesh writes, not the image's defaults' (#143) from feat/searxng-settings-as-a-file into main 2026-09-29 20:45:13 +00:00
jschoubben 63a255c5cb searxng: bind its route where its state now lives
The route binding still named /var/lib/searxng-module, the directory the
previous commit placed elsewhere — the host would have written it into a
directory nothing declares. Same shape as gitea and nextcloud.
2026-09-29 22:41:56 +02:00
jschoubben 7ad1fbd5c6 searxng: its settings are a file the mesh writes, not the image's defaults
The module ran searxng on the image's built-in settings, which serve html
only — so the module's own search tool (format=json) was refused by the
software it fronts. And there was no way to configure it per machine: the
only file settings reach was the sidecar's.

settings.yml is now the module's one mergeable file (JSON is YAML): generic
defaults in the manifest (json format on, limiter and image proxy off,
valkey wired), and whatever differs per machine — base_url, method,
autocomplete, suspended times — set as the assignment's settings. The
secret key is filled on the machine through ${secret:secret}, so the
secrets-in-environment exception and the env file go. Directories are
placed. Image pinned to 2026.9.20, what ace runs today (the old pin was
older, 2026.9.1).

The sidecar's config.json is no longer mergeable: settings merge into every
mergeable file of a module, and the sidecar would have received searxng's
keys. It only ever read an optional url, which its env already carries.

Verified on ace: the pinned image serves html and json from a read-only,
root-owned 0600 JSON settings.yml.
2026-09-29 22:33:38 +02:00
jschoubben 67f5f4cffd Merge pull request 'ca-trust: a machine trusts the mesh's authority because a module put its root there' (#142) from feat/ca-trust into main 2026-09-29 14:06:36 +00:00
jschoubben 8797335fbc ca-trust: a machine trusts the mesh's authority because a module put its root there
novox/hq ADR 0147, issue 129. Every internal HTTPS name fails verification
on every machine: the certificates are genuine and nothing on a machine has
ever been told what issued them. The proxy's fetch answers for the proxy and
for nothing else — a browser, git over HTTPS and every module calling another
by an internal name read the machine's own trust store.

The module requires internal-acme-ca, fetches the root over the mesh's own
network (no prior trust to have; that is what this establishes), installs it
among the machine's anchors and refreshes the extracted bundles. Being
unassigned stops the unit, and stopping it takes the anchor away and
refreshes them again.

Arch's layout is named out loud: a machine that keeps anchors elsewhere fails
visibly rather than writing a file nothing reads.
2026-09-29 15:07:40 +02:00
mesh-admin 53dc108603 Merge pull request 'Remove the network-checker module: it does not do what was decided' (#141) from chore/remove-the-network-checker-module into main 2026-09-29 12:43:34 +00:00
jschoubben ebf5ba2d4c Remove the network-checker module: it does not do what was decided
What was in the catalogue was the first thing I built, not the thing ADR 0146
describes. It dialled raw ports on machine addresses from one hosting form and
emitted nothing, so findings would have sat in a file on the machine — the exact
thing issue 145 is about. It was never registered, never assigned, and never ran.

0146 says names per hosting form, fetched over TLS with the certificate verified,
and machines discovered over the bus. That shares nothing with this but the word
checker, so it goes rather than being bent into shape. Recorded as work to be
analysed and built deliberately.

Connectivity is checked by hand in the meantime, against the services the mesh
already runs.
2026-09-29 14:43:25 +02:00
mesh-admin bbac08a7d2 Merge pull request 'A network-checker module: dial what the mesh claims, from where the callers are' (#140) from feat/a-network-checker-module into main 2026-09-29 11:44:37 +00:00
jschoubben 784a5a6514 A network-checker module: dial what the mesh claims, from where the callers are
The mesh asserts three things are callable (ADR 0144) — what runs on the same
machine, another machine's service exposed to the private network, and another
machine's service exposed publicly — and has never checked any of them. The first
was broken for eleven hours while the mesh reported every machine healthy.

This runs on every machine, on the cadence the mesh already has, in its own
container: the same position every other module calls from. Not the host and not
the control plane, both of which reach these addresses by paths no ordinary caller
uses and would have passed throughout that outage.

**Its probe is its own endpoint, and that is the point.** Declared reachable over
the private network like any other service, so it is admitted by exactly the rule
that governs every internally-exposed service and fails when that rule is wrong.
The tempting target is a service every machine has, and those are the ones never
closed — ssh above all — which would have passed while the thing that actually
broke was a service exposed to the private network.

It resolves before it dials and says which failed, because a name that does not
resolve and a port that does not answer have different owners. One failure is not
a fault: a machine rebooting is ordinary, so a path is broken after consecutive
runs and the count travels with the result. It reports and repairs nothing.

novox/hq ADR 0145. Eight tests; the consecutive-failure logic proved by reverting
it once. Not yet registered or assigned.
2026-09-29 13:44:16 +02:00
mesh-admin 0c31499fb0 Merge pull request 'Every module names its endpoints, and every route names the one it serves' (#139) from feat/modules-name-their-endpoints into main 2026-09-29 09:51:56 +00:00
jschoubben f118344246 Every module names its endpoints, and every route names the one it serves
75 endpoints across 50 modules, named from what each one is for rather than by a
rule: mail's seven protocol ports are smtp, imaps, submission and the rest; unifi's
nine are inform, stun, discovery, the two portal ports and syslog; minio's two are
s3 and console; the resolver's two are dns-udp and dns-tcp.

And 35 route contributions name the endpoint they serve instead of repeating its
port. A route and a listen both carried a port and nothing said they were the same
thing; now one of them does. gitea's path-level deny rule names neither, because it
is a rule about a name rather than an endpoint.

novox/hq ADR 0138. The words shipped a release ahead in mesh-controller #138 and
#139, and the control plane running today is built from that merge — checked before
this was written, because an unknown manifest key is refused and a catalogue using
one against an older control plane would stop resolving.
2026-09-29 11:51:39 +02:00
mesh-admin 822df220ab Merge pull request 'A routed module listens from the mesh, not from anywhere' (#138) from fix/a-routed-module-listens-from-the-mesh into main 2026-09-29 00:58:39 +00:00
jschoubben 9eb1265bc8 A routed module listens from the mesh, not from anywhere
umami declared its port reachable from anywhere, reasoning that the collection
endpoint tracked browsers POST to must be public. That is true of the name and
not of the port: both its surfaces are served through the proxy by name, so the
port is how the proxy reaches it and nothing else (ADR 0045).

Measured, which is how this was found: with the port open to the internet, the
dashboard's login page was served over plain HTTP directly on the machine's port,
bypassing every rule the proxy applies by path. The route stays exactly as it was,
so the collection endpoint keeps working.
2026-09-29 02:54:10 +02:00
mesh-admin 41cfc70b53 Merge pull request 'The resolver declares both protocols it answers on' (#137) from fix/the-resolver-declares-both-protocols into main 2026-09-28 22:04:19 +00:00
jschoubben acedc5d9d9 The resolver declares both protocols it answers on
It declared udp/53 only. The daemon listens on tcp/53 as well, and a resolver is
asked over tcp whenever an answer will not fit in a datagram — so on every
converged machine that port is closed while the service reports itself healthy
and the manifest reads as though the resolver were fully declared.

The same fault as issue 136 in miniature: the declaration covers part of what the
service does, and the gap is silent because nothing compares the two.
2026-09-28 23:53:03 +02:00
mesh-admin 521a8dd1e2 Merge pull request 'sshd: the daemon it owns starts at boot' (#136) from fix/sshd-declares-the-daemon-it-owns into main 2026-09-28 19:43:29 +00:00
jschoubben e145e2236c sshd: the daemon it owns starts at boot
The module said the service must be running and nothing about boot, so the
machine's own way back in was enabled only because something before the mesh
had enabled it. All four machines happen to be enabled today; none of them is
enabled because the mesh says so, and a machine adopted tomorrow would run ssh
until its first reboot.

Not `state: running` alone for the same reason the module exists: this is the
one daemon whose absence cannot be fixed remotely.
2026-09-28 21:43:27 +02:00
mesh-admin 4d7e37e319 Merge pull request 'fail2ban bans through an action every machine has' (#135) from fix/fail2ban-bans-through-what-every-machine-has into main 2026-09-28 18:48:26 +00:00
jschoubben 026421fd6e fail2ban: ban through an action every machine has
jail.local named ufw as the ban action. Two machines on this mesh have no ufw,
and fail2ban does not check: it starts, the jail reads the log, counts the
attempts, runs the ban command, gets 127 -- 'ufw: command not found' -- and
logs an error nobody reads. The service is active, the mesh reports the module
applied, and the machine is not protected. Proven by banning a documentation
address on such a machine today.

The replacement is this module's own dualchain action, already used by the
recidive jail on all four machines, so it is not a new dependency. It bans in
DOCKER-USER as well as INPUT, which ufw's action did not, and it bans all
ports, which ufw's action did.
2026-09-28 20:48:24 +02:00
mesh-admin af89bb11ff Merge pull request 'fail2ban declares the log its own recidive jail reads' (#134) from fix/fail2ban-declares-the-log-its-own-jail-reads into main 2026-09-28 18:45:34 +00:00
jschoubben 7c18cdbd39 fail2ban: declare the log its own recidive jail reads
The recidive jail bans whoever keeps coming back by reading fail2ban's own
log, and fail2ban checks every jail's log file while it configures itself --
before it has created that log. On a machine where the file is not there
already, no jail is found for recidive, configuration fails, and the whole
service refuses to start, taking the sshd jail with it. Two machines assigned
this module today came up failed for exactly that reason; the two where it
worked had a log from years of the service running.

Declared create-once: the mesh puts an empty file there when it is absent and
never touches it again, because what grows in it is fail2ban's, and the
logrotate file this module already ships is what keeps it small.

This also reverts the previous two commits' fail2ban.local. It declared a
logtarget that the package already sets to the same path on every machine
here -- pacman reports the config pristine -- so it fixed nothing and said
something untrue about why.
2026-09-28 20:45:32 +02:00
mesh-admin 4fb16b2e6b Merge pull request 'fail2ban restarts when the log declaration changes' (#133) from fix/fail2ban-restarts-on-its-log-target into main 2026-09-28 18:42:44 +00:00
jschoubben c5af8635c8 fail2ban: restart when the log declaration changes
The file that says where fail2ban logs was not in restart-on, so a change to
it would sit on disk with the running service unaware of it -- the same shape
as any other jail file this module already restarts for.
2026-09-28 20:42:42 +02:00
mesh-admin 87366c5f36 Merge pull request 'fail2ban declares where it logs, so the recidive jail has a file to read' (#132) from fix/fail2ban-declares-where-it-logs into main 2026-09-28 18:41:16 +00:00
jschoubben f8ca36aacf fail2ban: declare where it logs, so the recidive jail has a file to read
The recidive jail reads /var/log/fail2ban.log and this module ships the
logrotate file for it, but nothing ever told fail2ban to write there. Where
the package default stands, fail2ban logs to the journal, the recidive jail
finds no log file, and the whole service refuses to start -- taking the sshd
jail with it. Two machines assigned this module today came up failed; the two
where it worked had /etc/fail2ban/fail2ban.conf edited by hand, which a
package upgrade would have undone.

Declared in fail2ban.local, because fail2ban.conf belongs to the package.
2026-09-28 20:41:09 +02:00
mesh-admin 812355bf31 Merge pull request 'The catalogue hears what it missed' (#131) from feat/the-catalogue-hears-what-it-missed into main 2026-09-28 14:08:48 +00:00
jschoubben 016ddb2b3a The catalogue hears what it missed
It asks what it missed on every start and the answer never arrived: the control plane replayed each
build it held as a module's event from a module called "control-plane", which does not exist, so its
own account refused the publish and the graph kept the gap. The control plane now states those under
the seat it holds (novox/hq ADR 0134, mesh-controller #129), so this consumes that too — one handler,
because what a build means for the graph is the same whether the build machine says it as it happens
or the mesh says what it already held.
2026-09-28 16:08:46 +02:00
mesh-admin ea17bf46d2 Merge pull request 'The catalogue prepares its own schema instead of migrating at start' (#130) from feat/the-catalogue-prepares-its-own-schema into main 2026-09-28 13:40:30 +00:00
jschoubben 4258f01614 The catalogue prepares its own schema instead of migrating at start
It brought its schema up inside its runtime, on every start. That made a schema it could not reach a
crash loop rather than a stop, with the module graph keeping a gap and nothing saying so — which is
how a whole morning's builds went unrecorded. The mesh now prepares this module's state before it
starts this version and does not start it if that failed (novox/hq ADR 0135): the work moves to an
entrypoint the image names in MESH_PREPARE, beside the entrypoints it already names.

The reason it was at start — that a step blocking the apply would block the very apply bringing the
overlay up — stopped being true when a step's failure became its module's business rather than the
machine's (ADR 0136).
2026-09-28 15:40:28 +02:00
mesh-admin 4d9b4fdfa6 Merge pull request 'The catalogue declares the event it emits on starting' (#129) from fix/the-catalogue-declares-the-event-it-emits into main 2026-09-28 07:49:06 +00:00
jschoubben eff11b1d4d The catalogue declares the event it emits on starting
Its runtime announces that it has just started and may have missed builds — the event the control
plane follows to replay them — and its manifest did not declare it. A module's authority on the bus
is derived from what it declares, so the publish was refused and the runtime died on start, in a
loop, with the mesh's graph never catching up.
2026-09-28 09:49:03 +02:00
mesh-admin 4ead13d4d4 Merge pull request 'A merge says which files it changed' (#128) from feat/a-merge-rebuilds-what-it-changed into main 2026-09-28 07:20:05 +00:00
jschoubben 3b77dde666 A merge says which files it changed
Every module built from a repository was rebuilt for a change to any of them: one merge in this
repository meant twenty-six builds, which is what exhausted a public registry's pull limit. The
forge lists the files a merge changed and the event carries them, from the watcher and from the
merge tool alike; a merge that changed more files than were asked for says so, and the mesh then
treats the whole repository as changed rather than guessing.
2026-09-28 09:17:38 +02:00
mesh-admin 5ea4961980 Merge pull request 'A merge older than the watching is history, not news' (#127) from fix/a-merge-older-than-the-watching-is-history into main 2026-09-28 03:08:22 +00:00
jschoubben 2b8a668d06 A merge older than the watching is history, not news
An old merge past the first page of the forge's listing surfaced as newer pull requests were
updated, and was announced as if it had just happened; the mesh then rebuilt everything built from
that repository, once per old merge. The moment the watching began is kept with the record, and
only a merge made since is announced.
2026-09-28 05:08:20 +02:00
mesh-admin 719fb1e025 Merge pull request 'A merge the forge announces is said in its log' (#126) from fix/a-merge-announced-is-said into main 2026-09-28 02:44:19 +00:00
jschoubben ac5630bee2 A merge the forge announces is said in its log
A trigger that fires silently is indistinguishable from one that did not fire (novox/hq issue
131); the announcement is now one line an operator can read.
2026-09-28 04:44:17 +02:00
mesh-admin 1c995fa9fc Merge pull request 'The forge watches every repository, not the administrator's own' (#125) from fix/the-forge-watches-every-repository into main 2026-09-28 02:31:25 +00:00
jschoubben d70cb18ea0 The forge watches every repository, not the administrator's own
/user/repos lists what the token's user owns, which for the mesh's administrator is nothing — so
the forge module watched an empty list and never announced a merge. It reads the forge's whole
view through the search endpoint, every page.
2026-09-28 04:31:20 +02:00
74 changed files with 1708 additions and 139 deletions
+2 -1
View File
@@ -14,7 +14,7 @@
},
"route": {
"label": "baserow",
"port": 80
"endpoint": "web"
}
},
"binds": {
@@ -30,6 +30,7 @@
},
"listens": [
{
"name": "web",
"port": 80,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -13,6 +13,7 @@
},
"listens": [
{
"name": "web",
"port": 6767,
"protocol": "tcp",
"from": "mesh",
@@ -110,7 +111,7 @@
"contributes": {
"route": {
"label": "subs",
"port": 6767
"endpoint": "web"
}
},
"binds": {
+2 -1
View File
@@ -15,6 +15,7 @@
},
"listens": [
{
"name": "web",
"port": 8787,
"protocol": "tcp",
"from": "mesh",
@@ -87,7 +88,7 @@
"contributes": {
"route": {
"label": "books",
"port": 8787
"endpoint": "web"
}
},
"binds": {
+56
View File
@@ -0,0 +1,56 @@
{
"module": "ca-trust",
"version": "1",
"slug": "catrust",
"capabilities": [
"service-manager"
],
"requires": [
"internal-acme-ca"
],
"seats": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"claims": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "anchor",
"type": "file",
"path": "${dir:state}/anchor",
"mode": "0755",
"content": "#!/bin/sh\n# The mesh's internal certificate authority, trusted by this machine.\n#\n# Written by the mesh from the ca-trust module's manifest (novox/hq ADR 0147).\n# Editing it here lasts until the next apply.\n#\n# There is no prior trust to verify the fetch against \u2014 this is the thing that\n# establishes it \u2014 so it is made over the mesh's own private network, which is\n# what authenticates it (novox/hq ADR 0098, the same reasoning that lets the\n# route proxy fetch this root for itself). What comes back is checked here: a\n# body that is not a certificate is refused now, rather than believed and then\n# failed by whatever reads the trust store next.\nset -eu\n\nROOTS='https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:roots}'\nANCHORS=/etc/ca-certificates/trust-source/anchors\nANCHOR=\"$ANCHORS/mesh-internal-ca.crt\"\n\n# Arch's layout, said out loud rather than assumed: a machine that keeps its\n# anchors elsewhere fails here, visibly, instead of writing a file nothing\n# reads. That failure is the signal that this belongs in the host, where one\n# operating system's difference lives (novox/hq ADR 0147, option 2).\n[ -d \"$ANCHORS\" ] || {\n\techo \"this machine keeps no trust anchors in $ANCHORS; ca-trust is written for that layout\" >&2\n\texit 1\n}\n\ncase \"${1:-}\" in\ninstall)\n\ttmp=$(mktemp)\n\ttrap 'rm -f \"$tmp\"' EXIT\n\t# The authority may still be starting, or this machine may have come up\n\t# before it: two minutes of asking, then an honest failure.\n\tn=0\n\twhile [ \"$n\" -lt 60 ]; do\n\t\tif curl --fail --silent --show-error --insecure --max-time 10 \\\n\t\t\t--output \"$tmp\" \"$ROOTS\" &&\n\t\t\tgrep -q 'BEGIN CERTIFICATE' \"$tmp\"; then\n\t\t\tinstall -m 0644 \"$tmp\" \"$ANCHOR\"\n\t\t\tupdate-ca-trust\n\t\t\texit 0\n\t\tfi\n\t\tn=$((n + 1))\n\t\tsleep 2\n\tdone\n\techo \"the authority at $ROOTS did not serve a certificate within two minutes\" >&2\n\texit 1\n\t;;\nremove)\n\t# What stopping the unit does, and therefore what being unassigned does.\n\trm -f \"$ANCHOR\"\n\tupdate-ca-trust\n\t;;\n*)\n\techo \"usage: $(basename \"$0\") install|remove\" >&2\n\texit 2\n\t;;\nesac\n"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-ca-trust.service",
"mode": "0644",
"content": "[Unit]\nDescription=The mesh's internal certificate authority, trusted by this machine\n# novox/hq ADR 0147. Starting this unit places the mesh's root among this\n# machine's trust anchors; stopping it takes the root away again, which is what\n# the host does when the module is no longer assigned here.\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/anchor install\nExecStop=${dir:state}/anchor remove\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"id": "trust",
"type": "service",
"unit": "mesh-ca-trust.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"anchor",
"unit"
]
}
]
}
+2 -1
View File
@@ -11,7 +11,7 @@
"contributes": {
"route": {
"label": "de-spiegel",
"port": 35621
"endpoint": "web"
}
},
"binds": {
@@ -23,6 +23,7 @@
},
"listens": [
{
"name": "web",
"port": 35621,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -29,6 +29,7 @@
},
"listens": [
{
"name": "registry",
"port": 5000,
"protocol": "tcp",
"from": "mesh",
+9
View File
@@ -22,11 +22,20 @@
],
"listens": [
{
"name": "dns-udp",
"port": 53,
"protocol": "udp",
"from": "mesh",
"why": "every name for this machine and what it runs \u2014 the mesh's own answered here, the rest forwarded",
"fixed": true
},
{
"name": "dns-tcp",
"port": 53,
"protocol": "tcp",
"from": "mesh",
"why": "the same names over tcp, which a resolver answers on as well and is asked for whenever an answer will not fit in a datagram. Declared because the daemon serves it: a declaration that covers one of the two protocols its own service listens on leaves the other closed while everything reports success",
"fixed": true
}
],
"resources": [
+9 -1
View File
@@ -33,7 +33,7 @@
"type": "file",
"path": "/etc/fail2ban/jail.local",
"mode": "0644",
"content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}\n\nbantime = 10m\nfindtime = 10m\nmaxretry = 5\n\nbanaction = ufw\nbanaction_allports = iptables-allports\n\n[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
"content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}\n\nbantime = 10m\nfindtime = 10m\nmaxretry = 5\n\n# Ban through iptables, not through a firewall front-end the machine may not have. ufw is\n# installed on two of this mesh's machines and absent on the other two, and fail2ban finds out\n# only at ban time: the service reports healthy, the jail counts the attempt, the ban command\n# exits 127, and nothing is blocked. Proven on 2026-09-28 -- 'ufw: command not found' on a\n# machine the mesh reported as protected.\n#\n# The action below is this module's own, already used by the recidive jail on every machine\n# here, and it bans in DOCKER-USER as well as INPUT, so a container's published port is\n# covered too.\nbanaction = iptables-allports-dualchain\nbanaction_allports = iptables-allports-dualchain\n\n[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
},
{
"id": "jail-sshd",
@@ -42,6 +42,14 @@
"mode": "0644",
"content": "[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\nmaxretry = 5\n"
},
{
"id": "log",
"type": "file",
"path": "/var/log/fail2ban.log",
"mode": "0640",
"create-once": true,
"content": ""
},
{
"id": "jail-recidive",
"type": "file",
+27 -2
View File
@@ -121,9 +121,23 @@ export class GiteaClient {
// ---- Repositories ----
/** Every repository this token can see, one page. `/user/repos` is only what the token's own
* user owns — for the mesh's administrator that is nothing, which is how the forge watched an
* empty list and announced no merge (2026-09-28). The search endpoint is the forge's whole view. */
async listRepos(page = 1, limit = 20): Promise<GiteaRepo[]> {
const repos = await this.request<any[]>(`/user/repos?page=${page}&limit=${limit}`);
return (repos ?? []).map(GiteaClient.mapRepo);
const found = await this.request<{ data?: any[] }>(`/repos/search?page=${page}&limit=${limit}`);
return (found?.data ?? []).map(GiteaClient.mapRepo);
}
/** Every repository, all pages. */
async listAllRepos(): Promise<GiteaRepo[]> {
const all: GiteaRepo[] = [];
for (let page = 1; page < 100; page++) {
const batch = await this.listRepos(page, 50);
all.push(...batch);
if (batch.length < 50) break;
}
return all;
}
async createRepo(data: {
@@ -215,6 +229,17 @@ export class GiteaClient {
return GiteaClient.mapPull(await this.request<any>(`/repos/${owner}/${repo}/pulls/${index}`));
}
/** The files a merged pull request changed, as paths from the repository's root.
*
* `limit` is what is asked for, and a merge that changed more says so rather than being read
* page by page: what the mesh does with a partial list is treat the whole repository as changed,
* so more pages would buy nothing. */
async listPullFiles(owner: string, repo: string, index: number, limit = 100): Promise<{ paths: string[]; truncated: boolean }> {
const files = await this.request<any[]>(`/repos/${owner}/${repo}/pulls/${index}/files?limit=${limit}`);
const paths = (files ?? []).map((f) => String(f?.filename ?? "")).filter((p) => p !== "");
return { paths, truncated: paths.length >= limit };
}
async createPullRequest(
owner: string,
repo: string,
+26 -5
View File
@@ -31,7 +31,7 @@ try {
const seen = new Set<string>();
let primed = false;
async function pollRepos(client: GiteaClient): Promise<void> {
const repos = await client.listRepos(1, 50);
const repos = await client.listAllRepos();
for (const repo of repos) {
if (!seen.has(repo.full_name)) {
if (primed) {
@@ -61,9 +61,17 @@ import { join } from "node:path";
const mergedRecord = process.env.MESH_GITEA_STATE_DIR ? join(process.env.MESH_GITEA_STATE_DIR, "merged-announced.json") : null;
const announced = new Set<string>();
let primedMerges = false;
// since is the moment the watching began: a merge made before it is history, whatever page of the
// forge's listing it surfaces on. Without it, an old merge past the first page — pushed into view
// as newer pull requests were updated — was announced as if it had just happened, and the mesh
// rebuilt everything built from that repository, once per old merge (2026-09-28).
let since = "";
if (mergedRecord && existsSync(mergedRecord)) {
try {
for (const sha of JSON.parse(readFileSync(mergedRecord, "utf8")) as string[]) announced.add(sha);
const kept = JSON.parse(readFileSync(mergedRecord, "utf8")) as string[] | { announced: string[]; since: string };
const list = Array.isArray(kept) ? kept : kept.announced;
for (const sha of list) announced.add(sha);
since = Array.isArray(kept) ? new Date().toISOString() : kept.since;
primedMerges = true;
} catch {
// An unreadable record is treated as no record: prime again rather than re-announce history.
@@ -73,17 +81,24 @@ function keepAnnounced(): void {
if (!mergedRecord) return;
mkdirSync(join(mergedRecord, ".."), { recursive: true });
const tmp = mergedRecord + ".tmp";
writeFileSync(tmp, JSON.stringify([...announced].slice(-2000)));
writeFileSync(tmp, JSON.stringify({ announced: [...announced].slice(-2000), since }));
renameSync(tmp, mergedRecord);
}
async function pollMerged(client: GiteaClient): Promise<void> {
const repos = await client.listRepos(1, 50);
const repos = await client.listAllRepos();
let changed = false;
for (const repo of repos) {
const pulls = await client.listPullRequests(repo.owner, repo.name, { state: "closed", sort: "recentupdate", limit: "20" });
for (const pull of pulls) {
if (!pull.merged || !pull.merge_commit_sha || announced.has(pull.merge_commit_sha)) continue;
if (primedMerges) {
// Announced only if merged since the watching began; recorded either way, so it is looked
// at once.
const fresh = !!pull.merged_at && !!since && pull.merged_at > since;
if (primedMerges && fresh) {
// What it changed, asked for only now: a module is rebuilt because a file inside its own
// directory moved, and without this every module built from a repository is rebuilt for a
// change to any of them (novox/hq 04-ISSUES/131).
const changed = await client.listPullFiles(repo.owner, repo.name, pull.number);
await emit("pull.merged", {
owner: repo.owner,
repo: repo.name,
@@ -95,12 +110,18 @@ async function pollMerged(client: GiteaClient): Promise<void> {
merged_at: pull.merged_at,
clone_url: repo.clone_url,
html_url: pull.html_url,
paths: changed.paths,
paths_truncated: changed.truncated,
});
// Said, because a trigger that fires silently is indistinguishable from one that did not
// fire (novox/hq 04-ISSUES/131) — this line is how an operator knows the mesh was told.
console.log(`[gitea] announced merge ${repo.full_name}#${pull.number} (${pull.merge_commit_sha.slice(0, 8)}) into ${pull.base}`);
}
announced.add(pull.merge_commit_sha);
changed = true;
}
}
if (!primedMerges) since = new Date().toISOString();
if (!primedMerges || changed) keepAnnounced();
primedMerges = true;
}
+3 -1
View File
@@ -13,7 +13,7 @@
"route": {
"web": {
"label": "git",
"port": 3000
"endpoint": "web"
},
"internal-api-refused": {
"label": "git",
@@ -44,12 +44,14 @@
],
"listens": [
{
"name": "web",
"port": 3000,
"protocol": "tcp",
"from": "mesh",
"why": "the forge, over http"
},
{
"name": "ssh",
"port": 22,
"protocol": "tcp",
"from": "mesh",
+5
View File
@@ -233,6 +233,9 @@ export function getGiteaTools(gitea: GiteaClient): ToolDefinition[] {
await gitea.mergePullRequest(owner, repo, number, method, deleteBranch);
// Read it again: the merge commit only exists now, and it is what a build is made from.
const merged = await gitea.getPullRequest(owner, repo, number);
// And what it changed, so the mesh rebuilds the modules whose own files moved rather than
// every module built from the repository (novox/hq 04-ISSUES/131).
const changed = await gitea.listPullFiles(owner, repo, number);
await emit("pull.merged", {
owner,
repo,
@@ -244,6 +247,8 @@ export function getGiteaTools(gitea: GiteaClient): ToolDefinition[] {
merged_at: merged.merged_at,
method,
html_url: pull.html_url,
paths: changed.paths,
paths_truncated: changed.truncated,
});
return { merged: true, number, method, deleted_branch: deleteBranch };
},
+76 -17
View File
@@ -5,7 +5,7 @@
"alert.firing"
],
"own-secrets": {
"admin": "/var/lib/grafana-module/admin.secret",
"admin": "/var/lib/mesh/grafana/admin",
"broker": "/var/lib/mesh/grafana/broker"
},
"capabilities": [
@@ -13,6 +13,7 @@
],
"listens": [
{
"name": "web",
"port": 3000,
"protocol": "tcp",
"from": "mesh",
@@ -29,45 +30,87 @@
{
"id": "state",
"type": "directory",
"path": "/var/lib/grafana-module",
"mode": "0700"
"mode": "0700",
"place": "."
},
{
"id": "data",
"type": "directory",
"path": "/services/grafana/data",
"mode": "0700",
"owner": "472:472"
},
{
"id": "server-env",
"id": "admin-secret",
"type": "file",
"path": "/var/lib/grafana-module/server.env",
"mode": "0600",
"content": "GF_SECURITY_ADMIN_PASSWORD=${secret:admin}\n"
"path": "${dir:state}/admin.secret",
"mode": "0400",
"owner": "472:472",
"content": "${secret:admin}"
},
{
"id": "oidc-secret",
"type": "file",
"path": "${dir:state}/oidc-client.secret",
"mode": "0400",
"owner": "472:472",
"content": "${secret:oidc-client}"
},
{
"id": "oidc-env",
"type": "file",
"path": "${dir:state}/oidc.env",
"mode": "0644",
"content": "GF_SERVER_ROOT_URL=https://${bound:route:name}\nGF_AUTH_GENERIC_OAUTH_ENABLED=true\nGF_AUTH_GENERIC_OAUTH_NAME=Keycloak\nGF_AUTH_GENERIC_OAUTH_CLIENT_ID=${bound:oidc-client:as}\nGF_AUTH_GENERIC_OAUTH_CLIENT_SECRET__FILE=/run/secrets/oidc-client\nGF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile roles\nGF_AUTH_GENERIC_OAUTH_AUTH_URL=${bound:oidc-client:issuer}${bound:oidc-client:authorization-path}\nGF_AUTH_GENERIC_OAUTH_TOKEN_URL=${bound:oidc-client:issuer}${bound:oidc-client:token-path}\nGF_AUTH_GENERIC_OAUTH_API_URL=${bound:oidc-client:issuer}${bound:oidc-client:userinfo-path}\nGF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'admin') && 'Admin' || contains(realm_access.roles[*], 'admin') && 'Admin' || 'Viewer'\nGF_AUTH_GENERIC_OAUTH_USE_PKCE=true\nGF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true\nGF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true\n"
},
{
"id": "influxdb-secret",
"type": "file",
"path": "${dir:state}/influxdb-api.secret",
"mode": "0400",
"owner": "472:472",
"content": "${secret:influxdb-api}"
},
{
"id": "influxdb-datasource",
"type": "file",
"path": "${dir:state}/datasource-influxdb.yaml",
"mode": "0644",
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding; grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: ${bound:influxdb-api:scheme}://${bound:influxdb-api:at}:${bound:influxdb-api:port}\n user: ${bound:influxdb-api:as}\n isDefault: false\n editable: false\n jsonData:\n dbName: ${bound:influxdb-api:bucket}\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb-api}\n"
},
{
"id": "server",
"type": "container",
"name": "grafana",
"image": "grafana/grafana@sha256:f772d434e8fab0049deb2b1b30abd43342bcfca1537614aa8d36080232cf4283",
"image": "grafana/grafana@sha256:ac461fb352abc50da10a51c7d02462e9c05488f11f53f14b3ad79a8145f638a0",
"ports": [
"3000"
],
"volumes": [
"/services/grafana/data:/var/lib/grafana"
"${dir:data}:/var/lib/grafana",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"${dir:state}/oidc-client.secret:/run/secrets/oidc-client:ro",
"${dir:state}/influxdb-api.secret:/run/secrets/influxdb-api:ro",
"${dir:state}/datasource-influxdb.yaml:/etc/grafana/provisioning/datasources/mesh-influxdb.yaml:ro"
],
"env": {
"GF_SECURITY_ADMIN_PASSWORD__FILE": "/run/secrets/admin"
},
"env-file": [
"/var/lib/grafana-module/server.env"
"${dir:state}/oidc.env"
],
"secrets-in-environment": "grafana honours GF_SECURITY_ADMIN_PASSWORD__FILE; convertible, awaiting a bed that exercises the admin password (assigned-grafana serves tools only)"
"restart-on": [
"oidc-env",
"oidc-secret",
"influxdb-datasource",
"influxdb-secret"
]
},
{
"id": "runtime-config",
"type": "file",
"path": "/var/lib/mesh/grafana/config.json",
"mode": "0600",
"content": "{}\n",
"content": "{\n \"user\": \"admin\",\n \"password\": \"${secret:admin}\"\n}\n",
"merge": "json"
},
{
@@ -81,7 +124,7 @@
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_GRAFANA_URL": "http://127.0.0.1:3000",
"MESH_GRAFANA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GRAFANA_CONFIG_FILE": "/run/config/config.json"
},
"restart-on": [
@@ -91,16 +134,32 @@
}
],
"requires": [
"route"
"route",
"oidc-client",
"influxdb-api"
],
"contributes": {
"route": {
"label": "grafana",
"port": 3000
"endpoint": "web"
},
"oidc-client": {
"label": "grafana",
"endpoint": "web",
"callback": "/login/generic_oauth"
},
"influxdb-api": {
"access": "read"
}
},
"binds": {
"route": "/var/lib/mesh/grafana/route.json"
"route": "${dir:state}/route.json",
"oidc-client": "${dir:state}/oidc.json",
"influxdb-api": "${dir:state}/influxdb.json"
},
"secrets": {
"oidc-client": "/var/lib/mesh/grafana/oidc-client",
"influxdb-api": "/var/lib/mesh/grafana/influxdb-api"
},
"build": {
"on": [
+2 -1
View File
@@ -11,7 +11,7 @@
"contributes": {
"route": {
"label": "hello",
"port": 8080
"endpoint": "web"
}
},
"binds": {
@@ -19,6 +19,7 @@
},
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -14,6 +14,7 @@
},
"listens": [
{
"name": "web",
"port": 8123,
"protocol": "tcp",
"from": "mesh",
@@ -85,7 +86,7 @@
"contributes": {
"route": {
"label": "home-assistant",
"port": 8123
"endpoint": "web"
}
},
"binds": {
+1
View File
@@ -13,6 +13,7 @@
},
"listens": [
{
"name": "stream",
"port": 8000,
"protocol": "tcp",
"from": "mesh",
+2 -2
View File
@@ -13,7 +13,7 @@ ARG RUNTIME_BASE
FROM ${BUILD_BASE} AS build
WORKDIR /app/modules/influxdb
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts tools/index.ts \
RUN node /app/node_modules/typescript/bin/tsc client.ts grants.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
@@ -21,4 +21,4 @@ COPY --from=build /app/modules/influxdb/dist /app/modules/influxdb/dist
# Every serve-time entrypoint, loaded by the runtime in serve mode: tools and events serve, and a
# provider's provisioner runs its reconcile loop in the same process, with the broker connected —
# the convention novox/hq issues 060/061 settled.
ENV MESH_TOOL_MODULES=/app/modules/influxdb/dist/tools/index.js
ENV MESH_TOOL_MODULES=/app/modules/influxdb/dist/tools/index.js,/app/modules/influxdb/dist/provisioner/index.js
+118 -3
View File
@@ -17,6 +17,25 @@ export interface InfluxBucket {
retentionSeconds?: number;
}
/** One permission of an authorization, as InfluxDB represents it: an action on a resource type,
* in one org, optionally narrowed to one resource by id (no id = every resource of that type). */
export interface InfluxPermission {
action: "read" | "write";
resource: { type: string; orgID?: string; id?: string; name?: string; org?: string };
}
/** A v1-compatibility ("legacy") authorization: a username (InfluxDB calls it `token`) and a
* password the caller chooses, scoped by permissions. The one credential InfluxDB 2.x lets a
* caller set to a value it did not generate — which is what a mesh-minted password needs. */
export interface LegacyAuthorization {
id: string;
token: string;
orgID: string;
status?: "active" | "inactive";
description?: string;
permissions: InfluxPermission[];
}
/** The settings-merged config the mesh delivers (novox/hq ADR 0046): { url, apiKey, token, password, user, ... }. */
function meshConfig(file?: string): Record<string, string> {
if (!file) return {};
@@ -24,13 +43,20 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/** A secret delivered as a file, trimmed; undefined when there is none, so the caller can fall back. */
function tokenFromFile(file?: string): string | undefined {
if (!file) return undefined;
try { return readFileSync(file, "utf8").trim() || undefined; }
catch { return undefined; }
}
export class InfluxDBClient {
readonly baseUrl: string;
constructor(
url: string,
private readonly token: string,
private readonly org: string,
readonly org: string,
) {
this.baseUrl = url.replace(/\/$/, "");
}
@@ -43,8 +69,10 @@ export class InfluxDBClient {
static fromEnv(env: NodeJS.ProcessEnv = process.env): InfluxDBClient {
const cfg = meshConfig(env.MESH_INFLUXDB_CONFIG_FILE);
const url = cfg.url ?? env.MESH_INFLUXDB_URL ?? `http://127.0.0.1:${env.INFLUXDB_PORT ?? "8086"}`;
const token = cfg.token ?? env.MESH_INFLUXDB_TOKEN;
if (!token) throw new Error("no InfluxDB token — set MESH_INFLUXDB_TOKEN");
// The token reaches the process as a file (novox/hq ADR 0086); the environment variable stays
// only for a workstation running the tools by hand.
const token = cfg.token ?? tokenFromFile(env.MESH_INFLUXDB_TOKEN_FILE) ?? env.MESH_INFLUXDB_TOKEN;
if (!token) throw new Error("no InfluxDB token — set MESH_INFLUXDB_TOKEN_FILE");
const org = cfg.org ?? env.MESH_INFLUXDB_ORG ?? "mesh";
return new InfluxDBClient(url, token, org);
}
@@ -61,6 +89,93 @@ export class InfluxDBClient {
return res;
}
/** Like request, but the answer is returned whatever its status, for the caller to read. */
private async raw(path: string, init?: RequestInit): Promise<Response> {
return fetch(`${this.baseUrl}${path}`, {
...init,
headers: { Authorization: `Token ${this.token}`, ...(init?.headers ?? {}) },
});
}
private async send(path: string, method: string, body?: unknown): Promise<Response> {
return this.request(path, {
method,
headers: { "Content-Type": "application/json" },
body: body === undefined ? undefined : JSON.stringify(body),
});
}
/** The id of the org of this name, or undefined when there is none. */
async orgID(name: string): Promise<string | undefined> {
const res = await this.raw(`/api/v2/orgs?org=${encodeURIComponent(name)}`);
if (res.status === 404) return undefined;
if (!res.ok) throw new Error(`InfluxDB API /api/v2/orgs: ${res.status} ${await res.text()}`);
const body = (await res.json()) as { orgs?: { id: string; name: string }[] };
return body.orgs?.find((o) => o.name === name)?.id;
}
/** The bucket of exactly this name in the org, or undefined. */
async findBucket(orgID: string, name: string): Promise<InfluxBucket | undefined> {
const res = await this.raw(`/api/v2/buckets?orgID=${encodeURIComponent(orgID)}&name=${encodeURIComponent(name)}`);
if (res.status === 404) return undefined;
if (!res.ok) throw new Error(`InfluxDB API /api/v2/buckets: ${res.status} ${await res.text()}`);
const body = (await res.json()) as { buckets?: { id: string; name: string; orgID?: string }[] };
const b = body.buckets?.find((x) => x.name === name);
return b ? { id: b.id, name: b.name, orgID: b.orgID } : undefined;
}
/** Create a bucket that keeps its data for ever — retention is the operator's choice, never the mesh's. */
async createBucket(orgID: string, name: string, description: string): Promise<InfluxBucket> {
const b = (await (await this.send("/api/v2/buckets", "POST", {
orgID, name, description, retentionRules: [],
})).json()) as { id: string; name: string; orgID?: string };
return { id: b.id, name: b.name, orgID: b.orgID };
}
/** The v1 authorization whose username is exactly this, or undefined. */
async findLegacy(username: string): Promise<LegacyAuthorization | undefined> {
const path = `/private/legacy/authorizations?token=${encodeURIComponent(username)}`;
const res = await this.raw(path);
// InfluxDB answers a filter matching nothing with 404, not an empty list.
if (res.status === 404) return undefined;
if (!res.ok) throw new Error(`InfluxDB API ${path}: ${res.status} ${await res.text()}`);
const body = (await res.json()) as { authorizations?: LegacyAuthorization[] };
return body.authorizations?.find((a) => a.token === username);
}
async createLegacy(a: Omit<LegacyAuthorization, "id">): Promise<LegacyAuthorization> {
return (await (await this.send("/private/legacy/authorizations", "POST", a)).json()) as LegacyAuthorization;
}
/** Set a v1 authorization's password. InfluxDB keeps only a hash of it, so it can be set, never read. */
async setLegacyPassword(id: string, password: string): Promise<void> {
await this.send(`/private/legacy/authorizations/${encodeURIComponent(id)}/password`, "POST", { password });
}
async updateLegacy(id: string, patch: { status?: "active" | "inactive"; description?: string }): Promise<void> {
await this.send(`/private/legacy/authorizations/${encodeURIComponent(id)}`, "PATCH", patch);
}
async deleteLegacy(id: string): Promise<void> {
await this.send(`/private/legacy/authorizations/${encodeURIComponent(id)}`, "DELETE");
}
/**
* Whether this username and password sign in on the v1 API — the consumer's own view. Asked with
* a statement that reads nothing (`SHOW DATABASES` lists only what the credential may read), sent
* with Basic auth so the password is never in a URL. 401 is a wrong password or no such user;
* anything else that is not a server error means InfluxDB knew who was asking.
*/
async legacySignsIn(username: string, password: string): Promise<boolean> {
const res = await fetch(`${this.baseUrl}/query?q=${encodeURIComponent("SHOW DATABASES")}`, {
headers: { Authorization: `Basic ${Buffer.from(`${username}:${password}`).toString("base64")}` },
});
await res.arrayBuffer();
if (res.status === 401) return false;
if (res.status >= 500) throw new Error(`InfluxDB v1 /query: ${res.status}`);
return true;
}
/** Server health — the one endpoint that needs no token, but we send it anyway. */
async health(): Promise<InfluxHealth> {
return (await (await this.request("/health")).json()) as InfluxHealth;
+186
View File
@@ -0,0 +1,186 @@
// What the `influxdb-api` provision means in InfluxDB: one v1-compatibility authorization per
// consumer, in the org this module serves, under the username and password the mesh gave both ends,
// allowed exactly the access the consumer contributed. The provisioner (provisioner/index.ts) is the
// sdk harness calling these; they are here, apart from it, so they can be exercised against a fake
// InfluxDB without a broker or a contributions file.
//
// **Why a v1 authorization and not a v2 API token.** The mesh mints the consumer's password and
// hands it to both ends (novox/hq ADR 0048); the provider sets it, and never hands one back. An
// InfluxDB 2.x API token is generated by the server — `POST /api/v2/authorizations` ignores a token
// the caller sends — so a token could only ever be the operator's to accept, one per pair, by hand.
// A v1 authorization is a username and a password the caller chooses (8–72 characters; the mesh
// mints 40), stored hashed, and it reads and writes through InfluxQL (`/query`) and line protocol
// (`/write`), which every bucket answers under its own name as a database (InfluxDB maps each
// bucket to a database of the same name by itself). That is what grafana's InfluxDB data source
// speaks, and what Node-RED's influxdb nodes speak in their 1.x mode — so the mesh can make every
// consumer's credential, rotate it and withdraw it, with no person in the loop.
//
// **What a consumer contributes.** `access`: "read" (the default), "write" or "read-write".
// `buckets`: the buckets it may use, by name. A reader that names none may read every bucket of the
// org — a dashboard is pointed at data, it does not own it. A writer must name its buckets: writing
// everywhere, the org's system buckets included, is never what a consumer means. A named bucket
// that does not exist is created, keeping its data for ever; the mesh never deletes a bucket.
//
// **Only what the mesh made is touched.** An authorization this module creates is named with the
// mesh's identity prefix and its description starts with MARK. One with the same username that
// lacks the mark is somebody else's: it is refused, never adopted, never updated, never deleted.
// Every other authorization, token, user and bucket in the instance is left exactly as it was.
import type { InfluxDBClient, InfluxPermission, LegacyAuthorization } from "./client.js";
/** How a description marks an authorization as the mesh's own work. */
export const MARK = "[mesh]";
/** The prefix the mesh gives every consumer identity (novox/hq ADR 0049). */
const IDENTITY_PREFIX = "mesh_";
/** One consumer, as the harness hands it over. */
export interface ApiGrant {
readonly as: string;
readonly password: string;
readonly values: Readonly<Record<string, unknown>>;
readonly consumer?: string;
}
export type Access = "read" | "write" | "read-write";
/** What a contribution asks for, checked. Refused when it cannot be served as asked. */
export function askedFor(values: Readonly<Record<string, unknown>>): { access: Access; buckets: string[] } {
const access = values.access ?? "read";
if (access !== "read" && access !== "write" && access !== "read-write") {
throw new Error(`contributes an access of ${JSON.stringify(access)} — it is "read", "write" or "read-write"`);
}
const raw = values.buckets ?? [];
if (!Array.isArray(raw) || raw.some((b) => typeof b !== "string" || b.trim() === "")) {
throw new Error(`contributes buckets of ${JSON.stringify(raw)} — a list of bucket names`);
}
const buckets = [...new Set((raw as string[]).map((b) => b.trim()))].sort();
if (access !== "read" && buckets.length === 0) {
throw new Error(`asks to write and names no bucket (\`buckets\`) — a writer names what it writes to`);
}
if (buckets.some((b) => b.startsWith("_"))) {
throw new Error(`names a system bucket (${buckets.filter((b) => b.startsWith("_")).join(", ")}) — those are InfluxDB's own`);
}
return { access: access as Access, buckets };
}
/** The permissions a grant resolves to, given each named bucket's id. */
export function permissionsFor(orgID: string, access: Access, bucketIDs: string[]): InfluxPermission[] {
const actions: ("read" | "write")[] = access === "read-write" ? ["read", "write"] : [access];
const out: InfluxPermission[] = [];
for (const action of actions) {
if (bucketIDs.length === 0) {
out.push({ action, resource: { type: "buckets", orgID } });
continue;
}
for (const id of bucketIDs) out.push({ action, resource: { type: "buckets", orgID, id } });
}
return out;
}
/** A permission as a comparable string: what InfluxDB answers carries names and links besides. */
function key(p: InfluxPermission): string {
return `${p.action}:${p.resource.type}:${p.resource.orgID ?? ""}:${p.resource.id ?? "*"}`;
}
function samePermissions(a: readonly InfluxPermission[], b: readonly InfluxPermission[]): boolean {
const x = a.map(key).sort();
const y = b.map(key).sort();
return x.length === y.length && x.every((v, i) => v === y[i]);
}
export function marked(a: Pick<LegacyAuthorization, "token" | "description">): boolean {
return a.token.startsWith(IDENTITY_PREFIX) && (a.description ?? "").startsWith(MARK);
}
function describe(g: ApiGrant): string {
return `${MARK} made by the mesh for ${g.consumer ? `a module on ${g.consumer}` : "a consumer"} — do not edit; it is reset`;
}
export class ApiGrants {
constructor(private readonly influx: InfluxDBClient, readonly org: string) {}
private async orgID(): Promise<string> {
const id = await this.influx.orgID(this.org);
if (!id) throw new Error(`InfluxDB has no org ${JSON.stringify(this.org)} — the org this module serves must exist`);
return id;
}
/** The ids of the named buckets, creating any that are missing when `create` says so. Undefined
* when one is missing and may not be created (a read-only question). */
private async bucketIDs(orgID: string, names: string[], create: ApiGrant | undefined): Promise<string[] | undefined> {
const ids: string[] = [];
for (const name of names) {
let b = await this.influx.findBucket(orgID, name);
if (!b) {
if (!create) return undefined;
b = await this.influx.createBucket(orgID, name, `${MARK} made by the mesh for ${create.as}; the mesh never deletes it`);
}
ids.push(b.id);
}
return ids.sort();
}
/** Create the consumer's authorization, or bring the mesh's existing one back to what the grant
* says. Idempotent: a second apply of the same grant changes nothing beyond re-asserting the
* password, which InfluxDB can be told but never asked. */
async ensure(g: ApiGrant): Promise<"created" | "updated" | "unchanged"> {
if (!g.as.startsWith(IDENTITY_PREFIX)) {
throw new Error(`${g.as} is not a mesh identity — the mesh names every consumer ${IDENTITY_PREFIX}<node>_<module>`);
}
const { access, buckets } = askedFor(g.values);
const orgID = await this.orgID();
const found = await this.influx.findLegacy(g.as);
if (found && !marked(found)) {
throw new Error(
`InfluxDB already has a v1 authorization ${g.as} the mesh did not make — left alone; ` +
`delete it if the mesh should own that name`);
}
const want = permissionsFor(orgID, access, (await this.bucketIDs(orgID, buckets, g))!);
if (found && found.orgID === orgID && samePermissions(found.permissions, want)) {
// Only what differs is written. The password cannot be read back, so it is tried instead.
let changed = false;
if (found.status === "inactive") {
await this.influx.updateLegacy(found.id, { status: "active" });
changed = true;
}
if (!(await this.influx.legacySignsIn(g.as, g.password))) {
await this.influx.setLegacyPassword(found.id, g.password);
changed = true;
}
return changed ? "updated" : "unchanged";
}
// InfluxDB cannot change an authorization's permissions in place, so the mesh's own is made
// again. Only ever one the mesh made: a foreign one was refused above.
if (found) await this.influx.deleteLegacy(found.id);
const made = await this.influx.createLegacy({
token: g.as, orgID, status: "active", description: describe(g), permissions: want,
});
await this.influx.setLegacyPassword(made.id, g.password);
return found ? "updated" : "created";
}
/** Whether InfluxDB still holds this consumer's authorization exactly as the grant says: present,
* the mesh's, active, allowed what was asked and nothing more, and signing in with the mesh's
* password. Reads only — a missing bucket is "not held", never created here. */
async holds(g: ApiGrant): Promise<boolean> {
const { access, buckets } = askedFor(g.values);
const orgID = await this.influx.orgID(this.org);
if (!orgID) return false;
const found = await this.influx.findLegacy(g.as);
if (!found || !marked(found) || found.status === "inactive" || found.orgID !== orgID) return false;
const ids = await this.bucketIDs(orgID, buckets, undefined);
if (!ids || !samePermissions(found.permissions, permissionsFor(orgID, access, ids))) return false;
return this.influx.legacySignsIn(g.as, g.password);
}
/** Withdraw a consumer's authorization — only one the mesh made. Its buckets and their data stay. */
async remove(as: string): Promise<"removed" | "absent" | "not ours"> {
const found = await this.influx.findLegacy(as);
if (!found) return "absent";
if (!marked(found)) return "not ours";
await this.influx.deleteLegacy(found.id);
return "removed";
}
}
+63 -29
View File
@@ -1,6 +1,12 @@
{
"module": "influxdb",
"version": "1",
"provides": [
{
"name": "influxdb-api",
"scope": "mesh"
}
],
"capabilities": [
"container-runtime"
],
@@ -9,12 +15,27 @@
},
"listens": [
{
"name": "api",
"port": 8086,
"protocol": "tcp",
"from": "mesh",
"why": "queries and writes, over http"
"why": "queries, writes and the web UI, over http; consumers granted influxdb-api sign in with the mesh's credential, and a name is a route grant"
}
],
"serves": {
"influxdb-api": {
"scheme": "http",
"port": 8086,
"org": "mesh",
"bucket": "default"
}
},
"receives": {
"influxdb-api": "${dir:grants}/mesh.json"
},
"grants": {
"influxdb-api": "${dir:grants}"
},
"resources": [
{
"id": "mesh-state",
@@ -25,46 +46,50 @@
{
"id": "state",
"type": "directory",
"path": "/var/lib/influxdb-module",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/influxdb-module/server.env",
"mode": "0600",
"content": "DOCKER_INFLUXDB_INIT_MODE=setup\nDOCKER_INFLUXDB_INIT_USERNAME=admin\nDOCKER_INFLUXDB_INIT_PASSWORD=${secret:admin}\nDOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${secret:admin-token}\nDOCKER_INFLUXDB_INIT_ORG=mesh\nDOCKER_INFLUXDB_INIT_BUCKET=default\n"
"mode": "0700",
"place": "."
},
{
"id": "data",
"type": "directory",
"path": "/services/influxdb/data",
"mode": "0700",
"owner": "1000:1000"
},
{
"id": "config",
"type": "directory",
"path": "/services/influxdb/config",
"mode": "0700",
"owner": "1000:1000"
},
{
"id": "grants",
"type": "directory",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "${dir:state}/server.env",
"mode": "0600",
"content": "DOCKER_INFLUXDB_INIT_MODE=setup\nDOCKER_INFLUXDB_INIT_USERNAME=admin\nDOCKER_INFLUXDB_INIT_PASSWORD_FILE=/run/secrets/admin\nDOCKER_INFLUXDB_INIT_ADMIN_TOKEN_FILE=/run/secrets/admin-token\nDOCKER_INFLUXDB_INIT_ORG=mesh\nDOCKER_INFLUXDB_INIT_BUCKET=default\n"
},
{
"id": "server",
"type": "container",
"name": "influxdb",
"image": "influxdb@sha256:f75e48af0598e8aec7986e991a848d19a119101a7d563a2e5db1dfaac9c45daa",
"env-file": [
"/var/lib/influxdb-module/server.env"
"${dir:state}/server.env"
],
"ports": [
"8086"
],
"volumes": [
"/services/influxdb/data:/var/lib/influxdb2",
"/services/influxdb/config:/etc/influxdb2"
],
"secrets-in-environment": "the image honours DOCKER_INFLUXDB_INIT_PASSWORD_FILE and _ADMIN_TOKEN_FILE; convertible, awaiting a bed that proves it"
"${dir:data}:/var/lib/influxdb2",
"${dir:config}:/etc/influxdb2",
"${dir:state}/admin.secret:/run/secrets/admin:ro",
"${dir:state}/admin-token.secret:/run/secrets/admin-token:ro"
]
},
{
"id": "runtime-config",
@@ -82,13 +107,15 @@
"volumes": [
"/var/lib/mesh/influxdb/broker:/run/secrets/broker:ro",
"/var/lib/mesh/influxdb/config.json:/run/config/config.json:ro",
"/services/influxdb/config:/var/lib/influxdb/config:ro"
"${dir:state}/admin-token.secret:/run/secrets/admin-token:ro",
"${dir:grants}:${dir:grants}:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_INFLUXDB_URL": "http://127.0.0.1:8086",
"MESH_INFLUXDB_URL": "http://127.0.0.1:${port:8086}",
"MESH_INFLUXDB_CONFIG_FILE": "/run/config/config.json",
"MESH_INFLUXDB_CONFIG_DIR": "/var/lib/influxdb/config"
"MESH_INFLUXDB_TOKEN_FILE": "/run/secrets/admin-token",
"MESH_RECEIVES": "${dir:grants}/mesh.json"
},
"restart-on": [
"runtime-config"
@@ -96,6 +123,22 @@
"artifact": "runtime"
}
],
"requires": [
"route",
"secret"
],
"contributes": {
"route": {
"label": "influxdb",
"endpoint": "api"
}
},
"secrets": {
"secret": {
"admin": "${dir:state}/admin.secret",
"admin-token": "${dir:state}/admin-token.secret"
}
},
"build": {
"on": [
{
@@ -116,14 +159,5 @@
"from": "Dockerfile"
}
]
},
"requires": [
"secret"
],
"secrets": {
"secret": {
"admin": "/var/lib/influxdb-module/admin.secret",
"admin-token": "/var/lib/influxdb-module/admin-token.secret"
}
}
}
+6 -1
View File
@@ -1,9 +1,14 @@
{
"name": "@novox/module-influxdb",
"version": "0.1.0",
"description": "influxdb — time-series database. Its API client and tools live here (novox/hq ADR 0039).",
"description": "influxdb — time-series database; provides the mesh influxdb-api interface. Its API client, provisioner and tools live here (novox/hq ADR 0039).",
"type": "module",
"private": true,
"scripts": {
"build": "tsc client.ts grants.ts provisioner/index.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
"typecheck": "tsc -p tsconfig.json",
"test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.0"
},
+54
View File
@@ -0,0 +1,54 @@
// influxdb's provisioner — the adapter that makes influxdb a provider of the mesh `influxdb-api`
// interface. The reconcile loop, the contributions file and reading the mesh's minted secret are the
// sdk harness's; this writes only the per-service half: how InfluxDB creates, checks and removes a
// consumer's credential (novox/hq ADR 0039/0040/0048). What that credential is, and why it is a v1
// authorization, is in ../grants.ts.
//
// The `influxdb-api` interface: a consumer reaches `${bound:influxdb-api:scheme}://…:at:…:port`,
// signs in as `${bound:influxdb-api:as}` with the password the mesh minted for the pair, and reads
// or writes the org's buckets as databases of the same name — `${bound:influxdb-api:bucket}` being
// the one this instance serves by default. The org and the default bucket are the assignment's
// settings, which reach both what is served and this module's config.json, so the org a consumer is
// told and the org its credential is made in cannot disagree.
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
import { InfluxDBClient } from "../client.js";
import { ApiGrants } from "../grants.js";
let grants: ApiGrants | undefined;
try {
const influx = InfluxDBClient.fromEnv();
grants = new ApiGrants(influx, influx.org);
} catch (err) {
// No admin token: nothing can be provisioned, and the tools loaded beside this must still serve.
console.error(`[provisioner:influxdb-api] not started: ${err instanceof Error ? err.message : err}`);
}
if (grants) serve(grants);
function serve(grants: ApiGrants): void {
runProvisioner("influxdb-api", {
async create(p: Provision): Promise<void> {
const done = await grants.ensure(p);
if (done !== "unchanged") {
console.log(`[provisioner:influxdb-api] ${done} v1 authorization ${p.as} in org ${grants.org}`);
}
},
async remove(p: { as: string }): Promise<void> {
const done = await grants.remove(p.as);
if (done === "not ours") {
console.error(`[provisioner:influxdb-api] ${p.as}: an authorization of that name exists that the mesh did not make — left alone`);
} else if (done === "removed") {
console.log(`[provisioner:influxdb-api] removed v1 authorization ${p.as}; its buckets and their data stay`);
}
},
// Asked every minute by the harness: whether InfluxDB still holds this consumer's authorization
// exactly as the mesh gave it, so one deleted, disabled or re-passworded behind the mesh's back is
// made whole again (hq issue 120).
async holds(p: Provision): Promise<boolean> {
return grants.holds(p);
},
});
}
+246
View File
@@ -0,0 +1,246 @@
// What holds influxdb to the `influxdb-api` provision (grants.ts): one v1 authorization per consumer,
// under the username and password the mesh gave, allowed only what the consumer contributed; made
// once and brought back on every apply; buckets created when missing and never deleted; and an
// authorization the mesh did not make — same name or not — never adopted, changed or deleted.
//
// InfluxDB is a fake: the routes the module touches, answering with the status codes and shapes
// InfluxDB 2.9 gives (a filter matching nothing is a 404, a password outside 8–72 characters a 400,
// an inactive authorization or a wrong password a 401 on /query). Run against the compiled module
// (npm test builds first), the way the runtime loads it.
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { InfluxDBClient } from "../dist/client.js";
import { ApiGrants, MARK, askedFor, marked } from "../dist/grants.js";
type Rec = Record<string, any>;
const ADMIN = "operator-token";
const orgs = new Map<string, string>([["zurag", "org1"]]);
let buckets: Rec[] = [];
let auths: Rec[] = [];
let calls: string[] = [];
let seq = 0;
function body(req: IncomingMessage): Promise<any> {
return new Promise((resolve) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => resolve(raw ? JSON.parse(raw) : undefined));
});
}
function send(res: ServerResponse, status: number, value?: unknown): void {
res.writeHead(status, { "Content-Type": "application/json" });
res.end(value === undefined ? "" : JSON.stringify(value));
}
const server = createServer(async (req, res) => {
const url = new URL(req.url!, "http://fake");
const p = url.pathname;
calls.push(`${req.method} ${p}`);
if (p === "/query") {
const basic = (req.headers.authorization ?? "").replace(/^Basic /, "");
const [u, pw] = Buffer.from(basic, "base64").toString().split(":");
const a = auths.find((x) => x.token === u);
if (!a || a.status !== "active" || a.password === undefined || a.password !== pw) {
return send(res, 401, { code: "unauthorized", message: "Unauthorized" });
}
return send(res, 200, { results: [{ statement_id: 0 }] });
}
if (req.headers.authorization !== `Token ${ADMIN}`) return send(res, 401, { code: "unauthorized" });
if (p === "/api/v2/orgs") {
const id = orgs.get(url.searchParams.get("org") ?? "");
if (!id) return send(res, 404, { code: "not found", message: "organization name not found" });
return send(res, 200, { orgs: [{ id, name: url.searchParams.get("org") }] });
}
if (p === "/api/v2/buckets" && req.method === "GET") {
const found = buckets.filter((b) => b.orgID === url.searchParams.get("orgID") && b.name === url.searchParams.get("name"));
if (found.length === 0) return send(res, 404, { code: "not found", message: "bucket not found" });
return send(res, 200, { buckets: found });
}
if (p === "/api/v2/buckets" && req.method === "POST") {
const b = { ...(await body(req)), id: `b${++seq}` };
buckets.push(b);
return send(res, 201, b);
}
if (p === "/private/legacy/authorizations" && req.method === "GET") {
const found = auths.filter((a) => a.token === url.searchParams.get("token"));
if (found.length === 0) return send(res, 404, { code: "not found", message: "authorization not found" });
// Never answers with the password: InfluxDB keeps only its hash.
return send(res, 200, { authorizations: found.map(({ password, ...a }) => ({ ...a, links: {} })) });
}
if (p === "/private/legacy/authorizations" && req.method === "POST") {
const a = await body(req);
if (auths.some((x) => x.token === a.token)) return send(res, 409, { code: "conflict", message: "token already exists" });
const made = { ...a, id: `a${++seq}`, status: a.status ?? "active" };
auths.push(made);
return send(res, 201, made);
}
const m = /^\/private\/legacy\/authorizations\/([^/]+)(\/password)?$/.exec(p);
const a = m && auths.find((x) => x.id === m[1]);
if (!a) return send(res, 404, { code: "not found" });
if (m![2] && req.method === "POST") {
const { password } = await body(req);
if (typeof password !== "string" || password.length < 8 || password.length > 72) {
return send(res, 400, { code: "invalid", message: "passwords must be between 8 and 72 characters long" });
}
a.password = password;
return send(res, 204);
}
if (req.method === "PATCH") {
Object.assign(a, await body(req));
return send(res, 200, a);
}
if (req.method === "DELETE") {
auths = auths.filter((x) => x !== a);
return send(res, 204);
}
send(res, 405);
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
after(() => server.close());
const port = (server.address() as { port: number }).port;
const grants = new ApiGrants(new InfluxDBClient(`http://127.0.0.1:${port}`, ADMIN, "zurag"), "zurag");
const PW = "mesh-minted-password-of-forty-characters";
/** Grafana on ace, as the mesh hands it to the provisioner. */
function grafana(password = PW, values: Record<string, unknown> = { access: "read" }) {
return { as: "mesh_ace_grafana", password, consumer: "ace", values };
}
/** Node-RED on ace: writes one bucket. */
function nodered(password = PW, values: Record<string, unknown> = { access: "write", buckets: ["zurag"] }) {
return { as: "mesh_ace_nodered", password, consumer: "ace", values };
}
function only(token: string): Rec {
const found = auths.filter((a) => a.token === token);
assert.equal(found.length, 1, `exactly one authorization ${token}, found ${found.length}`);
return found[0];
}
function perms(a: Rec): string[] {
return a.permissions.map((p: Rec) => `${p.action}:${p.resource.type}:${p.resource.id ?? "*"}`).sort();
}
beforeEach(() => {
buckets = [{ id: "zb", orgID: "org1", name: "zurag" }];
auths = [];
calls = [];
});
test("what a contribution may ask for, and what is refused", () => {
assert.deepEqual(askedFor({}), { access: "read", buckets: [] });
assert.deepEqual(askedFor({ access: "read-write", buckets: ["b", "a", "a"] }), { access: "read-write", buckets: ["a", "b"] });
assert.throws(() => askedFor({ access: "admin" }), /access/);
assert.throws(() => askedFor({ access: "write" }), /names no bucket/);
assert.throws(() => askedFor({ buckets: "zurag" }), /list of bucket names/);
assert.throws(() => askedFor({ access: "write", buckets: ["_monitoring"] }), /system bucket/);
});
test("a reader is given one authorization, reading every bucket of the org, under the mesh's password", async () => {
assert.equal(await grants.ensure(grafana()), "created");
const a = only("mesh_ace_grafana");
assert.equal(a.orgID, "org1");
assert.equal(a.status, "active");
assert.ok(a.description.startsWith(MARK));
assert.deepEqual(perms(a), ["read:buckets:*"]);
assert.equal(a.password, PW);
assert.equal(await grants.holds(grafana()), true);
});
test("a writer is allowed its own buckets only, and a missing one is made — never deleted", async () => {
assert.equal(await grants.ensure(nodered(PW, { access: "write", buckets: ["zurag", "printer"] })), "created");
const made = buckets.find((b) => b.name === "printer");
assert.ok(made, "the missing bucket was created");
assert.deepEqual(made!.retentionRules, [], "kept for ever: retention is the operator's choice");
assert.deepEqual(perms(only("mesh_ace_nodered")), [`write:buckets:${made!.id}`, "write:buckets:zb"]);
assert.equal(await grants.remove("mesh_ace_nodered"), "removed");
assert.equal(buckets.length, 2, "withdrawing the consumer leaves every bucket and its data");
});
test("applying the same grant again writes nothing", async () => {
await grants.ensure(grafana());
calls = [];
assert.equal(await grants.ensure(grafana()), "unchanged");
assert.ok(calls.every((c) => c.startsWith("GET")), `only reads: ${calls.join(", ")}`);
only("mesh_ace_grafana");
});
test("a rotated password is set in place; a changed access remakes only the mesh's own", async () => {
await grants.ensure(nodered());
const id = only("mesh_ace_nodered").id;
assert.equal(await grants.holds(nodered("rotated-password-0123456789")), false);
assert.equal(await grants.ensure(nodered("rotated-password-0123456789")), "updated");
assert.equal(only("mesh_ace_nodered").id, id, "updated, not replaced");
assert.equal(await grants.holds(nodered("rotated-password-0123456789")), true);
await grants.ensure(nodered(PW, { access: "read-write", buckets: ["zurag"] }));
assert.deepEqual(perms(only("mesh_ace_nodered")), ["read:buckets:zb", "write:buckets:zb"]);
assert.equal(await grants.holds(nodered(PW, { access: "read-write", buckets: ["zurag"] })), true);
});
test("an authorization disabled, re-passworded or deleted behind the mesh's back is not held, and is made whole", async () => {
await grants.ensure(grafana());
only("mesh_ace_grafana").status = "inactive";
assert.equal(await grants.holds(grafana()), false);
assert.equal(await grants.ensure(grafana()), "updated");
assert.equal(await grants.holds(grafana()), true);
only("mesh_ace_grafana").password = "somebody-else-set-this";
assert.equal(await grants.holds(grafana()), false);
await grants.ensure(grafana());
assert.equal(await grants.holds(grafana()), true);
auths = [];
assert.equal(await grants.holds(grafana()), false);
assert.equal(await grants.ensure(grafana()), "created");
});
test("holds only reads, and a bucket gone missing is not held rather than made", async () => {
await grants.ensure(nodered());
buckets = [];
calls = [];
assert.equal(await grants.holds(nodered()), false);
assert.ok(calls.every((c) => c.startsWith("GET")), `only reads: ${calls.join(", ")}`);
assert.equal(buckets.length, 0);
});
test("an authorization of the same name the mesh did not make is refused, and left exactly as it was", async () => {
auths = [{ id: "theirs", token: "mesh_ace_grafana", orgID: "org1", status: "active", description: "hand-made",
permissions: [{ action: "write", resource: { type: "buckets", orgID: "org1" } }], password: "their-password" }];
const before = JSON.stringify(auths);
await assert.rejects(grants.ensure(grafana()), /did not make/);
assert.equal(JSON.stringify(auths), before);
assert.ok(calls.every((c) => c.startsWith("GET")), `only reads: ${calls.join(", ")}`);
assert.equal(await grants.holds(grafana()), false);
assert.equal(await grants.remove("mesh_ace_grafana"), "not ours");
assert.equal(auths.length, 1, "never deleted");
});
test("the predecessor's own v1 users and tokens are never touched", async () => {
auths = [{ id: "hal", token: "grafana", orgID: "org1", status: "active", description: "",
permissions: [{ action: "read", resource: { type: "buckets", orgID: "org1" } }], password: "old-password" }];
await grants.ensure(grafana());
assert.equal(auths.find((a) => a.id === "hal")!.password, "old-password");
assert.equal(await grants.remove("grafana"), "not ours");
assert.equal(marked({ token: "grafana", description: `${MARK} x` }), false, "the mark needs the mesh's name too");
});
test("an org the instance does not have, or a non-mesh name, makes nothing", async () => {
const elsewhere = new ApiGrants(new InfluxDBClient(`http://127.0.0.1:${port}`, ADMIN, "nope"), "nope");
await assert.rejects(elsewhere.ensure(grafana()), /no org "nope"/);
await assert.rejects(grants.ensure({ ...grafana(), as: "grafana" }), /not a mesh identity/);
assert.equal(auths.length, 0);
});
test("a withdrawn consumer's authorization is removed, and an absent one is not an error", async () => {
await grants.ensure(grafana());
assert.equal(await grants.remove("mesh_ace_grafana"), "removed");
assert.equal(auths.length, 0);
assert.equal(await grants.remove("mesh_ace_grafana"), "absent");
});
+6 -1
View File
@@ -8,5 +8,10 @@
"skipLibCheck": true,
"noEmit": true
},
"include": ["client.ts", "tools/index.ts"]
"include": [
"client.ts",
"grants.ts",
"provisioner/index.ts",
"tools/index.ts"
]
}
+4 -2
View File
@@ -17,11 +17,11 @@
"route": {
"site": {
"label": "invoicing",
"port": 80
"endpoint": "web"
},
"api": {
"label": "invoicing-api",
"port": 9000
"endpoint": "api"
}
}
},
@@ -36,12 +36,14 @@
},
"listens": [
{
"name": "web",
"port": 80,
"protocol": "tcp",
"from": "mesh",
"why": "the invoicing web frontend; a public name is a route grant later"
},
{
"name": "api",
"port": 9000,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -6,6 +6,7 @@
],
"listens": [
{
"name": "web",
"port": 9117,
"protocol": "tcp",
"from": "mesh",
@@ -82,7 +83,7 @@
"contributes": {
"route": {
"label": "indexers",
"port": 9117
"endpoint": "web"
}
},
"binds": {
+2 -2
View File
@@ -17,7 +17,7 @@ FROM ${BUILD_BASE} AS build
# resolved away.
WORKDIR /app/modules/keycloak
COPY . .
RUN node /app/node_modules/typescript/bin/tsc client.ts index.ts tools/index.ts \
RUN node /app/node_modules/typescript/bin/tsc client.ts oidc.ts index.ts provisioner/index.ts tools/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
FROM ${RUNTIME_BASE}
@@ -27,4 +27,4 @@ COPY --from=build /app/modules/keycloak/dist /app/modules/keycloak/dist
# the convention novox/hq issues 060/061 settled. A container that instead ran only its
# provisioner (`run`) served no tools and emitted no events; a container that named no command
# ran no provisioner at all.
ENV MESH_TOOL_MODULES=/app/modules/keycloak/dist/index.js,/app/modules/keycloak/dist/tools/index.js
ENV MESH_TOOL_MODULES=/app/modules/keycloak/dist/index.js,/app/modules/keycloak/dist/tools/index.js,/app/modules/keycloak/dist/provisioner/index.js
+90 -2
View File
@@ -12,6 +12,45 @@ function meshConfig(file?: string): Record<string, string> {
catch { return {}; }
}
/** A secret file's value, trailing newline trimmed; undefined when unset or unreadable. */
function secretFile(file?: string): string | undefined {
if (!file) return undefined;
try { return readFileSync(file, "utf8").replace(/\n$/, "") || undefined; }
catch { return undefined; }
}
/** A client as the admin API represents it — only the fields this module reads or writes are typed;
* the rest travel through untouched, so an update never drops what somebody else set. */
export interface ClientRepresentation {
id?: string;
clientId: string;
name?: string;
enabled?: boolean;
protocol?: string;
publicClient?: boolean;
clientAuthenticatorType?: string;
secret?: string;
rootUrl?: string;
baseUrl?: string;
redirectUris?: string[];
webOrigins?: string[];
standardFlowEnabled?: boolean;
implicitFlowEnabled?: boolean;
directAccessGrantsEnabled?: boolean;
serviceAccountsEnabled?: boolean;
attributes?: Record<string, string>;
protocolMappers?: ProtocolMapperRepresentation[];
[other: string]: unknown;
}
export interface ProtocolMapperRepresentation {
id?: string;
name: string;
protocol: string;
protocolMapper: string;
config: Record<string, string>;
}
export class KeycloakClient {
readonly baseUrl: string;
readonly defaultRealm: string;
@@ -40,8 +79,13 @@ export class KeycloakClient {
const cfg = meshConfig(env.MESH_KEYCLOAK_CONFIG_FILE);
const url = cfg.url ?? env.MESH_KEYCLOAK_URL ?? `http://127.0.0.1:${env.KEYCLOAK_PORT ?? "8080"}`;
const adminUser = cfg.user ?? env.MESH_KEYCLOAK_ADMIN ?? env.KEYCLOAK_ADMIN ?? "admin";
const adminPass = cfg.password ?? env.MESH_KEYCLOAK_PASSWORD ?? env.KEYCLOAK_ADMIN_PASSWORD;
if (!adminPass) throw new Error("no Keycloak admin password — set MESH_KEYCLOAK_PASSWORD");
// The admin password reaches the runtime as a file (novox/hq ADR 0086): the module's own `admin`
// secret, mounted read-only. The environment forms stay for a co-located server that has them.
const adminPass = cfg.password ?? secretFile(env.MESH_KEYCLOAK_PASSWORD_FILE)
?? env.MESH_KEYCLOAK_PASSWORD ?? env.KEYCLOAK_ADMIN_PASSWORD;
if (!adminPass) {
throw new Error("no Keycloak admin password — set MESH_KEYCLOAK_PASSWORD_FILE (or MESH_KEYCLOAK_PASSWORD)");
}
const realm = cfg.realm ?? env.MESH_KEYCLOAK_REALM ?? "master";
return new KeycloakClient(url, adminUser, adminPass, realm);
}
@@ -159,6 +203,50 @@ export class KeycloakClient {
return client.id as string;
}
/** The one client with exactly this clientId, or undefined. The admin API's `clientId` filter is an
* exact match unless `search=true` is asked for. */
async findClient(realm: string, clientId: string): Promise<ClientRepresentation | undefined> {
const found = await this.request<ClientRepresentation[]>(
`/${realm}/clients?clientId=${encodeURIComponent(clientId)}`);
return found.find((c) => c.clientId === clientId);
}
async createClientFrom(realm: string, rep: ClientRepresentation): Promise<void> {
await this.request(`/${realm}/clients`, { method: "POST", body: JSON.stringify(rep) });
}
/** Replace a client's representation, addressed by its internal id. */
async updateClient(realm: string, id: string, rep: ClientRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}`, { method: "PUT", body: JSON.stringify(rep) });
}
async deleteClientById(realm: string, id: string): Promise<void> {
await this.request(`/${realm}/clients/${id}`, { method: "DELETE" });
}
async clientSecretById(realm: string, id: string): Promise<string | undefined> {
const result = await this.request<{ value?: string }>(`/${realm}/clients/${id}/client-secret`);
return result.value;
}
async listClientMappers(realm: string, id: string): Promise<ProtocolMapperRepresentation[]> {
return this.request(`/${realm}/clients/${id}/protocol-mappers/models`);
}
async addClientMapper(realm: string, id: string, mapper: ProtocolMapperRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}/protocol-mappers/models`, {
method: "POST",
body: JSON.stringify(mapper),
});
}
async updateClientMapper(realm: string, id: string, mapper: ProtocolMapperRepresentation): Promise<void> {
await this.request(`/${realm}/clients/${id}/protocol-mappers/models/${mapper.id}`, {
method: "PUT",
body: JSON.stringify(mapper),
});
}
async deleteClient(realm: string, clientId: string): Promise<void> {
await this.request(`/${realm}/clients/${await this.resolveClientId(realm, clientId)}`, { method: "DELETE" });
}
+46 -6
View File
@@ -1,6 +1,12 @@
{
"module": "keycloak",
"version": "1",
"provides": [
{
"name": "oidc-client",
"scope": "mesh"
}
],
"requires": [
"postgres-database",
"route"
@@ -11,7 +17,7 @@
},
"route": {
"label": "keycloak",
"port": 8080
"endpoint": "web"
}
},
"binds": {
@@ -34,12 +40,26 @@
],
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
"why": "anything the mesh runs that authenticates a person"
}
],
"serves": {
"oidc-client": {
"authorization-path": "/protocol/openid-connect/auth",
"token-path": "/protocol/openid-connect/token",
"userinfo-path": "/protocol/openid-connect/userinfo"
}
},
"receives": {
"oidc-client": "/var/lib/keycloak/grants/mesh.json"
},
"grants": {
"oidc-client": "/var/lib/keycloak/grants"
},
"own-secrets": {
"admin": "/var/lib/keycloak/admin.secret",
"broker": "/var/lib/mesh/keycloak/broker"
@@ -57,6 +77,12 @@
"path": "/var/lib/keycloak",
"mode": "0700"
},
{
"id": "grants",
"type": "directory",
"path": "/var/lib/keycloak/grants",
"mode": "0700"
},
{
"id": "admin-env",
"type": "file",
@@ -76,6 +102,13 @@
"type": "network",
"name": "keycloak"
},
{
"id": "hostname",
"type": "file",
"path": "/var/lib/keycloak/hostname.env",
"mode": "0644",
"content": "KC_HOSTNAME=https://${bound:route:name}\n"
},
{
"id": "server",
"type": "container",
@@ -89,17 +122,20 @@
"KC_DB": "postgres",
"KC_HTTP_ENABLED": "true",
"KC_HEALTH_ENABLED": "true",
"KC_HOSTNAME": "https://keycloak.novox.be",
"KC_PROXY_HEADERS": "xforwarded"
},
"env-file": [
"/var/lib/keycloak/admin.env",
"/var/lib/keycloak/database.env"
"/var/lib/keycloak/database.env",
"/var/lib/keycloak/hostname.env"
],
"ports": [
"8080"
],
"secrets-in-environment": "KC_DB_PASSWORD is convertible through a generated keycloak.conf (db-password=); KEYCLOAK_ADMIN_PASSWORD is env-only before Keycloak 26; not yet converted"
"secrets-in-environment": "KC_DB_PASSWORD is convertible through a generated keycloak.conf (db-password=); KEYCLOAK_ADMIN_PASSWORD is env-only before Keycloak 26; not yet converted",
"restart-on": [
"hostname"
]
},
{
"id": "runtime-config",
@@ -116,12 +152,16 @@
"network": "host",
"volumes": [
"/var/lib/mesh/keycloak/broker:/run/secrets/broker:ro",
"/var/lib/mesh/keycloak/config.json:/run/config/config.json:ro"
"/var/lib/mesh/keycloak/config.json:/run/config/config.json:ro",
"/var/lib/keycloak/admin.secret:/run/secrets/admin:ro",
"/var/lib/keycloak/grants:/var/lib/keycloak/grants:ro"
],
"env": {
"MESH_BROKER_FILE": "/run/secrets/broker",
"MESH_KEYCLOAK_URL": "http://127.0.0.1:${port:8080}",
"MESH_KEYCLOAK_CONFIG_FILE": "/run/config/config.json"
"MESH_KEYCLOAK_CONFIG_FILE": "/run/config/config.json",
"MESH_KEYCLOAK_PASSWORD_FILE": "/run/secrets/admin",
"MESH_RECEIVES": "/var/lib/keycloak/grants/mesh.json"
},
"restart-on": [
"runtime-config"
+185
View File
@@ -0,0 +1,185 @@
// What the `oidc-client` provision means in Keycloak: one confidential OpenID Connect client per
// consumer, in the realm this module serves, under the name and secret the mesh gave both ends.
// The provisioner (provisioner/index.ts) is the sdk harness calling these; they are here, apart from
// it, so they can be exercised against a fake admin API without a broker or a contributions file.
//
// **The client id and the secret are the mesh's, not Keycloak's (novox/hq ADR 0048).** The mesh
// derives the consumer's identity (`as`, e.g. `mesh_ace_grafana`) and hands it to both ends — the
// consumer names it as its client id through `${bound:oidc-client:as}` — and mints the secret, which
// this sets as the client's secret. Keycloak generates neither.
//
// **Where the consumer's browser comes back to is the consumer's to say.** Its contribution carries
// `callback` (a path, e.g. `/login/generic_oauth`) and the `label`/`endpoint` of the endpoint it is
// reached on; the mesh composes that endpoint's names into `name` (public) and `internal-name`
// (private network) exactly as it does for a route (novox/hq ADR 0056, 0138), so the redirect URI
// registered here is built from the same names the proxy serves the consumer under.
//
// **Only what the mesh made is touched.** A client this module creates carries the attribute
// `mesh.provisioned=true`, and its id starts with the mesh's own prefix. A client with the same id
// that lacks the mark is somebody else's: it is refused, never adopted, never updated, never deleted.
import type { ClientRepresentation, KeycloakClient, ProtocolMapperRepresentation } from "./client.js";
/** The attribute marking a client as the mesh's own work. */
export const MARK = "mesh.provisioned";
/** The mapper every mesh client carries: realm roles as a flat `roles` claim in the id token, the
* access token and userinfo — what a consumer maps its own roles from (grafana's role path reads
* `roles[*]`), and what the predecessor added to its hand-made clients by hand. */
export const ROLES_MAPPER: ProtocolMapperRepresentation = {
name: "realm roles",
protocol: "openid-connect",
protocolMapper: "oidc-usermodel-realm-role-mapper",
config: {
"claim.name": "roles",
"jsonType.label": "String",
multivalued: "true",
"id.token.claim": "true",
"access.token.claim": "true",
"userinfo.token.claim": "true",
},
};
/** One consumer, as the harness hands it over. */
export interface OidcGrant {
readonly as: string;
readonly password: string;
readonly values: Readonly<Record<string, unknown>>;
readonly consumer?: string;
}
/** The realm named by an issuer URL — `https://id.example/realms/Novox` is realm `Novox`. The issuer is
* the one value an assignment sets (it is also what consumers are served), so the realm is read
* out of it rather than set a second time where the two could disagree. */
export function realmOf(issuer: string): string {
let path: string;
try {
path = new URL(issuer).pathname;
} catch {
throw new Error(`the issuer ${JSON.stringify(issuer)} is not a URL`);
}
const m = /\/realms\/([^/]+)\/?$/.exec(path);
if (!m) throw new Error(`the issuer ${JSON.stringify(issuer)} does not end in /realms/<realm>`);
return decodeURIComponent(m[1]);
}
/** The redirect URIs a consumer's contribution asks for: its callback under each name the mesh
* composed for its endpoint. Refused when there is nothing to register — a client that accepts no
* redirect is a client nobody can log in through, and one that accepts any is worse. */
export function redirectsOf(values: Readonly<Record<string, unknown>>): { root: string; redirects: string[] } {
const callback = values.callback;
if (typeof callback !== "string" || !callback.startsWith("/")) {
throw new Error(`contributes no callback path (\`callback\`, starting with "/"): ${JSON.stringify(callback)}`);
}
const names: string[] = [];
for (const key of ["name", "internal-name"]) {
const n = values[key];
if (typeof n === "string" && n.trim() !== "" && !names.includes(n.trim())) names.push(n.trim());
}
if (names.length === 0) {
throw new Error("has no name the mesh composed (`name` / `internal-name`) — contribute a `label` and the `endpoint` it is reached on");
}
return { root: `https://${names[0]}`, redirects: names.map((n) => `https://${n}${callback}`) };
}
/** The fields the mesh owns on a client it made. Everything else on the client is left as found. */
function wanted(g: OidcGrant): ClientRepresentation {
const { root, redirects } = redirectsOf(g.values);
return {
clientId: g.as,
name: g.as,
description: `made by the mesh for ${g.consumer ? `a module on ${g.consumer}` : "a consumer"} — do not edit; it is reset`,
enabled: true,
protocol: "openid-connect",
publicClient: false,
clientAuthenticatorType: "client-secret",
secret: g.password,
rootUrl: root,
baseUrl: root,
redirectUris: redirects,
standardFlowEnabled: true,
implicitFlowEnabled: false,
directAccessGrantsEnabled: false,
serviceAccountsEnabled: false,
};
}
function sameSet(a: readonly string[] | undefined, b: readonly string[]): boolean {
const x = [...(a ?? [])].sort();
const y = [...b].sort();
return x.length === y.length && x.every((v, i) => v === y[i]);
}
function marked(c: ClientRepresentation): boolean {
return c.attributes?.[MARK] === "true";
}
export class OidcClients {
constructor(private readonly kc: KeycloakClient, readonly realm: string) {}
/** Create the consumer's client, or bring the mesh's existing one back to what the grant says.
* Returns whether it was newly created. Idempotent: applying the same grant twice changes nothing
* the second time beyond re-asserting it. */
async ensure(g: OidcGrant): Promise<"created" | "updated"> {
const want = wanted(g);
const found = await this.kc.findClient(this.realm, g.as);
if (found && !marked(found)) {
throw new Error(
`realm ${this.realm} already has a client ${g.as} the mesh did not make — left alone; ` +
`delete or rename it if the mesh should own that id`);
}
if (!found) {
await this.kc.createClientFrom(this.realm, {
...want,
attributes: { [MARK]: "true" },
protocolMappers: [ROLES_MAPPER],
});
return "created";
}
// Overlay what the mesh owns on what is there, so a field Keycloak added or an operator set on a
// field the mesh does not own survives the update.
await this.kc.updateClient(this.realm, found.id!, {
...found,
...want,
attributes: { ...(found.attributes ?? {}), [MARK]: "true" },
});
await this.ensureMapper(found.id!);
return "updated";
}
private async ensureMapper(id: string): Promise<void> {
const mappers = await this.kc.listClientMappers(this.realm, id);
const have = mappers.find((m) => m.name === ROLES_MAPPER.name);
if (!have) {
await this.kc.addClientMapper(this.realm, id, ROLES_MAPPER);
return;
}
const drifted =
have.protocolMapper !== ROLES_MAPPER.protocolMapper ||
Object.entries(ROLES_MAPPER.config).some(([k, v]) => have.config?.[k] !== v);
if (drifted) {
await this.kc.updateClientMapper(this.realm, id, { ...ROLES_MAPPER, id: have.id });
}
}
/** Whether Keycloak still holds this consumer's client exactly as the grant says: present, the
* mesh's, enabled, confidential, with the mesh's secret and the redirects asked for. Reads only. */
async holds(g: OidcGrant): Promise<boolean> {
const want = wanted(g);
const found = await this.kc.findClient(this.realm, g.as);
if (!found || !marked(found) || found.enabled === false || found.publicClient) return false;
if (!sameSet(found.redirectUris, want.redirectUris!)) return false;
const mappers = await this.kc.listClientMappers(this.realm, found.id!);
if (!mappers.some((m) => m.name === ROLES_MAPPER.name)) return false;
return (await this.kc.clientSecretById(this.realm, found.id!)) === g.password;
}
/** Withdraw a consumer's client — only one the mesh made. Returns what happened, for the log. */
async remove(as: string): Promise<"removed" | "absent" | "not ours"> {
const found = await this.kc.findClient(this.realm, as);
if (!found) return "absent";
if (!marked(found)) return "not ours";
await this.kc.deleteClientById(this.realm, found.id!);
return "removed";
}
}
+6 -2
View File
@@ -1,11 +1,15 @@
{
"name": "@novox/module-keycloak",
"version": "0.1.0",
"description": "keycloak — identity and access. Its admin API client, tools and events live here (novox/hq ADR 0039).",
"description": "keycloak — identity and access; provides the mesh oidc-client interface. Its admin API client, provisioner, tools and events live here (novox/hq ADR 0039).",
"type": "module",
"private": true,
"scripts": {
"build": "tsc client.ts oidc.ts index.ts provisioner/index.ts tools/index.ts --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist",
"test": "npm run build && node --test --experimental-strip-types 'test/*.test.ts'"
},
"dependencies": {
"@novox/mesh-sdk": "^0.1.0"
"@novox/mesh-sdk": "^0.1.1"
},
"devDependencies": {
"@types/node": "^22.0.0",
+73
View File
@@ -0,0 +1,73 @@
// keycloak's provisioner — the adapter that makes keycloak a provider of the mesh `oidc-client`
// interface. The reconcile loop, the contributions file and reading the mesh's minted secret are the
// sdk harness's; this writes only the per-service half: how Keycloak creates, checks and removes a
// consumer's client (novox/hq ADR 0039/0040/0048). What a client is, and which ones are the mesh's,
// is in ../oidc.ts.
//
// The `oidc-client` interface: a consumer logs people in through the realm this module serves, as
// the confidential client `as` with the secret the mesh minted, and is redirected back to the
// callback it contributed under the names the mesh composed for its endpoint. What it is served —
// the issuer and the endpoint paths under it — is in the manifest's `serves`, settled with the
// assignment's settings.
//
// **The realm is read out of the issuer**, the one value an assignment sets (settings reach both the
// served facts and this module's config.json): a realm set in one place and an issuer in another
// would let the consumer be told one realm while its client is made in another.
import { runProvisioner, type Provision } from "@novox/mesh-sdk/provisioner";
import { emit } from "@novox/mesh-sdk/events";
import { readFileSync } from "node:fs";
import { KeycloakClient } from "../client.js";
import { OidcClients, realmOf } from "../oidc.js";
/** The issuer this assignment serves, from the settings-merged config the mesh delivers. */
function issuer(): string {
const file = process.env.MESH_KEYCLOAK_CONFIG_FILE;
let cfg: Record<string, unknown> = {};
if (file) {
try {
cfg = JSON.parse(readFileSync(file, "utf8")) as Record<string, unknown>;
} catch {
// Absent or unreadable: fall through to the environment, and refuse below if that is empty too.
}
}
const said = typeof cfg.issuer === "string" ? cfg.issuer : process.env.MESH_KEYCLOAK_ISSUER;
if (!said) throw new Error("no issuer — the module's config.json carries none and MESH_KEYCLOAK_ISSUER is unset");
return said;
}
const clients = new OidcClients(KeycloakClient.fromEnv(), realmOf(issuer()));
/** Emit a lifecycle event without letting a broker hiccup fail the provisioning itself. */
async function announce(type: string, body: Record<string, string>): Promise<void> {
try {
await emit(type, body);
} catch (err) {
console.error(`[provisioner:oidc-client] emit ${type} failed: ${err}`);
}
}
runProvisioner("oidc-client", {
async create(p: Provision): Promise<void> {
const done = await clients.ensure(p);
if (done === "created") {
console.log(`[provisioner:oidc-client] created client ${p.as} in realm ${clients.realm}`);
await announce("client.created", { realm: clients.realm, clientId: p.as, consumer: p.consumer ?? "" });
}
},
async remove(p: { as: string }): Promise<void> {
const done = await clients.remove(p.as);
if (done === "not ours") {
console.error(`[provisioner:oidc-client] ${p.as}: a client of that id exists that the mesh did not make — left alone`);
} else if (done === "removed") {
console.log(`[provisioner:oidc-client] removed client ${p.as} from realm ${clients.realm}`);
}
},
// Asked every minute by the harness: whether Keycloak still holds this consumer's client exactly as
// the mesh gave it, so a client deleted or edited behind the mesh's back is made again (hq issue 120).
async holds(p: Provision): Promise<boolean> {
return clients.holds(p);
},
});
+239
View File
@@ -0,0 +1,239 @@
// What holds keycloak to the `oidc-client` provision (oidc.ts): one confidential client per consumer,
// under the id and secret the mesh gave, redirecting only to the consumer's own callback under the
// names the mesh composed; made once and brought back on every apply; and a client the mesh did not
// make — same id or not — never adopted, changed or deleted.
//
// Keycloak is a fake: the admin routes the module touches, answering with the status codes and the
// shapes Keycloak gives. Run against the compiled module (npm test builds first), the way the runtime
// loads it.
import { test, after } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { randomUUID } from "node:crypto";
import { KeycloakClient } from "../dist/client.js";
import { MARK, OidcClients, ROLES_MAPPER, realmOf, redirectsOf } from "../dist/oidc.js";
type Client = Record<string, any>;
/** The realm's clients, by internal id, and what the fake was asked. */
const realm = "Novox";
const clients = new Map<string, Client>();
const calls: string[] = [];
function body(req: IncomingMessage): Promise<any> {
return new Promise((resolve) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => resolve(raw ? JSON.parse(raw) : undefined));
});
}
function send(res: ServerResponse, status: number, value?: unknown): void {
res.writeHead(status, { "Content-Type": "application/json" });
res.end(value === undefined ? "" : JSON.stringify(value));
}
const server = createServer(async (req, res) => {
const url = new URL(req.url!, "http://fake");
calls.push(`${req.method} ${url.pathname}`);
if (url.pathname === "/realms/master/protocol/openid-connect/token") {
return send(res, 200, { access_token: "t", expires_in: 300 });
}
const base = `/admin/realms/${realm}/clients`;
if (!url.pathname.startsWith(base)) return send(res, 404, { error: "Realm not found." });
const rest = url.pathname.slice(base.length).split("/").filter(Boolean);
if (rest.length === 0 && req.method === "GET") {
const want = url.searchParams.get("clientId");
return send(res, 200, [...clients.values()].filter((c) => !want || c.clientId === want));
}
if (rest.length === 0 && req.method === "POST") {
const rep = await body(req);
if ([...clients.values()].some((c) => c.clientId === rep.clientId)) {
return send(res, 409, { errorMessage: `Client ${rep.clientId} already exists` });
}
const id = randomUUID();
const mappers = (rep.protocolMappers ?? []).map((m: Client) => ({ ...m, id: randomUUID() }));
clients.set(id, { ...rep, id, protocolMappers: mappers });
return send(res, 201);
}
const c = clients.get(rest[0]);
if (!c) return send(res, 404, { error: "Could not find client" });
if (rest.length === 1 && req.method === "PUT") {
// Keycloak ignores protocolMappers on a client update: they have their own endpoints.
const rep = await body(req);
clients.set(c.id, { ...rep, id: c.id, protocolMappers: c.protocolMappers });
return send(res, 204);
}
if (rest.length === 1 && req.method === "DELETE") {
clients.delete(c.id);
return send(res, 204);
}
if (rest[1] === "client-secret" && req.method === "GET") {
return send(res, 200, { type: "secret", value: c.secret });
}
if (rest[1] === "protocol-mappers") {
if (req.method === "GET") return send(res, 200, c.protocolMappers ?? []);
if (req.method === "POST") {
c.protocolMappers = [...(c.protocolMappers ?? []), { ...(await body(req)), id: randomUUID() }];
return send(res, 201);
}
if (req.method === "PUT") {
const m = await body(req);
c.protocolMappers = c.protocolMappers.map((x: Client) => (x.id === rest[4] ? m : x));
return send(res, 204);
}
}
send(res, 405);
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
after(() => server.close());
const port = (server.address() as { port: number }).port;
const oidc = new OidcClients(new KeycloakClient(`http://127.0.0.1:${port}`, "admin", "pw"), realm);
/** Grafana on ace, as the mesh hands it to the provisioner. */
function grafana(secret = "s3cret", values: Record<string, unknown> = {}) {
return {
as: "mesh_ace_grafana",
password: secret,
consumer: "ace",
values: {
label: "grafana", endpoint: "web", port: 20010, callback: "/login/generic_oauth",
name: "grafana.zurag.be", "internal-name": "grafana.ace.internal", ...values,
},
};
}
function only(clientId: string): Client {
const found = [...clients.values()].filter((c) => c.clientId === clientId);
assert.equal(found.length, 1, `exactly one client ${clientId}, found ${found.length}`);
return found[0];
}
test("the realm is read out of the issuer, and an issuer that names none is refused", () => {
assert.equal(realmOf("https://keycloak.novox.be/realms/Novox"), "Novox");
assert.equal(realmOf("https://keycloak.novox.be/realms/Novox/"), "Novox");
assert.equal(realmOf("http://127.0.0.1:18500/realms/master"), "master");
assert.throws(() => realmOf("https://keycloak.novox.be"), /realms/);
assert.throws(() => realmOf("keycloak"), /not a URL/);
});
test("the redirect is the consumer's callback under every name the mesh composed for it", () => {
assert.deepEqual(redirectsOf(grafana().values), {
root: "https://grafana.zurag.be",
redirects: ["https://grafana.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"],
});
// A route reaching only the private network has only the internal name, and that is enough.
assert.deepEqual(redirectsOf({ callback: "/cb", "internal-name": "x.ace.internal" }).redirects,
["https://x.ace.internal/cb"]);
assert.throws(() => redirectsOf({ name: "grafana.zurag.be" }), /callback/);
assert.throws(() => redirectsOf({ name: "grafana.zurag.be", callback: "login" }), /callback/);
assert.throws(() => redirectsOf({ callback: "/cb" }), /label/);
});
test("a consumer is given one confidential client, under its id and the mesh's secret", async () => {
clients.clear();
assert.equal(await oidc.ensure(grafana()), "created");
const c = only("mesh_ace_grafana");
assert.equal(c.publicClient, false);
assert.equal(c.clientAuthenticatorType, "client-secret");
assert.equal(c.secret, "s3cret");
assert.equal(c.enabled, true);
assert.equal(c.standardFlowEnabled, true);
assert.equal(c.directAccessGrantsEnabled, false);
assert.equal(c.implicitFlowEnabled, false);
assert.deepEqual(c.redirectUris, [
"https://grafana.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"]);
assert.equal(c.attributes[MARK], "true");
assert.deepEqual(c.protocolMappers.map((m: Client) => m.name), [ROLES_MAPPER.name]);
assert.equal(await oidc.holds(grafana()), true);
});
test("applying the same grant again makes no second client", async () => {
clients.clear();
await oidc.ensure(grafana());
assert.equal(await oidc.ensure(grafana()), "updated");
assert.equal(await oidc.ensure(grafana()), "updated");
only("mesh_ace_grafana");
assert.equal(only("mesh_ace_grafana").protocolMappers.length, 1, "the roles mapper is not added twice");
});
test("a new secret or a moved name is applied in place, and what the mesh does not own survives", async () => {
clients.clear();
await oidc.ensure(grafana());
const id = only("mesh_ace_grafana").id;
// Something the mesh does not own, set on the client after it was made.
clients.get(id)!.consentRequired = true;
clients.get(id)!.attributes["post.logout.redirect.uris"] = "+";
assert.equal(await oidc.holds(grafana("rotated")), false, "a rotated secret is not held until applied");
await oidc.ensure(grafana("rotated", { name: "dash.zurag.be" }));
const c = only("mesh_ace_grafana");
assert.equal(c.id, id, "updated, not replaced");
assert.equal(c.secret, "rotated");
assert.deepEqual(c.redirectUris, [
"https://dash.zurag.be/login/generic_oauth", "https://grafana.ace.internal/login/generic_oauth"]);
assert.equal(c.rootUrl, "https://dash.zurag.be");
assert.equal(c.consentRequired, true);
assert.equal(c.attributes["post.logout.redirect.uris"], "+");
assert.equal(c.attributes[MARK], "true");
assert.equal(await oidc.holds(grafana("rotated", { name: "dash.zurag.be" })), true);
});
test("a client lost or edited behind the mesh's back is not held, and is made whole again", async () => {
clients.clear();
await oidc.ensure(grafana());
const c = only("mesh_ace_grafana");
c.redirectUris = ["*"];
assert.equal(await oidc.holds(grafana()), false, "a widened redirect is not what the mesh gave");
await oidc.ensure(grafana());
assert.equal(await oidc.holds(grafana()), true);
only("mesh_ace_grafana").protocolMappers = [];
assert.equal(await oidc.holds(grafana()), false, "a client without its roles mapper is not held");
await oidc.ensure(grafana());
assert.equal(await oidc.holds(grafana()), true);
clients.clear();
assert.equal(await oidc.holds(grafana()), false);
});
test("a client of the same id the mesh did not make is refused, and left exactly as it was", async () => {
clients.clear();
clients.set("theirs", { id: "theirs", clientId: "mesh_ace_grafana", secret: "their-secret", redirectUris: ["*"] });
const before = JSON.stringify(clients.get("theirs"));
const writes = calls.length;
await assert.rejects(oidc.ensure(grafana()), /did not make/);
assert.equal(JSON.stringify(clients.get("theirs")), before);
assert.ok(calls.slice(writes).every((c) => c.startsWith("GET") || c.startsWith("POST /realms/master")),
`only reads were made: ${calls.slice(writes).join(", ")}`);
assert.equal(await oidc.holds(grafana()), false);
assert.equal(await oidc.remove("mesh_ace_grafana"), "not ours");
assert.ok(clients.has("theirs"), "a client the mesh did not make is never deleted");
});
test("the predecessor's hand-made client is never touched: the mesh's has its own id", async () => {
clients.clear();
clients.set("hal", { id: "hal", clientId: "grafana", secret: "old", redirectUris: ["https://grafana.zurag.be/*"] });
await oidc.ensure(grafana());
assert.equal(clients.get("hal")!.secret, "old");
only("mesh_ace_grafana");
assert.equal(await oidc.remove("grafana"), "not ours");
assert.ok(clients.has("hal"));
});
test("a withdrawn consumer's client is removed, and an absent one is not an error", async () => {
clients.clear();
await oidc.ensure(grafana());
assert.equal(await oidc.remove("mesh_ace_grafana"), "removed");
assert.equal([...clients.values()].length, 0);
assert.equal(await oidc.remove("mesh_ace_grafana"), "absent");
});
test("a contribution with no callback makes no client at all", async () => {
clients.clear();
await assert.rejects(oidc.ensure({ ...grafana(), values: { name: "grafana.zurag.be" } }), /callback/);
assert.equal(clients.size, 0);
});
+1 -1
View File
@@ -8,5 +8,5 @@
"skipLibCheck": true,
"noEmit": true
},
"include": ["client.ts", "index.ts", "tools/index.ts"]
"include": ["client.ts", "oidc.ts", "index.ts", "provisioner/index.ts", "tools/index.ts"]
}
+1
View File
@@ -24,6 +24,7 @@
},
"listens": [
{
"name": "web",
"port": 8283,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -14,6 +14,7 @@
},
"listens": [
{
"name": "web",
"port": 8686,
"protocol": "tcp",
"from": "mesh",
@@ -86,7 +87,7 @@
"contributes": {
"route": {
"label": "lidarr",
"port": 8686
"endpoint": "web"
}
},
"binds": {
+15 -5
View File
@@ -16,27 +16,27 @@
"route": {
"web": {
"label": "mail",
"port": 7443,
"endpoint": "web-tls",
"scheme": "https",
"insecure": true
},
"acme": {
"label": "mail",
"path": "/.well-known/acme-challenge",
"port": 7080,
"endpoint": "web",
"priority": 100
},
"autoconfig": {
"label": "autoconfig",
"port": 4243
"endpoint": "autoconfig"
},
"autodiscover": {
"label": "autodiscover",
"port": 4243
"endpoint": "autoconfig"
},
"automx": {
"label": "automx",
"port": 4243
"endpoint": "autoconfig"
}
}
},
@@ -60,6 +60,7 @@
],
"listens": [
{
"name": "smtp",
"port": 25,
"protocol": "tcp",
"from": "anywhere",
@@ -67,6 +68,7 @@
"fixed": true
},
{
"name": "pop3",
"port": 110,
"protocol": "tcp",
"from": "anywhere",
@@ -74,6 +76,7 @@
"fixed": true
},
{
"name": "imap",
"port": 143,
"protocol": "tcp",
"from": "anywhere",
@@ -81,6 +84,7 @@
"fixed": true
},
{
"name": "smtps",
"port": 465,
"protocol": "tcp",
"from": "anywhere",
@@ -88,6 +92,7 @@
"fixed": true
},
{
"name": "submission",
"port": 587,
"protocol": "tcp",
"from": "anywhere",
@@ -95,6 +100,7 @@
"fixed": true
},
{
"name": "imaps",
"port": 993,
"protocol": "tcp",
"from": "anywhere",
@@ -102,6 +108,7 @@
"fixed": true
},
{
"name": "pop3s",
"port": 995,
"protocol": "tcp",
"from": "anywhere",
@@ -109,18 +116,21 @@
"fixed": true
},
{
"name": "web",
"port": 7080,
"protocol": "tcp",
"from": "mesh",
"why": "the web front over http; only the ACME HTTP-01 passthrough is routed here \u2014 everything else 301s to https and would loop a proxy"
},
{
"name": "web-tls",
"port": 7443,
"protocol": "tcp",
"from": "mesh",
"why": "the web front over its own TLS (admin, webmail, API); the public name mail.novox.be is a route grant reaching it here"
},
{
"name": "autoconfig",
"port": 4243,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -6,6 +6,7 @@
],
"listens": [
{
"name": "api",
"port": 59125,
"protocol": "tcp",
"from": "mesh",
+6 -1
View File
@@ -22,7 +22,7 @@ COPY . .
# The compiler is invoked by its real path rather than through node_modules/.bin, whose entries are
# symlinks to a launcher that requires its library relatively — resolved away when the base image
# was assembled.
RUN node /app/node_modules/typescript/bin/tsc pg.d.ts store.ts index.ts tools/index.ts \
RUN node /app/node_modules/typescript/bin/tsc pg.d.ts store.ts index.ts tools/index.ts prepare/index.ts \
--module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist
# **A module may need something the base image does not carry.** The base holds what every module
@@ -48,3 +48,8 @@ COPY --from=build /deps/node_modules /app/modules/mesh-catalog/node_modules
# to listen for what the builder announces. Serve binds the broker first, then imports these, so
# `on()` has something to subscribe to.
ENV MESH_TOOL_MODULES=/app/modules/mesh-catalog/dist/index.js,/app/modules/mesh-catalog/dist/tools/index.js
# And what prepares this module's state, for the runtime's `prepare` mode (novox/hq ADR 0135). Named
# here, beside the entrypoints above, because the module knows which of its files prepares its state
# and nothing else could: the mesh asks one word and this says what answers it.
ENV MESH_PREPARE=/app/modules/mesh-catalog/dist/prepare/index.js
+20 -6
View File
@@ -14,10 +14,12 @@ import { Graph, type Made } from "./store.js";
const graph = Graph.fromEnv();
// Before subscribing, and idempotent. The runtime is restarted until its store is reachable, which
// is the same arrangement model-usage uses: a schema step that had to reach the provider over the
// overlay would block the very apply that brings the overlay up.
await graph.migrate();
// The schema is not brought up here. The mesh prepares this module's state before it starts this
// version, and does not start it if that failed (novox/hq ADR 0135) — see prepare/index.ts. Doing it
// at start made a schema that could not be reached a crash loop instead of a stop, with the graph
// keeping a gap and nothing saying so. The reason it used to be here — that a step blocking the apply
// would block the very apply that brings the overlay up — stopped being true when a step's failure
// became this module's business and not the machine's (ADR 0136).
/** What the builder says when it has built something. */
interface Built {
@@ -47,7 +49,15 @@ interface Built {
replay?: boolean;
}
await on("mesh-build-machine.built", async (event) => {
/**
* What a build means for the graph, wherever it came from.
*
* Two emitters say the same thing and neither is a mistake: the build machine says it as it happens,
* and the control plane says what it already held when this module asks what it missed
* (novox/hq ADR 0134). A replay is marked as one in its body, so nothing acts on a module that moved
* months ago — see `replay` above.
*/
const placeTheBuild = async (event: { body: unknown }): Promise<void> => {
const body = event.body as Built;
if (!body.module || !body.commit) {
// Said rather than dropped: a build that announced itself without saying what it built is a
@@ -89,7 +99,11 @@ await on("mesh-build-machine.built", async (event) => {
because: next.because,
});
}
});
};
// As it happens, and what the mesh already held when this module asked what it missed.
await on("mesh-build-machine.built", placeTheBuild);
await on("mesh-controller.built-before", placeTheBuild);
// **And ask for what was built before this catalogue existed** (novox/hq 04-ISSUES/050).
//
+5 -2
View File
@@ -29,13 +29,16 @@
"broker": "/var/lib/mesh/mesh-catalog/broker"
},
"consumes": [
"mesh-build-machine.built"
"mesh-build-machine.built",
"mesh-controller.built-before"
],
"emits": [
"registered",
"upgraded",
"rebuild-needed"
"rebuild-needed",
"catching-up"
],
"prepares": true,
"resources": [
{
"id": "mesh-state",
+17
View File
@@ -0,0 +1,17 @@
// The catalogue's state, brought to the shape this version needs (novox/hq ADR 0135).
//
// **The mesh runs this before the version that needs it, and does not start that version if it
// fails** — and the refusal reaches this module and nothing else on the machine
// (novox/hq ADR 0136). That is the whole difference from where this used to happen: at start, inside
// the runtime, a schema that could not be brought up was a crash loop, the graph kept a gap, and
// nothing anywhere said so.
//
// Nothing here connects to the broker. Preparation runs before the version that would use it, so
// there is nothing yet to talk to; the runtime's `prepare` mode imports this and awaits it, and this
// process exiting non-zero is how the host knows not to start the runtime.
import { Graph } from "../store.js";
const graph = Graph.fromEnv();
await graph.migrate();
console.log("[mesh-catalog] the module graph's schema is what this version needs");
await graph.close();
+2 -1
View File
@@ -12,6 +12,7 @@
"pg.d.ts",
"store.ts",
"index.ts",
"tools/index.ts"
"tools/index.ts",
"prepare/index.ts"
]
}
+4 -2
View File
@@ -14,11 +14,11 @@
"route": {
"api": {
"label": "files-api",
"port": 9000
"endpoint": "s3"
},
"console": {
"label": "files",
"port": 9001
"endpoint": "console"
}
}
},
@@ -31,12 +31,14 @@
],
"listens": [
{
"name": "s3",
"port": 9000,
"protocol": "tcp",
"from": "mesh",
"why": "the S3 endpoint"
},
{
"name": "console",
"port": 9001,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -20,6 +20,7 @@
],
"listens": [
{
"name": "database",
"port": 27017,
"protocol": "tcp",
"from": "mesh",
+2
View File
@@ -34,12 +34,14 @@
},
"listens": [
{
"name": "mqtt",
"port": 1883,
"protocol": "tcp",
"from": "mesh",
"why": "modules on any machine that were granted a topic namespace"
},
{
"name": "mqtt-websockets",
"port": 8081,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -20,6 +20,7 @@
],
"listens": [
{
"name": "database",
"port": 4848,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -14,7 +14,7 @@
},
"route": {
"label": "n8n",
"port": 5682
"endpoint": "web"
}
},
"binds": {
@@ -29,6 +29,7 @@
},
"listens": [
{
"name": "web",
"port": 5682,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -21,6 +21,7 @@
"consumes": [],
"listens": [
{
"name": "bus",
"port": 4222,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -13,7 +13,7 @@
},
"route": {
"label": "drive",
"port": 80
"endpoint": "web"
}
},
"binds": {
@@ -38,6 +38,7 @@
],
"listens": [
{
"name": "web",
"port": 80,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -12,6 +12,7 @@
],
"listens": [
{
"name": "web",
"port": 1880,
"protocol": "tcp",
"from": "mesh",
@@ -81,7 +82,7 @@
"contributes": {
"route": {
"label": "nodered",
"port": 1880
"endpoint": "web"
}
},
"binds": {
+2 -1
View File
@@ -10,7 +10,7 @@
"contributes": {
"route": {
"label": "@",
"port": 4000
"endpoint": "web"
}
},
"binds": {
@@ -18,6 +18,7 @@
},
"listens": [
{
"name": "web",
"port": 4000,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -15,6 +15,7 @@
},
"listens": [
{
"name": "web",
"port": 6789,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -12,6 +12,7 @@
],
"listens": [
{
"name": "api",
"port": 11434,
"protocol": "tcp",
"from": "machine",
+2 -1
View File
@@ -14,6 +14,7 @@
},
"listens": [
{
"name": "web",
"port": 3579,
"protocol": "tcp",
"from": "mesh",
@@ -89,7 +90,7 @@
"contributes": {
"route": {
"label": "ombi",
"port": 3579
"endpoint": "web"
}
},
"binds": {
+2 -1
View File
@@ -11,7 +11,7 @@
"contributes": {
"route": {
"label": "office",
"port": 9070
"endpoint": "web"
}
},
"binds": {
@@ -22,6 +22,7 @@
},
"listens": [
{
"name": "web",
"port": 9070,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -11,7 +11,7 @@
"contributes": {
"route": {
"label": "eef",
"port": 4012
"endpoint": "web"
}
},
"binds": {
@@ -19,6 +19,7 @@
},
"listens": [
{
"name": "web",
"port": 4012,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -11,7 +11,7 @@
"contributes": {
"route": {
"label": "filip",
"port": 4013
"endpoint": "web"
}
},
"binds": {
@@ -19,6 +19,7 @@
},
"listens": [
{
"name": "web",
"port": 4013,
"protocol": "tcp",
"from": "mesh",
+3 -1
View File
@@ -18,7 +18,7 @@
},
"route": {
"label": "photos",
"port": 4001
"endpoint": "web"
}
},
"binds": {
@@ -32,12 +32,14 @@
},
"listens": [
{
"name": "api",
"port": 9000,
"protocol": "tcp",
"from": "mesh",
"why": "the photos backend API; the client sites on the module network call it"
},
{
"name": "web",
"port": 4001,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -18,6 +18,7 @@
},
"listens": [
{
"name": "stream",
"port": 32400,
"protocol": "tcp",
"from": "mesh",
+3 -1
View File
@@ -7,12 +7,14 @@
],
"listens": [
{
"name": "web",
"port": 9090,
"protocol": "tcp",
"from": "mesh",
"why": "the dashboard over http; portainer.novox.be is a route grant and the proxy reaches it here \u2014 the machine side of 9090:9000, the predecessor's number"
},
{
"name": "web-tls",
"port": 9443,
"protocol": "tcp",
"from": "mesh",
@@ -103,7 +105,7 @@
"contributes": {
"route": {
"label": "portainer",
"port": 9090
"endpoint": "web"
}
},
"binds": {
+1
View File
@@ -26,6 +26,7 @@
],
"listens": [
{
"name": "database",
"port": 5432,
"protocol": "tcp",
"from": "mesh",
+1
View File
@@ -16,6 +16,7 @@
},
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -14,6 +14,7 @@
},
"listens": [
{
"name": "web",
"port": 7878,
"protocol": "tcp",
"from": "mesh",
@@ -86,7 +87,7 @@
"contributes": {
"route": {
"label": "movies",
"port": 7878
"endpoint": "web"
}
},
"binds": {
+1
View File
@@ -40,6 +40,7 @@
},
"listens": [
{
"name": "cache",
"port": 6379,
"protocol": "tcp",
"from": "mesh",
+2
View File
@@ -27,12 +27,14 @@
},
"listens": [
{
"name": "http",
"port": 80,
"protocol": "tcp",
"from": "anywhere",
"why": "public HTTP, and the ACME HTTP-01 challenge answered at the name being certified"
},
{
"name": "https",
"port": 443,
"protocol": "tcp",
"from": "anywhere",
+23 -21
View File
@@ -5,11 +5,12 @@
"container-runtime"
],
"own-secrets": {
"secret": "/var/lib/searxng-module/secret.secret",
"secret": "/var/lib/mesh/searxng/secret",
"broker": "/var/lib/mesh/searxng/broker"
},
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
@@ -26,22 +27,14 @@
{
"id": "state",
"type": "directory",
"path": "/var/lib/searxng-module",
"mode": "0700"
"mode": "0700",
"place": "."
},
{
"id": "valkey-data",
"type": "directory",
"path": "/var/lib/searxng-module/valkey-data",
"mode": "0700"
},
{
"id": "server-env",
"type": "file",
"path": "/var/lib/searxng-module/server.env",
"mode": "0600",
"content": "SEARXNG_SECRET=${secret:secret}\nSEARXNG_VALKEY_URL=valkey://valkey:6379/0\n"
},
{
"id": "net",
"type": "network",
@@ -62,30 +55,39 @@
"warning"
],
"volumes": [
"/var/lib/searxng-module/valkey-data:/data"
"${dir:valkey-data}:/data"
]
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.yml",
"mode": "0600",
"merge": "json",
"content": "{\n \"use_default_settings\": true,\n \"server\": {\n \"secret_key\": \"${secret:secret}\",\n \"base_url\": false,\n \"limiter\": false,\n \"image_proxy\": false,\n \"public_instance\": false\n },\n \"search\": {\n \"formats\": [\"html\", \"json\"]\n },\n \"valkey\": {\n \"url\": \"valkey://valkey:6379/0\"\n }\n}\n"
},
{
"id": "server",
"type": "container",
"name": "searxng",
"image": "searxng/searxng@sha256:c7cc75852051bf6254afda6ed1b920dd1677d8efe4ab141bf558f02e582f4371",
"image": "searxng/searxng@sha256:cd8812607ab73730a0b1a0dc4990223fe1b9e383f6f35947114d0bef7f8bb441",
"network": "searxng",
"env-file": [
"/var/lib/searxng-module/server.env"
],
"ports": [
"8080"
],
"secrets-in-environment": "SEARXNG_SECRET is env-only, but settings.yml carries server.secret_key; convertible by mounting a generated settings.yml, not yet done"
"volumes": [
"${dir:state}/settings.yml:/etc/searxng/settings.yml:ro"
],
"restart-on": [
"settings"
]
},
{
"id": "runtime-config",
"type": "file",
"path": "/var/lib/mesh/searxng/config.json",
"mode": "0600",
"content": "{}\n",
"merge": "json"
"content": "{}\n"
},
{
"id": "runtime",
@@ -113,11 +115,11 @@
"contributes": {
"route": {
"label": "searxng",
"port": 8080
"endpoint": "web"
}
},
"binds": {
"route": "/var/lib/searxng-module/route.json"
"route": "${dir:state}/route.json"
},
"build": {
"on": [
+1
View File
@@ -43,6 +43,7 @@
],
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -14,6 +14,7 @@
},
"listens": [
{
"name": "web",
"port": 8989,
"protocol": "tcp",
"from": "mesh",
@@ -91,7 +92,7 @@
"contributes": {
"route": {
"label": "series",
"port": 8989
"endpoint": "web"
}
},
"binds": {
+2
View File
@@ -7,6 +7,7 @@
],
"listens": [
{
"name": "ssh",
"port": 22,
"protocol": "tcp",
"from": "anywhere",
@@ -31,6 +32,7 @@
"type": "service",
"unit": "sshd.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"config"
]
+1
View File
@@ -26,6 +26,7 @@
},
"listens": [
{
"name": "acme",
"port": 9000,
"protocol": "tcp",
"from": "mesh",
+2 -1
View File
@@ -12,6 +12,7 @@
],
"listens": [
{
"name": "web",
"port": 8181,
"protocol": "tcp",
"from": "mesh",
@@ -85,7 +86,7 @@
"contributes": {
"route": {
"label": "tautulli",
"port": 8181
"endpoint": "web"
}
},
"binds": {
+4 -3
View File
@@ -15,7 +15,7 @@
},
"route": {
"label": "umami",
"port": 3000
"endpoint": "web"
}
},
"binds": {
@@ -49,10 +49,11 @@
},
"listens": [
{
"name": "web",
"port": 3000,
"protocol": "tcp",
"from": "anywhere",
"why": "one port serves two surfaces: the dashboard (the proxy gates it to the mesh) and the public collection endpoint that the browsers of every tracked site POST to \u2014 so the port itself must be reachable from anywhere"
"from": "mesh",
"why": "one port serves two surfaces \u2014 the dashboard and the collection endpoint that the browsers of every tracked site POST to. Both are reached through the proxy, by name, so the port is how the proxy reaches this module and nothing else (novox/hq ADR 0045). It said \"anywhere\" and gave the reason that the collection endpoint must be public, which is true of the name and not of the port: opened, the machine-side port served the dashboard over plain HTTP to the internet, bypassing every rule the proxy applies by path"
}
],
"resources": [
+9
View File
@@ -6,54 +6,63 @@
],
"listens": [
{
"name": "web",
"port": 8443,
"protocol": "tcp",
"from": "mesh",
"why": "the controller web UI, over its own self-signed tls; reaching it from outside is a route grant later"
},
{
"name": "inform",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
"why": "device inform \u2014 how APs and switches check in and are adopted"
},
{
"name": "stun",
"port": 3478,
"protocol": "udp",
"from": "mesh",
"why": "STUN, so managed devices can find the controller through NAT"
},
{
"name": "discovery",
"port": 10001,
"protocol": "udp",
"from": "mesh",
"why": "device discovery \u2014 the controller finds unadopted devices on the network"
},
{
"name": "discovery-l2",
"port": 1902,
"protocol": "udp",
"from": "mesh",
"why": "layer-2 (UBNT) discovery broadcasts; published on 1902, the container listens on 1900"
},
{
"name": "portal-tls",
"port": 8843,
"protocol": "tcp",
"from": "mesh",
"why": "the guest captive portal over https"
},
{
"name": "portal",
"port": 8880,
"protocol": "tcp",
"from": "mesh",
"why": "the guest captive portal over http"
},
{
"name": "speedtest",
"port": 6789,
"protocol": "tcp",
"from": "mesh",
"why": "mobile-app speed-test throughput measurement"
},
{
"name": "syslog",
"port": 5514,
"protocol": "udp",
"from": "mesh",