Compare commits

...
Author SHA1 Message Date
jschoubben 204bfbbaf9 dnsmasq: answer a container's query, which arrives on the runtime's bridge
dnsmasq admits a query by the interface it arrives on when told interface=,
and by the address it is sent to when told listen-address=. A container's
query is sent to the machine's private address but arrives on docker0, so
interface=mesh0 dropped it silently on every machine (novox/hq issue 110).
Name the address, not the interface.
2026-09-30 14:31:26 +02:00
jschoubben e0c09f46b6 Merge pull request 'dnsmasq: the runtime is reloaded when its dns file changes, and may then be restarted safely' (#175) from fix/110-the-runtime-reads-its-dns into main 2026-09-30 12:22:10 +00:00
jschoubben e0faf012be dnsmasq: the runtime is reloaded when its dns file changes, and may then be restarted safely
novox/hq 04-ISSUES/110. The module writes the runtime's `dns` key and
deliberately ordered no restart, because a restart stops every container.
It also ordered no reload, and the key holds only for containers created
after the runtime next starts. On two of four machines the runtime
predated the file — one since August — so every container there was
handed a public resolver and no mesh name resolved, while everything read
as fine. The third machine works only because its runtime happened to
restart later.

The file now also sets live-restore, which the runtime reads on a reload,
and the module declares the runtime reloaded when the file changes (ADR
0102: reload, don't restart). A reload still does not make `dns` take
effect; what it does is make the one restart that key needs keep every
container running. That restart stays the operator's, once per machine,
and is harmless from the second time on.

The mesh restarts nothing here. Undeclared, the runtime's unit goes back
to the state it was found in (ADR 0118).
2026-09-30 14:22:03 +02:00
mesh-admin 1b19c79d63 Merge pull request 'postgres: the provisioner dials the port the mesh gave the store' (#174) from fix/postgres-provisioner-dials-the-port-it-was-given into main 2026-09-30 12:06:25 +00:00
jschoubben 4ec2ae1f7f postgres: the provisioner dials the port the mesh gave the store
MESH_PROVISION_POSTGRES named 127.0.0.1:5432 literally; the seat twin
(${seat:mesh-store:5432}) corrected it only on the machine holding the
mesh-store seat. On any other machine — ace, where the module provides
postgres-database without the seat — the twin is empty and the sidecar
would have dialled whatever else holds 5432 (HAL's postgres). ${port:5432}
is the machine port on every node; on novox the twin still says the same
6852, so nothing moves there.
2026-09-30 14:01:53 +02:00
mesh-admin 8064e5da8f Merge pull request 'searxng: its settings are a file the mesh writes, not the image's defaults' (#143) from feat/searxng-settings-as-a-file into main 2026-09-29 20:45:13 +00:00
jschoubben 63a255c5cb searxng: bind its route where its state now lives
The route binding still named /var/lib/searxng-module, the directory the
previous commit placed elsewhere — the host would have written it into a
directory nothing declares. Same shape as gitea and nextcloud.
2026-09-29 22:41:56 +02:00
jschoubben 7ad1fbd5c6 searxng: its settings are a file the mesh writes, not the image's defaults
The module ran searxng on the image's built-in settings, which serve html
only — so the module's own search tool (format=json) was refused by the
software it fronts. And there was no way to configure it per machine: the
only file settings reach was the sidecar's.

settings.yml is now the module's one mergeable file (JSON is YAML): generic
defaults in the manifest (json format on, limiter and image proxy off,
valkey wired), and whatever differs per machine — base_url, method,
autocomplete, suspended times — set as the assignment's settings. The
secret key is filled on the machine through ${secret:secret}, so the
secrets-in-environment exception and the env file go. Directories are
placed. Image pinned to 2026.9.20, what ace runs today (the old pin was
older, 2026.9.1).

The sidecar's config.json is no longer mergeable: settings merge into every
mergeable file of a module, and the sidecar would have received searxng's
keys. It only ever read an optional url, which its env already carries.

Verified on ace: the pinned image serves html and json from a read-only,
root-owned 0600 JSON settings.yml.
2026-09-29 22:33:38 +02:00
jschoubben 67f5f4cffd Merge pull request 'ca-trust: a machine trusts the mesh's authority because a module put its root there' (#142) from feat/ca-trust into main 2026-09-29 14:06:36 +00:00
jschoubben 8797335fbc ca-trust: a machine trusts the mesh's authority because a module put its root there
novox/hq ADR 0147, issue 129. Every internal HTTPS name fails verification
on every machine: the certificates are genuine and nothing on a machine has
ever been told what issued them. The proxy's fetch answers for the proxy and
for nothing else — a browser, git over HTTPS and every module calling another
by an internal name read the machine's own trust store.

The module requires internal-acme-ca, fetches the root over the mesh's own
network (no prior trust to have; that is what this establishes), installs it
among the machine's anchors and refreshes the extracted bundles. Being
unassigned stops the unit, and stopping it takes the anchor away and
refreshes them again.

Arch's layout is named out loud: a machine that keeps anchors elsewhere fails
visibly rather than writing a file nothing reads.
2026-09-29 15:07:40 +02:00
mesh-admin 53dc108603 Merge pull request 'Remove the network-checker module: it does not do what was decided' (#141) from chore/remove-the-network-checker-module into main 2026-09-29 12:43:34 +00:00
jschoubben ebf5ba2d4c Remove the network-checker module: it does not do what was decided
What was in the catalogue was the first thing I built, not the thing ADR 0146
describes. It dialled raw ports on machine addresses from one hosting form and
emitted nothing, so findings would have sat in a file on the machine — the exact
thing issue 145 is about. It was never registered, never assigned, and never ran.

0146 says names per hosting form, fetched over TLS with the certificate verified,
and machines discovered over the bus. That shares nothing with this but the word
checker, so it goes rather than being bent into shape. Recorded as work to be
analysed and built deliberately.

Connectivity is checked by hand in the meantime, against the services the mesh
already runs.
2026-09-29 14:43:25 +02:00
mesh-admin bbac08a7d2 Merge pull request 'A network-checker module: dial what the mesh claims, from where the callers are' (#140) from feat/a-network-checker-module into main 2026-09-29 11:44:37 +00:00
jschoubben 784a5a6514 A network-checker module: dial what the mesh claims, from where the callers are
The mesh asserts three things are callable (ADR 0144) — what runs on the same
machine, another machine's service exposed to the private network, and another
machine's service exposed publicly — and has never checked any of them. The first
was broken for eleven hours while the mesh reported every machine healthy.

This runs on every machine, on the cadence the mesh already has, in its own
container: the same position every other module calls from. Not the host and not
the control plane, both of which reach these addresses by paths no ordinary caller
uses and would have passed throughout that outage.

**Its probe is its own endpoint, and that is the point.** Declared reachable over
the private network like any other service, so it is admitted by exactly the rule
that governs every internally-exposed service and fails when that rule is wrong.
The tempting target is a service every machine has, and those are the ones never
closed — ssh above all — which would have passed while the thing that actually
broke was a service exposed to the private network.

It resolves before it dials and says which failed, because a name that does not
resolve and a port that does not answer have different owners. One failure is not
a fault: a machine rebooting is ordinary, so a path is broken after consecutive
runs and the count travels with the result. It reports and repairs nothing.

novox/hq ADR 0145. Eight tests; the consecutive-failure logic proved by reverting
it once. Not yet registered or assigned.
2026-09-29 13:44:16 +02:00
mesh-admin 0c31499fb0 Merge pull request 'Every module names its endpoints, and every route names the one it serves' (#139) from feat/modules-name-their-endpoints into main 2026-09-29 09:51:56 +00:00
4 changed files with 91 additions and 25 deletions
+56
View File
@@ -0,0 +1,56 @@
{
"module": "ca-trust",
"version": "1",
"slug": "catrust",
"capabilities": [
"service-manager"
],
"requires": [
"internal-acme-ca"
],
"seats": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"claims": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "anchor",
"type": "file",
"path": "${dir:state}/anchor",
"mode": "0755",
"content": "#!/bin/sh\n# The mesh's internal certificate authority, trusted by this machine.\n#\n# Written by the mesh from the ca-trust module's manifest (novox/hq ADR 0147).\n# Editing it here lasts until the next apply.\n#\n# There is no prior trust to verify the fetch against \u2014 this is the thing that\n# establishes it \u2014 so it is made over the mesh's own private network, which is\n# what authenticates it (novox/hq ADR 0098, the same reasoning that lets the\n# route proxy fetch this root for itself). What comes back is checked here: a\n# body that is not a certificate is refused now, rather than believed and then\n# failed by whatever reads the trust store next.\nset -eu\n\nROOTS='https://${bound:internal-acme-ca:at}:${bound:internal-acme-ca:port}${bound:internal-acme-ca:roots}'\nANCHORS=/etc/ca-certificates/trust-source/anchors\nANCHOR=\"$ANCHORS/mesh-internal-ca.crt\"\n\n# Arch's layout, said out loud rather than assumed: a machine that keeps its\n# anchors elsewhere fails here, visibly, instead of writing a file nothing\n# reads. That failure is the signal that this belongs in the host, where one\n# operating system's difference lives (novox/hq ADR 0147, option 2).\n[ -d \"$ANCHORS\" ] || {\n\techo \"this machine keeps no trust anchors in $ANCHORS; ca-trust is written for that layout\" >&2\n\texit 1\n}\n\ncase \"${1:-}\" in\ninstall)\n\ttmp=$(mktemp)\n\ttrap 'rm -f \"$tmp\"' EXIT\n\t# The authority may still be starting, or this machine may have come up\n\t# before it: two minutes of asking, then an honest failure.\n\tn=0\n\twhile [ \"$n\" -lt 60 ]; do\n\t\tif curl --fail --silent --show-error --insecure --max-time 10 \\\n\t\t\t--output \"$tmp\" \"$ROOTS\" &&\n\t\t\tgrep -q 'BEGIN CERTIFICATE' \"$tmp\"; then\n\t\t\tinstall -m 0644 \"$tmp\" \"$ANCHOR\"\n\t\t\tupdate-ca-trust\n\t\t\texit 0\n\t\tfi\n\t\tn=$((n + 1))\n\t\tsleep 2\n\tdone\n\techo \"the authority at $ROOTS did not serve a certificate within two minutes\" >&2\n\texit 1\n\t;;\nremove)\n\t# What stopping the unit does, and therefore what being unassigned does.\n\trm -f \"$ANCHOR\"\n\tupdate-ca-trust\n\t;;\n*)\n\techo \"usage: $(basename \"$0\") install|remove\" >&2\n\texit 2\n\t;;\nesac\n"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-ca-trust.service",
"mode": "0644",
"content": "[Unit]\nDescription=The mesh's internal certificate authority, trusted by this machine\n# novox/hq ADR 0147. Starting this unit places the mesh's root among this\n# machine's trust anchors; stopping it takes the root away again, which is what\n# the host does when the module is no longer assigned here.\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=${dir:state}/anchor install\nExecStop=${dir:state}/anchor remove\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"id": "trust",
"type": "service",
"unit": "mesh-ca-trust.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"anchor",
"unit"
]
}
]
}
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -105,7 +105,7 @@
"/var/lib/postgres/superuser.secret:/run/secrets/superuser:ro" "/var/lib/postgres/superuser.secret:/run/secrets/superuser:ro"
], ],
"env": { "env": {
"MESH_PROVISION_POSTGRES": "postgres://postgres@127.0.0.1:5432/postgres?sslmode=disable", "MESH_PROVISION_POSTGRES": "postgres://postgres@127.0.0.1:${port:5432}/postgres?sslmode=disable",
"MESH_PROVISION_POSTGRES_PORT": "${seat:mesh-store:5432}", "MESH_PROVISION_POSTGRES_PORT": "${seat:mesh-store:5432}",
"MESH_PROVISION_PASSWORD_FILE": "/run/secrets/superuser", "MESH_PROVISION_PASSWORD_FILE": "/run/secrets/superuser",
"MESH_BROKER_FILE": "/run/secrets/broker", "MESH_BROKER_FILE": "/run/secrets/broker",
+21 -20
View File
@@ -5,7 +5,7 @@
"container-runtime" "container-runtime"
], ],
"own-secrets": { "own-secrets": {
"secret": "/var/lib/searxng-module/secret.secret", "secret": "/var/lib/mesh/searxng/secret",
"broker": "/var/lib/mesh/searxng/broker" "broker": "/var/lib/mesh/searxng/broker"
}, },
"listens": [ "listens": [
@@ -27,22 +27,14 @@
{ {
"id": "state", "id": "state",
"type": "directory", "type": "directory",
"path": "/var/lib/searxng-module", "mode": "0700",
"mode": "0700" "place": "."
}, },
{ {
"id": "valkey-data", "id": "valkey-data",
"type": "directory", "type": "directory",
"path": "/var/lib/searxng-module/valkey-data",
"mode": "0700" "mode": "0700"
}, },
{
"id": "server-env",
"type": "file",
"path": "/var/lib/searxng-module/server.env",
"mode": "0600",
"content": "SEARXNG_SECRET=${secret:secret}\nSEARXNG_VALKEY_URL=valkey://valkey:6379/0\n"
},
{ {
"id": "net", "id": "net",
"type": "network", "type": "network",
@@ -63,30 +55,39 @@
"warning" "warning"
], ],
"volumes": [ "volumes": [
"/var/lib/searxng-module/valkey-data:/data" "${dir:valkey-data}:/data"
] ]
}, },
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.yml",
"mode": "0600",
"merge": "json",
"content": "{\n \"use_default_settings\": true,\n \"server\": {\n \"secret_key\": \"${secret:secret}\",\n \"base_url\": false,\n \"limiter\": false,\n \"image_proxy\": false,\n \"public_instance\": false\n },\n \"search\": {\n \"formats\": [\"html\", \"json\"]\n },\n \"valkey\": {\n \"url\": \"valkey://valkey:6379/0\"\n }\n}\n"
},
{ {
"id": "server", "id": "server",
"type": "container", "type": "container",
"name": "searxng", "name": "searxng",
"image": "searxng/searxng@sha256:c7cc75852051bf6254afda6ed1b920dd1677d8efe4ab141bf558f02e582f4371", "image": "searxng/searxng@sha256:cd8812607ab73730a0b1a0dc4990223fe1b9e383f6f35947114d0bef7f8bb441",
"network": "searxng", "network": "searxng",
"env-file": [
"/var/lib/searxng-module/server.env"
],
"ports": [ "ports": [
"8080" "8080"
], ],
"secrets-in-environment": "SEARXNG_SECRET is env-only, but settings.yml carries server.secret_key; convertible by mounting a generated settings.yml, not yet done" "volumes": [
"${dir:state}/settings.yml:/etc/searxng/settings.yml:ro"
],
"restart-on": [
"settings"
]
}, },
{ {
"id": "runtime-config", "id": "runtime-config",
"type": "file", "type": "file",
"path": "/var/lib/mesh/searxng/config.json", "path": "/var/lib/mesh/searxng/config.json",
"mode": "0600", "mode": "0600",
"content": "{}\n", "content": "{}\n"
"merge": "json"
}, },
{ {
"id": "runtime", "id": "runtime",
@@ -118,7 +119,7 @@
} }
}, },
"binds": { "binds": {
"route": "/var/lib/searxng-module/route.json" "route": "${dir:state}/route.json"
}, },
"build": { "build": {
"on": [ "on": [