A network-checker module: dial what the mesh claims, from where the callers are

The mesh asserts three things are callable (ADR 0144) — what runs on the same
machine, another machine's service exposed to the private network, and another
machine's service exposed publicly — and has never checked any of them. The first
was broken for eleven hours while the mesh reported every machine healthy.

This runs on every machine, on the cadence the mesh already has, in its own
container: the same position every other module calls from. Not the host and not
the control plane, both of which reach these addresses by paths no ordinary caller
uses and would have passed throughout that outage.

**Its probe is its own endpoint, and that is the point.** Declared reachable over
the private network like any other service, so it is admitted by exactly the rule
that governs every internally-exposed service and fails when that rule is wrong.
The tempting target is a service every machine has, and those are the ones never
closed — ssh above all — which would have passed while the thing that actually
broke was a service exposed to the private network.

It resolves before it dials and says which failed, because a name that does not
resolve and a port that does not answer have different owners. One failure is not
a fault: a machine rebooting is ordinary, so a path is broken after consecutive
runs and the count travels with the result. It reports and repairs nothing.

novox/hq ADR 0145. Eight tests; the consecutive-failure logic proved by reverting
it once. Not yet registered or assigned.

modules/baserow/module.json

@@ -14,7 +14,7 @@
     },
     "route": {
       "label": "baserow",
-      "port": 80
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -30,6 +30,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 80,
       "protocol": "tcp",
       "from": "mesh",

modules/bazarr/module.json

@@ -13,6 +13,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 6767,
       "protocol": "tcp",
       "from": "mesh",
@@ -110,7 +111,7 @@
   "contributes": {
     "route": {
       "label": "subs",
-      "port": 6767
+      "endpoint": "web"
     }
   },
   "binds": {

modules/bookshelf/module.json

@@ -15,6 +15,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8787,
       "protocol": "tcp",
       "from": "mesh",
@@ -87,7 +88,7 @@
   "contributes": {
     "route": {
       "label": "books",
-      "port": 8787
+      "endpoint": "web"
     }
   },
   "binds": {

modules/de-spiegel/module.json

@@ -11,7 +11,7 @@
   "contributes": {
     "route": {
       "label": "de-spiegel",
-      "port": 35621
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -23,6 +23,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 35621,
       "protocol": "tcp",
       "from": "mesh",

modules/distribution/module.json

@@ -29,6 +29,7 @@
   },
   "listens": [
     {
+      "name": "registry",
       "port": 5000,
       "protocol": "tcp",
       "from": "mesh",

modules/dnsmasq/module.json

@@ -22,11 +22,20 @@
   ],
   "listens": [
     {
+      "name": "dns-udp",
       "port": 53,
       "protocol": "udp",
       "from": "mesh",
       "why": "every name for this machine and what it runs \u2014 the mesh's own answered here, the rest forwarded",
       "fixed": true
+    },
+    {
+      "name": "dns-tcp",
+      "port": 53,
+      "protocol": "tcp",
+      "from": "mesh",
+      "why": "the same names over tcp, which a resolver answers on as well and is asked for whenever an answer will not fit in a datagram. Declared because the daemon serves it: a declaration that covers one of the two protocols its own service listens on leaves the other closed while everything reports success",
+      "fixed": true
     }
   ],
   "resources": [

modules/fail2ban/module.json

@@ -28,19 +28,12 @@
       "path": "/etc/fail2ban/action.d",
       "mode": "0755"
     },
-    {
-      "id": "fail2ban-local",
-      "type": "file",
-      "path": "/etc/fail2ban/fail2ban.local",
-      "mode": "0644",
-      "content": "[Definition]\n\n# Where fail2ban writes its own log, declared rather than assumed. The recidive jail reads\n# this file to ban whoever keeps coming back, and the logrotate file this module ships\n# rotates it -- but nothing told fail2ban to write there. Where the package default stands,\n# fail2ban logs to the journal, the recidive jail finds no log file, and the whole service\n# refuses to start, taking the sshd jail with it.\n#\n# In .local, not in fail2ban.conf: that file belongs to the package.\nlogtarget = /var/log/fail2ban.log\n"
-    },
     {
       "id": "jail-local",
       "type": "file",
       "path": "/etc/fail2ban/jail.local",
       "mode": "0644",
-      "content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}\n\nbantime  = 10m\nfindtime = 10m\nmaxretry = 5\n\nbanaction = ufw\nbanaction_allports = iptables-allports\n\n[sshd]\nenabled = true\nport    = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
+      "content": "[INCLUDES]\n\nbefore = paths-arch.conf\n\n[DEFAULT]\n\n# Never act on the machine itself or on a tunnel peer: the mesh's private range is\n# ${machine:mesh-range}, named here rather than written as a value the module cannot\n# know (novox/hq ADR 0112). Without this, fail2ban could ban the mesh's own nodes.\nignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}\n\nbantime  = 10m\nfindtime = 10m\nmaxretry = 5\n\n# Ban through iptables, not through a firewall front-end the machine may not have. ufw is\n# installed on two of this mesh's machines and absent on the other two, and fail2ban finds out\n# only at ban time: the service reports healthy, the jail counts the attempt, the ban command\n# exits 127, and nothing is blocked. Proven on 2026-09-28 -- 'ufw: command not found' on a\n# machine the mesh reported as protected.\n#\n# The action below is this module's own, already used by the recidive jail on every machine\n# here, and it bans in DOCKER-USER as well as INPUT, so a container's published port is\n# covered too.\nbanaction = iptables-allports-dualchain\nbanaction_allports = iptables-allports-dualchain\n\n[sshd]\nenabled = true\nport    = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\n"
     },
     {
       "id": "jail-sshd",
@@ -49,6 +42,14 @@
       "mode": "0644",
       "content": "[sshd]\nenabled = true\nport    = ssh\nlogpath = %(sshd_log)s\nbackend = %(sshd_backend)s\nmaxretry = 5\n"
     },
+    {
+      "id": "log",
+      "type": "file",
+      "path": "/var/log/fail2ban.log",
+      "mode": "0640",
+      "create-once": true,
+      "content": ""
+    },
     {
       "id": "jail-recidive",
       "type": "file",

modules/gitea/module.json

@@ -13,7 +13,7 @@
     "route": {
       "web": {
         "label": "git",
-        "port": 3000
+        "endpoint": "web"
       },
       "internal-api-refused": {
         "label": "git",
@@ -44,12 +44,14 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 3000,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the forge, over http"
     },
     {
+      "name": "ssh",
       "port": 22,
       "protocol": "tcp",
       "from": "mesh",

modules/grafana/module.json

@@ -13,6 +13,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 3000,
       "protocol": "tcp",
       "from": "mesh",
@@ -96,7 +97,7 @@
   "contributes": {
     "route": {
       "label": "grafana",
-      "port": 3000
+      "endpoint": "web"
     }
   },
   "binds": {

modules/hello-web/module.json

@@ -11,7 +11,7 @@
   "contributes": {
     "route": {
       "label": "hello",
-      "port": 8080
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -19,6 +19,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",

modules/home-assistant/module.json

@@ -14,6 +14,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8123,
       "protocol": "tcp",
       "from": "mesh",
@@ -85,7 +86,7 @@
   "contributes": {
     "route": {
       "label": "home-assistant",
-      "port": 8123
+      "endpoint": "web"
     }
   },
   "binds": {

modules/icecast/module.json

@@ -13,6 +13,7 @@
   },
   "listens": [
     {
+      "name": "stream",
       "port": 8000,
       "protocol": "tcp",
       "from": "mesh",

modules/influxdb/module.json

@@ -9,6 +9,7 @@
   },
   "listens": [
     {
+      "name": "api",
       "port": 8086,
       "protocol": "tcp",
       "from": "mesh",

modules/invoicing/module.json

@@ -17,11 +17,11 @@
     "route": {
       "site": {
         "label": "invoicing",
-        "port": 80
+        "endpoint": "web"
       },
       "api": {
         "label": "invoicing-api",
-        "port": 9000
+        "endpoint": "api"
       }
     }
   },
@@ -36,12 +36,14 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 80,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the invoicing web frontend; a public name is a route grant later"
     },
     {
+      "name": "api",
       "port": 9000,
       "protocol": "tcp",
       "from": "mesh",

modules/jackett/module.json

@@ -6,6 +6,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 9117,
       "protocol": "tcp",
       "from": "mesh",
@@ -82,7 +83,7 @@
   "contributes": {
     "route": {
       "label": "indexers",
-      "port": 9117
+      "endpoint": "web"
     }
   },
   "binds": {

modules/keycloak/module.json

@@ -11,7 +11,7 @@
     },
     "route": {
       "label": "keycloak",
-      "port": 8080
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -34,6 +34,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",

modules/letta/module.json

@@ -24,6 +24,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8283,
       "protocol": "tcp",
       "from": "mesh",

modules/lidarr/module.json

@@ -14,6 +14,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8686,
       "protocol": "tcp",
       "from": "mesh",
@@ -86,7 +87,7 @@
   "contributes": {
     "route": {
       "label": "lidarr",
-      "port": 8686
+      "endpoint": "web"
     }
   },
   "binds": {

modules/mailu/module.json

@@ -16,27 +16,27 @@
     "route": {
       "web": {
         "label": "mail",
-        "port": 7443,
+        "endpoint": "web-tls",
         "scheme": "https",
         "insecure": true
       },
       "acme": {
         "label": "mail",
         "path": "/.well-known/acme-challenge",
-        "port": 7080,
+        "endpoint": "web",
         "priority": 100
       },
       "autoconfig": {
         "label": "autoconfig",
-        "port": 4243
+        "endpoint": "autoconfig"
       },
       "autodiscover": {
         "label": "autodiscover",
-        "port": 4243
+        "endpoint": "autoconfig"
       },
       "automx": {
         "label": "automx",
-        "port": 4243
+        "endpoint": "autoconfig"
       }
     }
   },
@@ -60,6 +60,7 @@
   ],
   "listens": [
     {
+      "name": "smtp",
       "port": 25,
       "protocol": "tcp",
       "from": "anywhere",
@@ -67,6 +68,7 @@
       "fixed": true
     },
     {
+      "name": "pop3",
       "port": 110,
       "protocol": "tcp",
       "from": "anywhere",
@@ -74,6 +76,7 @@
       "fixed": true
     },
     {
+      "name": "imap",
       "port": 143,
       "protocol": "tcp",
       "from": "anywhere",
@@ -81,6 +84,7 @@
       "fixed": true
     },
     {
+      "name": "smtps",
       "port": 465,
       "protocol": "tcp",
       "from": "anywhere",
@@ -88,6 +92,7 @@
       "fixed": true
     },
     {
+      "name": "submission",
       "port": 587,
       "protocol": "tcp",
       "from": "anywhere",
@@ -95,6 +100,7 @@
       "fixed": true
     },
     {
+      "name": "imaps",
       "port": 993,
       "protocol": "tcp",
       "from": "anywhere",
@@ -102,6 +108,7 @@
       "fixed": true
     },
     {
+      "name": "pop3s",
       "port": 995,
       "protocol": "tcp",
       "from": "anywhere",
@@ -109,18 +116,21 @@
       "fixed": true
     },
     {
+      "name": "web",
       "port": 7080,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the web front over http; only the ACME HTTP-01 passthrough is routed here \u2014 everything else 301s to https and would loop a proxy"
     },
     {
+      "name": "web-tls",
       "port": 7443,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the web front over its own TLS (admin, webmail, API); the public name mail.novox.be is a route grant reaching it here"
     },
     {
+      "name": "autoconfig",
       "port": 4243,
       "protocol": "tcp",
       "from": "mesh",

modules/marrytts/module.json

@@ -6,6 +6,7 @@
   ],
   "listens": [
     {
+      "name": "api",
       "port": 59125,
       "protocol": "tcp",
       "from": "mesh",

modules/minio/module.json

@@ -14,11 +14,11 @@
     "route": {
       "api": {
         "label": "files-api",
-        "port": 9000
+        "endpoint": "s3"
       },
       "console": {
         "label": "files",
-        "port": 9001
+        "endpoint": "console"
       }
     }
   },
@@ -31,12 +31,14 @@
   ],
   "listens": [
     {
+      "name": "s3",
       "port": 9000,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the S3 endpoint"
     },
     {
+      "name": "console",
       "port": 9001,
       "protocol": "tcp",
       "from": "mesh",

modules/mongodb/module.json

@@ -20,6 +20,7 @@
   ],
   "listens": [
     {
+      "name": "database",
       "port": 27017,
       "protocol": "tcp",
       "from": "mesh",

modules/mosquitto/module.json

@@ -34,12 +34,14 @@
   },
   "listens": [
     {
+      "name": "mqtt",
       "port": 1883,
       "protocol": "tcp",
       "from": "mesh",
       "why": "modules on any machine that were granted a topic namespace"
     },
     {
+      "name": "mqtt-websockets",
       "port": 8081,
       "protocol": "tcp",
       "from": "mesh",

modules/mssql/module.json

@@ -20,6 +20,7 @@
   ],
   "listens": [
     {
+      "name": "database",
       "port": 4848,
       "protocol": "tcp",
       "from": "mesh",

modules/n8n/module.json

@@ -14,7 +14,7 @@
     },
     "route": {
       "label": "n8n",
-      "port": 5682
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -29,6 +29,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 5682,
       "protocol": "tcp",
       "from": "mesh",

modules/nats/module.json

@@ -21,6 +21,7 @@
   "consumes": [],
   "listens": [
     {
+      "name": "bus",
       "port": 4222,
       "protocol": "tcp",
       "from": "mesh",

modules/network-checker/check/index.ts

@@ -0,0 +1,84 @@
+// Dial everything the mesh claims is reachable, and say what was found (novox/hq ADR 0145).
+//
+// Runs on a cadence, from this machine, in this module's own container — the same position every other
+// module on the machine calls from. That is the whole point: a check run by the host or by the control
+// plane reaches these addresses by a path no ordinary caller uses, and would have passed throughout the
+// outage that produced this module (novox/hq 04-ISSUES/145).
+//
+// It reports and does nothing else. A checker that repaired things would be a second control plane.
+
+import { readFileSync, writeFileSync, mkdirSync, renameSync } from "node:fs";
+import { dirname, join } from "node:path";
+
+import { dial, tally, targetsFor, type Counts, type Result, type Roster } from "../reach.js";
+
+/** Where the mesh renders this machine's view of the others, and where the counts are kept between runs. */
+const rosterFile = process.env.MESH_NETWORK_CHECKER_ROSTER ?? "/run/config/roster.json";
+const stateDir = process.env.MESH_NETWORK_CHECKER_STATE ?? "/run/state";
+const probePort = Number(process.env.MESH_NETWORK_CHECKER_PORT ?? "9876");
+const publicPort = process.env.MESH_NETWORK_CHECKER_PUBLIC_PORT
+  ? Number(process.env.MESH_NETWORK_CHECKER_PUBLIC_PORT)
+  : undefined;
+const timeoutMs = Number(process.env.MESH_NETWORK_CHECKER_TIMEOUT_MS ?? "4000");
+const threshold = Number(process.env.MESH_NETWORK_CHECKER_THRESHOLD ?? "2");
+
+/** read is a JSON file or a stated failure — never a silent default, which is how a checker comes to
+ *  report that everything is fine because it read nothing. */
+function read<T>(path: string, whenMissing: T | null): T {
+  try {
+    return JSON.parse(readFileSync(path, "utf8")) as T;
+  } catch (err) {
+    if (whenMissing !== null) return whenMissing;
+    console.error(`network-checker: cannot read ${path}: ${(err as Error).message}`);
+    process.exit(1);
+  }
+}
+
+function writeAtomically(path: string, body: string): void {
+  mkdirSync(dirname(path), { recursive: true });
+  const temp = `${path}.writing`;
+  writeFileSync(temp, body);
+  renameSync(temp, path);
+}
+
+async function main(): Promise<void> {
+  const roster = read<Roster>(rosterFile, null);
+  if (!roster.machines?.length) {
+    console.error("network-checker: the roster names no machines; nothing to check");
+    process.exit(1);
+  }
+
+  const targets = targetsFor(roster, probePort, publicPort);
+  // In parallel, because a machine that is away should not delay the rest: a run that takes
+  // machines × timeout would outlast its own cadence on a mesh of any size.
+  const results: Result[] = await Promise.all(targets.map((t) => dial(t, timeoutMs)));
+
+  const countsFile = join(stateDir, "consecutive.json");
+  const { counts, broken } = tally(results, read<Counts>(countsFile, {}), threshold);
+  writeAtomically(countsFile, JSON.stringify(counts, null, 1));
+
+  // Written whole, every run: a reader asking "what does this machine reach" gets an answer about now
+  // rather than the last time something changed.
+  writeAtomically(join(stateDir, "reach.json"), JSON.stringify({
+    node: roster.node,
+    at: new Date().toISOString(),
+    checked: results.length,
+    broken: broken.length,
+    results,
+  }, null, 1));
+
+  for (const b of broken) {
+    console.error(
+      `network-checker: ${roster.node} cannot reach ${b.machine} (${b.claim}) at ${b.at}:${b.port} — ` +
+      `${b.failed} failed${b.detail ? `: ${b.detail}` : ""}, ${b.consecutive} run(s) running`);
+  }
+  if (broken.length === 0) {
+    console.log(`network-checker: ${roster.node} reaches all ${results.length} checked path(s)`);
+  }
+
+  // A broken path is not this process failing. It did its job; exiting non-zero would make the mesh
+  // read the checker as the fault, and a scheduled step that fails is retried rather than believed.
+  process.exit(0);
+}
+
+void main();

modules/network-checker/module.json

@@ -0,0 +1,61 @@
+{
+  "module": "network-checker",
+  "version": "1",
+  "slug": "netcheck",
+  "listens": [
+    {
+      "name": "probe",
+      "port": 9876,
+      "protocol": "tcp",
+      "from": "mesh",
+      "why": "what the other machines' checkers dial. Deliberately this module's own endpoint and nothing else's: it is admitted by exactly the rule that governs every internally-exposed service, so it fails when that rule is wrong. A probe on a port that is never closed — ssh, say — would have passed throughout the outage this module exists to catch (novox/hq ADR 0145)"
+    }
+  ],
+  "facts": {
+    "roster": {
+      "path": "/var/lib/network-checker/roster.json",
+      "template": "{\n \"generated\": \"by the mesh — do not edit; replaced whenever a machine joins or leaves\",\n \"node\": \"{{.Node}}\",\n \"machines\": [{{range $i, $m := .Machines}}{{if $i}},{{end}}\n  { \"name\": \"{{$m.Name}}\", \"fqdn\": \"{{$m.FQDN}}\", \"address\": \"{{$m.Address}}\" }{{end}}\n ]\n}\n"
+    }
+  },
+  "build": {
+    "artifacts": [
+      {
+        "name": "code",
+        "kind": "bundle",
+        "language": "typescript",
+        "entrypoints": ["probe/index.js", "check/index.js"]
+      }
+    ]
+  },
+  "resources": [
+    {
+      "id": "state",
+      "type": "directory",
+      "path": "/var/lib/network-checker",
+      "mode": "0700"
+    },
+    {
+      "id": "probe",
+      "type": "container",
+      "name": "mesh-network-checker-probe",
+      "args": ["run", "/app/modules/network-checker/dist/probe/index.js"],
+      "ports": ["9876"],
+      "state": "running",
+      "env": { "MESH_NETWORK_CHECKER_PORT": "9876" }
+    },
+    {
+      "id": "check",
+      "type": "container",
+      "name": "mesh-network-checker-check",
+      "network": "host",
+      "schedule": "*/5 * * * *",
+      "args": ["run", "/app/modules/network-checker/dist/check/index.js"],
+      "volumes": ["/var/lib/network-checker:/run/state"],
+      "env": {
+        "MESH_NETWORK_CHECKER_ROSTER": "/run/state/roster.json",
+        "MESH_NETWORK_CHECKER_STATE": "/run/state",
+        "MESH_NETWORK_CHECKER_PORT": "${port:9876}"
+      }
+    }
+  ]
+}

modules/network-checker/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "@novox/module-network-checker",
+  "version": "0.1.0",
+  "description": "network-checker — dials what the mesh claims is reachable, from where the callers are, and says what it found.",
+  "type": "module",
+  "private": true,
+  "devDependencies": { "@types/node": "^22.0.0", "typescript": "^5.6.0" }
+}

modules/network-checker/probe/index.ts

@@ -0,0 +1,31 @@
+// The endpoint the other machines' checkers dial (novox/hq ADR 0145).
+//
+// **This module's own endpoint is the instrument.** It is declared reachable over the private network
+// like any other service, so it is admitted by exactly the rule that governs every internally-exposed
+// service and it fails when that rule is wrong. A probe on a port that is never closed — ssh, say —
+// would have passed throughout the outage this module exists to catch.
+//
+// It accepts a connection and closes it. Answering anything would make this a protocol, and then the
+// question would be whether the protocol worked rather than whether the path did.
+
+import { createServer } from "node:net";
+
+const port = Number(process.env.MESH_NETWORK_CHECKER_PORT ?? "9876");
+
+const server = createServer((socket) => {
+  // Written before closing so a person dialling it by hand sees something, and so a half-open
+  // connection is not mistaken for a working path by a client that only checks the handshake.
+  socket.end("mesh network-checker\n");
+});
+
+server.on("error", (err: Error) => {
+  // Said and fatal: a probe that cannot listen must not look like a probe that nothing dialled.
+  console.error(`network-checker: cannot serve the probe on ${port}: ${err.message}`);
+  process.exit(1);
+});
+
+server.listen(port, () => console.log(`network-checker: probe listening on ${port}`));
+
+for (const signal of ["SIGTERM", "SIGINT"] as const) {
+  process.on(signal, () => server.close(() => process.exit(0)));
+}

modules/network-checker/reach.ts

@@ -0,0 +1,155 @@
+// What the mesh claims is reachable, and how to find out (novox/hq ADR 0145).
+//
+// The mesh asserts three things are callable (ADR 0144): what runs on the same machine, another
+// machine's service exposed to the private network, and another machine's service exposed publicly.
+// This decides what to dial for each and reads the answers. It opens connections and nothing more —
+// the module that owns a service is the one that knows whether it is working.
+//
+// **The target is this module's own endpoint, and that is deliberate.** The obvious thing to dial is a
+// service every machine has, and the services every machine has are the ones never closed — ssh above
+// all. Dialling one of those would have passed throughout the outage this exists to catch, because what
+// broke was a service exposed to the private network and ssh is admitted unconditionally. A probe on a
+// port that cannot fail measures nothing.
+
+import { connect } from "node:net";
+import { lookup } from "node:dns";
+
+/** One machine as the mesh's roster describes it. */
+export interface Machine {
+  name: string;
+  fqdn: string;
+  address: string;
+  /** The name this machine is reached by from outside, where it has one. Absent for most machines, and
+   *  a machine with no public face has no public claim to check. */
+  public?: string;
+}
+
+/** The roster the mesh renders for this module: who this machine is, and who the others are. */
+export interface Roster {
+  node: string;
+  machines: Machine[];
+}
+
+/** Which of the mesh's three claims a check is about, so a failure says which one broke. */
+export type Claim = "this machine" | "the private network" | "the public network";
+
+/** One thing to dial. */
+export interface Target {
+  claim: Claim;
+  machine: string;
+  /** What to dial — a name where the point is that names resolve, an address where it is not. */
+  at: string;
+  port: number;
+  /** Whether `at` is a name that must resolve first, so a resolution failure is reported as one. */
+  byName: boolean;
+}
+
+/** What one dial found. */
+export interface Result extends Target {
+  ok: boolean;
+  /** Which step failed, so a reader is sent to the right place: the resolver, or the filter. */
+  failed?: "resolution" | "connection";
+  detail?: string;
+  ms: number;
+}
+
+/**
+ * targetsFor is everything this machine should be able to reach, from the roster it was given.
+ *
+ * Its own machine first, because that is the case that distinguishes a caller on the machine from a
+ * caller in one of its containers — the one that broke. Then every other machine over the private
+ * network. The public claim is only checked where a public address is known for a machine, because a
+ * machine with no public face has nothing to fail.
+ */
+export function targetsFor(roster: Roster, probePort: number, publicPort?: number): Target[] {
+  const out: Target[] = [];
+  for (const m of roster.machines) {
+    const own = m.name === roster.node;
+    out.push({
+      claim: own ? "this machine" : "the private network",
+      machine: m.name,
+      at: m.address,
+      port: probePort,
+      byName: false,
+    });
+    // And by name, because a name that does not resolve and a port that does not answer are different
+    // faults with different owners.
+    out.push({
+      claim: own ? "this machine" : "the private network",
+      machine: m.name,
+      at: m.fqdn,
+      port: probePort,
+      byName: true,
+    });
+  }
+  if (publicPort !== undefined) {
+    for (const m of roster.machines) {
+      if (!m.public) continue;
+      out.push({
+        claim: "the public network",
+        machine: m.name,
+        at: m.public,
+        port: publicPort,
+        byName: true,
+      });
+    }
+  }
+  return out;
+}
+
+/** dial opens a connection and closes it. Whether the port accepts is the whole of what is asked. */
+export function dial(target: Target, timeoutMs: number): Promise<Result> {
+  const began = Date.now();
+  const done = (ok: boolean, failed?: Result["failed"], detail?: string): Result => ({
+    ...target, ok, failed, detail, ms: Date.now() - began,
+  });
+
+  return new Promise<Result>((resolve) => {
+    const open = () => {
+      const socket = connect({ host: target.at, port: target.port });
+      const finish = (r: Result) => { socket.destroy(); resolve(r); };
+      socket.setTimeout(timeoutMs);
+      socket.once("connect", () => finish(done(true)));
+      socket.once("timeout", () => finish(done(false, "connection", "timed out")));
+      socket.once("error", (err: Error) => finish(done(false, "connection", err.message)));
+    };
+
+    if (!target.byName) { open(); return; }
+    // Resolved first and reported separately: a checker that says "unreachable" for a name the
+    // resolver never answered sends a reader to the filter, which is not where the fault is.
+    lookup(target.at, (err) => {
+      if (err) { resolve(done(false, "resolution", err.message)); return; }
+      open();
+    });
+  });
+}
+
+/** A path's running count of consecutive failures, keyed so it survives between runs. */
+export type Counts = Record<string, number>;
+
+/** keyOf names one path, stably, so a count follows it across runs. */
+export function keyOf(t: Target): string {
+  return `${t.claim}|${t.machine}|${t.at}|${t.port}`;
+}
+
+/**
+ * tally folds this run's results into the counts carried from the last one.
+ *
+ * **One failure is not a fault.** A machine rebooting is ordinary, and a checker that cries at the
+ * first missed dial trains a reader to ignore it — which is worse than not checking (ADR 0145). A path
+ * is broken once it has failed on consecutive runs, and the count travels with the result so a reader
+ * can tell "briefly away" from "never worked".
+ */
+export function tally(results: Result[], before: Counts, threshold: number): {
+  counts: Counts; broken: Array<Result & { consecutive: number }>;
+} {
+  const counts: Counts = {};
+  const broken: Array<Result & { consecutive: number }> = [];
+  for (const r of results) {
+    const key = keyOf(r);
+    const n = r.ok ? 0 : (before[key] ?? 0) + 1;
+    if (n > 0) counts[key] = n;
+    if (n >= threshold) broken.push({ ...r, consecutive: n });
+  }
+  return { counts, broken };
+}

modules/network-checker/test/reach.test.ts

@@ -0,0 +1,72 @@
+import { strict as assert } from "node:assert";
+import test from "node:test";
+
+import { keyOf, tally, targetsFor, type Result, type Roster } from "../reach.js";
+
+const roster: Roster = {
+  node: "here",
+  machines: [
+    { name: "here", fqdn: "here.internal", address: "10.0.0.1" },
+    { name: "there", fqdn: "there.internal", address: "10.0.0.2", public: "there.example.test" },
+  ],
+};
+
+test("its own machine is checked, which is the case that distinguishes a caller on it from one in a container", () => {
+  const own = targetsFor(roster, 9876).filter((t) => t.claim === "this machine");
+  assert.equal(own.length, 2, "its own machine by address and by name");
+  assert.ok(own.some((t) => t.at === "10.0.0.1" && !t.byName));
+  assert.ok(own.some((t) => t.at === "here.internal" && t.byName));
+});
+
+test("every other machine is checked over the private network", () => {
+  const other = targetsFor(roster, 9876).filter((t) => t.claim === "the private network");
+  assert.deepEqual(other.map((t) => t.machine), ["there", "there"]);
+});
+
+test("the public claim is only checked where a machine has a public name", () => {
+  const pub = targetsFor(roster, 9876, 443).filter((t) => t.claim === "the public network");
+  assert.equal(pub.length, 1, "only the machine with a public name");
+  assert.equal(pub[0]!.at, "there.example.test");
+  assert.equal(pub[0]!.port, 443);
+});
+
+test("no public claim is made when no public port was given", () => {
+  assert.equal(targetsFor(roster, 9876).filter((t) => t.claim === "the public network").length, 0);
+});
+
+const failed = (at: string): Result => ({
+  claim: "this machine", machine: "here", at, port: 9876, byName: false,
+  ok: false, failed: "connection", ms: 1,
+});
+const passed = (at: string): Result => ({
+  claim: "this machine", machine: "here", at, port: 9876, byName: false, ok: true, ms: 1,
+});
+
+test("one failure is not a fault — a machine rebooting is ordinary", () => {
+  const { counts, broken } = tally([failed("10.0.0.1")], {}, 2);
+  assert.equal(broken.length, 0, "one missed dial says nothing");
+  assert.equal(counts[keyOf(failed("10.0.0.1"))], 1, "and is remembered");
+});
+
+test("a path that keeps failing is broken, and the count travels with it", () => {
+  const first = tally([failed("10.0.0.1")], {}, 2);
+  const second = tally([failed("10.0.0.1")], first.counts, 2);
+  assert.equal(second.broken.length, 1);
+  assert.equal(second.broken[0]!.consecutive, 2, "so a reader can tell briefly away from never worked");
+});
+
+test("a path that recovers stops being counted", () => {
+  const first = tally([failed("10.0.0.1")], {}, 2);
+  const second = tally([passed("10.0.0.1")], first.counts, 2);
+  assert.equal(second.broken.length, 0);
+  assert.deepEqual(second.counts, {}, "nothing carried forward for a path that works");
+});
+
+test("a count follows one path and not another", () => {
+  const a = failed("10.0.0.1");
+  const b = failed("10.0.0.2");
+  const first = tally([a, b], {}, 2);
+  const second = tally([a], first.counts, 2);
+  assert.equal(second.broken.length, 1, "only the path dialled this run is judged");
+  assert.equal(second.broken[0]!.at, "10.0.0.1");
+});

modules/network-checker/tsconfig.json

@@ -0,0 +1,12 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "noEmit": true
+  },
+  "include": ["reach.ts", "probe/index.ts", "check/index.ts", "test/*.ts"]
+}

modules/nextcloud/module.json

@@ -13,7 +13,7 @@
     },
     "route": {
       "label": "drive",
-      "port": 80
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -38,6 +38,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 80,
       "protocol": "tcp",
       "from": "mesh",

modules/nodered/module.json

@@ -12,6 +12,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 1880,
       "protocol": "tcp",
       "from": "mesh",
@@ -81,7 +82,7 @@
   "contributes": {
     "route": {
       "label": "nodered",
-      "port": 1880
+      "endpoint": "web"
     }
   },
   "binds": {

modules/novox.be/module.json

@@ -10,7 +10,7 @@
   "contributes": {
     "route": {
       "label": "@",
-      "port": 4000
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -18,6 +18,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 4000,
       "protocol": "tcp",
       "from": "mesh",

modules/nzbget/module.json

@@ -15,6 +15,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 6789,
       "protocol": "tcp",
       "from": "mesh",

modules/ollama/module.json

@@ -12,6 +12,7 @@
   ],
   "listens": [
     {
+      "name": "api",
       "port": 11434,
       "protocol": "tcp",
       "from": "machine",

modules/ombi/module.json

@@ -14,6 +14,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 3579,
       "protocol": "tcp",
       "from": "mesh",
@@ -89,7 +90,7 @@
   "contributes": {
     "route": {
       "label": "ombi",
-      "port": 3579
+      "endpoint": "web"
     }
   },
   "binds": {

modules/only-office/module.json

@@ -11,7 +11,7 @@
   "contributes": {
     "route": {
       "label": "office",
-      "port": 9070
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -22,6 +22,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 9070,
       "protocol": "tcp",
       "from": "mesh",

modules/photos-eef/module.json

@@ -11,7 +11,7 @@
   "contributes": {
     "route": {
       "label": "eef",
-      "port": 4012
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -19,6 +19,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 4012,
       "protocol": "tcp",
       "from": "mesh",

modules/photos-filip/module.json

@@ -11,7 +11,7 @@
   "contributes": {
     "route": {
       "label": "filip",
-      "port": 4013
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -19,6 +19,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 4013,
       "protocol": "tcp",
       "from": "mesh",

modules/photos/module.json

@@ -18,7 +18,7 @@
     },
     "route": {
       "label": "photos",
-      "port": 4001
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -32,12 +32,14 @@
   },
   "listens": [
     {
+      "name": "api",
       "port": 9000,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the photos backend API; the client sites on the module network call it"
     },
     {
+      "name": "web",
       "port": 4001,
       "protocol": "tcp",
       "from": "mesh",

modules/plex/module.json

@@ -18,6 +18,7 @@
   },
   "listens": [
     {
+      "name": "stream",
       "port": 32400,
       "protocol": "tcp",
       "from": "mesh",

modules/portainer/module.json

@@ -7,12 +7,14 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 9090,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the dashboard over http; portainer.novox.be is a route grant and the proxy reaches it here \u2014 the machine side of 9090:9000, the predecessor's number"
     },
     {
+      "name": "web-tls",
       "port": 9443,
       "protocol": "tcp",
       "from": "mesh",
@@ -103,7 +105,7 @@
   "contributes": {
     "route": {
       "label": "portainer",
-      "port": 9090
+      "endpoint": "web"
     }
   },
   "binds": {

modules/postgres/module.json

@@ -26,6 +26,7 @@
   ],
   "listens": [
     {
+      "name": "database",
       "port": 5432,
       "protocol": "tcp",
       "from": "mesh",

modules/qbittorrent/module.json

@@ -16,6 +16,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",

modules/radarr/module.json

@@ -14,6 +14,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 7878,
       "protocol": "tcp",
       "from": "mesh",
@@ -86,7 +87,7 @@
   "contributes": {
     "route": {
       "label": "movies",
-      "port": 7878
+      "endpoint": "web"
     }
   },
   "binds": {

modules/redis/module.json

@@ -40,6 +40,7 @@
   },
   "listens": [
     {
+      "name": "cache",
       "port": 6379,
       "protocol": "tcp",
       "from": "mesh",

modules/route-proxy/module.json

@@ -27,12 +27,14 @@
   },
   "listens": [
     {
+      "name": "http",
       "port": 80,
       "protocol": "tcp",
       "from": "anywhere",
       "why": "public HTTP, and the ACME HTTP-01 challenge answered at the name being certified"
     },
     {
+      "name": "https",
       "port": 443,
       "protocol": "tcp",
       "from": "anywhere",

modules/searxng/module.json

@@ -10,6 +10,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",
@@ -113,7 +114,7 @@
   "contributes": {
     "route": {
       "label": "searxng",
-      "port": 8080
+      "endpoint": "web"
     }
   },
   "binds": {

modules/showcase/module.json

@@ -43,6 +43,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",

modules/sonarr/module.json

@@ -14,6 +14,7 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 8989,
       "protocol": "tcp",
       "from": "mesh",
@@ -91,7 +92,7 @@
   "contributes": {
     "route": {
       "label": "series",
-      "port": 8989
+      "endpoint": "web"
     }
   },
   "binds": {

modules/sshd/module.json

@@ -7,6 +7,7 @@
   ],
   "listens": [
     {
+      "name": "ssh",
       "port": 22,
       "protocol": "tcp",
       "from": "anywhere",
@@ -31,6 +32,7 @@
       "type": "service",
       "unit": "sshd.service",
       "state": "running",
+      "boot": "enabled",
       "restart-on": [
         "config"
       ]

modules/step-ca/module.json

@@ -26,6 +26,7 @@
   },
   "listens": [
     {
+      "name": "acme",
       "port": 9000,
       "protocol": "tcp",
       "from": "mesh",

modules/tautulli/module.json

@@ -12,6 +12,7 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 8181,
       "protocol": "tcp",
       "from": "mesh",
@@ -85,7 +86,7 @@
   "contributes": {
     "route": {
       "label": "tautulli",
-      "port": 8181
+      "endpoint": "web"
     }
   },
   "binds": {

modules/umami/module.json

@@ -15,7 +15,7 @@
     },
     "route": {
       "label": "umami",
-      "port": 3000
+      "endpoint": "web"
     }
   },
   "binds": {
@@ -49,10 +49,11 @@
   },
   "listens": [
     {
+      "name": "web",
       "port": 3000,
       "protocol": "tcp",
-      "from": "anywhere",
-      "why": "one port serves two surfaces: the dashboard (the proxy gates it to the mesh) and the public collection endpoint that the browsers of every tracked site POST to \u2014 so the port itself must be reachable from anywhere"
+      "from": "mesh",
+      "why": "one port serves two surfaces \u2014 the dashboard and the collection endpoint that the browsers of every tracked site POST to. Both are reached through the proxy, by name, so the port is how the proxy reaches this module and nothing else (novox/hq ADR 0045). It said \"anywhere\" and gave the reason that the collection endpoint must be public, which is true of the name and not of the port: opened, the machine-side port served the dashboard over plain HTTP to the internet, bypassing every rule the proxy applies by path"
     }
   ],
   "resources": [

modules/unifi/module.json

@@ -6,54 +6,63 @@
   ],
   "listens": [
     {
+      "name": "web",
       "port": 8443,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the controller web UI, over its own self-signed tls; reaching it from outside is a route grant later"
     },
     {
+      "name": "inform",
       "port": 8080,
       "protocol": "tcp",
       "from": "mesh",
       "why": "device inform \u2014 how APs and switches check in and are adopted"
     },
     {
+      "name": "stun",
       "port": 3478,
       "protocol": "udp",
       "from": "mesh",
       "why": "STUN, so managed devices can find the controller through NAT"
     },
     {
+      "name": "discovery",
       "port": 10001,
       "protocol": "udp",
       "from": "mesh",
       "why": "device discovery \u2014 the controller finds unadopted devices on the network"
     },
     {
+      "name": "discovery-l2",
       "port": 1902,
       "protocol": "udp",
       "from": "mesh",
       "why": "layer-2 (UBNT) discovery broadcasts; published on 1902, the container listens on 1900"
     },
     {
+      "name": "portal-tls",
       "port": 8843,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the guest captive portal over https"
     },
     {
+      "name": "portal",
       "port": 8880,
       "protocol": "tcp",
       "from": "mesh",
       "why": "the guest captive portal over http"
     },
     {
+      "name": "speedtest",
       "port": 6789,
       "protocol": "tcp",
       "from": "mesh",
       "why": "mobile-app speed-test throughput measurement"
     },
     {
+      "name": "syslog",
       "port": 5514,
       "protocol": "udp",
       "from": "mesh",