jschoubben
784a5a6514
The mesh asserts three things are callable (ADR 0144) — what runs on the same machine, another machine's service exposed to the private network, and another machine's service exposed publicly — and has never checked any of them. The first was broken for eleven hours while the mesh reported every machine healthy. This runs on every machine, on the cadence the mesh already has, in its own container: the same position every other module calls from. Not the host and not the control plane, both of which reach these addresses by paths no ordinary caller uses and would have passed throughout that outage. **Its probe is its own endpoint, and that is the point.** Declared reachable over the private network like any other service, so it is admitted by exactly the rule that governs every internally-exposed service and fails when that rule is wrong. The tempting target is a service every machine has, and those are the ones never closed — ssh above all — which would have passed while the thing that actually broke was a service exposed to the private network. It resolves before it dials and says which failed, because a name that does not resolve and a port that does not answer have different owners. One failure is not a fault: a machine rebooting is ordinary, so a path is broken after consecutive runs and the count travels with the result. It reports and repairs nothing. novox/hq ADR 0145. Eight tests; the consecutive-failure logic proved by reverting it once. Not yet registered or assigned.
62 lines
2.1 KiB
JSON
62 lines
2.1 KiB
JSON
{
|
|
"module": "network-checker",
|
|
"version": "1",
|
|
"slug": "netcheck",
|
|
"listens": [
|
|
{
|
|
"name": "probe",
|
|
"port": 9876,
|
|
"protocol": "tcp",
|
|
"from": "mesh",
|
|
"why": "what the other machines' checkers dial. Deliberately this module's own endpoint and nothing else's: it is admitted by exactly the rule that governs every internally-exposed service, so it fails when that rule is wrong. A probe on a port that is never closed — ssh, say — would have passed throughout the outage this module exists to catch (novox/hq ADR 0145)"
|
|
}
|
|
],
|
|
"facts": {
|
|
"roster": {
|
|
"path": "/var/lib/network-checker/roster.json",
|
|
"template": "{\n \"generated\": \"by the mesh — do not edit; replaced whenever a machine joins or leaves\",\n \"node\": \"{{.Node}}\",\n \"machines\": [{{range $i, $m := .Machines}}{{if $i}},{{end}}\n { \"name\": \"{{$m.Name}}\", \"fqdn\": \"{{$m.FQDN}}\", \"address\": \"{{$m.Address}}\" }{{end}}\n ]\n}\n"
|
|
}
|
|
},
|
|
"build": {
|
|
"artifacts": [
|
|
{
|
|
"name": "code",
|
|
"kind": "bundle",
|
|
"language": "typescript",
|
|
"entrypoints": ["probe/index.js", "check/index.js"]
|
|
}
|
|
]
|
|
},
|
|
"resources": [
|
|
{
|
|
"id": "state",
|
|
"type": "directory",
|
|
"path": "/var/lib/network-checker",
|
|
"mode": "0700"
|
|
},
|
|
{
|
|
"id": "probe",
|
|
"type": "container",
|
|
"name": "mesh-network-checker-probe",
|
|
"args": ["run", "/app/modules/network-checker/dist/probe/index.js"],
|
|
"ports": ["9876"],
|
|
"state": "running",
|
|
"env": { "MESH_NETWORK_CHECKER_PORT": "9876" }
|
|
},
|
|
{
|
|
"id": "check",
|
|
"type": "container",
|
|
"name": "mesh-network-checker-check",
|
|
"network": "host",
|
|
"schedule": "*/5 * * * *",
|
|
"args": ["run", "/app/modules/network-checker/dist/check/index.js"],
|
|
"volumes": ["/var/lib/network-checker:/run/state"],
|
|
"env": {
|
|
"MESH_NETWORK_CHECKER_ROSTER": "/run/state/roster.json",
|
|
"MESH_NETWORK_CHECKER_STATE": "/run/state",
|
|
"MESH_NETWORK_CHECKER_PORT": "${port:9876}"
|
|
}
|
|
}
|
|
]
|
|
}
|