mesh-watcher
The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): what tells the operator when the parts that would tell them are what failed.
- Assigned to one machine that is not the control node. It reads where the controller and the
bus run (
mesh-controller.seats) every ten minutes; on the same machine its status saysMISPLACED. - Watches two signals.
- The self-check: the controller's
doctorheartbeat,mesh-controller.doctor-heartbeat. Bound: twice the interval the heartbeat says doctor runs at (five minutes when it says none). Heard by the time the heartbeat says it was made, so a backlog delivered after a restart is not a sign of life. The heartbeat is read in one place,cmd/mesh-watcher/heartbeat.go, which states every assumption it makes about its shape. - The bus: a round trip through the bus server — its own
watcher_pingon its own machine — every minute. Bound: three minutes.
- The self-check: the controller's
- Silent past its bound: it sends to Telegram directly over HTTPS, not through the bus, once; once more an hour later if still silent; and again when the signal returns. Before a signal is first heard it counts from the watcher's start: a heartbeat that never comes is what this is for.
- Its own health is visible:
watcher_statusleads with whether it can tell the operator anything at all (BLINDwithout a token or chat id, or while Telegram fails), whether it is misplaced, and whether heartbeats reach it. A message it could not send is owed and tried every minute.
What the operator gives
- The bot token, as this module's own secret:
secret accept <machine> mesh-watcher telegram-token, then push that machine. The same bot as the operator-channel's is fine; it is one more machine holding it (ADR 0227, accepted for this one case). - The chat id, as a setting:
settingsformesh-watcherwith{"telegram-chat-id": "<id>"}.
Tools
| tool | does |
|---|---|
watcher_status |
its own health, then each signal: last heard, bound, silent, owed |
watcher_last_heard |
when each signal was last heard, and what it sent lately |
watcher_test |
a test message to Telegram now, directly |
watcher_ping |
the bus round trip's other end |