mesh-catalog/modules/mesh-watcher

mesh-watcher

The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): what tells the operator when the parts that would tell them are what failed.

• Assigned to one machine that is not the control node. It reads where the controller and the bus run (mesh-controller.seats) every ten minutes; on the same machine its status says MISPLACED.
• Watches two signals.
  • The self-check: the controller's doctor heartbeat, mesh-controller.doctor-heartbeat. Bound: twice the interval the heartbeat says doctor runs at (five minutes when it says none). Heard by the time the heartbeat says it was made, so a backlog delivered after a restart is not a sign of life. The heartbeat is read in one place, cmd/mesh-watcher/heartbeat.go, which states every assumption it makes about its shape.
  • The bus: a round trip through the bus server — its own watcher_ping on its own machine — every minute. Bound: three minutes.
• Silent past its bound: it sends to Telegram directly over HTTPS, not through the bus, once; once more an hour later if still silent; and again when the signal returns. Before a signal is first heard it counts from the watcher's start: a heartbeat that never comes is what this is for.
• Its own health is visible: watcher_status leads with whether it can tell the operator anything at all (BLIND without a token or chat id, or while Telegram fails), whether it is misplaced, and whether heartbeats reach it. A message it could not send is owed and tried every minute.

What the operator gives

1. The bot token, as this module's own secret: secret accept <machine> mesh-watcher telegram-token, then push that machine. The same bot as the operator-channel's is fine; it is one more machine holding it (ADR 0227, accepted for this one case).
2. The chat id, as a setting: settings for mesh-watcher with {"telegram-chat-id": "<id>"}.

Tools

tool	does
watcher_status	its own health, then each signal: last heard, bound, silent, owed
watcher_last_heard	when each signal was last heard, and what it sent lately
watcher_test	a test message to Telegram now, directly
watcher_ping	the bus round trip's other end

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

• ${dir:state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).