# mesh-watcher

The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): what tells the operator
when the parts that would tell them are what failed.

- **Assigned to one machine that is not the control node.** It reads where the controller and the
  bus run (`mesh-controller.seats`) every ten minutes; on the same machine its status says
  `MISPLACED`.
- **Watches two signals.**
  - **The self-check:** the controller's `doctor` heartbeat, `mesh-controller.doctor-heartbeat`.
    Bound: twice the interval the heartbeat says doctor runs at (five minutes when it says none).
    Heard by the time the heartbeat says it was made, so a backlog delivered after a restart is not a
    sign of life. The heartbeat is read in one place, `cmd/mesh-watcher/heartbeat.go`, which states
    every assumption it makes about its shape.
  - **The bus:** a round trip through the bus server — its own `watcher_ping` on its own machine —
    every minute. Bound: three minutes.
- **Silent past its bound:** it sends to Telegram **directly over HTTPS, not through the bus**, once;
  once more an hour later if still silent; and again when the signal returns. Before a signal is
  first heard it counts from the watcher's start: a heartbeat that never comes is what this is for.
- **Its own health is visible:** `watcher_status` leads with whether it can tell the operator anything
  at all (`BLIND` without a token or chat id, or while Telegram fails), whether it is misplaced,
  and whether heartbeats reach it. A message it could not send is owed and tried every minute.

## What the operator gives

1. **The bot token**, as this module's own secret: `secret accept <machine> mesh-watcher telegram-token`,
   then push that machine. The same bot as the operator-channel's is fine; it is one more machine
   holding it (ADR 0227, accepted for this one case).
2. **The chat id**, as a setting: `settings` for `mesh-watcher` with `{"telegram-chat-id": "<id>"}`.

## Tools

| tool | does |
|---|---|
| `watcher_status` | its own health, then each signal: last heard, bound, silent, owed |
| `watcher_last_heard` | when each signal was last heard, and what it sent lately |
| `watcher_test` | a test message to Telegram now, directly |
| `watcher_ping` | the bus round trip's other end |