jochen
0ae7933d54
The mesh noticed 48 core failures in six days and told nobody (ADR 0227). messenger holds the operator-channel seat: it consumes the controller's condition events and sends them to Telegram and the desktop notifier, deduplicated by key, reminded once, edited on clear, capped at 20 an hour with the rest folded, and refusing anything carrying an address, a path or a secret. mesh-watcher, on a machine other than the control node, sends to Telegram directly when the self-check heartbeat or the bus goes silent.
39 lines
2.2 KiB
Markdown
39 lines
2.2 KiB
Markdown
# mesh-watcher
|
|
|
|
The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): what tells the operator
|
|
when the parts that would tell them are what failed.
|
|
|
|
- **Assigned to one machine that is not the control node.** It reads where the controller and the
|
|
bus run (`mesh-controller.seats`) every ten minutes; on the same machine its status says
|
|
`MISPLACED`.
|
|
- **Watches two signals.**
|
|
- **The self-check:** the controller's `doctor` heartbeat, `mesh-controller.doctor-heartbeat`.
|
|
Bound: twice the interval the heartbeat says doctor runs at (five minutes when it says none).
|
|
Heard by the time the heartbeat says it was made, so a backlog delivered after a restart is not a
|
|
sign of life. The heartbeat is read in one place, `cmd/mesh-watcher/heartbeat.go`, which states
|
|
every assumption it makes about its shape.
|
|
- **The bus:** a round trip through the bus server — its own `watcher_ping` on its own machine —
|
|
every minute. Bound: three minutes.
|
|
- **Silent past its bound:** it sends to Telegram **directly over HTTPS, not through the bus**, once;
|
|
once more an hour later if still silent; and again when the signal returns. Before a signal is
|
|
first heard it counts from the watcher's start: a heartbeat that never comes is what this is for.
|
|
- **Its own health is visible:** `watcher_status` leads with whether it can tell the operator anything
|
|
at all (`BLIND` without a token or chat id, or while Telegram fails), whether it is misplaced,
|
|
and whether heartbeats reach it. A message it could not send is owed and tried every minute.
|
|
|
|
## What the operator gives
|
|
|
|
1. **The bot token**, as this module's own secret: `secret accept <machine> mesh-watcher telegram-token`,
|
|
then push that machine. The same bot as the operator-channel's is fine; it is one more machine
|
|
holding it (ADR 0227, accepted for this one case).
|
|
2. **The chat id**, as a setting: `settings` for `mesh-watcher` with `{"telegram-chat-id": "<id>"}`.
|
|
|
|
## Tools
|
|
|
|
| tool | does |
|
|
|---|---|
|
|
| `watcher_status` | its own health, then each signal: last heard, bound, silent, owed |
|
|
| `watcher_last_heard` | when each signal was last heard, and what it sent lately |
|
|
| `watcher_test` | a test message to Telegram now, directly |
|
|
| `watcher_ping` | the bus round trip's other end |
|