Tell the operator what the mesh finds wrong, and watch the watcher (hq to-be 45 phase 1)
The mesh noticed 48 core failures in six days and told nobody (ADR 0227). messenger holds the operator-channel seat: it consumes the controller's condition events and sends them to Telegram and the desktop notifier, deduplicated by key, reminded once, edited on clear, capped at 20 an hour with the rest folded, and refusing anything carrying an address, a path or a secret. mesh-watcher, on a machine other than the control node, sends to Telegram directly when the self-check heartbeat or the bus goes silent.
This commit is contained in:
jochen
@@ -0,0 +1,38 @@
|
||||
# mesh-watcher
|
||||
|
||||
The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): what tells the operator
|
||||
when the parts that would tell them are what failed.
|
||||
|
||||
- **Assigned to one machine that is not the control node.** It reads where the controller and the
|
||||
bus run (`mesh-controller.seats`) every ten minutes; on the same machine its status says
|
||||
`MISPLACED`.
|
||||
- **Watches two signals.**
|
||||
- **The self-check:** the controller's `doctor` heartbeat, `mesh-controller.doctor-heartbeat`.
|
||||
Bound: twice the interval the heartbeat says doctor runs at (five minutes when it says none).
|
||||
Heard by the time the heartbeat says it was made, so a backlog delivered after a restart is not a
|
||||
sign of life. The heartbeat is read in one place, `cmd/mesh-watcher/heartbeat.go`, which states
|
||||
every assumption it makes about its shape.
|
||||
- **The bus:** a round trip through the bus server — its own `watcher_ping` on its own machine —
|
||||
every minute. Bound: three minutes.
|
||||
- **Silent past its bound:** it sends to Telegram **directly over HTTPS, not through the bus**, once;
|
||||
once more an hour later if still silent; and again when the signal returns. Before a signal is
|
||||
first heard it counts from the watcher's start: a heartbeat that never comes is what this is for.
|
||||
- **Its own health is visible:** `watcher_status` leads with whether it can tell the operator anything
|
||||
at all (`BLIND` without a token or chat id, or while Telegram fails), whether it is misplaced,
|
||||
and whether heartbeats reach it. A message it could not send is owed and tried every minute.
|
||||
|
||||
## What the operator gives
|
||||
|
||||
1. **The bot token**, as this module's own secret: `secret accept <machine> mesh-watcher telegram-token`,
|
||||
then push that machine. The same bot as the operator-channel's is fine; it is one more machine
|
||||
holding it (ADR 0227, accepted for this one case).
|
||||
2. **The chat id**, as a setting: `settings` for `mesh-watcher` with `{"telegram-chat-id": "<id>"}`.
|
||||
|
||||
## Tools
|
||||
|
||||
| tool | does |
|
||||
|---|---|
|
||||
| `watcher_status` | its own health, then each signal: last heard, bound, silent, owed |
|
||||
| `watcher_last_heard` | when each signal was last heard, and what it sent lately |
|
||||
| `watcher_test` | a test message to Telegram now, directly |
|
||||
| `watcher_ping` | the bus round trip's other end |
|
||||
@@ -0,0 +1,114 @@
|
||||
package main
|
||||
|
||||
// The self-check's heartbeat, read in this one place (novox/hq to-be 45 §4, S10).
|
||||
//
|
||||
// **The contract this reads, and every assumption it makes**, because the controller's side was built
|
||||
// at the same time from the same design, which says only "every run ends with a heartbeat event
|
||||
// carrying the run's id and counts":
|
||||
//
|
||||
// - The event is the mesh-controller seat's own, `doctor-heartbeat`, consumed as
|
||||
// `mesh-controller.doctor-heartbeat` (subject `mesh.seat.mesh-controller.event.doctor-heartbeat`).
|
||||
// - The body is one JSON object: `run` (the run's id), `at` (when the run ended, RFC 3339),
|
||||
// `interval-seconds` (how often doctor runs), and `counts` ({pass, fail, failed-to-run}).
|
||||
// `finished`/`time` are read for `at`; `interval_seconds`, `interval` (seconds, or a duration such
|
||||
// as "5m") for the interval.
|
||||
// - Without a time, the heartbeat is taken as made when it arrived (said in the status as such).
|
||||
// Without an interval, the design's five minutes stand.
|
||||
// - The counts are kept for the status only. A run that found failures is still a heartbeat: the
|
||||
// watcher watches that the checker runs; what it finds is the controller's to raise.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// HeartbeatEvent is the event's local name under the controller's seat.
|
||||
const (
|
||||
HeartbeatEvent = "doctor-heartbeat"
|
||||
ControllerSeat = "mesh-controller"
|
||||
)
|
||||
|
||||
// Beat is one heartbeat.
|
||||
type Beat struct {
|
||||
Run string
|
||||
At time.Time
|
||||
Interval time.Duration
|
||||
Counts map[string]int
|
||||
}
|
||||
|
||||
// isHeartbeat says whether an envelope's key is the controller's heartbeat.
|
||||
func isHeartbeat(key string) bool { return key == ControllerSeat+"."+HeartbeatEvent }
|
||||
|
||||
// DecodeBeat reads one heartbeat's body.
|
||||
func DecodeBeat(body []byte) (Beat, error) {
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &raw); err != nil || raw == nil {
|
||||
return Beat{}, fmt.Errorf("%s: the body is not a JSON object", HeartbeatEvent)
|
||||
}
|
||||
var b Beat
|
||||
if v, ok := raw["run"]; ok {
|
||||
var s string
|
||||
var n float64
|
||||
switch {
|
||||
case json.Unmarshal(v, &s) == nil:
|
||||
b.Run = s
|
||||
case json.Unmarshal(v, &n) == nil:
|
||||
b.Run = fmt.Sprintf("%.0f", n)
|
||||
default:
|
||||
return Beat{}, fmt.Errorf("%s: run is neither a string nor a number", HeartbeatEvent)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"at", "finished", "time"} {
|
||||
v, ok := raw[name]
|
||||
if !ok || string(v) == "null" {
|
||||
continue
|
||||
}
|
||||
var s string
|
||||
if json.Unmarshal(v, &s) != nil {
|
||||
return Beat{}, fmt.Errorf("%s: %s is not a string", HeartbeatEvent, name)
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339Nano, s)
|
||||
if err != nil {
|
||||
return Beat{}, fmt.Errorf("%s: %s is not an RFC 3339 time: %q", HeartbeatEvent, name, s)
|
||||
}
|
||||
b.At = t
|
||||
break
|
||||
}
|
||||
for _, name := range []string{"interval-seconds", "interval_seconds", "interval"} {
|
||||
v, ok := raw[name]
|
||||
if !ok || string(v) == "null" {
|
||||
continue
|
||||
}
|
||||
var n float64
|
||||
var s string
|
||||
switch {
|
||||
case json.Unmarshal(v, &n) == nil:
|
||||
b.Interval = time.Duration(n * float64(time.Second))
|
||||
case json.Unmarshal(v, &s) == nil:
|
||||
d, err := time.ParseDuration(strings.TrimSpace(s))
|
||||
if err != nil {
|
||||
return Beat{}, fmt.Errorf("%s: %s is not a duration: %q", HeartbeatEvent, name, s)
|
||||
}
|
||||
b.Interval = d
|
||||
default:
|
||||
return Beat{}, fmt.Errorf("%s: %s is neither seconds nor a duration", HeartbeatEvent, name)
|
||||
}
|
||||
if b.Interval < 0 || b.Interval > 24*time.Hour {
|
||||
return Beat{}, fmt.Errorf("%s: an interval of %s is not one doctor runs at", HeartbeatEvent, b.Interval)
|
||||
}
|
||||
break
|
||||
}
|
||||
if v, ok := raw["counts"]; ok && string(v) != "null" {
|
||||
var counts map[string]float64
|
||||
if json.Unmarshal(v, &counts) != nil {
|
||||
return Beat{}, fmt.Errorf("%s: counts is not an object of numbers", HeartbeatEvent)
|
||||
}
|
||||
b.Counts = map[string]int{}
|
||||
for k, n := range counts {
|
||||
b.Counts[k] = int(n)
|
||||
}
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
// mesh-watcher: the watcher's watcher (novox/hq to-be 45 §5, S10, ADR 0227 rule 6). A Go bundle the
|
||||
// node's runtime launches on one machine that is not the control node. It hears the controller's
|
||||
// self-check heartbeat and makes a round trip through the bus every minute; when either goes silent
|
||||
// past its bound, it tells the operator on Telegram directly over HTTPS — the one sender that does not
|
||||
// pass through the control node — and tells them again when it returns. stdout is the MCP channel;
|
||||
// what this module says, it says on stderr.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
|
||||
|
||||
func readChatID(path string) (string, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var s map[string]any
|
||||
if err := json.Unmarshal(raw, &s); err != nil {
|
||||
return "", fmt.Errorf("the settings file is not JSON: %v", err)
|
||||
}
|
||||
switch v := s["telegram-chat-id"].(type) {
|
||||
case string:
|
||||
return strings.TrimSpace(v), nil
|
||||
case float64:
|
||||
return fmt.Sprintf("%.0f", v), nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
type listening struct {
|
||||
mu sync.Mutex
|
||||
now string
|
||||
}
|
||||
|
||||
func (l *listening) set(s string) { l.mu.Lock(); l.now = s; l.mu.Unlock() }
|
||||
func (l *listening) get() string { l.mu.Lock(); defer l.mu.Unlock(); return l.now }
|
||||
|
||||
func main() {
|
||||
settings := os.Getenv("MESH_WATCHER_SETTINGS")
|
||||
var said sync.Mutex
|
||||
lastSaid := ""
|
||||
tg := NewTelegram(TelegramConfig{
|
||||
TokenFile: os.Getenv("MESH_WATCHER_TELEGRAM_TOKEN_FILE"),
|
||||
ChatID: func() string {
|
||||
id, err := readChatID(settings)
|
||||
said.Lock()
|
||||
defer said.Unlock()
|
||||
if err != nil && err.Error() != lastSaid {
|
||||
logf("[mesh-watcher] settings cannot be read: %v", err)
|
||||
}
|
||||
lastSaid = ""
|
||||
if err != nil {
|
||||
lastSaid = err.Error()
|
||||
}
|
||||
return id
|
||||
},
|
||||
})
|
||||
node := os.Getenv("MESH_NODE")
|
||||
w := NewWatcher(tg, time.Now, logf, node)
|
||||
l := &listening{now: "not yet: starting"}
|
||||
go run(w, l, node)
|
||||
if err := stdio.Serve("", tools(w, l)); err != nil {
|
||||
logf("%v", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// run keeps time, makes the round trips, reads where the controller is, and listens for heartbeats.
|
||||
// The clock runs whatever the bus does: it is what notices the bus is gone.
|
||||
func run(w *Watcher, l *listening, node string) {
|
||||
if err := w.Telegram.Ready(); err != nil {
|
||||
logf("[mesh-watcher] BLIND until given: %v", err)
|
||||
}
|
||||
go func() {
|
||||
for range time.Tick(time.Minute) {
|
||||
go func() { w.BusAnswered(roundTrip(node)) }()
|
||||
w.Tick()
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
time.Sleep(2 * time.Second)
|
||||
for {
|
||||
if nodes, err := controlNodes(); err == nil {
|
||||
w.Placed(nodes)
|
||||
} else {
|
||||
logf("[mesh-watcher] cannot read where the controller runs (%v); asking again in ten minutes", err)
|
||||
}
|
||||
time.Sleep(10 * time.Minute)
|
||||
}
|
||||
}()
|
||||
handle := func(e stdio.Envelope) error {
|
||||
if !isHeartbeat(e.Key) {
|
||||
return nil
|
||||
}
|
||||
b, err := DecodeBeat(e.Body)
|
||||
if err != nil {
|
||||
w.BadHeartbeat(err)
|
||||
return nil
|
||||
}
|
||||
w.Heartbeat(b)
|
||||
return nil
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
|
||||
err := stdio.Subscribe(ControllerSeat+".*", handle)
|
||||
if err == nil {
|
||||
l.set("listening")
|
||||
logf("[mesh-watcher] listening for the self-check's heartbeat; watching from %s", node)
|
||||
return
|
||||
}
|
||||
l.set("not yet: " + err.Error())
|
||||
logf("[mesh-watcher] not hearing heartbeats yet (%v); asking again in %s", err, wait)
|
||||
time.Sleep(wait)
|
||||
}
|
||||
}
|
||||
|
||||
// roundTrip asks this module's own ping on this machine, through the bus server: an answer is the bus
|
||||
// carrying a request and its reply.
|
||||
func roundTrip(node string) error {
|
||||
key := "mesh-watcher.watcher_ping"
|
||||
if node != "" {
|
||||
key += "@" + node
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := stdio.Ask(key, map[string]any{})
|
||||
done <- err
|
||||
}()
|
||||
select {
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-time.After(30 * time.Second):
|
||||
return fmt.Errorf("no answer in 30 s")
|
||||
}
|
||||
}
|
||||
|
||||
// controlNodes reads which machines hold the controller and the bus.
|
||||
func controlNodes() ([]string, error) {
|
||||
raw, err := stdio.Ask("seat:mesh-controller.seats", map[string]any{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return holdersOf(raw, "mesh-controller", "mesh-broker")
|
||||
}
|
||||
|
||||
// holdersOf reads the seats verb's answer, bare or inside a tool reply, for the named seats' machines.
|
||||
func holdersOf(raw json.RawMessage, seats ...string) ([]string, error) {
|
||||
var answer struct {
|
||||
Seats []struct {
|
||||
Seat string `json:"seat"`
|
||||
Holders []struct {
|
||||
Node string `json:"node"`
|
||||
} `json:"holders"`
|
||||
} `json:"seats"`
|
||||
Output string `json:"output"`
|
||||
Content []struct {
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &answer); err != nil {
|
||||
var s string
|
||||
if json.Unmarshal(raw, &s) == nil {
|
||||
return holdersOf(json.RawMessage(s), seats...)
|
||||
}
|
||||
return nil, fmt.Errorf("the seats answer is not JSON")
|
||||
}
|
||||
if len(answer.Seats) == 0 {
|
||||
switch {
|
||||
case answer.Output != "":
|
||||
return holdersOf(json.RawMessage(answer.Output), seats...)
|
||||
case len(answer.Content) > 0:
|
||||
return holdersOf(json.RawMessage(answer.Content[0].Text), seats...)
|
||||
}
|
||||
return nil, fmt.Errorf("the seats answer lists no seat")
|
||||
}
|
||||
var out []string
|
||||
seen := map[string]bool{}
|
||||
for _, s := range answer.Seats {
|
||||
for _, want := range seats {
|
||||
if s.Seat != want {
|
||||
continue
|
||||
}
|
||||
for _, h := range s.Holders {
|
||||
if !seen[h.Node] {
|
||||
seen[h.Node] = true
|
||||
out = append(out, h.Node)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func tools(w *Watcher, l *listening) []stdio.Tool {
|
||||
return []stdio.Tool{
|
||||
{Name: "watcher_status",
|
||||
Description: "The watcher's own health first — whether it can tell the operator anything (the Telegram token and " +
|
||||
"chat id), whether it runs on the machine it watches, whether heartbeats reach it — then each watched " +
|
||||
"signal (the controller's self-check heartbeat, a round trip through the bus): last heard, how long ago, " +
|
||||
"its bound, whether it is said silent, and any message not sent yet.",
|
||||
Run: func(map[string]any) (any, error) { return w.Status(l.get()), nil }},
|
||||
{Name: "watcher_last_heard",
|
||||
Description: "When each watched signal was last heard, and what the watcher sent to Telegram lately, newest first.",
|
||||
Run: func(map[string]any) (any, error) { return w.LastHeard(), nil }},
|
||||
{Name: "watcher_test",
|
||||
Description: "Send a test message to Telegram now, directly: proves the watcher can reach the operator when the " +
|
||||
"mesh cannot. Answers sent, or why not.",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
err := w.send("test", Message{Title: "TEST: mesh-watcher on " + w.Node + " reaches you directly",
|
||||
Body: "nothing is wrong; this was asked for"})
|
||||
if err != nil {
|
||||
return map[string]string{"telegram": "not sent: " + err.Error()}, nil
|
||||
}
|
||||
return map[string]string{"telegram": "sent"}, nil
|
||||
}},
|
||||
{Name: "watcher_ping",
|
||||
Description: "Answers at once: the round trip the watcher makes through the bus every minute to know the bus carries a request and its reply.",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
return map[string]string{"pong": time.Now().UTC().Format(time.RFC3339)}, nil
|
||||
}},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The manifest says what the code does: it consumes the heartbeat and nothing else, calls its own
|
||||
// ping and the controller's seats verb, holds no seat (it must not be the controller's), keeps its
|
||||
// Telegram token as its own secret, lists its own tools — and names nothing of one installation.
|
||||
|
||||
func TestTheManifestSaysWhatTheCodeDoes(t *testing.T) {
|
||||
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var m struct {
|
||||
Module string `json:"module"`
|
||||
Consumes []string `json:"consumes"`
|
||||
Invokes []string `json:"invokes"`
|
||||
Claims []any `json:"claims"`
|
||||
Own map[string]string `json:"own-secrets"`
|
||||
Tools []string `json:"tools"`
|
||||
Build struct {
|
||||
Artifacts []map[string]any `json:"artifacts"`
|
||||
} `json:"build"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(m.Consumes, []string{ControllerSeat + "." + HeartbeatEvent}) {
|
||||
t.Fatalf("consumes %v", m.Consumes)
|
||||
}
|
||||
if !reflect.DeepEqual(m.Invokes, []string{m.Module + ".watcher_ping", "seat:mesh-controller.seats"}) {
|
||||
t.Fatalf("invokes %v", m.Invokes)
|
||||
}
|
||||
if len(m.Claims) != 0 {
|
||||
t.Fatalf("a watcher holds no seat: %v", m.Claims)
|
||||
}
|
||||
env, _ := m.Build.Artifacts[0]["env"].(map[string]any)
|
||||
if m.Own["telegram-token"] == "" || env["MESH_WATCHER_TELEGRAM_TOKEN_FILE"] != m.Own["telegram-token"] {
|
||||
t.Fatalf("token: own %v, env %v", m.Own, env)
|
||||
}
|
||||
var served []string
|
||||
for _, tool := range tools(NewWatcher(&fakeTelegram{}, time.Now, t.Logf, ""), &listening{}) {
|
||||
served = append(served, tool.Name)
|
||||
if strings.TrimSpace(tool.Description) == "" {
|
||||
t.Errorf("%s has no description", tool.Name)
|
||||
}
|
||||
}
|
||||
listed := append([]string(nil), m.Tools...)
|
||||
sort.Strings(served)
|
||||
sort.Strings(listed)
|
||||
if !reflect.DeepEqual(served, listed) {
|
||||
t.Fatalf("serves %v, lists %v", served, listed)
|
||||
}
|
||||
for _, never := range []string{"/home/", "jochen", "g14", "shanks", "novox", "zurag", "api.telegram"} {
|
||||
if strings.Contains(strings.ToLower(string(raw)), never) {
|
||||
t.Errorf("module.json names %q", never)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
package main
|
||||
|
||||
// Telegram, sent to directly over HTTPS — never through the bus or the control node (novox/hq to-be 45
|
||||
// §5): the one sender that does not pass through the control node. The same client as the
|
||||
// operator-channel's holder (module messenger), kept here so the watcher depends on nothing it
|
||||
// watches. The bot token is this module's own secret, accepted from the operator; the
|
||||
// chat id is a setting. Neither is ever said: an error from the HTTP client carries the URL, and the
|
||||
// URL carries the token, so every error is rebuilt here from its kind before it leaves this file.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TelegramAPI is where the bot API answers; a test points it elsewhere.
|
||||
var TelegramAPI = "https://api.telegram.org"
|
||||
|
||||
var (
|
||||
tokenShape = regexp.MustCompile(`^\d{5,}:[A-Za-z0-9_-]{30,}$`)
|
||||
chatIDShape = regexp.MustCompile(`^(-?\d{1,20}|@[A-Za-z][A-Za-z0-9_]{4,})$`)
|
||||
)
|
||||
|
||||
// TelegramConfig is where the token is and what the chat is; read each time it is used, so a secret
|
||||
// accepted or a setting changed takes effect at the next message without a restart.
|
||||
type TelegramConfig struct {
|
||||
TokenFile string
|
||||
ChatID func() string
|
||||
}
|
||||
|
||||
// Telegram sends to one chat.
|
||||
type Telegram struct {
|
||||
Config TelegramConfig
|
||||
Client *http.Client
|
||||
}
|
||||
|
||||
func NewTelegram(cfg TelegramConfig) *Telegram {
|
||||
return &Telegram{Config: cfg, Client: &http.Client{Timeout: 20 * time.Second}}
|
||||
}
|
||||
|
||||
func (t *Telegram) Name() string { return "telegram" }
|
||||
|
||||
// Ready says whether the channel can send, in words.
|
||||
func (t *Telegram) Ready() error {
|
||||
_, _, err := t.ready()
|
||||
return err
|
||||
}
|
||||
|
||||
// ready says whether the channel can send, and when not, what the operator must give. The words name
|
||||
// the setting and the secret, never a value.
|
||||
func (t *Telegram) ready() (string, string, error) {
|
||||
token, err := t.token()
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
chat := strings.TrimSpace(t.Config.ChatID())
|
||||
if chat == "" {
|
||||
return "", "", errors.New("no chat to send to: the setting telegram-chat-id is not given " +
|
||||
"(`settings` for mesh-watcher with {\"telegram-chat-id\": \"<the chat's id>\"})")
|
||||
}
|
||||
if !chatIDShape.MatchString(chat) {
|
||||
return "", "", errors.New("the setting telegram-chat-id is not a chat id (a number, or @name of a channel)")
|
||||
}
|
||||
return token, chat, nil
|
||||
}
|
||||
|
||||
func (t *Telegram) token() (string, error) {
|
||||
if t.Config.TokenFile == "" {
|
||||
return "", errors.New("no bot token: this module was started without a token file")
|
||||
}
|
||||
raw, err := os.ReadFile(t.Config.TokenFile)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return "", errors.New("no bot token: the own secret telegram-token is not on this machine yet " +
|
||||
"(`secret accept <machine> mesh-watcher telegram-token`, then push the machine)")
|
||||
}
|
||||
if err != nil {
|
||||
return "", errors.New("the own secret telegram-token cannot be read: " + plainError(err))
|
||||
}
|
||||
token := strings.TrimSpace(string(raw))
|
||||
if token == "" {
|
||||
return "", errors.New("no bot token: the own secret telegram-token is empty")
|
||||
}
|
||||
if !tokenShape.MatchString(token) {
|
||||
// The mesh mints a random value for an own secret nobody accepted; that is not a bot token.
|
||||
return "", errors.New("the own secret telegram-token is not a bot token (digits, a colon, then the key " +
|
||||
"BotFather gave) — most likely the mesh made it because none was accepted: " +
|
||||
"`secret accept <machine> mesh-watcher telegram-token`, then push the machine")
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
|
||||
// Send posts a message and answers its message id.
|
||||
func (t *Telegram) Send(m Message) (string, error) {
|
||||
text := m.Text()
|
||||
token, chat, err := t.ready()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var out struct {
|
||||
MessageID int64 `json:"message_id"`
|
||||
}
|
||||
if err := t.call(token, "sendMessage", map[string]any{
|
||||
"chat_id": chat, "text": text, "disable_web_page_preview": true,
|
||||
}, &out); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprint(out.MessageID), nil
|
||||
}
|
||||
|
||||
// Edit replaces the text of a message sent before: how a cleared condition is said (to-be 45 §5).
|
||||
func (t *Telegram) Edit(id string, m Message) error {
|
||||
text := m.Text()
|
||||
token, chat, err := t.ready()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return t.call(token, "editMessageText", map[string]any{
|
||||
"chat_id": chat, "message_id": json.Number(id), "text": text, "disable_web_page_preview": true,
|
||||
}, nil)
|
||||
}
|
||||
|
||||
// CanEdit: Telegram edits a message in place.
|
||||
func (t *Telegram) CanEdit() bool { return true }
|
||||
|
||||
// Who asks the bot API who it is: the check that the token works, with no message sent.
|
||||
func (t *Telegram) Who() (string, error) {
|
||||
token, _, err := t.ready()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var me struct {
|
||||
Username string `json:"username"`
|
||||
}
|
||||
if err := t.call(token, "getMe", map[string]any{}, &me); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return me.Username, nil
|
||||
}
|
||||
|
||||
func (t *Telegram) call(token, method string, body map[string]any, into any) error {
|
||||
raw, _ := json.Marshal(body)
|
||||
req, err := http.NewRequest(http.MethodPost, TelegramAPI+"/bot"+token+"/"+method, bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return errors.New("telegram " + method + ": the request could not be made")
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := t.Client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("telegram %s: %s", method, plainError(err))
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var answer struct {
|
||||
OK bool `json:"ok"`
|
||||
ErrorCode int `json:"error_code"`
|
||||
Description string `json:"description"`
|
||||
Result json.RawMessage `json:"result"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&answer); err != nil {
|
||||
return fmt.Errorf("telegram %s: HTTP %d with an answer that is not the bot API's", method, resp.StatusCode)
|
||||
}
|
||||
if !answer.OK {
|
||||
d := strings.ReplaceAll(answer.Description, token, "<token>")
|
||||
return fmt.Errorf("telegram %s refused: %d %s", method, answer.ErrorCode, d)
|
||||
}
|
||||
if into != nil && len(answer.Result) > 0 {
|
||||
_ = json.Unmarshal(answer.Result, into)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// plainError is an error in words, without the URL a transport error carries.
|
||||
func plainError(err error) string {
|
||||
var ue *url.Error
|
||||
if errors.As(err, &ue) {
|
||||
err = ue.Err
|
||||
}
|
||||
var ne net.Error
|
||||
switch {
|
||||
case errors.As(err, &ne) && ne.Timeout():
|
||||
return "no answer in time"
|
||||
case errors.Is(err, os.ErrPermission):
|
||||
return "permission denied"
|
||||
}
|
||||
var dns *net.DNSError
|
||||
if errors.As(err, &dns) {
|
||||
return "the bot API's name does not resolve"
|
||||
}
|
||||
var op *net.OpError
|
||||
if errors.As(err, &op) {
|
||||
return "cannot connect (" + op.Op + ")"
|
||||
}
|
||||
s := err.Error()
|
||||
if strings.Contains(s, "/bot") || strings.Contains(s, "://") {
|
||||
return "the request failed"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const goodToken = "123456789:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsawQ"
|
||||
|
||||
func tokenFile(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "telegram-token")
|
||||
if content != "" {
|
||||
if err := os.WriteFile(p, []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func TestTelegramSaysWhatItLacks(t *testing.T) {
|
||||
for _, c := range []struct{ token, chat, says string }{
|
||||
{"", "42", "not on this machine yet"},
|
||||
{"r4nd0mlyMadeByTheMeshBecauseNobodyAcceptedOne", "42", "not a bot token"},
|
||||
{goodToken, "", "telegram-chat-id is not given"},
|
||||
{goodToken, "not a chat", "is not a chat id"},
|
||||
} {
|
||||
tg := NewTelegram(TelegramConfig{TokenFile: tokenFile(t, c.token), ChatID: func() string { return c.chat }})
|
||||
err := tg.Ready()
|
||||
if err == nil || !strings.Contains(err.Error(), c.says) {
|
||||
t.Errorf("%+v: %v", c, err)
|
||||
}
|
||||
if err != nil && strings.Contains(err.Error(), goodToken) {
|
||||
t.Errorf("the token is in the words")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTelegramSendsEditsAndNeverSaysItsToken(t *testing.T) {
|
||||
var asked []string
|
||||
var bodies []map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
asked = append(asked, r.URL.Path)
|
||||
raw, _ := io.ReadAll(r.Body)
|
||||
var b map[string]any
|
||||
_ = json.Unmarshal(raw, &b)
|
||||
bodies = append(bodies, b)
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/sendMessage"):
|
||||
_, _ = w.Write([]byte(`{"ok":true,"result":{"message_id":77}}`))
|
||||
case strings.HasSuffix(r.URL.Path, "/editMessageText"):
|
||||
_, _ = w.Write([]byte(`{"ok":false,"error_code":400,"description":"Bad Request: message to edit not found"}`))
|
||||
default:
|
||||
_, _ = w.Write([]byte(`{"ok":true,"result":{"username":"mesh_bot"}}`))
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
old := TelegramAPI
|
||||
TelegramAPI = srv.URL
|
||||
defer func() { TelegramAPI = old }()
|
||||
tg := NewTelegram(TelegramConfig{TokenFile: tokenFile(t, goodToken+"\n"), ChatID: func() string { return "-1001" }})
|
||||
id, err := tg.Send(Message{Title: "URGENT: x", Body: "key: a.b.c"})
|
||||
if err != nil || id != "77" {
|
||||
t.Fatalf("%q %v", id, err)
|
||||
}
|
||||
if asked[0] != "/bot"+goodToken+"/sendMessage" || bodies[0]["chat_id"] != "-1001" || bodies[0]["text"] != "URGENT: x\nkey: a.b.c" {
|
||||
t.Fatalf("%v %v", asked, bodies[0])
|
||||
}
|
||||
err = tg.Edit("77", Message{Title: "CLEARED"})
|
||||
if err == nil || !strings.Contains(err.Error(), "message to edit not found") || strings.Contains(err.Error(), goodToken) {
|
||||
t.Fatalf("edit: %v", err)
|
||||
}
|
||||
if bodies[1]["message_id"] != float64(77) {
|
||||
t.Fatalf("message id: %v", bodies[1]["message_id"])
|
||||
}
|
||||
if who, err := tg.Who(); err != nil || who != "mesh_bot" {
|
||||
t.Fatalf("%q %v", who, err)
|
||||
}
|
||||
// Nothing answers: the error is in words, without the URL that carries the token.
|
||||
srv.Close()
|
||||
_, err = tg.Send(Message{Title: "x"})
|
||||
if err == nil || strings.Contains(err.Error(), goodToken) || strings.Contains(err.Error(), "/bot") {
|
||||
t.Fatalf("unreachable: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,392 @@
|
||||
package main
|
||||
|
||||
// The watcher's watcher (novox/hq to-be 45 §5, signal S10, ADR 0227 rule 6): on a machine that is not
|
||||
// the control node, it listens for the self-check's heartbeat and for the bus itself. When either has
|
||||
// been silent past its bound it sends to Telegram directly over HTTPS — not through the bus — says so
|
||||
// once more an hour later if it still is, and says so again when it returns.
|
||||
//
|
||||
// Two signals:
|
||||
//
|
||||
// - self-check: the controller's `doctor` heartbeat. Bound: twice the interval the heartbeat says
|
||||
// it runs at, else twice five minutes (to-be 45 §3, S10). Heard by the time the heartbeat says it
|
||||
// was made, not by when it arrived: a backlog delivered after a restart is old news, not a sign of
|
||||
// life.
|
||||
// - bus: a round trip through the bus server — this module asking its own tool on its own machine
|
||||
// — every minute. Bound: three minutes.
|
||||
//
|
||||
// Before a signal is first heard it counts from when the watcher started: a heartbeat that never comes
|
||||
// is exactly what this is for.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
SelfCheck = "self-check"
|
||||
Bus = "bus"
|
||||
DefaultInterval = 5 * time.Minute
|
||||
BusBound = 3 * time.Minute
|
||||
RemindAfter = time.Hour
|
||||
// Skew is how far in the future a heartbeat's time may be and still be believed.
|
||||
Skew = 2 * time.Minute
|
||||
)
|
||||
|
||||
// Message is one thing said to the operator.
|
||||
type Message struct {
|
||||
Title string
|
||||
Body string
|
||||
}
|
||||
|
||||
func (m Message) Text() string {
|
||||
if m.Body == "" {
|
||||
return m.Title
|
||||
}
|
||||
return m.Title + "\n" + m.Body
|
||||
}
|
||||
|
||||
// Sender is the Telegram channel.
|
||||
type Sender interface {
|
||||
Ready() error
|
||||
Send(Message) (string, error)
|
||||
}
|
||||
|
||||
type signal struct {
|
||||
name string
|
||||
lastHeard time.Time
|
||||
bound time.Duration
|
||||
silentAt time.Time // when it was said silent; zero while it is heard
|
||||
heardBefore time.Time // the last word before it went silent
|
||||
reminded bool
|
||||
owed *Message // a message that could not be sent yet
|
||||
lastRun string // the heartbeat's run id, for the status
|
||||
lastErr string // the bus probe's last error
|
||||
}
|
||||
|
||||
// Watcher watches.
|
||||
type Watcher struct {
|
||||
Telegram Sender
|
||||
Now func() time.Time
|
||||
Logf func(string, ...any)
|
||||
Node string
|
||||
|
||||
mu sync.Mutex
|
||||
started time.Time
|
||||
signals map[string]*signal
|
||||
sent []sentNote
|
||||
sendErr string
|
||||
sendOK time.Time
|
||||
misplace string // why this machine is the wrong one to watch from, or ""
|
||||
control string // the machine the controller is on, as last read
|
||||
badBeats int
|
||||
lastBad string
|
||||
}
|
||||
|
||||
type sentNote struct {
|
||||
At time.Time `json:"at"`
|
||||
Signal string `json:"signal"`
|
||||
What string `json:"what"`
|
||||
Outcome string `json:"outcome"`
|
||||
}
|
||||
|
||||
func NewWatcher(tg Sender, now func() time.Time, logf func(string, ...any), node string) *Watcher {
|
||||
w := &Watcher{Telegram: tg, Now: now, Logf: logf, Node: node, started: now(), signals: map[string]*signal{
|
||||
SelfCheck: {name: SelfCheck, bound: 2 * DefaultInterval},
|
||||
Bus: {name: Bus, bound: BusBound},
|
||||
}}
|
||||
return w
|
||||
}
|
||||
|
||||
// Heartbeat takes one self-check heartbeat.
|
||||
func (w *Watcher) Heartbeat(b Beat) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
now := w.Now()
|
||||
s := w.signals[SelfCheck]
|
||||
at := b.At
|
||||
if at.IsZero() {
|
||||
at = now
|
||||
}
|
||||
if at.After(now.Add(Skew)) {
|
||||
w.badBeats++
|
||||
w.lastBad = fmt.Sprintf("a heartbeat from the future (%s), not believed", at.UTC().Format(time.RFC3339))
|
||||
w.Logf("[mesh-watcher] %s", w.lastBad)
|
||||
return
|
||||
}
|
||||
if at.After(s.lastHeard) {
|
||||
s.lastHeard = at
|
||||
s.lastRun = b.Run
|
||||
}
|
||||
if b.Interval > 0 {
|
||||
s.bound = 2 * b.Interval
|
||||
}
|
||||
}
|
||||
|
||||
// BadHeartbeat counts a heartbeat that could not be read: said, never read as a sign of life.
|
||||
func (w *Watcher) BadHeartbeat(err error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.badBeats++
|
||||
w.lastBad = err.Error()
|
||||
w.Logf("[mesh-watcher] a heartbeat could not be read (not counted as heard): %v", err)
|
||||
}
|
||||
|
||||
// BusAnswered takes the outcome of one round trip through the bus.
|
||||
func (w *Watcher) BusAnswered(err error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
s := w.signals[Bus]
|
||||
if err != nil {
|
||||
if s.lastErr != err.Error() {
|
||||
w.Logf("[mesh-watcher] the bus did not answer a round trip: %v", err)
|
||||
}
|
||||
s.lastErr = err.Error()
|
||||
return
|
||||
}
|
||||
s.lastErr = ""
|
||||
s.lastHeard = w.Now()
|
||||
}
|
||||
|
||||
// Placed records where the controller runs, and says when that is this machine.
|
||||
func (w *Watcher) Placed(controlNodes []string) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.control = ""
|
||||
was := w.misplace
|
||||
w.misplace = ""
|
||||
for _, n := range controlNodes {
|
||||
if w.control != "" {
|
||||
w.control += ", "
|
||||
}
|
||||
w.control += n
|
||||
if n == w.Node && w.Node != "" {
|
||||
w.misplace = "this machine holds " + n + "'s controller or bus: a watcher here goes down with what it watches; assign mesh-watcher to another machine"
|
||||
}
|
||||
}
|
||||
if w.misplace != "" && was == "" {
|
||||
w.Logf("[mesh-watcher] %s", w.misplace)
|
||||
}
|
||||
}
|
||||
|
||||
// Tick decides what is silent, what returned, and sends what is owed.
|
||||
func (w *Watcher) Tick() {
|
||||
w.mu.Lock()
|
||||
now := w.Now()
|
||||
var out []struct {
|
||||
s *signal
|
||||
m Message
|
||||
w string
|
||||
}
|
||||
for _, name := range []string{SelfCheck, Bus} {
|
||||
s := w.signals[name]
|
||||
since := s.lastHeard
|
||||
if since.IsZero() {
|
||||
since = w.started
|
||||
}
|
||||
silent := now.Sub(since)
|
||||
switch {
|
||||
case s.silentAt.IsZero() && silent > s.bound:
|
||||
s.silentAt, s.reminded, s.heardBefore = now, false, since
|
||||
m := w.silentMessage(s, silent, false)
|
||||
s.owed = &m
|
||||
case !s.silentAt.IsZero() && silent <= s.bound:
|
||||
m := Message{
|
||||
Title: "CLEARED: " + w.what(s) + " heard again",
|
||||
Body: "silent for about " + roughly(s.lastHeard.Sub(s.heardBefore)) + "; told by mesh-watcher on " +
|
||||
w.Node + ", directly over HTTPS",
|
||||
}
|
||||
s.silentAt = time.Time{}
|
||||
s.owed = &m
|
||||
case !s.silentAt.IsZero() && !s.reminded && now.Sub(s.silentAt) >= RemindAfter:
|
||||
s.reminded = true
|
||||
m := w.silentMessage(s, silent, true)
|
||||
s.owed = &m
|
||||
}
|
||||
if s.owed != nil {
|
||||
out = append(out, struct {
|
||||
s *signal
|
||||
m Message
|
||||
w string
|
||||
}{s, *s.owed, name})
|
||||
}
|
||||
}
|
||||
w.mu.Unlock()
|
||||
for _, o := range out {
|
||||
err := w.send(o.w, o.m)
|
||||
w.mu.Lock()
|
||||
if err == nil {
|
||||
o.s.owed = nil
|
||||
}
|
||||
w.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) what(s *signal) string {
|
||||
if s.name == SelfCheck {
|
||||
return "the controller's self-check (doctor)"
|
||||
}
|
||||
return "the bus"
|
||||
}
|
||||
|
||||
func (w *Watcher) silentMessage(s *signal, silent time.Duration, again bool) Message {
|
||||
title := "URGENT: self-check-silent: " + w.what(s) + " has not been heard for " + roughly(silent)
|
||||
if s.name == Bus {
|
||||
title = "URGENT: bus-silent: " + w.what(s) + " has not answered a round trip for " + roughly(silent)
|
||||
}
|
||||
if again {
|
||||
title = "STILL SILENT: " + title[len("URGENT: "):]
|
||||
}
|
||||
body := "bound: " + roughly(s.bound) + "; told by mesh-watcher on " + w.Node + ", directly over HTTPS, not through the mesh"
|
||||
if s.lastHeard.IsZero() {
|
||||
body += "\nnot heard once since the watcher started"
|
||||
} else {
|
||||
body += "\nlast heard: " + s.lastHeard.UTC().Format("2006-01-02 15:04") + " UTC"
|
||||
}
|
||||
if s.name == SelfCheck {
|
||||
body += "\nthe controller, the control node or the bus may be down; the mesh's own messages pass through them"
|
||||
}
|
||||
return Message{Title: title, Body: body}
|
||||
}
|
||||
|
||||
func (w *Watcher) send(signalName string, m Message) error {
|
||||
_, err := w.Telegram.Send(m)
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
note := sentNote{At: w.Now(), Signal: signalName, What: m.Title, Outcome: "sent"}
|
||||
if err != nil {
|
||||
note.Outcome = "failed: " + err.Error()
|
||||
if w.sendErr != err.Error() {
|
||||
w.Logf("[mesh-watcher] cannot send to Telegram (tried again every minute): %v", err)
|
||||
}
|
||||
w.sendErr = err.Error()
|
||||
} else {
|
||||
if w.sendErr != "" {
|
||||
w.Logf("[mesh-watcher] Telegram sends again")
|
||||
}
|
||||
w.sendErr = ""
|
||||
w.sendOK = w.Now()
|
||||
w.Logf("[mesh-watcher] told the operator: %s", m.Title)
|
||||
}
|
||||
w.sent = append(w.sent, note)
|
||||
if len(w.sent) > 100 {
|
||||
w.sent = w.sent[len(w.sent)-100:]
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// SignalStatus is one signal as the status says it.
|
||||
type SignalStatus struct {
|
||||
Signal string `json:"signal"`
|
||||
LastHeard string `json:"last_heard"`
|
||||
Ago string `json:"ago"`
|
||||
Bound string `json:"bound"`
|
||||
Silent bool `json:"silent"`
|
||||
SaidSilent string `json:"said_silent_at,omitempty"`
|
||||
Owed string `json:"not_sent_yet,omitempty"`
|
||||
LastRun string `json:"last_run,omitempty"`
|
||||
LastFailure string `json:"last_failure,omitempty"`
|
||||
}
|
||||
|
||||
// Status is the watcher's account of itself, its own health first.
|
||||
type Status struct {
|
||||
Verdict string `json:"verdict"`
|
||||
Node string `json:"watching_from"`
|
||||
Control string `json:"controller_and_bus_on,omitempty"`
|
||||
Misplaced string `json:"misplaced,omitempty"`
|
||||
Telegram string `json:"telegram"`
|
||||
LastSent string `json:"last_sent,omitempty"`
|
||||
Signals []SignalStatus `json:"signals"`
|
||||
BadBeats int `json:"unreadable_heartbeats"`
|
||||
LastBad string `json:"last_unreadable,omitempty"`
|
||||
Started string `json:"started"`
|
||||
Listening string `json:"listening"`
|
||||
}
|
||||
|
||||
func (w *Watcher) Status(listening string) Status {
|
||||
ready := w.Telegram.Ready()
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
now := w.Now()
|
||||
st := Status{Node: w.Node, Control: w.control, Misplaced: w.misplace, Started: w.started.UTC().Format(time.RFC3339),
|
||||
BadBeats: w.badBeats, LastBad: w.lastBad, Listening: listening}
|
||||
switch {
|
||||
case ready != nil:
|
||||
st.Telegram = "CANNOT SEND: " + ready.Error()
|
||||
case w.sendErr != "":
|
||||
st.Telegram = "FAILING: " + w.sendErr
|
||||
default:
|
||||
st.Telegram = "ready"
|
||||
}
|
||||
if !w.sendOK.IsZero() {
|
||||
st.LastSent = w.sendOK.UTC().Format(time.RFC3339)
|
||||
}
|
||||
anySilent := false
|
||||
for _, name := range []string{SelfCheck, Bus} {
|
||||
s := w.signals[name]
|
||||
ss := SignalStatus{Signal: name, Bound: roughly(s.bound), Silent: !s.silentAt.IsZero(), LastRun: s.lastRun, LastFailure: s.lastErr}
|
||||
if s.lastHeard.IsZero() {
|
||||
ss.LastHeard = "never since start"
|
||||
ss.Ago = roughly(now.Sub(w.started)) + " since start"
|
||||
} else {
|
||||
ss.LastHeard = s.lastHeard.UTC().Format(time.RFC3339)
|
||||
ss.Ago = roughly(now.Sub(s.lastHeard))
|
||||
}
|
||||
if ss.Silent {
|
||||
ss.SaidSilent = s.silentAt.UTC().Format(time.RFC3339)
|
||||
anySilent = true
|
||||
}
|
||||
if s.owed != nil {
|
||||
ss.Owed = s.owed.Title
|
||||
}
|
||||
st.Signals = append(st.Signals, ss)
|
||||
}
|
||||
switch {
|
||||
case st.Telegram != "ready":
|
||||
st.Verdict = "BLIND: the watcher cannot tell the operator anything — " + st.Telegram
|
||||
case w.misplace != "":
|
||||
st.Verdict = "MISPLACED: " + w.misplace
|
||||
case listening != "listening":
|
||||
st.Verdict = "NOT LISTENING: heartbeats cannot reach the watcher — " + listening
|
||||
case anySilent:
|
||||
st.Verdict = "ALARM: a watched signal is silent — see signals"
|
||||
default:
|
||||
st.Verdict = "ok: watching"
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// LastHeard is each signal's last word, and the recent sends.
|
||||
func (w *Watcher) LastHeard() map[string]any {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
now := w.Now()
|
||||
out := map[string]any{}
|
||||
for name, s := range w.signals {
|
||||
if s.lastHeard.IsZero() {
|
||||
out[name] = "never since the watcher started " + roughly(now.Sub(w.started)) + " ago"
|
||||
} else {
|
||||
out[name] = s.lastHeard.UTC().Format(time.RFC3339) + " (" + roughly(now.Sub(s.lastHeard)) + " ago)"
|
||||
}
|
||||
}
|
||||
sent := make([]sentNote, 0, len(w.sent))
|
||||
for i := len(w.sent) - 1; i >= 0; i-- {
|
||||
sent = append(sent, w.sent[i])
|
||||
}
|
||||
out["sent"] = sent
|
||||
return out
|
||||
}
|
||||
|
||||
func roughly(d time.Duration) string {
|
||||
switch {
|
||||
case d < 0:
|
||||
return "a moment"
|
||||
case d < 2*time.Minute:
|
||||
return fmt.Sprintf("%d s", int(d.Seconds()))
|
||||
case d < 2*time.Hour:
|
||||
return fmt.Sprintf("%d min", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%.1f h", d.Hours())
|
||||
}
|
||||
return fmt.Sprintf("%d days", int(d.Hours()/24))
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type fakeTelegram struct {
|
||||
notReady error
|
||||
fail error
|
||||
sent []Message
|
||||
}
|
||||
|
||||
func (f *fakeTelegram) Ready() error { return f.notReady }
|
||||
func (f *fakeTelegram) Send(m Message) (string, error) {
|
||||
if f.fail != nil {
|
||||
return "", f.fail
|
||||
}
|
||||
f.sent = append(f.sent, m)
|
||||
return "1", nil
|
||||
}
|
||||
|
||||
type clock struct{ t time.Time }
|
||||
|
||||
func (c *clock) now() time.Time { return c.t }
|
||||
func (c *clock) pass(d time.Duration) { c.t = c.t.Add(d) }
|
||||
|
||||
func watcher(t *testing.T) (*Watcher, *fakeTelegram, *clock) {
|
||||
c := &clock{t: time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)}
|
||||
tg := &fakeTelegram{}
|
||||
return NewWatcher(tg, c.now, t.Logf, "ace"), tg, c
|
||||
}
|
||||
|
||||
// beatEvery has the controller's doctor run, and the bus answer, every five minutes up to d.
|
||||
func beatEvery(w *Watcher, c *clock, d time.Duration) {
|
||||
for end := c.t.Add(d); c.t.Before(end); {
|
||||
c.pass(time.Minute)
|
||||
if c.t.Minute()%5 == 0 {
|
||||
w.Heartbeat(Beat{Run: "r", At: c.t, Interval: 5 * time.Minute})
|
||||
}
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeardSignalsSayNothing(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
beatEvery(w, c, 3*time.Hour)
|
||||
if len(tg.sent) != 0 {
|
||||
t.Fatalf("sent %v", tg.sent)
|
||||
}
|
||||
if st := w.Status("listening"); st.Verdict != "ok: watching" {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSelfCheckSilentPastTwiceItsIntervalIsSentOnceThenOnceMoreThenItsReturn(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
beatEvery(w, c, 30*time.Minute)
|
||||
last := c.t
|
||||
// The controller stops; the bus still answers.
|
||||
for i := 0; i < 9; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 0 {
|
||||
t.Fatalf("said silent within its bound (9 min of 10): %v", tg.sent)
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 1 || !strings.Contains(tg.sent[0].Title, "self-check-silent") {
|
||||
t.Fatalf("not said silent past twice the interval: %v", tg.sent)
|
||||
}
|
||||
if !strings.Contains(tg.sent[0].Body, "directly over HTTPS") || !strings.Contains(tg.sent[0].Body, last.Format("15:04")) {
|
||||
t.Fatalf("body: %q", tg.sent[0].Body)
|
||||
}
|
||||
if st := w.Status("listening"); !strings.HasPrefix(st.Verdict, "ALARM") {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
for i := 0; i < 70; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 2 || !strings.HasPrefix(tg.sent[1].Title, "STILL SILENT") {
|
||||
t.Fatalf("not said once more after an hour: %v", tg.sent)
|
||||
}
|
||||
beatEvery(w, c, 10*time.Minute)
|
||||
if len(tg.sent) != 3 || !strings.HasPrefix(tg.sent[2].Title, "CLEARED") {
|
||||
t.Fatalf("its return not said: %v", tg.sent)
|
||||
}
|
||||
beatEvery(w, c, time.Hour)
|
||||
if len(tg.sent) != 3 {
|
||||
t.Fatalf("said again after it returned: %v", tg.sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeartbeatNeverHeardIsSilenceFromTheStart(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
for i := 0; i < 11; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 1 || !strings.Contains(tg.sent[0].Body, "not heard once since the watcher started") {
|
||||
t.Fatalf("%v", tg.sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplayedOldHeartbeatIsNotASignOfLife(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
beatEvery(w, c, 10*time.Minute)
|
||||
old := c.t
|
||||
for i := 0; i < 11; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
// The watcher's consumer delivers a backlog: heartbeats made before the silence.
|
||||
w.Heartbeat(Beat{Run: "old", At: old.Add(-time.Minute)})
|
||||
w.Heartbeat(Beat{Run: "older", At: old.Add(-10 * time.Minute)})
|
||||
c.pass(time.Minute)
|
||||
w.Tick()
|
||||
if len(tg.sent) != 1 {
|
||||
t.Fatalf("a replay read as a return: %v", tg.sent)
|
||||
}
|
||||
w.Heartbeat(Beat{Run: "future", At: c.t.Add(time.Hour)})
|
||||
c.pass(time.Minute)
|
||||
w.Tick()
|
||||
if len(tg.sent) != 1 || w.Status("listening").BadBeats != 1 {
|
||||
t.Fatalf("a heartbeat from the future believed: %v", tg.sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheBoundFollowsTheIntervalTheHeartbeatSays(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
w.Heartbeat(Beat{At: c.t, Interval: 15 * time.Minute})
|
||||
for i := 0; i < 29; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 0 {
|
||||
t.Fatalf("said silent inside 2 × 15 min: %v", tg.sent)
|
||||
}
|
||||
c.pass(2 * time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
if len(tg.sent) != 1 {
|
||||
t.Fatalf("not said past 2 × 15 min")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheBusSilentPastThreeMinutesIsSent(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
beatEvery(w, c, 10*time.Minute)
|
||||
for i := 0; i < 4; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(errors.New("timeout"))
|
||||
w.Tick()
|
||||
}
|
||||
if len(tg.sent) != 1 || !strings.Contains(tg.sent[0].Title, "bus-silent") {
|
||||
t.Fatalf("%v", tg.sent)
|
||||
}
|
||||
if st := w.Status("listening"); st.Signals[1].LastFailure != "timeout" {
|
||||
t.Fatalf("%+v", st.Signals[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestATelegramThatCannotSendIsSaidAndTheMessageIsOwed(t *testing.T) {
|
||||
w, tg, c := watcher(t)
|
||||
tg.fail = errors.New("telegram sendMessage: cannot connect (dial)")
|
||||
for i := 0; i < 11; i++ {
|
||||
c.pass(time.Minute)
|
||||
w.BusAnswered(nil)
|
||||
w.Tick()
|
||||
}
|
||||
st := w.Status("listening")
|
||||
if !strings.HasPrefix(st.Verdict, "BLIND") || st.Signals[0].Owed == "" {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
tg.fail = nil
|
||||
c.pass(time.Minute)
|
||||
w.Tick()
|
||||
if len(tg.sent) != 1 || w.Status("listening").Signals[0].Owed != "" {
|
||||
t.Fatalf("the owed message was not sent: %v", tg.sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotGivenATokenIsBlind(t *testing.T) {
|
||||
w, tg, _ := watcher(t)
|
||||
tg.notReady = errors.New("no bot token")
|
||||
if st := w.Status("listening"); st.Verdict != "BLIND: the watcher cannot tell the operator anything — CANNOT SEND: no bot token" {
|
||||
t.Fatalf("%q", st.Verdict)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnTheControlNodeItSaysItIsMisplaced(t *testing.T) {
|
||||
w, _, _ := watcher(t)
|
||||
w.Placed([]string{"novox"})
|
||||
if st := w.Status("listening"); st.Verdict != "ok: watching" || st.Control != "novox" {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
w.Placed([]string{"ace"})
|
||||
if st := w.Status("listening"); !strings.HasPrefix(st.Verdict, "MISPLACED") {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSeatsAnswerIsReadInEitherShape(t *testing.T) {
|
||||
bare := `{"seats":[{"seat":"mesh-controller","holders":[{"node":"n1","module":"mesh-controller"}]},
|
||||
{"seat":"mesh-broker","holders":[{"node":"n1"},{"node":"n2"}]},{"seat":"git","holders":[{"node":"n3"}]}]}`
|
||||
wrapped, _ := json.Marshal(map[string]any{"content": []map[string]string{{"type": "text", "text": bare}}})
|
||||
output, _ := json.Marshal(map[string]any{"ok": true, "output": bare})
|
||||
for _, raw := range []string{bare, string(wrapped), string(output)} {
|
||||
got, err := holdersOf(json.RawMessage(raw), "mesh-controller", "mesh-broker")
|
||||
if err != nil || strings.Join(got, ",") != "n1,n2" {
|
||||
t.Errorf("%s: %v %v", raw, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheHeartbeatAsTheDesignSaysIt(t *testing.T) {
|
||||
b, err := DecodeBeat([]byte(`{"run":"doctor-41","at":"2026-10-06T12:05:00Z","interval-seconds":300,
|
||||
"counts":{"pass":10,"fail":1,"failed-to-run":0}}`))
|
||||
if err != nil || b.Run != "doctor-41" || b.Interval != 5*time.Minute || b.Counts["fail"] != 1 ||
|
||||
!b.At.Equal(time.Date(2026, 10, 6, 12, 5, 0, 0, time.UTC)) {
|
||||
t.Fatalf("%+v %v", b, err)
|
||||
}
|
||||
b, err = DecodeBeat([]byte(`{"run":7,"finished":"2026-10-06T12:05:00Z","interval":"5m"}`))
|
||||
if err != nil || b.Run != "7" || b.Interval != 5*time.Minute || b.At.IsZero() {
|
||||
t.Fatalf("%+v %v", b, err)
|
||||
}
|
||||
if b, err := DecodeBeat([]byte(`{}`)); err != nil || !b.At.IsZero() {
|
||||
t.Fatalf("an empty heartbeat: %+v %v", b, err)
|
||||
}
|
||||
for body, says := range map[string]string{
|
||||
`[]`: "not a JSON object",
|
||||
`{"at":"yesterday"}`: "not an RFC 3339 time",
|
||||
`{"interval":"often"}`: "not a duration",
|
||||
`{"interval-seconds":-5}`: "not one doctor runs at",
|
||||
`{"counts":"many"}`: "counts",
|
||||
} {
|
||||
if _, err := DecodeBeat([]byte(body)); err == nil || !strings.Contains(err.Error(), says) {
|
||||
t.Errorf("%s: %v", body, err)
|
||||
}
|
||||
}
|
||||
if !isHeartbeat("mesh-controller.doctor-heartbeat") || isHeartbeat("mesh-controller.applied") {
|
||||
t.Fatalf("the heartbeat's key")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
module mesh-watcher
|
||||
|
||||
go 1.22
|
||||
|
||||
require git.novox.be/novox/mesh-sdk/go v0.1.7
|
||||
@@ -0,0 +1,2 @@
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.7 h1:C0sTQmtTiyYH7bnqZb7PusXnqA37gKuT7Nqjn9gG47w=
|
||||
git.novox.be/novox/mesh-sdk/go v0.1.7/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
|
||||
Executable
BIN
Binary file not shown.
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"module": "mesh-watcher",
|
||||
"version": "1",
|
||||
"slug": "watch",
|
||||
"consumes": [
|
||||
"mesh-controller.doctor-heartbeat"
|
||||
],
|
||||
"invokes": [
|
||||
"mesh-watcher.watcher_ping",
|
||||
"seat:mesh-controller.seats"
|
||||
],
|
||||
"own-secrets": {
|
||||
"telegram-token": "${dir:state}/telegram-token"
|
||||
},
|
||||
"tools": [
|
||||
"watcher_status",
|
||||
"watcher_last_heard",
|
||||
"watcher_test",
|
||||
"watcher_ping"
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"id": "state",
|
||||
"type": "directory",
|
||||
"mode": "0700",
|
||||
"place": "."
|
||||
},
|
||||
{
|
||||
"id": "settings",
|
||||
"type": "file",
|
||||
"path": "${dir:state}/settings.json",
|
||||
"mode": "0600",
|
||||
"merge": "json",
|
||||
"content": "{\n \"telegram-chat-id\": \"\"\n}\n"
|
||||
}
|
||||
],
|
||||
"build": {
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "tools",
|
||||
"kind": "bundle",
|
||||
"language": "go",
|
||||
"system": "arch",
|
||||
"from": "cmd/mesh-watcher",
|
||||
"binary": "mesh-watcher",
|
||||
"loads": [
|
||||
"mesh-watcher"
|
||||
],
|
||||
"env": {
|
||||
"MESH_WATCHER_SETTINGS": "${dir:state}/settings.json",
|
||||
"MESH_WATCHER_TELEGRAM_TOKEN_FILE": "${dir:state}/telegram-token"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user