diff --git a/01-RESEARCH/028-the-meshs-output-channel/00-overview.md b/01-RESEARCH/028-the-meshs-output-channel/00-overview.md new file mode 100644 index 0000000..b0f90db --- /dev/null +++ b/01-RESEARCH/028-the-meshs-output-channel/00-overview.md @@ -0,0 +1,71 @@ +--- +status: active +initiated: 2026-10-04 +touches: + - 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md + - 04-ISSUES/229-a-rollout-cannot-be-followed-through-the-meshs-tools/00-report.md + - 04-ISSUES/230-a-host-that-hands-over-to-a-newer-one-loses-its-report-and-a-plan-waits-for-ever/00-report.md + - 04-ISSUES/233-a-host-without-its-package-managers-configuration-refuses-the-declaration-that-would-restore-it/00-report.md + - 02-DECISIONS/0126-a-module-declares-its-own-seats.md + - 02-DECISIONS/0208-the-graphical-session-is-one-module-per-piece-on-the-meshs-seats.md + - 02-DECISIONS/0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md + - 03-DESIGN/01-to-be/32-what-a-module-declares.md +became: [] +--- + +# 028 — The mesh's output channel + +## What + +How the mesh tells its operator what it noticed. The operator's framing: sending notifications +is **an output channel for the mesh**. The mesh already knows when a machine stops answering, when +a failure repeats and will not fix itself, when a rollout waits for ever. Today it keeps that to +itself until someone asks. + +The effort looks at: + +- **the seat:** one, held once for the mesh, that every other part uses to say something to the + operator; +- **the channels**, each a module: a desktop notification on the machine the operator is at, + **Telegram**, a phone push service, chat, mail and others (see [02](02-the-channels.md)); +- **the routing**, by severity and by where the operator is; +- **the life of a message:** deduplicated while it holds, resolved when it stops, acknowledged or + silenced by the operator; +- **the watcher's watcher:** who tells the operator when the parts that would tell them are the + ones that failed. + +## Why + +[Issue 187](../../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md) is the +class: *the mesh tells nobody when it stops working*. [01](01-what-the-mesh-already-knows.md) +counts it. +- 15 of the 236 issue reports say the fault was found because a person happened to look. +- 74 describe something failing silently. + +On the day this effort opened, the mesh knew three things and told nobody: +- a workstation had refused every declaration for ninety minutes; +- the same workstation had been out of touch for ten minutes after an upgrade; +- one failure on the laptop had repeated thirteen times. + +Every one of them was in `status`, for whoever asked. + +The pieces exist. [To-be 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) already uses a +`telegram-sender` seat as its worked example of a work queue with retention. ADR 0208 made a +machine's desktop notifier a node seat with a `send` verb. What is missing is a seat that speaks +for the mesh, sources that call it, and channels that deliver. + +## What it touches + +- **The controller**, which would become the first source of what it already computes for `status`. +- **The node-notifier seat**, which would become one channel among several. +- **Issues 187, 229, 230 and 233**, each of which ends in "and nothing said so". +- **ADR 0210**, because a channel extends the output seat through a contribution, and therefore + depends on it. + +## Documents + +1. [What the mesh already knows](01-what-the-mesh-already-knows.md): the evidence, and the events + that exist. +2. [The channels](02-the-channels.md): the candidates, Telegram first, weighed on the same questions. +3. [Open questions](03-open-questions.md): the seat, routing, life of a message, the watcher's + watcher, what may leave the mesh. diff --git a/01-RESEARCH/028-the-meshs-output-channel/01-what-the-mesh-already-knows.md b/01-RESEARCH/028-the-meshs-output-channel/01-what-the-mesh-already-knows.md new file mode 100644 index 0000000..1dfd8e5 --- /dev/null +++ b/01-RESEARCH/028-the-meshs-output-channel/01-what-the-mesh-already-knows.md @@ -0,0 +1,60 @@ +# 01 — What the mesh already knows, and who hears it + +## The count + +Over the 236 issue reports in `04-ISSUES/`, on the day this effort opened: + +- **15** say, in some wording, that a person found the fault by looking: "nobody was told", + "nothing logged / said / alerted / emitted", "a person asked", "found by a person". Five of them + are still open. +- **74** describe something that failed silently. + +The search was a word match over the reports' text, so it undercounts reports that tell the same +story in other words. It never overcounts by much: each of the fifteen was read. + +The fifteen fall into three groups: + +- **The mesh knew, and kept it in a query.** The fault was in `status`, `plans` or a node's record, + for whoever asked. Examples: a rollout waiting for ever (230), a machine refusing every declaration + (233), a setting that cannot work stored and stopping the node (096). +- **The fault was in a log nothing reads.** Examples: the bus refusing the controller's publishes + (187), a dropped report (187, 230). +- **The fault was invisible to the mesh itself.** Examples: a resolver outside the mesh closed by its + filter (198), a port narrowed without saying (086). + +Only the first group is a matter of telling: the fact exists, and only delivery is missing. The +other two need a source first. This effort is about the first, and about giving the other two a +place to say something once they can. + +## What the controller computes and does not say + +Read from `status` and `node show` on the day this effort opened. Each line is a fact the controller +already holds: + +| fact | where it is today | example that day | +|---|---|---| +| a machine is out of touch | `node show`: "last heard from — out of touch 10m" | a workstation after an upgrade | +| a machine refused its declaration | `status`: "refused" with the reason | the same workstation, for 90 minutes | +| a failure repeats and will not fix itself | `status`: "stuck: the same failure N times since …" | 13 times on the laptop | +| machines run different hosts | `status`: the version table | after a host release | +| something runs that the mesh did not write | `node show`: strays | 16 containers on one machine | +| a filter rule the mesh did not write | `status` | one machine | +| a plan is waiting | `plans` | issue 230: "for 0s", for ever | +| an assignment does not compose | the `assign` answer only | issue 235 | + +None of these is published. The bus carries a seat's own events (a build's outcome), a module's +declared events, tool calls and declarations. It carries no event for any line above. + +## What exists to deliver with + +- **A machine's desktop:** the `node-notifier` seat (ADR 0208), held on the laptop. Its `send` verb + shows a notification, and `history` lists them. It was used through the console the day this effort + opened. +- **Mail:** a mail module provides `smtp` to the mesh. +- **Chat:** a Matrix server runs as a module on the home server. +- **Home automation:** a home-automation module runs there too, and its phone app can receive pushes. +- **A seat shape for exactly this:** to-be 32 §5 uses `telegram-sender` (`accepts: send`, + `retain 7d`, `emits: delivered, failed`, `serves: status`) as its worked example. A seat's stream + exists from registration, so work queues until a holder appears. + +No module sends to Telegram, a phone push service or SMS today. diff --git a/01-RESEARCH/028-the-meshs-output-channel/02-the-channels.md b/01-RESEARCH/028-the-meshs-output-channel/02-the-channels.md new file mode 100644 index 0000000..1419126 --- /dev/null +++ b/01-RESEARCH/028-the-meshs-output-channel/02-the-channels.md @@ -0,0 +1,116 @@ +# 02 — The channels + +Each channel is a candidate module that delivers what the output seat hands it. They are weighed on +the same questions: + +- **Reach:** does it reach the operator away from the machines (phone), or only at a desk? +- **Off-mesh:** does it still work when the mesh's own parts (the bus, the controller, the control + node's network) are what failed? +- **Two-way:** can the operator answer through it: acknowledge, silence, ask? +- **Where the words go:** does the message leave the operator's own machines, and to whom? +- **What it costs to hold:** a secret, a server, an account, money. + +## The candidates + +### Telegram (required by the operator) + +A bot created with Telegram's bot service sends to one chat: the operator's own, or a group. + +- **Reach:** the phone and every desktop, with push. +- **Off-mesh:** sending needs only outbound HTTPS from any machine. No inbound port, no server of the + mesh's own. A second machine can hold the same bot token and send when the first is the one that + failed. +- **Two-way:** yes. Inline buttons on a message (acknowledge, silence for an hour) and commands to + the bot, read by long polling over outbound HTTPS. This makes Telegram the strongest candidate for + answering, and the riskiest (see [03](03-open-questions.md), Q7). +- **Where the words go:** to Telegram's servers. Bot chats are not end-to-end encrypted. What a + message may contain is therefore a rule this effort must set. +- **Cost:** one secret (the bot token) and the chat's id. Free. Rate limits are far above what an + operator should receive. +- **Formatting:** short text with a little markup, buttons and links. Enough for a subject, a + machine role, a severity and one line of why. + +### The desktop notifier (exists) + +The `node-notifier` seat's `send` verb on the machine the operator is at. + +- **Reach:** only at that machine, only while a session is up. +- **Off-mesh:** no. It is reached through the mesh's tools. +- **Two-way:** dunst has actions, which a click can answer, but nothing reads them back yet. +- **Where the words go:** nowhere; it is local. +- **Cost:** none. +- **Its place:** the gentlest channel, for a warning while the operator is at a desk. "At a desk" is + itself a question: an unlocked session on a machine with recent input. + +### ntfy (or Gotify): a self-hosted phone push + +A small server publishes topics; its phone app subscribes. + +- **Reach:** the phone, with push. +- **Off-mesh:** only if the server runs outside what failed. On the control node it fails with it. +- **Two-way:** action buttons can call a URL, which is an inbound path to design. +- **Where the words go:** stays on the operator's machines when self-hosted. ntfy's iOS push passes + through an upstream relay unless configured otherwise. +- **Cost:** a module with a container and a routed name; a token per topic. + +### Matrix (a server exists as a module) + +A bot account posts to a room the operator is in. + +- **Reach:** phone and desktop through any Matrix client. +- **Off-mesh:** no, the server is one of the mesh's modules. +- **Two-way:** yes, by messages to the bot. +- **Where the words go:** stays on the operator's server, end-to-end encrypted if the bot supports it. +- **Cost:** a bot account, a secret. + +### Mail (a mail module provides `smtp`) + +- **Reach:** everywhere, without urgency. +- **Off-mesh:** no, if the mesh's own mail server sends. Yes, through an outside relay. +- **Two-way:** no, not usefully. +- **Its place:** the record and the digest: a daily summary of what was said and resolved, and the + fallback when nothing else acknowledged. + +### The home-automation companion app (a module exists) + +Its phone app takes pushes and actionable notifications, and the home has lights and speakers. + +- **Reach:** the phone, and the house itself: a light that turns a colour. +- **Off-mesh:** no, the home server is a node. +- **Its place:** a playful critical channel, not a primary one. + +### The bar on the desktop + +An `i3status-rust` block showing the count of open messages, red while one is critical. + +- **Reach:** the desk only, and silent. +- **Its place:** the ambient state. Nothing interrupts the operator, and they always see whether + something is open. + +### The console (an agent session) + +A message the next agent session opens with ("two things happened while you were away"). + +- **Its place:** context for the agent working on the mesh rather than an alert. It falls out of the + message store if the store is queryable. + +### Others, noted and not pursued now + +- **SMS or a voice call** through a paid gateway. It is the only channel that works with no data + connection, and the only one that costs per message. +- **Signal**, through an unofficial client: no bot API, and a registered number. +- **Discord or Slack** webhooks: the words go to a third party, as with Telegram, without its two-way + strength. +- **Pushover:** paid, closed, and a phone push service much like ntfy. +- **An external dead-man service** (a heartbeat URL that alerts when pings stop). It belongs to + [03](03-open-questions.md), Q6, as the watcher's watcher rather than as a channel. + +## A first reading + +- **Telegram** is the primary phone channel, and the only candidate that is cheap, off-mesh capable + and two-way at once. +- **The desktop notifier** is for the desk. +- **The bar** shows the ambient state. +- **Mail** carries the digest and the record. +- **ntfy and Matrix** are self-hosted alternatives for an operator who keeps words off third parties. + The seat must make that a choice, not a rewrite. diff --git a/01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md b/01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md new file mode 100644 index 0000000..4a0ddba --- /dev/null +++ b/01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md @@ -0,0 +1,127 @@ +# 03 — Open questions + +Each question names the options seen so far. None is decided here. + +## Q1. The seat + +**What speaks for the mesh to its operator?** + +- **a.** One seat in the mesh's own set, held once for the mesh. Working name: `operator-channel`. + - It **accepts** `notify` (a work queue, as to-be 32 §5 designs `telegram-sender`), so a message + waits until a holder appears. + - It **emits** `delivered`, `acknowledged` and `resolved`. + - It **serves** `open` (what is unresolved now) and `history`. +- **b.** No seat: every source calls every channel. Rejected in advance, because each source would + learn every channel. This is the inversion ADR 0126 exists to prevent. +- **c.** Each channel as its own seat, with routing in the sources. Same objection as b, one level up. + +Under a, the holder routes. The channels are modules that **contribute** themselves to the seat +(ADR 0210): a channel extends the seat, and so depends on it. Whether the holder is a module of its +own or part of the controller is open. A module keeps the controller small. The controller already +holds most of the facts. + +## Q2. What a message is + +The first shape seen: a **subject** (what it is about: a machine's role, a module, a plan), a +**kind** (out of touch, refused, stuck, late, …), a **severity**, a one-line **why**, a link to +the tool that shows more, and a **key** that makes it the same message the next time it is said. + +- **Severity:** two levels (needs you now / when you can), or three (critical / warning / info)? + Every extra level is a routing rule somebody must keep right. +- **The key** is what makes deduplication possible. "Machine X out of touch" said every minute is + one message, still open, not sixty. + +## Q3. The life of a message + +open → (acknowledged) → resolved. + +- **Deduplicate** by key while open. +- **Resolve** when the source stops saying it, or says it is over ("back in touch after 14 min"). A + channel that can edit its message (Telegram can) updates it in place rather than sending a second. +- **Acknowledge** from any channel that can answer, which stops escalation and repeats. +- **Repeat or escalate** an unacknowledged critical message after a while, to the next channel. +- **Where the open set lives:** the seat's own state, in a key-value bucket (to-be 32's `state:`), so + `open` answers after a restart. + +## Q4. Routing and presence + +- **By severity:** critical goes to every channel at once. A warning goes to the desk when the + operator is at one, otherwise to the phone, otherwise to the digest. +- **Presence:** "at a desk" needs a fact the mesh does not hold yet. Candidates: an unlocked + graphical session with recent input, read from the `node-lock-screen` and `node-login-manager` + seats' holders. Nothing more invasive. +- **Quiet hours:** a setting of the seat's holder (ADR 0174). Critical overrides it, or not, as the + operator chooses. +- **Rate:** a cap per hour per channel, with the excess folded into one summary, so a storm (a + network outage where every node is out of touch) arrives as one message naming many. + +## Q5. The sources + +The first sources are the facts in [01](01-what-the-mesh-already-knows.md), all in the controller +today: + +- a machine out of touch; +- a declaration refused; +- a stuck failure; +- a plan late (once issue 230 gives a wait an age); +- an assignment that does not compose (issue 235); +- a host version split. + +**How each becomes an event:** +- **a.** The controller emits an event per change of state, and the seat's holder consumes them. +- **b.** The controller calls `notify` itself. + +With a, the controller learns nothing about telling: other consumers (a board, a log) get the same +facts, and the holder decides what is worth a message. With b, the controller decides severity. + +**Modules as sources:** a module may `use` the seat to tell the operator something of its own +(a backup failed, a certificate is close to expiry), with the same message shape. + +## Q6. The watcher's watcher + +When the controller, the bus or the control node is what failed, nothing above runs. Options: + +- **A dead-man signal:** the seat's holder sends a heartbeat out of the mesh (a ping to an external + heartbeat service), which alerts the operator by its own means when pings stop. +- **A second holder of the Telegram channel on another machine** that sends directly, without the + bus, when it stops hearing the controller for longer than a bound. +- **Each host** sending a last message itself when it loses the mesh for longer than a bound. This + needs the channel's secret on every machine, a cost to weigh. + +The first is the cheapest and the only one that also covers "the whole house is offline". + +## Q7. Answering back + +Telegram, and Matrix, can carry the operator's answers. + +- **Acknowledge and silence** are safe: they change only the message's state. +- **Commands** ("push the workstation", "show status") turn a chat account into a door to the + controller. If it ever comes, it needs: + - its own record; + - a narrow verb set; + - a check that the answer came from the operator's own account and chat; + - and probably a confirmation step. + +The first version should probably answer with acknowledge and silence only. + +## Q8. What may leave the mesh + +Telegram, and any third-party channel, carries the words to someone else's servers. A message +names a machine, a module and a reason, which is operational detail. + +- **What may a message contain?** Roles rather than addresses; no secrets, tokens or paths; a reason + in words. The rule must be enforced by the seat's holder, not hoped for from each source. +- **Is a self-hosted channel required for anything above a severity?** +- **The bot token and chat id** are secrets of the channel's module, delivered as any module secret is. + +## Q9. How it is checked + +A rule this effort produces must say how it is verified. Candidates: + +- a message said twice with one key is one message; +- a resolved source resolves its message; +- a critical message reaches every channel within a bound; +- a message containing an address or a secret is refused; +- the dead-man signal fires when the holder is stopped. + +Each is a test of the holder, or a live drill: stop a machine's host and time the message.