novox/hq
6
0
Fork 0
Code Issues Pull Requests 2 Releases Wiki Activity

Research 028: the mesh's output channel — how the mesh tells its operator what it noticed

Browse Source
This commit is contained in:
jochen
2026-10-04 15:36:03 +02:00
parent 6ac936f170
commit 73e6400802
4 changed files with 374 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/028-the-meshs-output-channel/00-overview.md
+71
View File
@@ -0,0 +1,71 @@
---
status: active
initiated: 2026-10-04
touches:
- 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md
- 04-ISSUES/229-a-rollout-cannot-be-followed-through-the-meshs-tools/00-report.md
- 04-ISSUES/230-a-host-that-hands-over-to-a-newer-one-loses-its-report-and-a-plan-waits-for-ever/00-report.md
- 04-ISSUES/233-a-host-without-its-package-managers-configuration-refuses-the-declaration-that-would-restore-it/00-report.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
- 02-DECISIONS/0208-the-graphical-session-is-one-module-per-piece-on-the-meshs-seats.md
- 02-DECISIONS/0210-a-tools-configuration-is-its-seat-holders-and-every-other-module-extends-it-through-the-seat.md
- 03-DESIGN/01-to-be/32-what-a-module-declares.md
became: []
---
# 028 — The mesh's output channel
## What
How the mesh tells its operator what it noticed. The operator's framing: sending notifications
is **an output channel for the mesh**. The mesh already knows when a machine stops answering, when
a failure repeats and will not fix itself, when a rollout waits for ever. Today it keeps that to
itself until someone asks.
The effort looks at:
- **the seat:** one, held once for the mesh, that every other part uses to say something to the
operator;
- **the channels**, each a module: a desktop notification on the machine the operator is at,
**Telegram**, a phone push service, chat, mail and others (see [02](02-the-channels.md));
- **the routing**, by severity and by where the operator is;
- **the life of a message:** deduplicated while it holds, resolved when it stops, acknowledged or
silenced by the operator;
- **the watcher's watcher:** who tells the operator when the parts that would tell them are the
ones that failed.
## Why
[Issue 187](../../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md) is the
class: *the mesh tells nobody when it stops working*. [01](01-what-the-mesh-already-knows.md)
counts it.
- 15 of the 236 issue reports say the fault was found because a person happened to look.
- 74 describe something failing silently.
On the day this effort opened, the mesh knew three things and told nobody:
- a workstation had refused every declaration for ninety minutes;
- the same workstation had been out of touch for ten minutes after an upgrade;
- one failure on the laptop had repeated thirteen times.
Every one of them was in `status`, for whoever asked.
The pieces exist. [To-be 32](../../03-DESIGN/01-to-be/32-what-a-module-declares.md) already uses a
`telegram-sender` seat as its worked example of a work queue with retention. ADR 0208 made a
machine's desktop notifier a node seat with a `send` verb. What is missing is a seat that speaks
for the mesh, sources that call it, and channels that deliver.
## What it touches
- **The controller**, which would become the first source of what it already computes for `status`.
- **The node-notifier seat**, which would become one channel among several.
- **Issues 187, 229, 230 and 233**, each of which ends in "and nothing said so".
- **ADR 0210**, because a channel extends the output seat through a contribution, and therefore
depends on it.
## Documents
1. [What the mesh already knows](01-what-the-mesh-already-knows.md): the evidence, and the events
that exist.
2. [The channels](02-the-channels.md): the candidates, Telegram first, weighed on the same questions.
3. [Open questions](03-open-questions.md): the seat, routing, life of a message, the watcher's
watcher, what may leave the mesh.
01-RESEARCH/028-the-meshs-output-channel/01-what-the-mesh-already-knows.md
+60
View File
@@ -0,0 +1,60 @@
# 01 — What the mesh already knows, and who hears it
## The count
Over the 236 issue reports in `04-ISSUES/`, on the day this effort opened:
- **15** say, in some wording, that a person found the fault by looking: "nobody was told",
"nothing logged / said / alerted / emitted", "a person asked", "found by a person". Five of them
are still open.
- **74** describe something that failed silently.
The search was a word match over the reports' text, so it undercounts reports that tell the same
story in other words. It never overcounts by much: each of the fifteen was read.
The fifteen fall into three groups:
- **The mesh knew, and kept it in a query.** The fault was in `status`, `plans` or a node's record,
for whoever asked. Examples: a rollout waiting for ever (230), a machine refusing every declaration
(233), a setting that cannot work stored and stopping the node (096).
- **The fault was in a log nothing reads.** Examples: the bus refusing the controller's publishes
(187), a dropped report (187, 230).
- **The fault was invisible to the mesh itself.** Examples: a resolver outside the mesh closed by its
filter (198), a port narrowed without saying (086).
Only the first group is a matter of telling: the fact exists, and only delivery is missing. The
other two need a source first. This effort is about the first, and about giving the other two a
place to say something once they can.
## What the controller computes and does not say
Read from `status` and `node show` on the day this effort opened. Each line is a fact the controller
already holds:
| fact | where it is today | example that day |
|---|---|---|
| a machine is out of touch | `node show`: "last heard from — out of touch 10m" | a workstation after an upgrade |
| a machine refused its declaration | `status`: "refused" with the reason | the same workstation, for 90 minutes |
| a failure repeats and will not fix itself | `status`: "stuck: the same failure N times since …" | 13 times on the laptop |
| machines run different hosts | `status`: the version table | after a host release |
| something runs that the mesh did not write | `node show`: strays | 16 containers on one machine |
| a filter rule the mesh did not write | `status` | one machine |
| a plan is waiting | `plans` | issue 230: "for 0s", for ever |
| an assignment does not compose | the `assign` answer only | issue 235 |
None of these is published. The bus carries a seat's own events (a build's outcome), a module's
declared events, tool calls and declarations. It carries no event for any line above.
## What exists to deliver with
- **A machine's desktop:** the `node-notifier` seat (ADR 0208), held on the laptop. Its `send` verb
shows a notification, and `history` lists them. It was used through the console the day this effort
opened.
- **Mail:** a mail module provides `smtp` to the mesh.
- **Chat:** a Matrix server runs as a module on the home server.
- **Home automation:** a home-automation module runs there too, and its phone app can receive pushes.
- **A seat shape for exactly this:** to-be 32 §5 uses `telegram-sender` (`accepts: send`,
`retain 7d`, `emits: delivered, failed`, `serves: status`) as its worked example. A seat's stream
exists from registration, so work queues until a holder appears.
No module sends to Telegram, a phone push service or SMS today.
01-RESEARCH/028-the-meshs-output-channel/02-the-channels.md
+116
View File
@@ -0,0 +1,116 @@
# 02 — The channels
Each channel is a candidate module that delivers what the output seat hands it. They are weighed on
the same questions:
- **Reach:** does it reach the operator away from the machines (phone), or only at a desk?
- **Off-mesh:** does it still work when the mesh's own parts (the bus, the controller, the control
node's network) are what failed?
- **Two-way:** can the operator answer through it: acknowledge, silence, ask?
- **Where the words go:** does the message leave the operator's own machines, and to whom?
- **What it costs to hold:** a secret, a server, an account, money.
## The candidates
### Telegram (required by the operator)
A bot created with Telegram's bot service sends to one chat: the operator's own, or a group.
- **Reach:** the phone and every desktop, with push.
- **Off-mesh:** sending needs only outbound HTTPS from any machine. No inbound port, no server of the
mesh's own. A second machine can hold the same bot token and send when the first is the one that
failed.
- **Two-way:** yes. Inline buttons on a message (acknowledge, silence for an hour) and commands to
the bot, read by long polling over outbound HTTPS. This makes Telegram the strongest candidate for
answering, and the riskiest (see [03](03-open-questions.md), Q7).
- **Where the words go:** to Telegram's servers. Bot chats are not end-to-end encrypted. What a
message may contain is therefore a rule this effort must set.
- **Cost:** one secret (the bot token) and the chat's id. Free. Rate limits are far above what an
operator should receive.
- **Formatting:** short text with a little markup, buttons and links. Enough for a subject, a
machine role, a severity and one line of why.
### The desktop notifier (exists)
The `node-notifier` seat's `send` verb on the machine the operator is at.
- **Reach:** only at that machine, only while a session is up.
- **Off-mesh:** no. It is reached through the mesh's tools.
- **Two-way:** dunst has actions, which a click can answer, but nothing reads them back yet.
- **Where the words go:** nowhere; it is local.
- **Cost:** none.
- **Its place:** the gentlest channel, for a warning while the operator is at a desk. "At a desk" is
itself a question: an unlocked session on a machine with recent input.
### ntfy (or Gotify): a self-hosted phone push
A small server publishes topics; its phone app subscribes.
- **Reach:** the phone, with push.
- **Off-mesh:** only if the server runs outside what failed. On the control node it fails with it.
- **Two-way:** action buttons can call a URL, which is an inbound path to design.
- **Where the words go:** stays on the operator's machines when self-hosted. ntfy's iOS push passes
through an upstream relay unless configured otherwise.
- **Cost:** a module with a container and a routed name; a token per topic.
### Matrix (a server exists as a module)
A bot account posts to a room the operator is in.
- **Reach:** phone and desktop through any Matrix client.
- **Off-mesh:** no, the server is one of the mesh's modules.
- **Two-way:** yes, by messages to the bot.
- **Where the words go:** stays on the operator's server, end-to-end encrypted if the bot supports it.
- **Cost:** a bot account, a secret.
### Mail (a mail module provides `smtp`)
- **Reach:** everywhere, without urgency.
- **Off-mesh:** no, if the mesh's own mail server sends. Yes, through an outside relay.
- **Two-way:** no, not usefully.
- **Its place:** the record and the digest: a daily summary of what was said and resolved, and the
fallback when nothing else acknowledged.
### The home-automation companion app (a module exists)
Its phone app takes pushes and actionable notifications, and the home has lights and speakers.
- **Reach:** the phone, and the house itself: a light that turns a colour.
- **Off-mesh:** no, the home server is a node.
- **Its place:** a playful critical channel, not a primary one.
### The bar on the desktop
An `i3status-rust` block showing the count of open messages, red while one is critical.
- **Reach:** the desk only, and silent.
- **Its place:** the ambient state. Nothing interrupts the operator, and they always see whether
something is open.
### The console (an agent session)
A message the next agent session opens with ("two things happened while you were away").
- **Its place:** context for the agent working on the mesh rather than an alert. It falls out of the
message store if the store is queryable.
### Others, noted and not pursued now
- **SMS or a voice call** through a paid gateway. It is the only channel that works with no data
connection, and the only one that costs per message.
- **Signal**, through an unofficial client: no bot API, and a registered number.
- **Discord or Slack** webhooks: the words go to a third party, as with Telegram, without its two-way
strength.
- **Pushover:** paid, closed, and a phone push service much like ntfy.
- **An external dead-man service** (a heartbeat URL that alerts when pings stop). It belongs to
[03](03-open-questions.md), Q6, as the watcher's watcher rather than as a channel.
## A first reading
- **Telegram** is the primary phone channel, and the only candidate that is cheap, off-mesh capable
and two-way at once.
- **The desktop notifier** is for the desk.
- **The bar** shows the ambient state.
- **Mail** carries the digest and the record.
- **ntfy and Matrix** are self-hosted alternatives for an operator who keeps words off third parties.
The seat must make that a choice, not a rewrite.
01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md
+127
View File
@@ -0,0 +1,127 @@
# 03 — Open questions
Each question names the options seen so far. None is decided here.
## Q1. The seat
**What speaks for the mesh to its operator?**
- **a.** One seat in the mesh's own set, held once for the mesh. Working name: `operator-channel`.
- It **accepts** `notify` (a work queue, as to-be 32 §5 designs `telegram-sender`), so a message
waits until a holder appears.
- It **emits** `delivered`, `acknowledged` and `resolved`.
- It **serves** `open` (what is unresolved now) and `history`.
- **b.** No seat: every source calls every channel. Rejected in advance, because each source would
learn every channel. This is the inversion ADR 0126 exists to prevent.
- **c.** Each channel as its own seat, with routing in the sources. Same objection as b, one level up.
Under a, the holder routes. The channels are modules that **contribute** themselves to the seat
(ADR 0210): a channel extends the seat, and so depends on it. Whether the holder is a module of its
own or part of the controller is open. A module keeps the controller small. The controller already
holds most of the facts.
## Q2. What a message is
The first shape seen: a **subject** (what it is about: a machine's role, a module, a plan), a
**kind** (out of touch, refused, stuck, late, …), a **severity**, a one-line **why**, a link to
the tool that shows more, and a **key** that makes it the same message the next time it is said.
- **Severity:** two levels (needs you now / when you can), or three (critical / warning / info)?
Every extra level is a routing rule somebody must keep right.
- **The key** is what makes deduplication possible. "Machine X out of touch" said every minute is
one message, still open, not sixty.
## Q3. The life of a message
open → (acknowledged) → resolved.
- **Deduplicate** by key while open.
- **Resolve** when the source stops saying it, or says it is over ("back in touch after 14 min"). A
channel that can edit its message (Telegram can) updates it in place rather than sending a second.
- **Acknowledge** from any channel that can answer, which stops escalation and repeats.
- **Repeat or escalate** an unacknowledged critical message after a while, to the next channel.
- **Where the open set lives:** the seat's own state, in a key-value bucket (to-be 32's `state:`), so
`open` answers after a restart.
## Q4. Routing and presence
- **By severity:** critical goes to every channel at once. A warning goes to the desk when the
operator is at one, otherwise to the phone, otherwise to the digest.
- **Presence:** "at a desk" needs a fact the mesh does not hold yet. Candidates: an unlocked
graphical session with recent input, read from the `node-lock-screen` and `node-login-manager`
seats' holders. Nothing more invasive.
- **Quiet hours:** a setting of the seat's holder (ADR 0174). Critical overrides it, or not, as the
operator chooses.
- **Rate:** a cap per hour per channel, with the excess folded into one summary, so a storm (a
network outage where every node is out of touch) arrives as one message naming many.
## Q5. The sources
The first sources are the facts in [01](01-what-the-mesh-already-knows.md), all in the controller
today:
- a machine out of touch;
- a declaration refused;
- a stuck failure;
- a plan late (once issue 230 gives a wait an age);
- an assignment that does not compose (issue 235);
- a host version split.
**How each becomes an event:**
- **a.** The controller emits an event per change of state, and the seat's holder consumes them.
- **b.** The controller calls `notify` itself.
With a, the controller learns nothing about telling: other consumers (a board, a log) get the same
facts, and the holder decides what is worth a message. With b, the controller decides severity.
**Modules as sources:** a module may `use` the seat to tell the operator something of its own
(a backup failed, a certificate is close to expiry), with the same message shape.
## Q6. The watcher's watcher
When the controller, the bus or the control node is what failed, nothing above runs. Options:
- **A dead-man signal:** the seat's holder sends a heartbeat out of the mesh (a ping to an external
heartbeat service), which alerts the operator by its own means when pings stop.
- **A second holder of the Telegram channel on another machine** that sends directly, without the
bus, when it stops hearing the controller for longer than a bound.
- **Each host** sending a last message itself when it loses the mesh for longer than a bound. This
needs the channel's secret on every machine, a cost to weigh.
The first is the cheapest and the only one that also covers "the whole house is offline".
## Q7. Answering back
Telegram, and Matrix, can carry the operator's answers.
- **Acknowledge and silence** are safe: they change only the message's state.
- **Commands** ("push the workstation", "show status") turn a chat account into a door to the
controller. If it ever comes, it needs:
- its own record;
- a narrow verb set;
- a check that the answer came from the operator's own account and chat;
- and probably a confirmation step.
The first version should probably answer with acknowledge and silence only.
## Q8. What may leave the mesh
Telegram, and any third-party channel, carries the words to someone else's servers. A message
names a machine, a module and a reason, which is operational detail.
- **What may a message contain?** Roles rather than addresses; no secrets, tokens or paths; a reason
in words. The rule must be enforced by the seat's holder, not hoped for from each source.
- **Is a self-hosted channel required for anything above a severity?**
- **The bot token and chat id** are secrets of the channel's module, delivered as any module secret is.
## Q9. How it is checked
A rule this effort produces must say how it is verified. Candidates:
- a message said twice with one key is one message;
- a resolved source resolves its message;
- a critical message reaches every channel within a bound;
- a message containing an address or a secret is refused;
- the dead-man signal fires when the holder is stopped.
Each is a test of the holder, or a live drill: stop a machine's host and time the message.