End every problem message with whether a delivery was running on what it is about (ADR 0505 §1)
This commit is contained in:
@@ -190,17 +190,28 @@ the next minute's sweep.
|
||||
history. An event that cannot be read
|
||||
is refused by name, counted, and told to the operator once an hour.
|
||||
|
||||
## While a delivery walks (novox/hq ADR 0308)
|
||||
## While a delivery walks (novox/hq ADR 0308, ADR 0505)
|
||||
|
||||
The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`started`, `ended`), `touches`
|
||||
(`[{machine, modules}]`: what it moves and what a moved tool runner or node-engine carries), `outcome`
|
||||
(`[{machine, modules, declared}]`: what it moves and what a moved tool runner or node-engine carries, and, marked
|
||||
`declared`, a machine it sends nothing to whose declaration its merges change), `outcome`
|
||||
(`delivered`, `failed`, `stopped`), `bound_seconds` and `budget_class` (`cmd/messenger/walks.go`).
|
||||
|
||||
- **Held:** while a walk is open and within its bound, a message whose subject is a machine it touches, or a
|
||||
- **Held:** while a walk is open and within its bound and settle, a message whose subject is a machine it touches, or a
|
||||
module it touches (on a machine it touches, when the subject names one), is held: kept on the walk with when
|
||||
and what, and the history says `held`. A message several open walks touch is told when the last ends or passes
|
||||
its bound. At a walk's end, a later open walk that touches a message takes it over only as a first hold would:
|
||||
still open, not silenced, not said since, and not one that is never held.
|
||||
still open, not silenced, not said since, and not one that is never held. The controller's own lease, bus and
|
||||
health conditions (`core.controller.…`, `bus.controller.…`) are about the module `mesh-controller`, wherever a
|
||||
walk moves it (ADR 0505 §2).
|
||||
- **The settle (ADR 0505 §3, §6):** a walk that ended delivered holds on for the setting `settle-seconds`
|
||||
(120 by default): what it touched stays held, a condition raised then on what it touched is held too, and its
|
||||
end is told when the settle has passed. A walk that failed or stopped is told at once; for its settle after
|
||||
that, a condition on what it touched is sent at once, never held.
|
||||
- **The delivery line (ADR 0505 §1):** every problem's message (raised, asked, reopened, escalated, reminded; not
|
||||
a clearing, a silencing, a plain notice or an ask's own message) ends, after its since-when line, with "During
|
||||
the delivery of <modules> to <machines>.", "<n> after the delivery of <modules> to <machines>, which
|
||||
succeeded/failed/was stopped." or "No delivery running.", by the walks that touch its subject.
|
||||
- **Never held:** an ask waiting for the operator (and a condition an open ask is about), a secret given or
|
||||
replaced, a subject no open walk touches, and a walk's own failure (a plan, walk, delivery or build).
|
||||
- **The end is one message:** the outcome, the modules and machines reached, how long; each held message still
|
||||
@@ -209,7 +220,7 @@ The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`
|
||||
only once the message was sent there. Until then each is kept owed in the state (`owed`): a message lost with a
|
||||
restart before it was sent leaves them untold and owed, and the restarted router says each on its own. A
|
||||
record that clears owes nothing more.
|
||||
- **The bound:** at its start plus `bound_seconds` (900 when not given), what is held and still open is said,
|
||||
- **The bound:** at its start plus `bound_seconds` (900 when not given) plus its settle (ADR 0505 §7), what is held and still open is said,
|
||||
with "The walk … is still running, since HH:MM", and the cleared counted; the walk holds nothing more. A walk
|
||||
whose end never comes is forgotten a day after it started. A start heard after its end opens nothing: the
|
||||
controller's moments are compared on its own clock, never with one capped at the router's now.
|
||||
@@ -221,6 +232,7 @@ The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`
|
||||
said in it; not given, the zone of the machine this runs on.
|
||||
**The kinds that say only what needs the operator**, as a setting: `{"only-what-needs-me": ["<kind>"]}`. A
|
||||
walk that delivered with nothing held and nothing open is not told on them (ADR 0308 §4); empty by default.
|
||||
**The settle**, as a setting: `{"settle-seconds": 120}`: how long a delivered walk still holds (ADR 0505 §3).
|
||||
2. **The channels**: assign `telegram` and `desk-channel` (each says in its README what it needs). Linking
|
||||
an account as the operator is done on the channel, never by a setting.
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
@@ -245,3 +246,73 @@ func TestAnUrgentConditionOnAnUntouchedMachineIsSentAtOnceDuringAWalk(t *testing
|
||||
t.Fatalf("held, while no walk touches ace: %v", outcomes(h, k.Key))
|
||||
}
|
||||
}
|
||||
|
||||
// bodyOf is a held or open record's message as it would be sent now.
|
||||
func bodyOf(t *testing.T, h *Holder, key, what string) string {
|
||||
t.Helper()
|
||||
h.mu.Lock()
|
||||
r := h.open[key]
|
||||
h.mu.Unlock()
|
||||
if r == nil {
|
||||
t.Fatalf("%s is not open", key)
|
||||
}
|
||||
return h.say(r, what).Body
|
||||
}
|
||||
|
||||
// ADR 0505 §1: a problem's message ends, after its since-when line, with the delivery line: during a walk, in a
|
||||
// walk's settle, after a failed walk, and with no walk.
|
||||
func TestEveryProblemMessageEndsWithWhetherADeliveryRan(t *testing.T) {
|
||||
h, tg, dt, c, _ := newHolder(t)
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
if !strings.HasSuffix(tg.sends[0].Body, ".\nNo delivery running.") || !strings.Contains(tg.sends[0].Body, "\nSince ") {
|
||||
t.Fatalf("no walk: %q", tg.sends[0].Body)
|
||||
}
|
||||
walk(t, h, "plan-l", WalkStarted, "", c.now(), 600, novox("gitea", "mesh-issues"), shanksDeclared)
|
||||
if b := bodyOf(t, h, k.Key, "reminder"); !strings.HasSuffix(b, ".\nDuring the delivery of gitea, mesh-issues to novox.") {
|
||||
t.Fatalf("during a walk: %q", b)
|
||||
}
|
||||
runFor(h, c, time.Minute)
|
||||
walk(t, h, "plan-l", WalkEnded, WalkDelivered, c.now(), 0)
|
||||
c.pass(40 * time.Second)
|
||||
if b := bodyOf(t, h, k.Key, "raised"); !strings.HasSuffix(b, "\n40 s after the delivery of gitea, mesh-issues to novox, which succeeded.") {
|
||||
t.Fatalf("in the settle of a delivered walk: %q", b)
|
||||
}
|
||||
runFor(h, c, 3*time.Minute)
|
||||
if b := bodyOf(t, h, k.Key, "reminder"); !strings.HasSuffix(b, "\nNo delivery running.") {
|
||||
t.Fatalf("after the settle: %q", b)
|
||||
}
|
||||
walk(t, h, "plan-m", WalkStarted, "", c.now(), 600, novox("gitea"))
|
||||
walk(t, h, "plan-m", WalkEnded, WalkFailed, c.now(), 0)
|
||||
c.pass(30 * time.Second)
|
||||
if b := bodyOf(t, h, k.Key, "reminder"); !strings.HasSuffix(b, "\n30 s after the delivery of gitea to novox, which failed.") {
|
||||
t.Fatalf("after a failed walk: %q", b)
|
||||
}
|
||||
keyless(t, dt.sends...)
|
||||
// Not on a clearing, a silencing or a plain notice.
|
||||
for _, what := range []string{"cleared", "silenced", "tell"} {
|
||||
if carriesDeliveryLine(&Record{Origin: "condition"}, what) {
|
||||
t.Errorf("a %s message carries the delivery line", what)
|
||||
}
|
||||
}
|
||||
if carriesDeliveryLine(&Record{Origin: "ask"}, "raised") {
|
||||
t.Errorf("an ask's own message carries the delivery line")
|
||||
}
|
||||
}
|
||||
|
||||
// ADR 0505 §1: the line names at most six modules and counts the rest, and says no name the content rule refuses.
|
||||
func TestTheDeliveryLineNamesAtMostSixModulesAndNothingTheRuleRefuses(t *testing.T) {
|
||||
many := novox("a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8")
|
||||
if got := deliveryWords([]Touch{many}); got != "a1, a2, a3, a4, a5, a6 and 2 more to novox" {
|
||||
t.Fatalf("eight modules: %q", got)
|
||||
}
|
||||
h, _, _, c, _ := newHolder(t)
|
||||
walk(t, h, "plan-p", WalkStarted, "", c.now(), 600, novox("/var/lib/mesh-tools"))
|
||||
k := cond("machine.novox.tools-silent", Urgent, "the tools on novox are silent")
|
||||
k.Raised = c.now()
|
||||
h.Condition(EventRaised, k)
|
||||
if b := bodyOf(t, h, k.Key, "raised"); !strings.HasSuffix(b, "\nDuring a delivery.") {
|
||||
t.Fatalf("a refused name in the line: %q", b)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
package main
|
||||
|
||||
// The delivery line (novox/hq ADR 0505 §1): every message about a problem — a condition or a module's notice,
|
||||
// raised, asked, reopened, escalated or reminded — ends, after its since-when line, with whether a delivery was
|
||||
// running on what it is about:
|
||||
//
|
||||
// - "During the delivery of <modules> to <machines>." when an open walk touches its subject;
|
||||
// - "<n> after the delivery of <modules> to <machines>, which <outcome>." when a walk that touched it ended within
|
||||
// its settle (a delivered walk settling, a failed or stopped one in the two minutes after it, §6);
|
||||
// - "No delivery running." otherwise.
|
||||
//
|
||||
// The line is this router's, from the walks it keeps (ADR 0308 §1); no producer adds it. It names modules and
|
||||
// machines as the operator says them, never an id, and is judged by the content rule with the rest of the body.
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NoDeliveryLine is the delivery line of a message no walk touches.
|
||||
const NoDeliveryLine = "No delivery running."
|
||||
|
||||
// outcomeWords are a walk's outcome as the delivery line says it.
|
||||
var outcomeWords = map[string]string{WalkDelivered: "succeeded", WalkFailed: "failed", WalkStopped: "was stopped"}
|
||||
|
||||
// carriesDeliveryLine says whether a message of a record says the delivery line: a problem's message, and not its
|
||||
// clearing, its silencing, a plain notice or an ask's own message.
|
||||
func carriesDeliveryLine(r *Record, what string) bool {
|
||||
switch what {
|
||||
case "cleared", "silenced", "tell":
|
||||
return false
|
||||
}
|
||||
return r.Origin != "ask"
|
||||
}
|
||||
|
||||
// deliveryLine is the line for a record's message at a moment.
|
||||
func (h *Holder) deliveryLine(r *Record, now time.Time) string {
|
||||
s := subjectOf(r)
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
var during, after []*Walk
|
||||
for _, w := range h.walks {
|
||||
if !w.touches(s) {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case w.settling() && now.Before(w.SettleUntil) && !now.Before(w.Ended):
|
||||
after = append(after, w)
|
||||
case !w.settling():
|
||||
during = append(during, w) // open, within its bound or past it (ADR 0505 §7)
|
||||
}
|
||||
}
|
||||
if len(during) > 0 {
|
||||
sort.Slice(during, func(i, j int) bool { return during[i].Started.Before(during[j].Started) })
|
||||
return plainly("During the delivery of "+deliveryWords(during[0].Touches)+".", "During a delivery.")
|
||||
}
|
||||
if f := h.failedJustNow(s, now); f != nil {
|
||||
after = append(after, f)
|
||||
}
|
||||
if len(after) == 0 {
|
||||
return NoDeliveryLine
|
||||
}
|
||||
sort.Slice(after, func(i, j int) bool { return after[i].Ended.After(after[j].Ended) })
|
||||
w := after[0]
|
||||
ago := roughly(now.Sub(w.Ended))
|
||||
return plainly(ago+" after the delivery of "+deliveryWords(w.Touches)+", which "+outcomeWords[w.Outcome]+".",
|
||||
ago+" after a delivery, which "+outcomeWords[w.Outcome]+".")
|
||||
}
|
||||
|
||||
// plainly is a line when the content rule lets it leave the mesh, else the same said without the names that it
|
||||
// refused, so the line never withholds the message it ends.
|
||||
func plainly(line, without string) string {
|
||||
if _, ok := Check(line); ok {
|
||||
return line
|
||||
}
|
||||
return without
|
||||
}
|
||||
@@ -1053,11 +1053,21 @@ func (h *Holder) words(r *Record, what string) Message {
|
||||
// rule judges the message as it is sent.
|
||||
func (h *Holder) composeTracked(r *Record, what string, now time.Time, withhold int) (Message, string) {
|
||||
m := compose(r, what, now, h.zone(), withhold)
|
||||
if !h.tracksAsked() || withhold >= 2 {
|
||||
return m, ""
|
||||
line := ""
|
||||
if h.tracksAsked() && withhold < 2 {
|
||||
issues, known := h.trackedBy(r.Key)
|
||||
m, line = withTracking(m, r, what, issues, known, now, h.zone())
|
||||
}
|
||||
issues, known := h.trackedBy(r.Key)
|
||||
return withTracking(m, r, what, issues, known, now, h.zone())
|
||||
if withhold < 2 && carriesDeliveryLine(r, what) {
|
||||
// Last, after the since-when line, the delivery line (ADR 0505 §1): judged with the body by the content rule.
|
||||
dl := h.deliveryLine(r, now)
|
||||
if m.Body == "" {
|
||||
m.Body = dl
|
||||
} else {
|
||||
m.Body += "\n" + dl
|
||||
}
|
||||
}
|
||||
return m, line
|
||||
}
|
||||
|
||||
// allowedWords is the message the content rule leaves of a record: its own words when they may leave the
|
||||
|
||||
@@ -58,7 +58,7 @@ var examples = []example{{
|
||||
resolved: "openrazer delivery no longer waiting", popup: true,
|
||||
title: "Urgent: openrazer delivery waiting to start",
|
||||
body: "Needs you: start it, or stop it. The change to openrazer is merged and built, and mesh-delivery (the module " +
|
||||
"that decides when a delivery goes out) has not let it start for 5 hours, so mesh-delivery may be stuck.\nNo issue tracks this yet.\nSince 12:40.",
|
||||
"that decides when a delivery goes out) has not let it start for 5 hours, so mesh-delivery may be stuck.\nNo issue tracks this yet.\nSince 12:40.\nNo delivery running.",
|
||||
offers: "Details", choose: "Details",
|
||||
|
||||
cleared: "Resolved: openrazer delivery no longer waiting, after 5.0 h", openFor: 5 * time.Hour,
|
||||
@@ -80,7 +80,7 @@ var examples = []example{{
|
||||
title: "openrazer not working on g14",
|
||||
body: "Needs you: restart its service openrazer-daemon on g14; if it fails again, the details say why. openrazer " +
|
||||
"on g14 is not healthy: its service openrazer-daemon stopped with an error. It clears as soon as it runs " +
|
||||
"again.\nNo issue tracks this yet.\nSince 12:40.",
|
||||
"again.\nNo issue tracks this yet.\nSince 12:40.\nNo delivery running.",
|
||||
offers: "Details", choose: "Details",
|
||||
cleared: "Resolved: openrazer works again on g14, after 2.0 h", openFor: 2 * time.Hour,
|
||||
}, {
|
||||
@@ -104,7 +104,7 @@ var examples = []example{{
|
||||
title: "3 failed services on shanks",
|
||||
body: "Needs you: mend or remove them on shanks, or silence this if they do not matter. On shanks, mnt-recalbox " +
|
||||
"(a mount), storage-media (a mount) and greenclip failed. No module manages them, so the mesh does not repair " +
|
||||
"them.\nNo issue tracks this yet.\nSince 12:40.",
|
||||
"them.\nNo issue tracks this yet.\nSince 12:40.\nNo delivery running.",
|
||||
offers: "Details", choose: "Details",
|
||||
cleared: "Resolved: no failed services on shanks any more, after 45 min", openFor: 45 * time.Minute,
|
||||
}, {
|
||||
@@ -135,7 +135,7 @@ var examples = []example{{
|
||||
explanation: "Needs you: release it, or stop it. A delivery of hq has been held for 36 hours, past its limit.",
|
||||
resolved: "Delivery of hq is no longer held", popup: true,
|
||||
title: "Delivery of hq held for 36 hours",
|
||||
body: "Needs you: release it, or stop it. A delivery of hq has been held for 36 hours, past its limit.\nNo issue tracks this yet.\nSince 12:40.",
|
||||
body: "Needs you: release it, or stop it. A delivery of hq has been held for 36 hours, past its limit.\nNo issue tracks this yet.\nSince 12:40.\nNo delivery running.",
|
||||
offers: "Details", choose: "Details",
|
||||
|
||||
cleared: "Resolved: delivery of hq is no longer held, after 3 min", openFor: 3 * time.Minute,
|
||||
@@ -296,7 +296,7 @@ func TestAConditionWithoutPlainWordsIsSaidFromItsSummary(t *testing.T) {
|
||||
Summary: "ace has not been heard from: `push ace` sends it again", Raised: c.now()}
|
||||
h.Condition(EventRaised, k)
|
||||
m := tg.sends[0]
|
||||
if m.Title != "Urgent: Silent on ace" || m.Body != "Ace has not been heard from: a command sends it again\nSince 12:00." {
|
||||
if m.Title != "Urgent: Silent on ace" || m.Body != "Ace has not been heard from: a command sends it again\nSince 12:00.\nNo delivery running." {
|
||||
t.Errorf("%q / %q", m.Title, m.Body)
|
||||
}
|
||||
keyless(t, m)
|
||||
|
||||
Reference in New Issue
Block a user