diff --git a/.gitignore b/.gitignore index 0c8e31a7..18c2b03c 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,5 @@ modules/dnsmasq/dnsmasq modules/mesh-delivery/mesh-delivery modules/mesh-issues/cmd/mesh-issues/mesh-issues modules/mounts/cmd/mounts/mounts +modules/mesh-knowledge/cmd/mesh-knowledge/mesh-knowledge +modules/mesh-knowledge/mesh-knowledge diff --git a/modules/mesh-knowledge/README.md b/modules/mesh-knowledge/README.md new file mode 100644 index 00000000..ea173edf --- /dev/null +++ b/modules/mesh-knowledge/README.md @@ -0,0 +1,160 @@ +# mesh-knowledge — the knowledge base + +The mesh's decision records and research efforts, held whole in one module (novox/hq ADR 0290, to-be 55; +the first slice, hq issue 425). One instance on the control-node holds the seat `knowledge-base`, once for the +mesh. Each record keeps its number, kind, title, topic, status, date and deciders. It also keeps its typed links, +read from both ends, and its reasoning in sections. A decision also keeps its **clauses**: each numbered statement +it makes, with its own status and source. Every version of every record is kept, with who made it, when and why. + +An agent asks what holds and gets the clauses in force, the records they come from, what changed them since, and +how to read the reasoning, all in one call under 4 KiB. A record is proposed and checked when it is written, then +reviewed by someone who did not write it, then accepted. When its author says it needs the operator, it is +accepted only on the operator's Approve, given on a channel that proves who answered. + +## The verbs (seat `knowledge-base`) + +| Verb | Takes | Answers | +|---|---|---| +| `ask` | `question` (at most 500 characters), `bound` | the records and clauses bearing on the question, ranked; each clause says whether it holds, what changed it, or that it is only drafted; each record says what changed it since and how to read its reasoning | +| `get` | `number`, `kind`, `reasoning`, `section`, `version`, `bound` | one record: clauses, links both ways, what changed it since, versions, reviews, the operator's ask; its reasoning only when asked | +| `holds` | `topic` or `number`, `bound` | the clauses in force for a topic of the reading order or for one record; never a drafted clause | +| `links` | `number`, `kind` | the record's typed links, out and in | +| `history` | `number`, `kind`, `from`, `to`, `bound` | every version; with `from` and `to`, the difference between the two, line by line | +| `reserve` | `kind`, `for` | the next decision or research number, asked of the issue tracker (`issue-tracker.reserve`); spent whether or not its record is accepted | +| `propose` | the record whole | proposed, or refused naming every rule it breaks; an accepted record changes only by a progressive insight (`insight`) | +| `review` | `number`, `verdict` (approve or changes), `findings` | accepted, asked of the operator, or kept proposed with the findings | +| `import` | `dry-run`, `commit` | hq's decision records and research efforts at the commit the records module stands at, each whole at its own number | +| `return` | `number`, `reason` | drafted clauses sent back to be drafted again | + +`ask`, `get`, `holds` and `history` stay within `bound` bytes (4096 unless asked for, at most 65536) and say what they cut +and how to read the rest. The module's own tool `knowledge_health` answers its health (below). + +**Who acted** is the caller as the runtime proves it, never an argument: no verb takes one. Today the runtime +does not hand a module its caller. The SDK's tool call carries only a name and arguments, so proving the caller +needs an SDK change first (hq issue 365). Until then every change is recorded as by `unproven`. In this module +the caller is read in one place, `callerOf` in `main.go`, so the SDK change touches that one function. + +The rules about callers: + +- A proposal is revised only by its author, as the runtime proves them. +- A number reserved by a proven caller is proposed only by that caller. +- A review is refused from anyone who wrote any version of the record. +- A review is refused for a record that has a version whose writer was unproven: who wrote it cannot be told + from who reviews it, so its author proposes it again under a new number. + +Until the runtime proves callers, **every review is refused**. + +## What a record holds + +A clause is `proposed`, `in force`, `changed` (a later clause replaces part of it, named in `changed-by`) or +`superseded`. Its source is one of three: + +- `written`: written as a clause by its author; +- `drafted`: drafted from a prose Decision section by the agent's judgement at the import, and never served as in + force; +- `confirmed`: drafted, then confirmed by a second, separate judgement that saw only the record and the drafts. + +The links are `supersedes`, `changes` (to a clause: `0276 §4`), `extends`, `became` (from a research effort to a +decision record or design) and `cites` (to any record, an issue, or a design). Each link is stored once, on the +record it goes out from. + +## Writing a record + +1. `reserve` a number for its kind. +2. `propose` the record whole. It is refused, naming every reason at once, when it breaks one of the **29 rules + that judge one record or its prose**. They are hq's checks moved to the write; `PerRecordRules` in `rules.go` + lists each one with the check it comes from: + - 25 on the structure: kind, reserved number, title, topic, date, deciders, research status, at least one + clause, clause numbering, bold lead, clause size, the Context, Consequences and "How it is checked" sections, + a research effort's overview, link kind, link target, link fit, no link to itself, what it extends, what it + supersedes or changes is accepted, a changed clause is in force, a superseded record cited beside its + superseder, a graduated effort says what it became, and the links in its text resolve; + - 4 on the prose: no word the glossary retired, no daemon's line, a deferral names its tracking issue, and a + progressive insight in the dated, marked form. + + A rule holds a record from the day hq's checks began to hold it (`RuleFrom` in `rules.go`): + - retired words from 2026-10-07 and daemons' lines from 2026-10-09 for a research effort; for a decision record + both from 2026-10-11, since hq's words check never read a decision record; + - deferrals from 2026-10-09; + - the clauses from 2026-10-11; + - the decision record's sections and the live-citation rule, which no hq check held on a decision record, + from 2026-10-11, the day the knowledge base takes the write. + + A progressive insight keeps its record's date, so a correction to an older record is not refused by a rule + that never held it, and it keeps the links the accepted record holds. A new record is dated no earlier than the day its number was reserved. +3. Revise it by proposing again; every version is kept. +4. A `review` by someone else either approves it or asks for changes. On approve the record and its clauses are + accepted and in force. Every clause it changes moves to `changed`, and every record it supersedes moves to + `superseded`, each as a new version of that record. A record marked `needs-operator` is first asked of the + operator: an ask with Approve and Decline, both at the level approve, each bound to this version's digest. It is + accepted only on the router's warrant for Approve, read back from the router's own record and held to the ask + composed again. A record changed after it was asked finds the digests differ, and nothing is accepted. The + warrant is kept with the record. +5. After every acceptance, the **4 rules that judge the whole record set** run: + - `cited`: a decision accepted here is cited by another record or names a home in hq; + - `supersession`: both ends of a supersession agree; + - `partial`: both ends of a change agree; + - `rests-on-live`: no accepted record extends a superseded one. + + A failure is kept as a finding and makes the module unhealthy, so the mesh raises a condition naming the rule + and the records. The acceptance stands. + +## The import + +`import` reads hq through the records module at the commit its checkout stands at. With `commit`, it refuses a +checkout standing elsewhere. It writes every decision record and research effort whole, as version 1, naming the +commit and path it came from. It creates only what is absent and never overwrites. It never asks for a number, so +no counter moves. A second run changes nothing and says so. `dry-run` writes nothing. From a terminal, +`mesh-knowledge dry-run ` gives the same dry run over a checkout on disk, against an empty knowledge +base. + +A decision whose Decision section writes numbered statements with a bold lead gets them as `written` clauses. One +written as prose (226 of main's 278 on 2026-10-11) gets none at the import. A background worker then takes one +judgement at a time (`claude-code.claude_code_judge` on this node): it drafts the clauses, then confirms them in a +separate call. A clause the second judgement rejects stays `drafted`, with the reason. hq's `supersedes-in-part` +named a record, not a clause: which clause it changes is judged and then confirmed the same way, and only then is +the older clause marked `changed`. When every prose record of a topic has been through both judgements, the +operator is told once (`operator-channel.notify`, a warning) with the counts. `return` sends a record back to be +drafted again. + +## Where it keeps its records + +In a database in the store: the module requires a `postgres-database` provision named `knowledge`, as +`claude-licence-manager` does, and reaches it through the address in `${dir:state}/database.url`. Ranked search is +the database's full-text search (`ask`). The schema is numbered migrations (`cmd/mesh-knowledge/migrations/`), and +the consolidated schema for a fresh database is `schema.sql`. A test holds the two equal on a real postgres. + +## Its processes + +- **the bundle**: the verbs, `knowledge_health`, the operator's answers heard, and once a minute the router's + record read, reservations never proposed withdrawn, and the drafting worker. It never changes the schema, and + refuses to serve while the schema is behind. +- **`prepare`** (run once): brings the schema to this version. +- **`keep`** (judged by its unit): judges the whole record set every five minutes and keeps the findings. +- **`watch`** (judged by `knowledge_health`): every minute, reads whether the database serves and what the + findings say, and leaves its verdict in `${dir:state}/health.json`. That is the file the health tool answers + from. A verdict older than three minutes is unhealthy. + +## Settings + +| Setting | Default | What it does | +|---|---|---| +| `judge-model` | `sonnet` | the model the drafting and confirming judgements ask | +| `drafting` | `on` | `off` stops drafting and confirming clauses | +| `reserved-for` | `168h` | how long a reserved number waits for its proposal before it is withdrawn | + +## What it needs at release + +- assignment to the control-node, and a postgres provider for its `postgres-database` provision; +- the issue tracker serving `issue-tracker.reserve` (hq issue 424, merged as mesh-catalog#204) on the mesh; +- the SDK and the runtime proving the caller (hq issue 365) before any review is taken; +- the import run once, then checked against hq record by record (hq issue 426). + +## Tests + +`go test ./...` in this directory. The store tests run over memory and over a real postgres. That postgres is +either `MESH_TEST_POSTGRES` or a private server the tests start from the machine's `initdb` on a socket of its own, +listening on no port. A machine with neither skips the postgres half and says so. The fixtures are real hq files: +`testdata/hq` holds eleven records at the commit in its `COMMIT` file, and `testdata/hq-main.tar.gz` holds the +whole of main at `testdata/hq-main.commit`. The bus test starts the bundle as the runtime does and plays the +runtime over a real NATS server. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/answers.go b/modules/mesh-knowledge/cmd/mesh-knowledge/answers.go new file mode 100644 index 00000000..9a21a34e --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/answers.go @@ -0,0 +1,747 @@ +package main + +// The answers (novox/hq ADR 0290 rule 6, to-be 55 §2 and §3): ask, get, holds, links and history, each the shape +// an agent needs. ask, get and holds stay under a bound they state, 4 KiB unless the caller asks for more, and say +// when they were cut and how to read the rest. A drafted clause is marked as such and never served as in force. + +import ( + "context" + "encoding/json" + "fmt" + "sort" + "strconv" + "strings" +) + +// MostQuestion is the most characters a question to ask may hold. +const MostQuestion = 500 + +// The bound of an answer. +const ( + DefaultBound = 4 << 10 + MostBound = 64 << 10 + LeastBound = 1 << 10 +) + +func boundOf(b int) int { + switch { + case b <= 0: + return DefaultBound + case b < LeastBound: + return LeastBound + case b > MostBound: + return MostBound + } + return b +} + +func size(v any) int { + raw, _ := json.Marshal(v) + return len(raw) +} + +// fit is the largest n (from 0 to total) whose answer stays within the bound. +func fit(total, bound int, build func(n int) any) (any, int) { + lo, hi := 0, total + for lo < hi { + mid := (lo + hi + 1) / 2 + if size(build(mid)) <= bound { + lo = mid + } else { + hi = mid - 1 + } + } + return build(lo), lo +} + +func clip(s string, n int) string { + s = strings.Join(strings.Fields(s), " ") + if len([]rune(s)) <= n { + return s + } + return string([]rune(s)[:n-1]) + "…" +} + +// held says, for a clause, whether it holds and what changed it, in the words an agent reads. +func held(r *Record, c Clause) string { + switch { + case c.Source == SourceDrafted: + s := "drafted, not confirmed: not the record's word, never in force" + if c.Unconfirmed != "" { + s += " (the second judgement did not confirm it: " + c.Unconfirmed + ")" + } + return s + case r.Status == StatusProposed || c.Status == ClauseProposed: + return "proposed: not in force" + case c.Status == ClauseChanged: + return "changed by " + strings.Join(c.ChangedBy, ", ") + ": read both" + case c.Status == ClauseSuperseded: + if len(c.ChangedBy) > 0 { + return "superseded by " + strings.Join(c.ChangedBy, ", ") + } + return "superseded" + case c.Source == SourceConfirmed: + return "in force (drafted from prose, confirmed)" + } + return "in force" +} + +// Served says whether a clause may be served as holding: in force or changed, and not drafted. +func Served(r *Record, c Clause) bool { + return r.Accepted() && c.Source != SourceDrafted && (c.Status == ClauseInForce || c.Status == ClauseChanged) +} + +// AnswerClause is one clause in an answer. +type AnswerClause struct { + N int `json:"n"` + Text string `json:"text"` + Holds string `json:"holds"` + Source string `json:"source,omitempty"` +} + +// AnswerRecord is one record in an answer to ask. +type AnswerRecord struct { + Record string `json:"record"` + Title string `json:"title"` + Status string `json:"status"` + Clauses []AnswerClause `json:"clauses,omitempty"` + Note string `json:"note,omitempty"` + Since []string `json:"changed-since,omitempty"` + Read string `json:"read"` +} + +// Asked is the answer to ask. +type Asked struct { + Question string `json:"question"` + Records []AnswerRecord `json:"records"` + Cut string `json:"cut,omitempty"` + Bound int `json:"bound"` + Says string `json:"says,omitempty"` +} + +func readHow(k Key) string { + return fmt.Sprintf("get {\"kind\": %q, \"number\": %d, \"reasoning\": true}", k.Kind, k.Number) +} + +// since is what changed a record since it was accepted: the accepted records that supersede or change it. +func (k *KB) since(ctx context.Context, key Key) []string { + in, err := k.Store.Inbound(ctx, key) + if err != nil { + return nil + } + var out []string + for _, x := range in { + if x.Link.Kind != LinkSupersedes && x.Link.Kind != LinkChanges { + continue + } + from, err := k.Store.Get(ctx, x.From) + if err != nil || from == nil || !from.Current.Accepted() { + continue + } + s := x.From.String() + if x.Link.From > 0 { + s += fmt.Sprintf(" §%d", x.Link.From) + } + switch { + case x.Link.Kind == LinkSupersedes: + s += " supersedes it" + case x.Link.To.Clause > 0: + s += fmt.Sprintf(" changes §%d", x.Link.To.Clause) + default: + s += " changes it in part; which clause is not judged yet" + } + out = append(out, s) + } + return out +} + +// Ask is the records and clauses bearing on a question, ranked, each saying whether it holds or what changed it. +func (k *KB) Ask(question string, bound int) (*Asked, error) { + ctx := context.Background() + question = strings.TrimSpace(question) + if question == "" { + return nil, fmt.Errorf("ask needs a question, in plain words") + } + if len([]rune(question)) > MostQuestion { + return nil, fmt.Errorf("a question is at most %d characters; this one is %d: ask it in fewer words", MostQuestion, len([]rune(question))) + } + bound = boundOf(bound) + hits, err := k.Store.Search(ctx, question, 80) + if err != nil { + return nil, fmt.Errorf("the knowledge base cannot be searched (%v)", err) + } + type group struct { + key Key + clauses []int + record bool + // score is the record's own rank, its best clause's, and a little of each other clause's. + score, best, rest float64 + } + var order []*group + byKey := map[Key]*group{} + for _, h := range hits { + g := byKey[h.Key] + if g == nil { + g = &group{key: h.Key} + byKey[h.Key] = g + order = append(order, g) + } + if h.Clause > 0 { + g.clauses = append(g.clauses, h.Clause) + if h.Rank > g.best { + g.rest += g.best + g.best = h.Rank + } else { + g.rest += h.Rank + } + } else { + g.record = true + g.score += h.Rank + } + } + for _, g := range order { + g.score += g.best + g.rest/4 + } + sort.SliceStable(order, func(i, j int) bool { return order[i].score > order[j].score }) + var records []AnswerRecord + for _, g := range order { + e, err := k.Store.Get(ctx, g.key) + if err != nil || e == nil { + continue + } + r := e.Current + ar := AnswerRecord{Record: g.key.String(), Title: r.Title, Status: r.Status, Read: readHow(g.key), Since: k.since(ctx, g.key)} + sort.Ints(g.clauses) + for _, n := range g.clauses { + if c := r.Clause(n); c != nil { + ar.Clauses = append(ar.Clauses, AnswerClause{N: n, Text: clip(c.Text, 220), Holds: held(r, *c), Source: c.Source}) + } + } + if len(ar.Clauses) == 0 { + drafted := 0 + for _, c := range r.Clauses { + if c.Source == SourceDrafted { + drafted++ + } + } + switch { + case len(r.Clauses) == 0 && r.Kind == KindDecision: + ar.Note = "its decision is written as prose and its clauses are not drafted yet: read its Decision section" + case drafted > 0: + ar.Note = fmt.Sprintf("%d of its clauses are drafted and not confirmed: none of them is served as in force", drafted) + } + } + records = append(records, ar) + } + out := &Asked{Question: question, Bound: bound} + if len(records) == 0 { + out.Records = []AnswerRecord{} + out.Says = "nothing the knowledge base holds bears on that; ask in other words" + return out, nil + } + // Within the bound: the most records that fit; when not even one fits whole, its clauses shortened. + build := func(n int) any { + a := *out + a.Records = records[:n] + if n < len(records) { + names := []string{} + for _, r := range records[n:min(len(records), n+6)] { + names = append(names, r.Record) + } + a.Cut = fmt.Sprintf("%d more record(s) bear on it and are not shown within %d bytes (%s…): ask with a larger bound, or narrower words", + len(records)-n, bound, strings.Join(names, ", ")) + } + return &a + } + v, n := fit(len(records), bound, build) + if n == 0 { + first := records[0] + for i := range first.Clauses { + first.Clauses[i].Text = clip(first.Clauses[i].Text, 80) + } + records[0] = first + v, _ = fit(len(records), bound, build) + } + return v.(*Asked), nil +} + +// ---- get ----------------------------------------------------------------------------------------------------- + +// GetAsk is what get takes. +type GetAsk struct { + Kind string + Number int + Reasoning bool + Section string + Version int + Bound int +} + +// Got is the answer to get. +type Got struct { + Record string `json:"record"` + Title string `json:"title"` + Status string `json:"status"` + Topic string `json:"topic,omitempty"` + Date string `json:"date,omitempty"` + Deciders []string `json:"deciders,omitempty"` + Version int `json:"version"` + Versions int `json:"versions"` + Clauses []AnswerClause `json:"clauses,omitempty"` + Links []string `json:"links,omitempty"` + LinkedFrom []string `json:"linked-from,omitempty"` + Since []string `json:"changed-since,omitempty"` + Sections []string `json:"sections,omitempty"` + Reasoning []Section `json:"reasoning,omitempty"` + Insights []Insight `json:"insights,omitempty"` + Pending string `json:"pending,omitempty"` + Waiting string `json:"waiting,omitempty"` + Reviews []Review `json:"reviews,omitempty"` + Drafting *Drafting `json:"drafting,omitempty"` + Source string `json:"source,omitempty"` + Reserved string `json:"reserved,omitempty"` + NeedsOperat bool `json:"needs-operator,omitempty"` + Cut string `json:"cut,omitempty"` + Bound int `json:"bound"` + Read string `json:"read,omitempty"` +} + +// Get is one record: its clauses, its links both ways, what changed it since; its reasoning only when asked. +func (k *KB) Get(a GetAsk) (*Got, error) { + ctx := context.Background() + if a.Kind == "" { + a.Kind = KindDecision + } + key := Key{a.Kind, a.Number} + e, err := k.Store.Get(ctx, key) + if err != nil { + return nil, fmt.Errorf("the knowledge base cannot be read (%v)", err) + } + if e == nil { + return nil, fmt.Errorf("the knowledge base holds no %s", key) + } + r, version := e.Current, e.CurrentVersion + if a.Version > 0 { + vs, err := k.Store.Versions(ctx, key) + if err != nil { + return nil, err + } + if a.Version > len(vs) { + return nil, fmt.Errorf("%s has %d version(s); there is no version %d", key, len(vs), a.Version) + } + r, version = vs[a.Version-1].Record, a.Version + } + bound := boundOf(a.Bound) + g := &Got{Record: key.String(), Title: r.Title, Status: r.Status, Topic: r.Topic, Date: r.Date, Deciders: r.Deciders, + Version: version, Versions: e.Versions, Insights: r.Insights, Reviews: r.Reviews, Drafting: r.Drafting, Source: r.Source, + NeedsOperat: r.NeedsOperator, Bound: bound, Since: k.since(ctx, key)} + if r.Status == StatusReserved || r.Status == StatusWithdrawn { + g.Reserved = fmt.Sprintf("reserved by %s at %s for %q", r.ReservedBy, r.ReservedAt.Format("2006-01-02 15:04 UTC"), r.ReservedFor) + } + if e.Pending != nil && a.Version == 0 { + g.Pending = fmt.Sprintf("version %d, a progressive insight, waits for its review: history shows what it changes", e.PendingVersion) + } + if r.Ask != nil { + g.Waiting = "the operator's ask: " + r.Ask.Said + } + for _, c := range r.Clauses { + g.Clauses = append(g.Clauses, AnswerClause{N: c.N, Text: c.Text, Holds: held(r, c), Source: c.Source}) + } + for _, l := range r.Links { + g.Links = append(g.Links, l.String()) + } + if in, err := k.Store.Inbound(ctx, key); err == nil { + for _, x := range in { + s := x.From.String() + " " + if x.Link.From > 0 { + s += fmt.Sprintf("§%d ", x.Link.From) + } + s += x.Link.Kind + if x.Link.To.Clause > 0 { + s += fmt.Sprintf(" §%d", x.Link.To.Clause) + } + g.LinkedFrom = append(g.LinkedFrom, s) + } + } + for _, s := range r.Sections { + g.Sections = append(g.Sections, s.Name) + if a.Reasoning || (a.Section != "" && strings.EqualFold(a.Section, s.Name)) { + g.Reasoning = append(g.Reasoning, s) + } + } + if a.Section != "" && len(g.Reasoning) == 0 { + return nil, fmt.Errorf("%s has no section %q; it has %s", key, a.Section, strings.Join(g.Sections, ", ")) + } + if !a.Reasoning && a.Section == "" { + g.Read = "its reasoning: " + readHow(key) + ", or one section with section" + } + if size(g) <= bound { + return g, nil + } + // Cut: the reasoning first, section by section from the end, then the reviews but the last, then the links + // from the end, then the clauses' text, then clauses; each cut is said. Room is kept for saying it. + room := bound - 600 + var cut []string + if len(g.Reasoning) > 0 { + reasoning := g.Reasoning + v, n := fit(len(reasoning), room, func(n int) any { c := *g; c.Reasoning = reasoning[:n]; return &c }) + *g = *(v.(*Got)) + if n < len(reasoning) { + left := []string{} + for _, s := range reasoning[n:] { + left = append(left, s.Name) + } + cut = append(cut, fmt.Sprintf("the sections %s are not shown within %d bytes: ask for one with section, or raise bound", strings.Join(left, ", "), bound)) + if spare := room - size(g); n == 0 && spare > 400 { + s := reasoning[0] + s.Text = clip(s.Text, spare/2) + g.Reasoning = []Section{s} + cut[len(cut)-1] = fmt.Sprintf("the section %s is shortened, and %s not shown: ask for one with section, or raise bound", + s.Name, strings.Join(append([]string{}, left[1:]...), ", ")) + } + } + } + if size(g) > room && len(g.Reviews) > 1 { + cut = append(cut, fmt.Sprintf("%d earlier review(s) are not shown: history lists every version", len(g.Reviews)-1)) + g.Reviews = g.Reviews[len(g.Reviews)-1:] + } + for _, which := range []string{"linked-from", "links"} { + if size(g) <= room { + break + } + list := func(c *Got) *[]string { + if which == "links" { + return &c.Links + } + return &c.LinkedFrom + } + full := *list(g) + v, n := fit(len(full), room, func(n int) any { c := *g; *list(&c) = full[:n]; return &c }) + if n < len(full) { + *g = *(v.(*Got)) + cut = append(cut, fmt.Sprintf("%d of its %s are not shown: links lists them all", len(full)-n, which)) + } + } + if size(g) > room { + for i := range g.Clauses { + g.Clauses[i].Text = clip(g.Clauses[i].Text, 160) + } + cut = append(cut, "the clauses are shortened: raise bound to read them whole") + } + if size(g) > room { + clauses := g.Clauses + v, n := fit(len(clauses), room, func(n int) any { c := *g; c.Clauses = clauses[:n]; return &c }) + *g = *(v.(*Got)) + cut = append(cut, fmt.Sprintf("%d clause(s) are not shown: raise bound", len(clauses)-n)) + } + g.Cut = strings.Join(cut, "; ") + return g, nil +} + +// ---- holds --------------------------------------------------------------------------------------------------- + +// HeldClause is one clause in force, with the record it comes from. +type HeldClause struct { + Record string `json:"record"` + N int `json:"n"` + Text string `json:"text"` + Holds string `json:"holds"` +} + +// Holding is the answer to holds. +type Holding struct { + Of string `json:"of"` + Clauses []HeldClause `json:"clauses"` + NotYet string `json:"not-served,omitempty"` + Cut string `json:"cut,omitempty"` + Bound int `json:"bound"` +} + +// Holds is the clauses in force for a topic or a record: never a drafted one, and a confirmed one marked. +func (k *KB) Holds(topic string, key *Key, bound int) (*Holding, error) { + ctx := context.Background() + bound = boundOf(bound) + var records []*Record + of := "" + switch { + case key != nil: + e, err := k.Store.Get(ctx, *key) + if err != nil { + return nil, err + } + if e == nil { + return nil, fmt.Errorf("the knowledge base holds no %s", key) + } + records, of = []*Record{e.Current}, key.String() + case strings.TrimSpace(topic) != "": + all, err := k.Store.All(ctx) + if err != nil { + return nil, err + } + for _, e := range all { + if e.Current.Topic == topic { + records = append(records, e.Current) + } + } + of = "the topic " + topic + if len(records) == 0 { + return nil, fmt.Errorf("no record has the topic %q", topic) + } + default: + return nil, fmt.Errorf("holds takes a topic or a record's number") + } + out := &Holding{Of: of, Bound: bound, Clauses: []HeldClause{}} + drafted, prose := 0, 0 + for _, r := range records { + if r.Kind == KindDecision && r.Accepted() && len(r.Clauses) == 0 { + prose++ + } + for _, c := range r.Clauses { + if c.Source == SourceDrafted && r.Accepted() && c.Status != ClauseSuperseded { + drafted++ + } + if Served(r, c) { + out.Clauses = append(out.Clauses, HeldClause{Record: r.Key().String(), N: c.N, Text: c.Text, Holds: held(r, c)}) + } + } + } + var notes []string + if drafted > 0 { + notes = append(notes, fmt.Sprintf("%d drafted clause(s) are not served: they are not confirmed as the record's word", drafted)) + } + if prose > 0 { + notes = append(notes, fmt.Sprintf("%d accepted decision(s) write their decision as prose and have no clauses yet", prose)) + } + out.NotYet = strings.Join(notes, "; ") + clauses := out.Clauses + v, n := fit(len(clauses), bound, func(n int) any { + c := *out + c.Clauses = clauses[:n] + if n < len(clauses) { + c.Cut = fmt.Sprintf("%d more clause(s) are in force and not shown within %d bytes: raise bound, or ask holds of one record", len(clauses)-n, bound) + } + return &c + }) + if n == 0 && len(clauses) > 0 { + for i := range clauses { + clauses[i].Text = clip(clauses[i].Text, 160) + } + v, _ = fit(len(clauses), bound, func(n int) any { + c := *out + c.Clauses = clauses[:n] + c.Cut = fmt.Sprintf("the clauses are shortened, and %d not shown within %d bytes: raise bound", len(clauses)-n, bound) + return &c + }) + } + return v.(*Holding), nil +} + +// ---- links --------------------------------------------------------------------------------------------------- + +// Linked is the answer to links. +type Linked struct { + Record string `json:"record"` + Out []LinkRow `json:"out"` + In []LinkRow `json:"in"` +} + +// LinkRow is one link, read from either end. +type LinkRow struct { + Kind string `json:"kind"` + From string `json:"from"` + To string `json:"to"` + Source string `json:"source,omitempty"` + Unresolved bool `json:"unresolved,omitempty"` +} + +// Links is a record's links, both ways. +func (k *KB) Links(key Key) (*Linked, error) { + ctx := context.Background() + e, err := k.Store.Get(ctx, key) + if err != nil { + return nil, err + } + if e == nil { + return nil, fmt.Errorf("the knowledge base holds no %s", key) + } + out := &Linked{Record: key.String(), Out: []LinkRow{}, In: []LinkRow{}} + from := func(k Key, clause int) string { + if clause > 0 { + return k.String() + " §" + strconv.Itoa(clause) + } + return k.String() + } + for _, l := range e.Current.Links { + out.Out = append(out.Out, LinkRow{Kind: l.Kind, From: from(key, l.From), To: l.To.String(), Source: l.Source, Unresolved: l.Unresolved}) + } + in, err := k.Store.Inbound(ctx, key) + if err != nil { + return nil, err + } + for _, x := range in { + out.In = append(out.In, LinkRow{Kind: x.Link.Kind, From: from(x.From, x.Link.From), To: x.Link.To.String(), Source: x.Link.Source}) + } + return out, nil +} + +// ---- history ------------------------------------------------------------------------------------------------- + +// VersionRow is one version in a history. +type VersionRow struct { + N int `json:"n"` + By string `json:"by"` + At string `json:"at"` + Why string `json:"why"` + Status string `json:"status"` + Insight bool `json:"insight,omitempty"` + Served bool `json:"served,omitempty"` + Pending bool `json:"pending,omitempty"` +} + +// Historied is the answer to history. +type Historied struct { + Record string `json:"record"` + Versions []VersionRow `json:"versions"` + From int `json:"from,omitempty"` + To int `json:"to,omitempty"` + Diff []string `json:"diff,omitempty"` + Same bool `json:"same,omitempty"` + Cut string `json:"cut,omitempty"` + Bound int `json:"bound,omitempty"` +} + +// History is a record's versions, and the difference between two of them, line by line. +func (k *KB) History(key Key, from, to, bound int) (*Historied, error) { + ctx := context.Background() + e, err := k.Store.Get(ctx, key) + if err != nil { + return nil, err + } + if e == nil { + return nil, fmt.Errorf("the knowledge base holds no %s", key) + } + vs, err := k.Store.Versions(ctx, key) + if err != nil { + return nil, err + } + out := &Historied{Record: key.String()} + for _, v := range vs { + out.Versions = append(out.Versions, VersionRow{N: v.N, By: v.By, At: v.At.UTC().Format("2006-01-02 15:04:05 UTC"), Why: v.Why, + Status: v.Record.Status, Insight: v.Insight, Served: v.N == e.CurrentVersion, Pending: v.N == e.PendingVersion}) + } + if from == 0 && to == 0 { + out.Bound = boundOf(bound) + versions := out.Versions + v, m := fit(len(versions), out.Bound-300, func(m int) any { c := *out; c.Versions = versions[len(versions)-m:]; return &c }) + *out = *(v.(*Historied)) + if m < len(versions) { + out.Cut = fmt.Sprintf("the %d oldest versions are not listed within %d bytes: raise bound", len(versions)-m, out.Bound) + } + return out, nil + } + if from < 1 || to < 1 || from > len(vs) || to > len(vs) { + return nil, fmt.Errorf("%s has versions 1 to %d; from and to name two of them", key, len(vs)) + } + out.From, out.To, out.Bound = from, to, boundOf(bound) + diff := Diff(Render(vs[from-1].Record), Render(vs[to-1].Record)) + out.Same = len(diff) == 0 + // Within the bound: the most of the difference's lines that fit, cut at a hunk's line and said; and when the + // versions alone do not fit, the oldest of them left out first. + versions := out.Versions + v, n := fit(len(diff), out.Bound-300, func(n int) any { c := *out; c.Diff = diff[:n]; return &c }) + *out = *(v.(*Historied)) + var cut []string + if n < len(diff) { + cut = append(cut, fmt.Sprintf("%d of the difference's %d lines are not shown within %d bytes: raise bound, or compare nearer versions", + len(diff)-n, len(diff), out.Bound)) + } + if size(out) > out.Bound-300 { + v, m := fit(len(versions), out.Bound-300, func(m int) any { c := *out; c.Versions = versions[len(versions)-m:]; return &c }) + *out = *(v.(*Historied)) + cut = append(cut, fmt.Sprintf("the %d oldest versions are not listed", len(versions)-m)) + } + out.Cut = strings.Join(cut, "; ") + return out, nil +} + +// MostDiffWork bounds the work of a difference: the lines of the two texts' changed middles multiplied. Past it, +// the middle is given whole as removed and added — still exact, only less narrow. +const MostDiffWork = 4_000_000 + +// Diff is the difference of two texts, line by line: each hunk headed "@@ -a,n +b,m @@", its lines "-" removed +// and "+" added, by the longest common subsequence of their lines. +func Diff(a, b string) []string { + x, y := strings.Split(a, "\n"), strings.Split(b, "\n") + // The common head and tail cost nothing to find; only the middle between them is compared. + head := 0 + for head < len(x) && head < len(y) && x[head] == y[head] { + head++ + } + tail := 0 + for tail < len(x)-head && tail < len(y)-head && x[len(x)-1-tail] == y[len(y)-1-tail] { + tail++ + } + x, y = x[head:len(x)-tail], y[head:len(y)-tail] + n, m := len(x), len(y) + if n*m > MostDiffWork { + var out []string + if n+m > 0 { + out = append(out, fmt.Sprintf("@@ -%d,%d +%d,%d @@", head+1, n, head+1, m)) + } + for _, l := range x { + out = append(out, "-"+l) + } + for _, l := range y { + out = append(out, "+"+l) + } + return out + } + lcs := make([][]int, n+1) + for i := range lcs { + lcs[i] = make([]int, m+1) + } + for i := n - 1; i >= 0; i-- { + for j := m - 1; j >= 0; j-- { + if x[i] == y[j] { + lcs[i][j] = lcs[i+1][j+1] + 1 + } else { + lcs[i][j] = max(lcs[i+1][j], lcs[i][j+1]) + } + } + } + var out []string + var hunk []string + ha, hb, ca, cb := 0, 0, 0, 0 + flush := func() { + if len(hunk) > 0 { + out = append(out, fmt.Sprintf("@@ -%d,%d +%d,%d @@", ha, ca, hb, cb)) + out = append(out, hunk...) + hunk = nil + } + } + i, j := 0, 0 + for i < n || j < m { + switch { + case i < n && j < m && x[i] == y[j]: + flush() + i++ + j++ + case i < n && (j == m || lcs[i+1][j] >= lcs[i][j+1]): + if len(hunk) == 0 { + ha, hb, ca, cb = head+i+1, head+j+1, 0, 0 + } + hunk = append(hunk, "-"+x[i]) + ca++ + i++ + default: + if len(hunk) == 0 { + ha, hb, ca, cb = head+i+1, head+j+1, 0, 0 + } + hunk = append(hunk, "+"+y[j]) + cb++ + j++ + } + } + flush() + return out +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/bound_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/bound_test.go new file mode 100644 index 00000000..3f47ac3e --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/bound_test.go @@ -0,0 +1,132 @@ +package main + +import ( + "encoding/json" + "strings" + "testing" +) + +// to-be 55 §10: "An answer stays within its bound — the module's test over the full import of main at a recorded +// commit (testdata/hq-main.commit): every ask, get and holds of a fixed question list." + +var questions = []string{ + "what holds about the merge window, and why", + "how is a module's health judged", + "who hands out decision numbers", + "what is a seat", + "can the mesh create a symlink", + "how does a node reach the bus", + "how are secrets kept", + "what does the operator approve and on which channel", + "what is the knowledge base", + "how is a delivery walked across the nodes", + "what does the firewall allow", + "the", + "zebra quantum", +} + +func TestEveryAnswerStaysWithinItsBoundOverTheWholeOfMain(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := newWorld(t, s, hqMain(t)) + if _, err := w.k.Import("import@test", ImportAsk{}, hqMain(t)); err != nil { + t.Fatal(err) + } + // Draft and confirm a few prose records, so drafted and confirmed clauses are in the answers too. + for i := 0; i < 12; i++ { + if _, err := w.k.DraftStep(); err != nil { + t.Fatal(err) + } + } + within := func(what string, v any, bound int) { + t.Helper() + raw, _ := json.Marshal(v) + if len(raw) > bound { + t.Errorf("%s: %d bytes, over its bound of %d", what, len(raw), bound) + } + } + cuts := 0 + for _, q := range questions { + for _, bound := range []int{0, 2048, 16 << 10} { + a, err := w.k.Ask(q, bound) + if err != nil { + t.Fatalf("%q: %v", q, err) + } + within("ask "+q, a, boundOf(bound)) + if a.Cut != "" { + cuts++ + } + } + } + if cuts == 0 { + t.Error("no answer was cut: the bound was never tested") + } + all, _ := s.All(t.Context()) + topics := map[string]bool{} + for _, e := range all { + k := e.Current.Key() + for _, get := range []GetAsk{{Kind: k.Kind, Number: k.Number}, {Kind: k.Kind, Number: k.Number, Reasoning: true}} { + g, err := w.k.Get(get) + if err != nil { + t.Fatalf("get %s: %v", k, err) + } + within("get "+k.String(), g, DefaultBound) + if get.Reasoning && len(g.Sections) > 0 && g.Cut == "" && len(g.Reasoning) < len(g.Sections) { + t.Errorf("get %s left out reasoning without saying so", k) + } + } + h, err := w.k.Holds("", &k, 0) + if err != nil { + t.Fatal(err) + } + within("holds "+k.String(), h, DefaultBound) + topics[e.Current.Topic] = true + } + for topic := range topics { + h, err := w.k.Holds(topic, nil, 0) + if err != nil { + t.Fatal(err) + } + within("holds "+topic, h, DefaultBound) + } + }) +} + +// The acceptance sentence, over the whole of main: "what holds about the merge window, and why" answers, in one +// call under 4 KiB, the clauses in force of the record that decided it, what changed them since, and how to read +// its reasoning. +func TestTheMergeWindowQuestionIsAnsweredInOneCallUnderFourKiB(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := newWorld(t, s, hqMain(t)) + if _, err := w.k.Import("import@test", ImportAsk{}, hqMain(t)); err != nil { + t.Fatal(err) + } + a, err := w.k.Ask("what holds about the merge window, and why", 0) + if err != nil { + t.Fatal(err) + } + raw, _ := json.Marshal(a) + if len(raw) > 4096 || len(a.Records) == 0 { + t.Fatalf("%d bytes, %d records", len(raw), len(a.Records)) + } + var r276 *AnswerRecord + for i, r := range a.Records[:min(3, len(a.Records))] { + if r.Record == "ADR 0276" { + r276 = &a.Records[i] + } + } + if r276 == nil { + t.Fatalf("ADR 0276 is not among the first three: %s", raw) + } + if len(r276.Clauses) == 0 || !strings.Contains(r276.Clauses[0].Holds, "in force") || !strings.Contains(r276.Read, "reasoning") { + t.Fatalf("0276 as answered: %+v", r276) + } + changed := false + for _, s := range r276.Since { + changed = changed || strings.Contains(s, "ADR 0282") + } + if !changed { + t.Fatalf("what changed 0276 since is not said: %+v", r276.Since) + } + t.Logf("%d bytes: %s", len(raw), raw) + }) +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/bus_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/bus_test.go new file mode 100644 index 00000000..32b6df67 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/bus_test.go @@ -0,0 +1,390 @@ +package main + +import ( + "bufio" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/nats-io/nats-server/v2/server" + "github.com/nats-io/nats.go" + "github.com/nats-io/nats.go/jetstream" +) + +// Over a real bus: a JetStream server is linked into the test and started for it alone; the module is this test +// binary started again as the bundle, speaking MCP on its standard streams to a runtime played here, which does +// what the tool runner does with a bundle: a tool it calls is a request on the bus, an event a publish on the +// module's event subject, an ask a publish on the router's subject, the router's record a key of a bucket only +// the router writes. On the other side of the bus stand the issue tracker's reserve, the agent's judgement, the +// records module (reading the fixture), and the router, played by the test. + +// busWorld is one test's bus and what answers on it. +type busWorld struct { + t *testing.T + nc *nats.Conn + router jetstream.KeyValue + mu sync.Mutex + events []map[string]any + asked map[string][]json.RawMessage +} + +func newBusWorld(t *testing.T) *busWorld { + t.Helper() + s, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true}) + if err != nil { + t.Fatal(err) + } + go s.Start() + if !s.ReadyForConnections(30 * time.Second) { + t.Fatal("no bus within 30s") + } + t.Cleanup(func() { s.Shutdown(); s.WaitForShutdown() }) + nc, err := nats.Connect(s.ClientURL()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(nc.Close) + js, _ := jetstream.New(nc) + kv, err := js.CreateKeyValue(context.Background(), jetstream.KeyValueConfig{Bucket: "router"}) + if err != nil { + t.Fatal(err) + } + w := &busWorld{t: t, nc: nc, router: kv, asked: map[string][]json.RawMessage{}} + // The module's events, as anyone on the bus hears them. + if _, err := nc.Subscribe("mesh.mod.mesh-knowledge.event.>", func(m *nats.Msg) { + var body map[string]any + _ = json.Unmarshal(m.Data, &body) + body["event"] = strings.TrimPrefix(m.Subject, "mesh.mod.mesh-knowledge.event.") + w.mu.Lock() + w.events = append(w.events, body) + w.mu.Unlock() + }); err != nil { + t.Fatal(err) + } + // What stands on the other side: the issue tracker, the records module, the agent's judgement. + records := fixture() + answer := func(key string, do func(body json.RawMessage) (any, error)) { + if _, err := nc.Subscribe("ask."+key, func(m *nats.Msg) { + w.mu.Lock() + w.asked[key] = append(w.asked[key], m.Data) + w.mu.Unlock() + out, err := do(m.Data) + if err != nil { + _ = m.Respond([]byte(`{"error":` + strconv.Quote(err.Error()) + `}`)) + return + } + raw, _ := json.Marshal(map[string]any{"result": out}) + _ = m.Respond(raw) + }); err != nil { + t.Fatal(err) + } + } + next := 300 + answer("seat:issue-tracker.reserve", func(body json.RawMessage) (any, error) { + var a struct{ Kind, For string } + _ = json.Unmarshal(body, &a) + next++ + return map[string]any{"kind": a.Kind, "number": next - 1}, nil + }) + answer("records.records_status", func(json.RawMessage) (any, error) { + c, err := records.Standing() + return map[string]any{"commit": c}, err + }) + answer("records.records_list", func(body json.RawMessage) (any, error) { + var a struct{ Folder string } + _ = json.Unmarshal(body, &a) + f, d, err := records.List(a.Folder) + return map[string]any{"folders": f, "documents": d}, err + }) + answer("records.records_read", func(body json.RawMessage) (any, error) { + var a struct{ Path string } + _ = json.Unmarshal(body, &a) + c, commit, err := records.Read(a.Path) + return map[string]any{"content": c, "commit": commit}, err + }) + j := &judge{} + answer("claude-code.claude_code_judge@test", func(body json.RawMessage) (any, error) { + var a struct { + Question string `json:"question"` + Schema map[string]any `json:"schema"` + Model string `json:"model"` + } + _ = json.Unmarshal(body, &a) + out, by, err := j.judge(a.Question, a.Schema, a.Model) + return map[string]any{"answer": out, "model": a.Model, "node": "test", "by": by}, err + }) + return w +} + +func (w *busWorld) of(event string) []map[string]any { + w.mu.Lock() + defer w.mu.Unlock() + var out []map[string]any + for _, e := range w.events { + if e["event"] == event { + out = append(out, e) + } + } + return out +} + +// bundle is the knowledge base running as the runtime launches it, and the runtime's half of its conversation. +type bundle struct { + t *testing.T + w *busWorld + in io.WriteCloser + mu sync.Mutex + next int + wait map[string]chan json.RawMessage +} + +func (w *busWorld) start(t *testing.T) (*bundle, string) { + t.Helper() + dir := t.TempDir() + settings := filepath.Join(dir, "settings.json") + _ = os.WriteFile(settings, []byte(`{"judge-model": "sonnet", "drafting": "off"}`), 0o600) + status := filepath.Join(dir, "health.json") + cmd := exec.Command(os.Args[0], "-test.run=^$") + cmd.Env = append(os.Environ(), bundleVar+"=1", "MESH_KNOWLEDGE_SETTINGS="+settings, "MESH_KNOWLEDGE_STATUS="+status, + "MESH_SERVED_MODULE="+Module, "MESH_MODULE="+Module, "MESH_NODE=test") + in, _ := cmd.StdinPipe() + out, _ := cmd.StdoutPipe() + cmd.Stderr = logWriter{t} + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + b := &bundle{t: t, w: w, in: in, wait: map[string]chan json.RawMessage{}} + go b.read(out) + t.Cleanup(func() { _ = in.Close(); _ = cmd.Wait() }) + return b, status +} + +type logWriter struct{ t *testing.T } + +func (l logWriter) Write(p []byte) (int, error) { l.t.Logf("bundle: %s", p); return len(p), nil } + +func (b *bundle) say(m map[string]any) { + m["jsonrpc"] = "2.0" + raw, _ := json.Marshal(m) + b.mu.Lock() + defer b.mu.Unlock() + _, _ = b.in.Write(append(raw, '\n')) +} + +func (b *bundle) read(out io.Reader) { + sc := bufio.NewScanner(out) + sc.Buffer(make([]byte, 1<<20), 1<<24) + for sc.Scan() { + var m struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + Params json.RawMessage `json:"params"` + } + if json.Unmarshal(sc.Bytes(), &m) != nil { + continue + } + if m.Method == "" { + var id string + _ = json.Unmarshal(m.ID, &id) + b.mu.Lock() + ch := b.wait[id] + delete(b.wait, id) + b.mu.Unlock() + if ch != nil { + ch <- append([]byte(nil), sc.Bytes()...) + } + continue + } + go b.runtime(m.ID, m.Method, m.Params) + } +} + +// runtime is what the tool runner does for the bundle, over the bus. +func (b *bundle) runtime(id json.RawMessage, method string, params json.RawMessage) { + var p struct { + Key string `json:"key"` + Body json.RawMessage `json:"body"` + Subject string `json:"subject"` + Headers map[string]string `json:"headers"` + } + _ = json.Unmarshal(params, &p) + result, err := func() (any, error) { + switch method { + case "mesh/ask": + msg, err := b.w.nc.Request("ask."+p.Key, p.Body, 30*time.Second) + if err != nil { + return nil, fmt.Errorf("nothing answers %s: %v", p.Key, err) + } + var r struct { + Result json.RawMessage `json:"result"` + Error string `json:"error"` + } + _ = json.Unmarshal(msg.Data, &r) + if r.Error != "" { + return nil, errors.New(r.Error) + } + return r.Result, nil + case "mesh/publish": + if p.Headers["x-source"] != Module { + return nil, fmt.Errorf("an event not said as %s", Module) + } + return map[string]any{}, b.w.nc.Publish("mesh.mod."+Module+".event."+p.Key, p.Body) + case "mesh/seat.publish": + return map[string]any{}, b.w.nc.Publish(p.Subject, p.Body) + case "mesh/seat.record": + e, err := b.w.router.Get(context.Background(), p.Key) + if errors.Is(err, jetstream.ErrKeyNotFound) { + return nil, nil + } + if err != nil { + return nil, err + } + return map[string]any{"key": p.Key, "value": json.RawMessage(e.Value()), "revision": e.Revision()}, nil + case "mesh/subscribe": + return map[string]any{}, nil + } + return nil, fmt.Errorf("the runtime offers no %s", method) + }() + if err != nil { + b.say(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}}) + return + } + b.say(map[string]any{"id": id, "result": result}) +} + +// call is one tools/call, as the tool runner makes it. +func (b *bundle) call(tool string, args map[string]any) (json.RawMessage, error) { + b.mu.Lock() + b.next++ + id := "runtime-" + strconv.Itoa(b.next) + ch := make(chan json.RawMessage, 1) + b.wait[id] = ch + b.mu.Unlock() + b.say(map[string]any{"id": id, "method": "tools/call", "params": map[string]any{"name": tool, "arguments": args}}) + select { + case raw := <-ch: + var m struct { + Result struct { + Content []struct { + Text string `json:"text"` + } `json:"content"` + IsError bool `json:"isError"` + } `json:"result"` + } + if err := json.Unmarshal(raw, &m); err != nil || len(m.Result.Content) == 0 { + return nil, fmt.Errorf("an answer that is not one: %s", raw) + } + if m.Result.IsError { + return nil, errors.New(m.Result.Content[0].Text) + } + return json.RawMessage(m.Result.Content[0].Text), nil + case <-time.After(60 * time.Second): + return nil, fmt.Errorf("%s did not answer in 60 s", tool) + } +} + +func TestTheKnowledgeBaseOverTheBus(t *testing.T) { + w := newBusWorld(t) + b, _ := w.start(t) + must := func(raw json.RawMessage, err error) json.RawMessage { + t.Helper() + if err != nil { + t.Fatal(err) + } + return raw + } + // The import, read through the records module over the bus; run twice. + var imp Imported + _ = json.Unmarshal(must(b.call(Seat+".import", map[string]any{})), &imp) + if imp.Created != 11 || imp.Refused != 0 { + t.Fatalf("import over the bus: %+v", imp) + } + _ = json.Unmarshal(must(b.call(Seat+".import", map[string]any{})), &imp) + if imp.Created != 0 || !strings.Contains(imp.Said, "Nothing changed") { + t.Fatalf("a second import: %s", imp.Said) + } + // A number from the issue tracker's reserve, asked with its contract. + var r Reserved + _ = json.Unmarshal(must(b.call(Seat+".reserve", map[string]any{"kind": "decision", "for": "a merge window that closes sooner"})), &r) + if r.Number != 300 { + t.Fatalf("reserved %+v", r) + } + w.mu.Lock() + var asked map[string]any + _ = json.Unmarshal(w.asked["seat:issue-tracker.reserve"][0], &asked) + w.mu.Unlock() + if asked["kind"] != "decision" || asked["for"] != "a merge window that closes sooner" || len(asked) != 2 { + t.Fatalf("the tracker was asked %v", asked) + } + // A proposal, its sections and links as an agent sends them; refused naming the rule, then taken. + p := valid(300) + links := []map[string]any{} + for _, l := range p.Links { + links = append(links, map[string]any{"kind": l.Kind, "to": l.To, "from": l.From}) + } + sections := []map[string]any{} + for _, s := range p.Sections { + sections = append(sections, map[string]any{"name": s.Name, "text": s.Text}) + } + args := map[string]any{"number": 300, "title": p.Title, "topic": "gardening", "date": p.Date, "deciders": []string{"jochen"}, + "clauses": p.Clauses, "sections": sections, "links": links} + if _, err := b.call(Seat+".propose", args); err == nil || !strings.Contains(err.Error(), "- topic:") { + t.Fatalf("a proposal with an unknown topic: %v", err) + } + if _, err := b.call(Seat+".propose", map[string]any{"number": 300, "title": "x", "by": "someone@else"}); err == nil || + !strings.Contains(err.Error(), "does not take by") { + t.Fatalf("an argument naming the caller: %v", err) + } + args["topic"] = "the mesh" + var prop Proposed + _ = json.Unmarshal(must(b.call(Seat+".propose", args)), &prop) + if prop.Status != StatusProposed { + t.Fatalf("%+v", prop) + } + // The runtime proves no caller yet: the review is refused, and says why. + if _, err := b.call(Seat+".review", map[string]any{"number": 300, "verdict": "approve"}); err == nil || !strings.Contains(err.Error(), "issue 365") { + t.Fatalf("a review by an unproven caller over the bus: %v", err) + } + // What holds, in one call within its bound. + raw := must(b.call(Seat+".ask", map[string]any{"question": "what holds about the merge window, and why"})) + if len(raw) > DefaultBound || !strings.Contains(string(raw), "ADR 0276") { + t.Fatalf("%d bytes: %s", len(raw), raw) + } + var g Got + _ = json.Unmarshal(must(b.call(Seat+".get", map[string]any{"number": 300})), &g) + if g.Status != StatusProposed || len(g.Clauses) != 2 || g.Clauses[0].Holds != "proposed: not in force" { + t.Fatalf("get: %+v", g) + } + var h Historied + _ = json.Unmarshal(must(b.call(Seat+".history", map[string]any{"number": 300, "from": 1, "to": 2})), &h) + if len(h.Versions) != 2 || len(h.Diff) == 0 { + t.Fatalf("history: %+v", h) + } + // Every change said on the bus, as the module, with who acted. + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) && len(w.of(EventProposed)) == 0 { + time.Sleep(50 * time.Millisecond) + } + if len(w.of(EventImported)) != 11 || len(w.of(EventReserved)) != 1 || len(w.of(EventProposed)) != 1 { + t.Fatalf("events: %d imported, %d reserved, %d proposed", len(w.of(EventImported)), len(w.of(EventReserved)), len(w.of(EventProposed))) + } + if e := w.of(EventProposed)[0]; e["by"] != Unproven || e["record"] != "ADR 0300" { + t.Fatalf("the proposed event: %v", e) + } + // The health tool says what the watch last saw: here it has seen nothing. + var health HealthAnswer + _ = json.Unmarshal(must(b.call(HealthTool, map[string]any{})), &health) + if health.Healthy || !strings.Contains(health.Why, "no verdict") { + t.Fatalf("health: %+v", health) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/daemon-lines b/modules/mesh-knowledge/cmd/mesh-knowledge/daemon-lines new file mode 100644 index 00000000..be1c6592 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/daemon-lines @@ -0,0 +1,73 @@ +# The daemons' lines a record never writes (novox/hq ADR 0272): a copy of the catalogue's retired-lines, which +# the test TestTheDaemonLinesAreTheCataloguesCopy holds equal. One line each, ` `; a trailing ' …' +# marks a command that is a line only with an argument after it. +mesh-controller adopt +mesh-controller artifacts … +mesh-controller ask … +mesh-controller assign +mesh-controller broker +mesh-controller builds … +mesh-controller bus … +mesh-controller cancel +mesh-controller cleanup +mesh-controller collect +mesh-controller collection … +mesh-controller conditions +mesh-controller converge +mesh-controller data … +mesh-controller delivery … +mesh-controller doctor +mesh-controller durations +mesh-controller hand-act +mesh-controller hand-acts +mesh-controller healers +mesh-controller identity +mesh-controller images … +mesh-controller kill +mesh-controller licence +mesh-controller migrate +mesh-controller mirrors … +mesh-controller module … +mesh-controller node … +mesh-controller operator +mesh-controller overlay … +mesh-controller pause +mesh-controller pin +mesh-controller plan +mesh-controller plans … +mesh-controller prepare +mesh-controller push +mesh-controller queue … +mesh-controller rebuild +mesh-controller rehearse +mesh-controller replay +mesh-controller resume +mesh-controller retire +mesh-controller rollout +mesh-controller rotate +mesh-controller seat … +mesh-controller seats … +mesh-controller secret +mesh-controller settings +mesh-controller status +mesh-controller take … +mesh-controller token +mesh-controller unassign +mesh-controller unpin +mesh-controller upgrade +mesh-controller version … +mesh-host apply +mesh-host bundle +mesh-host enrol +mesh-host enroll +mesh-host hand-over +mesh-host help +mesh-host inventory +mesh-host key +mesh-host overlay +mesh-host owned +mesh-host profile +mesh-host reconcile +mesh-host run +mesh-host version +nox-mesh-host hand-over diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/dated_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/dated_test.go new file mode 100644 index 00000000..20bbbdec --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/dated_test.go @@ -0,0 +1,98 @@ +package main + +import ( + "strings" + "testing" +) + +// A rule holds a record from the day hq's checks began to hold it (the review of mesh-catalog#206): a dated +// correction to an older record is judged by the rules that held it then, never by one that came later. + +// insightTo is a progressive insight to a held record: the record as it stands, resent whole, with what stood +// before. +func insightTo(t *testing.T, w *world, k Key) ProposeAsk { + t.Helper() + r := mustGet(t, w.k.Store, k).Current + p := ProposeAsk{Kind: k.Kind, Number: k.Number, Title: r.Title, Topic: r.Topic, Date: r.Date, Deciders: r.Deciders, + Sections: r.Sections, Homes: r.Homes, Insight: "the count said 12 where 13 stood"} + for _, c := range r.Clauses { + p.Clauses = append(p.Clauses, c.Text) + } + for _, l := range r.Links { + p.Links = append(p.Links, LinkAsk{Kind: l.Kind, To: l.To.String(), From: l.From}) + } + return p +} + +// ADR 0011 (2026-04-03) is a real older record: its decision is prose, it has no section saying how it is checked, +// and it was written before the words and clause rules held. A correction of fact to it is taken. +func TestAnInsightToAnOlderRecordIsJudgedByTheRulesThatHeldIt(t *testing.T) { + w := newWorld(t, newMemStore(), hqMain(t)) + if _, err := w.k.Import("import@test", ImportAsk{}, hqMain(t)); err != nil { + t.Fatal(err) + } + p := insightTo(t, w, Key{KindDecision, 11}) + p.Sections[0].Text += "\n\nThe control plane wrote them." + if _, err := w.k.Propose("corrector@ace", p); err != nil { + t.Fatalf("an insight to ADR 0011 was refused by rules that never held it: %v", err) + } + // The same words in a record of today are refused. + n := w.reserved(t, "author@g14") + q := valid(n) + q.Sections[2].Text = "The control plane wrote them." + if _, err := w.k.Propose("author@g14", q); err == nil || !strings.Contains(err.Error(), "- retired-word:") { + t.Fatalf("a retired word in a record of today: %v", err) + } +} + +func TestEachDatedRuleHoldsFromItsDay(t *testing.T) { + for _, c := range []struct { + rule, before, from string + }{ + {"retired-word", "2026-10-06", "2026-10-07"}, + {"daemon-line", "2026-10-08", "2026-10-09"}, + {"deferral", "2026-10-08", "2026-10-09"}, + {"has-clause", "2026-10-10", "2026-10-11"}, + {"clause-lead", "2026-10-10", "2026-10-11"}, + {"checked", "2026-10-10", "2026-10-11"}, + } { + if Holds(c.rule, KindResearch, c.before) || !Holds(c.rule, KindResearch, c.from) { + t.Errorf("%s holds research of %s: %v, of %s: %v", c.rule, c.before, Holds(c.rule, KindResearch, c.before), c.from, Holds(c.rule, KindResearch, c.from)) + } + } + for _, rule := range []string{"retired-word", "daemon-line"} { + if Holds(rule, KindDecision, "2026-10-10") || !Holds(rule, KindDecision, "2026-10-11") { + t.Errorf("%s holds a decision from another day than 2026-10-11", rule) + } + } + if !Holds("title", KindDecision, "2026-01-01") { + t.Error("an undated rule does not hold an old record") + } +} + +// A new record cannot be dated back past the rules: it is of the day its number was reserved, or later. +func TestANewRecordIsNotDatedBeforeItsReservation(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@g14") + p := valid(n) + p.Date = "2026-01-01" + p.Sections[2].Text = "The control plane wrote them." + _, err := w.k.Propose("author@g14", p) + if err == nil || !strings.Contains(err.Error(), "- date:") { + t.Fatalf("a new record dated before its reservation: %v", err) + } +} + +// hq's words check never read a decision record: a decision keeps the words it was written with. A correction to +// ADR 0276 (2026-10-10), which writes words and lines those checks held only research and issues to, is taken. +func TestAnInsightToADecisionOfOctoberKeepsItsWords(t *testing.T) { + w := newWorld(t, newMemStore(), hqMain(t)) + if _, err := w.k.Import("import@test", ImportAsk{}, hqMain(t)); err != nil { + t.Fatal(err) + } + for _, n := range []int{276, 240, 244, 272} { + if _, err := w.k.Propose("corrector@ace", insightTo(t, w, Key{KindDecision, n})); err != nil { + t.Errorf("an insight to ADR %04d was refused: %v", n, err) + } + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/draft.go b/modules/mesh-knowledge/cmd/mesh-knowledge/draft.go new file mode 100644 index 00000000..2dda7547 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/draft.go @@ -0,0 +1,463 @@ +package main + +// Drafting and confirming clauses at the import (novox/hq ADR 0290 rule 8, to-be 55 §7). For a decision whose +// Decision section is prose, one judgement drafts its clauses: the section in, numbered statements out, each one +// sentence saying nothing the section does not. They are kept `drafted`, and never served as in force. A second +// judgement, a separate call that sees only the record and the drafts, answers per clause whether the record says +// exactly that: a clause it accepts becomes `confirmed`, one it rejects stays `drafted` with the reason. Which +// clause of an older record a supersedes-in-part changes is judged and confirmed the same way. When every record +// of a topic has been through both, the operator is told once, with the counts. `return` sends a record back to +// be drafted again. +// +// The judgements are the agent's on this node (claude-code.claude_code_judge, ADR 0278): no tools, no session, +// at most 25 seconds each, one at a time. The worker takes one step at a time in the background. + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strconv" + "strings" + "time" +) + +// DefaultJudgeModel is the model the judgements ask unless the settings name another. +const DefaultJudgeModel = "sonnet" + +// MostDraftTries is how many times a record's drafting is tried before it is left failed, said by its get. +const MostDraftTries = 3 + +var draftSchema = map[string]any{ + "type": "object", + "properties": map[string]any{ + "clauses": map[string]any{"type": "array", "items": map[string]any{ + "type": "object", + "properties": map[string]any{ + "lead": map[string]any{"type": "string", "description": "a few words naming what is decided, ending with a full stop"}, + "text": map[string]any{"type": "string", "description": "the one sentence of the statement, after the lead"}, + }, + "required": []string{"lead", "text"}, + }}, + }, + "required": []string{"clauses"}, +} + +var confirmSchema = map[string]any{ + "type": "object", + "properties": map[string]any{ + "verdicts": map[string]any{"type": "array", "items": map[string]any{ + "type": "object", + "properties": map[string]any{ + "clause": map[string]any{"type": "integer"}, + "exact": map[string]any{"type": "boolean", "description": "true when the record says exactly this, nothing more"}, + "reason": map[string]any{"type": "string", "description": "when not exact: what the record says instead"}, + }, + "required": []string{"clause", "exact"}, + }}, + }, + "required": []string{"verdicts"}, +} + +var whichSchema = map[string]any{ + "type": "object", + "properties": map[string]any{ + "clauses": map[string]any{"type": "array", "items": map[string]any{"type": "integer"}, + "description": "the numbers of the older record's clauses the newer record replaces part of"}, + "reason": map[string]any{"type": "string"}, + }, + "required": []string{"clauses", "reason"}, +} + +var whichConfirmSchema = map[string]any{ + "type": "object", + "properties": map[string]any{ + "exact": map[string]any{"type": "boolean"}, + "reason": map[string]any{"type": "string"}, + }, + "required": []string{"exact", "reason"}, +} + +// DraftQuestion is the first judgement's question: the record's Decision section, nothing else of the mesh. +func DraftQuestion(r *Record, section, returned string) string { + var b strings.Builder + fmt.Fprintf(&b, "Below is the Decision section of a decision record, %s: %q. It is written as prose. Write what it decides as numbered "+ + "clauses: each one statement, one sentence, with a lead of a few words naming what is decided. A clause says nothing the section "+ + "does not say: no inference, no context from elsewhere, nothing added. Every decision the section makes is in one clause; a "+ + "sentence that only explains is in none. Keep the section's own words where you can.\n", r.Key(), r.Title) + if returned != "" { + fmt.Fprintf(&b, "\nAn earlier draft was sent back, with this reason: %s\n", returned) + } + b.WriteString("\nThe Decision section:\n\n" + section) + return b.String() +} + +// ConfirmQuestion is the second judgement's question: the record and the drafts, nothing else. +func ConfirmQuestion(r *Record, section string, drafts []Clause) string { + var b strings.Builder + fmt.Fprintf(&b, "Below are the Decision section of a decision record, %s: %q, and clauses drafted from it by someone else. For each "+ + "clause, answer whether the section says exactly that: exact is true only when the clause says nothing the section does not, "+ + "and leaves out nothing that changes its meaning. When it is not exact, say in reason what the section says instead.\n\n"+ + "The Decision section:\n\n%s\n\nThe drafted clauses:\n\n", r.Key(), r.Title, section) + for _, c := range drafts { + fmt.Fprintf(&b, "%d. %s\n", c.N, c.Text) + } + return b.String() +} + +func (k *KB) judgeModel() string { + if m := strings.TrimSpace(k.settings().JudgeModel); m != "" { + return m + } + return DefaultJudgeModel +} + +// DraftStep takes one step of the drafting and confirming: the first record waiting to be drafted or confirmed, +// then the first changes link whose clause is not yet judged. It answers what it did, or "" when nothing waits. +func (k *KB) DraftStep() (string, error) { + if k.Judge == nil || strings.EqualFold(strings.TrimSpace(k.settings().Drafting), "off") { + return "", nil + } + ctx := context.Background() + all, err := k.Store.All(ctx) + if err != nil { + return "", err + } + for _, e := range all { + d := e.Current.Drafting + if d == nil || d.State == DraftDone || (d.State == DraftFailed && d.Tries >= MostDraftTries) { + continue + } + if d.State == DraftFailed && k.now().Sub(d.At) < time.Duration(d.Tries)*10*time.Minute { + continue + } + return k.draftOne(ctx, e.Current.Key()) + } + for _, e := range all { + if !e.Current.Accepted() { + continue + } + for i, l := range e.Current.Links { + if l.Kind == LinkChanges && (l.Source == LinkPending || l.Source == SourceDrafted) && !l.Unresolved { + return k.judgeChange(ctx, e.Current.Key(), i) + } + } + } + return "", nil +} + +// draftOne drafts or confirms one record's clauses, under the write lock only to read and to write. +func (k *KB) draftOne(ctx context.Context, key Key) (string, error) { + k.mu.Lock() + e, err := k.Store.Get(ctx, key) + k.mu.Unlock() + if err != nil || e == nil { + return "", err + } + r := e.Current.Clone() + section, ok := decisionSection(r) + if !ok { + r.Drafting = &Drafting{State: DraftDone, Reason: "it has no Decision section", At: k.now()} + return k.saveDraft(ctx, e, r, EventDrafted, "no Decision section to draft from") + } + model := k.judgeModel() + fail := func(step string, err error) (string, error) { + // Tried again after a pause that grows with each try, at the step it failed at, up to MostDraftTries. + r.Drafting.State, r.Drafting.Reason, r.Drafting.At = DraftFailed, step+": "+err.Error(), k.now() + r.Drafting.Tries++ + _, serr := k.saveDraft(ctx, e, r, EventDrafted, step+" failed: "+err.Error()) + if serr != nil { + return "", serr + } + return "", fmt.Errorf("%s %s: %v", step, key, err) + } + if r.Drafting == nil { + r.Drafting = &Drafting{State: DraftWaiting} + } + // A failed record goes back to the step it failed at: confirming when it holds drafts, drafting otherwise. + confirming := r.Drafting.State == DraftDrafted || (r.Drafting.State == DraftFailed && hasDrafts(r)) + if !confirming { + answer, by, err := k.Judge(DraftQuestion(r, section, r.Drafting.Returned), draftSchema, model) + if err != nil { + return fail("drafting", err) + } + var drafted struct { + Clauses []struct { + Lead string `json:"lead"` + Text string `json:"text"` + } `json:"clauses"` + } + if err := json.Unmarshal(answer, &drafted); err != nil || len(drafted.Clauses) == 0 { + return fail("drafting", errors.New("the judgement drafted no clause")) + } + r.Clauses = nil + for i, c := range drafted.Clauses { + lead := strings.TrimSpace(strings.Trim(c.Lead, "* ")) + if !strings.HasSuffix(lead, ".") { + lead += "." + } + r.Clauses = append(r.Clauses, Clause{N: i + 1, Text: "**" + lead + "** " + strings.TrimSpace(c.Text), + Status: clauseStatusOf(r.Status), Source: SourceDrafted}) + } + r.Drafting = &Drafting{State: DraftDrafted, Drafter: by, Returned: r.Drafting.Returned, Tries: r.Drafting.Tries, At: k.now()} + return k.saveDraft(ctx, e, r, EventDrafted, fmt.Sprintf("%d clauses drafted by %s", len(r.Clauses), by)) + } + answer, by, err := k.Judge(ConfirmQuestion(r, section, r.Clauses), confirmSchema, model) + if err != nil { + return fail("confirming", err) + } + var verdicts struct { + Verdicts []struct { + Clause int `json:"clause"` + Exact bool `json:"exact"` + Reason string `json:"reason"` + } `json:"verdicts"` + } + if err := json.Unmarshal(answer, &verdicts); err != nil { + return fail("confirming", errors.New("the judgement answered no verdicts")) + } + confirmed := 0 + for i := range r.Clauses { + r.Clauses[i].Unconfirmed = "the second judgement gave no verdict on it" + for _, v := range verdicts.Verdicts { + if v.Clause != r.Clauses[i].N { + continue + } + if v.Exact { + r.Clauses[i].Source, r.Clauses[i].Unconfirmed = SourceConfirmed, "" + confirmed++ + } else { + r.Clauses[i].Unconfirmed = strings.TrimSpace(v.Reason) + if r.Clauses[i].Unconfirmed == "" { + r.Clauses[i].Unconfirmed = "not exact" + } + } + } + } + r.Drafting.State, r.Drafting.Checker, r.Drafting.At, r.Drafting.Reason = DraftDone, by, k.now(), "" + said, err := k.saveDraft(ctx, e, r, EventConfirmed, fmt.Sprintf("%d of %d drafted clauses confirmed by %s", confirmed, len(r.Clauses), by)) + if err != nil { + return "", err + } + k.tellTopic(ctx, r.Topic) + return said, nil +} + +func hasDrafts(r *Record) bool { + for _, c := range r.Clauses { + if c.Source == SourceDrafted { + return true + } + } + return false +} + +func (k *KB) saveDraft(ctx context.Context, e *Entry, r *Record, event, why string) (string, error) { + k.mu.Lock() + defer k.mu.Unlock() + by := "the knowledge base" + if r.Drafting != nil && event == EventDrafted && r.Drafting.Drafter != "" { + by = r.Drafting.Drafter + } + if r.Drafting != nil && event == EventConfirmed && r.Drafting.Checker != "" { + by = r.Drafting.Checker + } + if err := k.Store.Save(ctx, Version{By: by, At: k.now(), Why: why, Record: r}, true, false, e.Revision); err != nil { + return "", fmt.Errorf("%s could not be kept (%v)", r.Key(), err) + } + k.emit(event, r, by, why, e.Versions+1) + return r.Key().String() + ": " + why, nil +} + +// tellTopic tells the operator, once, when every prose record of a topic has been drafted and confirmed. +func (k *KB) tellTopic(ctx context.Context, topic string) { + if k.Notify == nil || topic == "" { + return + } + all, err := k.Store.All(ctx) + if err != nil { + return + } + records, confirmed, drafted := 0, 0, 0 + for _, e := range all { + r := e.Current + if r.Topic != topic || r.Drafting == nil { + continue + } + if r.Drafting.State != DraftDone { + return + } + records++ + for _, c := range r.Clauses { + if c.Source == SourceConfirmed { + confirmed++ + } else if c.Source == SourceDrafted { + drafted++ + } + } + } + if records == 0 { + return + } + first, err := k.Store.Told(ctx, topic) + if err != nil || !first { + return + } + summary := fmt.Sprintf("The clauses of %d prose decisions on %s are drafted and checked: %d confirmed, %d still drafted.", records, topic, confirmed, drafted) + explanation := "A second judgement confirmed each clause against its record; a clause still drafted is never served as in force. " + + "You may send any record back with the knowledge base's return, and its clauses are drafted again. Nothing needs you otherwise." + if err := k.Notify("knowledge.clauses."+strings.ReplaceAll(topic, " ", "-"), summary, "Clauses drafted for "+topic, explanation); err != nil { + k.logf("[mesh-knowledge] the operator could not be told about %s's clauses: %v", topic, err) + } +} + +// judgeChange judges, then confirms, which clause of an older record a supersedes-in-part of hq changes. +func (k *KB) judgeChange(ctx context.Context, from Key, i int) (string, error) { + k.mu.Lock() + e, err := k.Store.Get(ctx, from) + var t *Entry + if err == nil && e != nil && i < len(e.Current.Links) { + if to, ok := e.Current.Links[i].To.Record(); ok { + t, err = k.Store.Get(ctx, to) + } + } + k.mu.Unlock() + if err != nil || e == nil || t == nil { + return "", err + } + r := e.Current.Clone() + l := &r.Links[i] + older := t.Current + served := 0 + for _, c := range older.Clauses { + if c.Source != SourceDrafted { + served++ + } + } + if served == 0 { + return "", nil // the older record's clauses are not confirmed yet: judged once they are + } + model := k.judgeModel() + newer, _ := decisionSection(r) + var b strings.Builder + fmt.Fprintf(&b, "%s says it replaces part of an older record, %s. Below are the clauses of %s and the Decision section of %s.\n\n", r.Key(), older.Key(), older.Key(), r.Key()) + for _, c := range older.Clauses { + fmt.Fprintf(&b, "%d. %s\n", c.N, c.Text) + } + fmt.Fprintf(&b, "\nThe Decision section of %s:\n\n%s\n", r.Key(), newer) + if l.Source == LinkPending { + answer, by, err := k.Judge(b.String()+"\nWhich of the older record's clauses does the newer record replace part of? Name their numbers.", whichSchema, model) + if err != nil { + return "", fmt.Errorf("judging which clause %s changes: %v", r.Key(), err) + } + var which struct { + Clauses []int `json:"clauses"` + Reason string `json:"reason"` + } + if json.Unmarshal(answer, &which) != nil || len(which.Clauses) == 0 || older.Clause(which.Clauses[0]) == nil { + return "", fmt.Errorf("judging which clause %s changes: the judgement named no clause of %s", r.Key(), older.Key()) + } + base := *l + l.To.Clause, l.Source = which.Clauses[0], SourceDrafted + for _, n := range which.Clauses[1:] { + if older.Clause(n) != nil { + extra := base + extra.To.Clause, extra.Source = n, SourceDrafted + r.Links = append(r.Links, extra) + } + } + return k.saveLinks(ctx, e, r, by, fmt.Sprintf("which clause of %s it changes drafted: §%s (%s)", older.Key(), joinInts(which.Clauses), which.Reason)) + } + answer, by, err := k.Judge(b.String()+fmt.Sprintf("\nA first judgement said %s replaces part of clause %d of %s. Is that exact?", r.Key(), l.To.Clause, older.Key()), whichConfirmSchema, model) + if err != nil { + return "", fmt.Errorf("confirming which clause %s changes: %v", r.Key(), err) + } + var v struct { + Exact bool `json:"exact"` + Reason string `json:"reason"` + } + if json.Unmarshal(answer, &v) != nil { + return "", fmt.Errorf("confirming which clause %s changes: no verdict", r.Key()) + } + if !v.Exact { + clause := l.To.Clause + l.To.Clause, l.Source = 0, SourceDrafted+": not confirmed" + return k.saveLinks(ctx, e, r, by, fmt.Sprintf("that it changes %s §%d was not confirmed: %s", older.Key(), clause, v.Reason)) + } + l.Source = SourceConfirmed + said, err := k.saveLinks(ctx, e, r, by, fmt.Sprintf("that it changes %s §%d confirmed by %s", older.Key(), l.To.Clause, by)) + if err != nil { + return "", err + } + // The older clause is changed now, as an acceptance would have made it. + k.mu.Lock() + defer k.mu.Unlock() + t, err = k.Store.Get(ctx, older.Key()) + if err != nil || t == nil { + return said, err + } + c := t.Current.Clone() + if cl := c.Clause(l.To.Clause); cl != nil && cl.Status == ClauseInForce { + cl.Status = ClauseChanged + cl.ChangedBy = append(cl.ChangedBy, r.Key().String()) + what := fmt.Sprintf("§%d changed by %s (as hq said, its clause judged and confirmed)", l.To.Clause, r.Key()) + if err := k.Store.Save(ctx, Version{By: by, At: k.now(), Why: what, Record: c}, true, false, t.Revision); err != nil { + return said, err + } + k.emit(EventChanged, c, by, what, t.Versions+1) + } + return said, nil +} + +func (k *KB) saveLinks(ctx context.Context, e *Entry, r *Record, by, why string) (string, error) { + k.mu.Lock() + defer k.mu.Unlock() + if err := k.Store.Save(ctx, Version{By: by, At: k.now(), Why: why, Record: r}, true, false, e.Revision); err != nil { + return "", fmt.Errorf("%s could not be kept (%v)", r.Key(), err) + } + event := EventDrafted + if strings.Contains(why, "confirmed") { + event = EventConfirmed + } + k.emit(event, r, by, why, e.Versions+1) + return r.Key().String() + ": " + why, nil +} + +func joinInts(ns []int) string { + s := make([]string, len(ns)) + for i, n := range ns { + s[i] = strconv.Itoa(n) + } + return strings.Join(s, ", §") +} + +// Return sends a record's drafted or confirmed clauses back to be drafted again, with the reason, which the next +// drafting reads. +func (k *KB) Return(caller string, key Key, reason string) (map[string]any, error) { + ctx := context.Background() + reason = strings.TrimSpace(reason) + if reason == "" { + return nil, errors.New("return needs reason: what is wrong with the clauses drafted") + } + k.mu.Lock() + defer k.mu.Unlock() + e, err := k.Store.Get(ctx, key) + if err != nil { + return nil, err + } + if e == nil { + return nil, fmt.Errorf("the knowledge base holds no %s", key) + } + r := e.Current.Clone() + if r.Drafting == nil { + return nil, fmt.Errorf("%s's clauses were written by its author, not drafted: a written clause is changed by a new record", key) + } + r.Clauses = nil + r.Drafting = &Drafting{State: DraftWaiting, Returned: reason, At: k.now()} + why := "returned to be drafted again: " + reason + if err := k.Store.Save(ctx, Version{By: caller, At: k.now(), Why: why, Record: r}, true, false, e.Revision); err != nil { + return nil, err + } + k.emit(EventReturned, r, caller, why, e.Versions+1) + return map[string]any{"record": key.String(), "says": fmt.Sprintf("%s's clauses are taken back and will be drafted again, with your reason; until then it serves none.", key)}, nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/draft_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/draft_test.go new file mode 100644 index 00000000..83d60a5b --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/draft_test.go @@ -0,0 +1,243 @@ +package main + +import ( + "context" + "strings" + "testing" +) + +// to-be 55 §10: "A drafted clause is never served as in force — the module's test: holds over a record with only +// drafted clauses answers none, and ask marks them." And §7: two separate judgements, the second seeing only the +// record and the drafts; the operator told once per topic; return drafts again. + +// draftUntil takes drafting steps until the record's drafting reaches a state, at most a few. +func draftUntil(t *testing.T, w *world, k Key, state string) *Record { + t.Helper() + for i := 0; i < 40; i++ { + r := mustGet(t, w.k.Store, k).Current + if r.Drafting != nil && r.Drafting.State == state { + return r + } + if _, err := w.k.DraftStep(); err != nil { + t.Fatal(err) + } + } + t.Fatalf("%s's drafting never reached %s", k, state) + return nil +} + +func TestADraftedClauseIsNeverServedAsInForce(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + k := Key{KindDecision, 240} + r := draftUntil(t, w, k, DraftDrafted) + if len(r.Clauses) == 0 || r.Clauses[0].Source != SourceDrafted { + t.Fatalf("drafted: %+v", r.Clauses) + } + h, err := w.k.Holds("", &k, 0) + if err != nil { + t.Fatal(err) + } + if len(h.Clauses) != 0 || !strings.Contains(h.NotYet, "drafted clause(s) are not served") { + t.Fatalf("holds over a record with only drafted clauses: %+v", h) + } + // The topic's holds leaves them out too. + ht, _ := w.k.Holds(r.Topic, nil, 64<<10) + for _, c := range ht.Clauses { + if c.Record == k.String() { + t.Fatalf("the topic's holds served a drafted clause: %+v", c) + } + } + a, err := w.k.Ask("a module says how it is healthy and the node-engine judges it", 0) + if err != nil { + t.Fatal(err) + } + marked := false + for _, rec := range a.Records { + for _, c := range rec.Clauses { + if rec.Record == k.String() { + if !strings.HasPrefix(c.Holds, "drafted, not confirmed") { + t.Fatalf("ask served a drafted clause as %q", c.Holds) + } + marked = true + } + } + if rec.Record == k.String() && len(rec.Clauses) == 0 && strings.Contains(rec.Note, "drafted") { + marked = true + } + } + if !marked { + t.Fatalf("ask did not show 0240's drafted clauses as drafted: %+v", a) + } + // get marks each one. + g, _ := w.k.Get(GetAsk{Kind: KindDecision, Number: 240}) + for _, c := range g.Clauses { + if !strings.HasPrefix(c.Holds, "drafted") { + t.Fatalf("get: %+v", c) + } + } + }) +} + +func TestClausesAreDraftedThenConfirmedByASeparateJudgementThatSeesOnlyTheRecordAndTheDrafts(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + k := Key{KindDecision, 240} + r := draftUntil(t, w, k, DraftDone) + var draftQ, confirmQ string + n := 0 + for _, q := range w.judge.questions { + if !strings.Contains(q, "ADR 0240") { + continue + } + n++ + if strings.Contains(q, "drafted from it by someone else") { + confirmQ = q + } else { + draftQ = q + } + } + if n != 2 || draftQ == "" || confirmQ == "" { + t.Fatalf("0240 was not drafted and confirmed by two separate judgements: %d questions", n) + } + section, _ := decisionSection(r) + if !strings.Contains(confirmQ, section) || !strings.Contains(confirmQ, r.Clauses[0].Text) { + t.Fatal("the confirming judgement was not shown the record and the drafts") + } + // It sees only those: nothing of the mesh but the record, and none of the first judgement's own words. + for _, other := range []string{"Write what it decides as numbered", "An earlier draft"} { + if strings.Contains(confirmQ, other) { + t.Fatalf("the confirming judgement was shown %q", other) + } + } + if !strings.Contains(draftQ, section) || strings.Contains(draftQ, "The drafted clauses") { + t.Fatal("the drafting judgement's question is not the section alone") + } + for _, c := range r.Clauses { + if c.Source != SourceConfirmed { + t.Fatalf("not confirmed: %+v", c) + } + } + if len(w.said.of(EventDrafted)) == 0 || len(w.said.of(EventConfirmed)) == 0 { + t.Fatalf("events: %v", w.said.events) + } + // Now confirmed, holds serves them, marked. + h, _ := w.k.Holds("", &k, 0) + if len(h.Clauses) != len(r.Clauses) || !strings.Contains(h.Clauses[0].Holds, "confirmed") { + t.Fatalf("holds after confirming: %+v", h) + } + }) +} + +func TestAClauseTheSecondJudgementRejectsStaysDraftedWithTheReason(t *testing.T) { + w := imported(t, newMemStore()) + k := Key{KindDecision, 240} + r := draftUntil(t, w, k, DraftDrafted) + ctx := context.Background() + e := mustGet(t, w.k.Store, k) + r = e.Current.Clone() + r.Clauses[0].Text += " NOT EXACT" + if err := w.k.Store.Save(ctx, Version{By: "test", Why: "a draft the checker will reject", Record: r}, true, false, e.Revision); err != nil { + t.Fatal(err) + } + r = draftUntil(t, w, k, DraftDone) + if r.Clauses[0].Source != SourceDrafted || r.Clauses[0].Unconfirmed == "" { + t.Fatalf("a rejected clause: %+v", r.Clauses[0]) + } + h, _ := w.k.Holds("", &k, 0) + for _, c := range h.Clauses { + if c.N == 1 { + t.Fatal("a clause the second judgement rejected is served") + } + } +} + +func TestTheOperatorIsToldOncePerTopicAndReturnDraftsAgain(t *testing.T) { + w := imported(t, newMemStore()) + for i := 0; i < 40; i++ { + said, err := w.k.DraftStep() + if err != nil { + t.Fatal(err) + } + if said == "" { + break + } + } + // Every prose record of the fixture's topics drafted and confirmed; each topic told once. + topics := map[string]int{} + for _, s := range w.told { + topics[s]++ + } + if len(w.told) == 0 { + t.Fatal("the operator was told of no topic") + } + for s, n := range topics { + if n != 1 { + t.Fatalf("told %d times: %s", n, s) + } + } + told := len(w.told) + k := Key{KindDecision, 240} + if _, err := w.k.Return("operator@shanks", Key{KindDecision, 276}, "x"); err == nil { + t.Fatal("a record whose author wrote its clauses was returned") + } + if _, err := w.k.Return("operator@shanks", k, "clause 2 says more than the section"); err != nil { + t.Fatal(err) + } + r := mustGet(t, w.k.Store, k).Current + if len(r.Clauses) != 0 || r.Drafting.State != DraftWaiting || len(w.said.of(EventReturned)) != 1 { + t.Fatalf("returned: %+v", r) + } + r = draftUntil(t, w, k, DraftDone) + last := "" + for _, q := range w.judge.questions { + if strings.Contains(q, "An earlier draft was sent back") { + last = q + } + } + if !strings.Contains(last, "clause 2 says more than the section") { + t.Fatal("the drafting after a return did not read its reason") + } + if len(w.told) != told { + t.Fatal("the operator was told of the topic again") + } +} + +// A supersedes-in-part hq wrote names the record; which clause it changes is judged, then confirmed, and only then +// is the older clause changed. +func TestWhichClauseAPartialSupersessionChangesIsJudgedThenConfirmed(t *testing.T) { + w := imported(t, newMemStore()) + for i := 0; i < 40; i++ { + if said, err := w.k.DraftStep(); err != nil { + t.Fatal(err) + } else if said == "" { + break + } + } + r := mustGet(t, w.k.Store, Key{KindDecision, 282}).Current + var l *Link + for i := range r.Links { + if r.Links[i].Kind == LinkChanges && r.Links[i].To.Number == 276 { + l = &r.Links[i] + } + } + if l == nil || l.Source != SourceConfirmed || l.To.Clause != 1 { + t.Fatalf("0282's change of 0276: %+v", l) + } + c := mustGet(t, w.k.Store, Key{KindDecision, 276}).Current.Clause(1) + if c.Status != ClauseChanged || c.ChangedBy[0] != "ADR 0282" { + t.Fatalf("0276 §1: %+v", c) + } +} + +func TestADraftingThatFailsIsSaidAndTriedAgainLater(t *testing.T) { + w := imported(t, newMemStore()) + w.judge.fail = true + if _, err := w.k.DraftStep(); err == nil { + t.Fatal("a failed judgement was not said") + } + r := mustGet(t, w.k.Store, Key{KindDecision, 11}).Current + if r.Drafting.State != DraftFailed || r.Drafting.Tries != 1 || !strings.Contains(r.Drafting.Reason, "licence") { + t.Fatalf("%+v", r.Drafting) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/history_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/history_test.go new file mode 100644 index 00000000..f17a0bf0 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/history_test.go @@ -0,0 +1,151 @@ +package main + +import ( + "fmt" + "reflect" + "strings" + "testing" +) + +// to-be 55 §10: "Every version is kept and comparable — the module's test: three versions, history lists them, +// the difference of the first and third is exact." +func TestThreeVersionsAreKeptAndTheFirstAndThirdCompareExactly(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + n := w.reserved(t, "author@g14") // version 1: the reservation + p := valid(n) + if _, err := w.k.Propose("author@g14", p); err != nil { // version 2 + t.Fatal(err) + } + p.Clauses[1] = "**A batch is delivered by one walk.** At most one commit per repository." + p.Why = "the bound named" + if _, err := w.k.Propose("author@g14", p); err != nil { // version 3 + t.Fatal(err) + } + h, err := w.k.History(Key{KindDecision, n}, 0, 0, 0) + if err != nil { + t.Fatal(err) + } + if len(h.Versions) != 3 || h.Versions[0].Status != StatusReserved || h.Versions[1].Why != "proposed" || + h.Versions[2].Why != "the bound named" || !h.Versions[2].Served || h.Versions[0].By != "author@g14" { + t.Fatalf("history: %+v", h.Versions) + } + d, err := w.k.History(Key{KindDecision, n}, 1, 3, 0) + if err != nil { + t.Fatal(err) + } + vs, _ := s.Versions(t.Context(), Key{KindDecision, n}) + // Exact: applying the difference to the first version's text gives the third's, line for line. + if got := applyDiff(Render(vs[0].Record), d.Diff); got != Render(vs[2].Record) { + t.Fatalf("the difference does not turn version 1 into version 3:\n%s\n---\n%s", got, Render(vs[2].Record)) + } + // And it is the difference: what changed between the reservation and the revised proposal, nothing else. + want := []string{"-status: reserved", "+status: proposed", "+topic: the mesh", "+date: 2026-10-11"} + for _, line := range want { + if !containsLine(d.Diff, line) { + t.Errorf("the difference lacks %q: %v", line, d.Diff) + } + } + if containsLine(d.Diff, "-# "+strings.TrimPrefix(itoaKey(n), "ADR ")+". a test record") == false { + t.Errorf("the title's change is not in the difference: %v", d.Diff) + } + // Versions 2 and 3 differ in clause 2 alone. + d23, _ := w.k.History(Key{KindDecision, n}, 2, 3, 0) + var changed []string + for _, l := range d23.Diff { + if !strings.HasPrefix(l, "@@") { + changed = append(changed, l) + } + } + if !reflect.DeepEqual(changed, []string{"-2. **A batch is delivered by one walk.** One commit per repository.", + "+2. **A batch is delivered by one walk.** At most one commit per repository."}) { + t.Fatalf("2 to 3: %v", d23.Diff) + } + if d22, _ := w.k.History(Key{KindDecision, n}, 2, 2, 0); !d22.Same || len(d22.Diff) != 0 { + t.Fatalf("a version against itself: %+v", d22) + } + }) +} + +func containsLine(lines []string, s string) bool { + for _, l := range lines { + if l == s { + return true + } + } + return false +} + +// applyDiff applies a difference as Diff writes it to the text it was taken from. +func applyDiff(a string, diff []string) string { + src := strings.Split(a, "\n") + var out []string + at := 0 + for i := 0; i < len(diff); i++ { + var oa, ca, ob, cb int + if _, err := fmt.Sscanf(diff[i], "@@ -%d,%d +%d,%d @@", &oa, &ca, &ob, &cb); err != nil { + continue + } + for at < oa-1 { + out = append(out, src[at]) + at++ + } + for i+1 < len(diff) && !strings.HasPrefix(diff[i+1], "@@") { + i++ + if strings.HasPrefix(diff[i], "-") { + at++ + } else { + out = append(out, diff[i][1:]) + } + } + } + out = append(out, src[at:]...) + return strings.Join(out, "\n") +} + +func TestDiffIsExactOnSmallTexts(t *testing.T) { + for _, c := range [][2]string{{"a\nb\nc", "a\nc"}, {"a", "b"}, {"", "x\ny"}, {"x\ny", ""}, {"a\nb\nc\nd", "b\nc\nd\ne"}} { + if got := applyDiff(c[0], Diff(c[0], c[1])); got != c[1] { + t.Errorf("%q to %q: %q (%v)", c[0], c[1], got, Diff(c[0], c[1])) + } + } +} + +// history is bounded like the other answers: its work, and its answer, with a cut notice (the review of #206). +func TestHistoryIsBounded(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@g14") + p := valid(n) + big := strings.Repeat("A line of reasoning that is long enough to count.\n", 3000) + p.Sections[0].Text = big + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatal(err) + } + p.Sections[0].Text = strings.ReplaceAll(big, "long enough", "longer than") + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatal(err) + } + h, err := w.k.History(Key{KindDecision, n}, 2, 3, 0) + if err != nil { + t.Fatal(err) + } + if size(h) > DefaultBound || h.Cut == "" { + t.Fatalf("a large difference: %d bytes, cut %q", size(h), h.Cut) + } + // The difference stays exact when it is not cut, however large: its work is bounded, not its truth. + whole, _ := w.k.History(Key{KindDecision, n}, 2, 3, MostBound) + vs, _ := w.k.Store.Versions(t.Context(), Key{KindDecision, n}) + if whole.Cut == "" && applyDiff(Render(vs[1].Record), whole.Diff) != Render(vs[2].Record) { + t.Fatal("the bounded difference is not exact") + } + if d := Diff(strings.Repeat("a\n", 5000)+"x", strings.Repeat("b\n", 5000)+"y"); applyDiff(strings.Repeat("a\n", 5000)+"x", d) != strings.Repeat("b\n", 5000)+"y" { + t.Fatal("a difference past the work bound is not exact") + } +} + +func TestAQuestionIsBounded(t *testing.T) { + w := imported(t, newMemStore()) + if _, err := w.k.Ask(strings.Repeat("merge window ", 200), 0); err == nil || !strings.Contains(err.Error(), "at most") { + t.Fatalf("a question of 2600 characters: %v", err) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/hq.go b/modules/mesh-knowledge/cmd/mesh-knowledge/hq.go new file mode 100644 index 00000000..fe0d8916 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/hq.go @@ -0,0 +1,128 @@ +package main + +// hq as the knowledge base reads it: through the records module over the bus (to-be 55 §1: "it holds no +// checkout of this repository; the import reads it through the records module, as the issue tracker's import +// does"), or a checkout on disk for the terminal's dry run and the tests. + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +// Records is hq as the records module serves it: where its checkout stands, what a folder holds, and one +// document whole with the commit it was read at. +type Records interface { + Standing() (string, error) + List(folder string) (folders, documents []string, err error) + Read(path string) (content, commit string, err error) +} + +// busRecords is Records over the bus: the records module's tools, called as this module. +type busRecords struct{} + +func (busRecords) Standing() (string, error) { + raw, err := stdio.Ask("records.records_status", map[string]any{}) + if err != nil { + return "", err + } + var s struct { + Commit string `json:"commit"` + LastError string `json:"lastError"` + } + if err := json.Unmarshal(raw, &s); err != nil { + return "", fmt.Errorf("records_status answered something else: %v", err) + } + if s.Commit == "" { + return "", fmt.Errorf("the record's checkout stands at no commit yet (%s)", s.LastError) + } + return s.Commit, nil +} + +func (busRecords) List(folder string) ([]string, []string, error) { + raw, err := stdio.Ask("records.records_list", map[string]any{"folder": folder}) + if err != nil { + return nil, nil, err + } + var f struct { + Folders []string `json:"folders"` + Documents []string `json:"documents"` + } + if err := json.Unmarshal(raw, &f); err != nil { + return nil, nil, fmt.Errorf("records_list answered something else: %v", err) + } + return f.Folders, f.Documents, nil +} + +func (busRecords) Read(path string) (string, string, error) { + raw, err := stdio.Ask("records.records_read", map[string]any{"path": path}) + if err != nil { + return "", "", err + } + var d struct { + Content string `json:"content"` + Commit string `json:"commit"` + Truncated bool `json:"truncated"` + } + if err := json.Unmarshal(raw, &d); err != nil { + return "", "", fmt.Errorf("records_read answered something else: %v", err) + } + if d.Truncated { + return "", "", fmt.Errorf("%s is longer than the records module answers whole", path) + } + return d.Content, d.Commit, nil +} + +// dirRecords is Records over a checkout on disk. Its commit is the file COMMIT at its root when there is one (a +// fixture copied from hq at a commit), else the checkout's HEAD, else `dir:` and the directory. +type dirRecords struct{ dir string } + +func (d dirRecords) Standing() (string, error) { + if st, err := os.Stat(d.dir); err != nil || !st.IsDir() { + return "", fmt.Errorf("%s is not a checkout", d.dir) + } + if raw, err := os.ReadFile(filepath.Join(d.dir, "COMMIT")); err == nil { + return strings.TrimSpace(string(raw)), nil + } + if out, err := os.ReadFile(filepath.Join(d.dir, ".git", "HEAD")); err == nil { + ref := strings.TrimSpace(string(out)) + if strings.HasPrefix(ref, "ref: ") { + if sha, err := os.ReadFile(filepath.Join(d.dir, ".git", strings.TrimPrefix(ref, "ref: "))); err == nil { + return strings.TrimSpace(string(sha)), nil + } + } else if ref != "" { + return ref, nil + } + } + return "dir:" + d.dir, nil +} + +func (d dirRecords) List(folder string) ([]string, []string, error) { + entries, err := os.ReadDir(filepath.Join(d.dir, filepath.FromSlash(folder))) + if err != nil { + return nil, nil, err + } + var folders, docs []string + for _, e := range entries { + switch { + case e.IsDir(): + folders = append(folders, e.Name()) + case strings.HasSuffix(e.Name(), ".md"): + docs = append(docs, e.Name()) + } + } + return folders, docs, nil +} + +func (d dirRecords) Read(path string) (string, string, error) { + raw, err := os.ReadFile(filepath.Join(d.dir, filepath.FromSlash(path))) + if err != nil { + return "", "", err + } + commit, _ := d.Standing() + return string(raw), commit, nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/import.go b/modules/mesh-knowledge/cmd/mesh-knowledge/import.go new file mode 100644 index 00000000..10973c07 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/import.go @@ -0,0 +1,289 @@ +package main + +// The import (novox/hq ADR 0290 rule 8, to-be 55 §8): every decision record and research effort of hq, read at +// one commit through the records module, written whole at its own number as its first version, naming the commit +// and path it came from. It creates what is absent and never overwrites; it never asks the issue tracker for a +// number, so no counter moves; a second run changes nothing and says so. A dry run reads everything, writes +// nothing, and says what it would do. A record whose Decision section is prose is written with no clauses and +// waits for the drafting and confirming of its clauses (draft.go). + +import ( + "context" + "errors" + "fmt" + "regexp" + "sort" + "strings" + "sync" +) + +// The folders of hq the import reads. +const ( + DecisionsFolder = "02-DECISIONS" + ResearchFolder = "01-RESEARCH" +) + +// LinkPending is the source of a `changes` link the import read from hq's supersedes-in-part, which named the +// record and not the clause: which clause it changes is drafted and confirmed by the judgement (to-be 55 §7). +const LinkPending = "pending" + +var commitPrefix = regexp.MustCompile(`^[0-9a-f]{7,40}$`) + +// ImportAsk is what an import takes. +type ImportAsk struct { + Commit string + DryRun bool +} + +// ImportRow is what the import did, or would do, with one record. +type ImportRow struct { + Record string `json:"record"` + Action string `json:"action"` + Status string `json:"status,omitempty"` + From string `json:"from"` + Why string `json:"why,omitempty"` +} + +// The actions of an import. +const ( + ImportCreate = "create" + ImportHeld = "held" + ImportRefused = "refused" + ImportWould = "would-create" +) + +// Imported is the answer to an import. +type Imported struct { + DryRun bool `json:"dry-run"` + Commit string `json:"commit"` + Read int `json:"read"` + Created int `json:"created"` + Held int `json:"held"` + Refused int `json:"refused"` + Clauses int `json:"written-clauses"` + ToDraft int `json:"prose-decisions"` + Rows []ImportRow `json:"rows"` + Odd []string `json:"odd,omitempty"` + Cut string `json:"cut,omitempty"` + Said string `json:"said"` + Findings []Finding `json:"whole-set-findings,omitempty"` + Unresolved int `json:"unresolved-links"` +} + +// MostImportRows is how many rows an import answers; the counts are always whole. +const MostImportRows = 60 + +const importReaders = 8 + +type importCandidate struct { + path string + parsed *ParsedRecord + err error +} + +// Import reads hq at its commit and writes every record it does not hold. +func (k *KB) Import(caller string, a ImportAsk, records Records) (*Imported, error) { + commit, err := records.Standing() + if err != nil { + return nil, fmt.Errorf("hq cannot be read through the records module (%v); nothing was imported", err) + } + want := strings.TrimSpace(a.Commit) + if want != "" && !commitPrefix.MatchString(want) { + return nil, errors.New("commit is a commit's id or a prefix of it, 7 to 40 hexadecimal digits; nothing was imported") + } + if want != "" && !strings.HasPrefix(commit, want) { + return nil, fmt.Errorf("the record stands at hq commit %s, not %s: bring it there (records_sync), or name the commit it "+ + "stands at. Nothing was imported", short(commit), want) + } + _, docs, err := records.List(DecisionsFolder) + if err != nil { + return nil, fmt.Errorf("hq's %s cannot be listed (%v); nothing was imported", DecisionsFolder, err) + } + efforts, _, err := records.List(ResearchFolder) + if err != nil { + return nil, fmt.Errorf("hq's %s cannot be listed (%v); nothing was imported", ResearchFolder, err) + } + var cands []*importCandidate + for _, d := range docs { + if adrFile.MatchString(d) { + cands = append(cands, &importCandidate{path: DecisionsFolder + "/" + d}) + } + } + for _, f := range efforts { + if regexp.MustCompile(`^\d{3}-`).MatchString(f) { + cands = append(cands, &importCandidate{path: ResearchFolder + "/" + f}) + } + } + // Read and parse, a few at a time, every document at the one commit. + var moved error + var movedMu sync.Mutex + read := func(path string) (string, error) { + text, at, err := records.Read(path) + if err != nil { + return "", err + } + if at != commit { + movedMu.Lock() + moved = fmt.Errorf("hq moved while it was read: it stood at %s and %s was read at %s. Run the import again", short(commit), path, short(at)) + movedMu.Unlock() + } + return text, nil + } + var wg sync.WaitGroup + sem := make(chan struct{}, importReaders) + for _, c := range cands { + wg.Add(1) + sem <- struct{}{} + go func(c *importCandidate) { + defer wg.Done() + defer func() { <-sem }() + if strings.HasPrefix(c.path, DecisionsFolder) { + text, err := read(c.path) + if err != nil { + c.err = fmt.Errorf("could not be read: %v", err) + return + } + c.parsed, c.err = ParseDecision(c.path, text) + return + } + _, names, err := records.List(c.path) + if err != nil { + c.err = fmt.Errorf("could not be listed: %v", err) + return + } + texts := map[string]string{} + for _, n := range names { + text, err := read(c.path + "/" + n) + if err != nil { + c.err = fmt.Errorf("%s could not be read: %v", n, err) + return + } + texts[n] = text + } + c.parsed, c.err = ParseResearch(docName(c.path), texts) + }(c) + } + wg.Wait() + if moved != nil { + return nil, fmt.Errorf("%v; nothing was imported", moved) + } + + out := &Imported{DryRun: a.DryRun, Commit: commit, Rows: []ImportRow{}} + byKey := map[Key][]*importCandidate{} + for _, c := range cands { + if c.parsed != nil { + byKey[c.parsed.Record.Key()] = append(byKey[c.parsed.Record.Key()], c) + } + } + for key, cs := range byKey { + if len(cs) > 1 { + paths := make([]string, len(cs)) + for i, c := range cs { + paths[i] = c.path + } + for _, c := range cs { + c.err = fmt.Errorf("%s is the number of %s: a number names one record", key, strings.Join(paths, " and ")) + } + } + } + sort.Slice(cands, func(i, j int) bool { return cands[i].path < cands[j].path }) + + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + known := map[Key]bool{} + for key := range byKey { + known[key] = true + } + all, err := k.Store.All(ctx) + if err != nil { + return nil, fmt.Errorf("the knowledge base cannot be read (%v); nothing was imported", err) + } + held := map[Key]bool{} + for _, e := range all { + held[e.Current.Key()] = true + known[e.Current.Key()] = true + } + at := k.now() + var rows []ImportRow + for _, c := range cands { + row := ImportRow{From: c.path} + if c.err != nil { + row.Action, row.Why = ImportRefused, c.err.Error() + row.Record = c.path + out.Refused++ + rows = append(rows, row) + continue + } + r := c.parsed.Record + row.Record, row.Status = r.Key().String(), r.Status + out.Read++ + out.Odd = append(out.Odd, c.parsed.Odd...) + if held[r.Key()] { + row.Action = ImportHeld + out.Held++ + continue + } + for i, l := range r.Links { + if to, ok := l.To.Record(); ok && !known[to] { + r.Links[i].Unresolved = true + out.Unresolved++ + out.Odd = append(out.Odd, fmt.Sprintf("%s %s %s, which hq does not hold at this commit", r.Key(), l.Kind, to)) + } + if l.Kind == LinkChanges && l.To.Clause == 0 { + r.Links[i].Source = LinkPending + } + } + r.Source = "hq@" + commit + ":" + c.path + out.Clauses += len(r.Clauses) + if r.Drafting != nil { + out.ToDraft++ + } + row.Action = ImportWould + if !a.DryRun { + err := k.Store.Create(ctx, Version{By: caller, At: at, Why: "imported from hq at " + short(commit) + ": " + c.path, Record: r}) + switch { + case errors.Is(err, ErrExists): + row.Action = ImportHeld + out.Held++ + continue + case err != nil: + row.Action, row.Why = ImportRefused, err.Error() + out.Refused++ + rows = append(rows, row) + continue + } + row.Action = ImportCreate + k.emit(EventImported, r, caller, "imported from hq at "+short(commit), 1) + } + out.Created++ + rows = append(rows, row) + } + if len(rows) > MostImportRows { + out.Cut = fmt.Sprintf("%d more row(s) are not shown; the counts are whole", len(rows)-MostImportRows) + rows = rows[:MostImportRows] + } + out.Rows = append(out.Rows, rows...) + if len(out.Odd) > 40 { + out.Odd = append(out.Odd[:40], fmt.Sprintf("and %d more", len(out.Odd)-40)) + } + verb := "created" + if a.DryRun { + verb = "would create" + } + out.Said = fmt.Sprintf("Read %d records of hq at commit %s: %s %d, already held %d, refused %d; %d clauses written as such, "+ + "%d decisions written as prose wait for their clauses to be drafted and confirmed.", + out.Read, short(commit), verb, out.Created, out.Held, out.Refused, out.Clauses, out.ToDraft) + switch { + case a.DryRun: + out.Said += " Nothing was written: a dry run." + case out.Created == 0 && out.Refused == 0: + out.Said += " Nothing changed: every record is already held." + case out.Created == 0: + out.Said += " Nothing changed: every record not refused is already held." + } + if !a.DryRun && out.Created > 0 { + out.Findings = k.judgeSet(ctx) + } + return out, nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/import_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/import_test.go new file mode 100644 index 00000000..9214e138 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/import_test.go @@ -0,0 +1,182 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" +) + +// to-be 55 §10: "The import changes nothing on a second run, and loses nothing — the module's test over a fixture +// repository." The fixture is real hq records copied at the commit testdata/hq/COMMIT names. + +func TestTheImportWritesEveryRecordWholeAndASecondRunChangesNothing(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := newWorld(t, s, fixture()) + ctx := context.Background() + dry, err := w.k.Import("import@test", ImportAsk{DryRun: true}, fixture()) + if err != nil { + t.Fatal(err) + } + if all, _ := s.All(ctx); len(all) != 0 || dry.Created != 11 || !strings.Contains(dry.Said, "a dry run") { + t.Fatalf("the dry run wrote %d records, or would create %d: %s", len(all), dry.Created, dry.Said) + } + first, err := w.k.Import("import@test", ImportAsk{}, fixture()) + if err != nil { + t.Fatal(err) + } + if first.Created != 11 || first.Refused != 0 || first.Held != 0 || first.ToDraft != 5 { + t.Fatalf("first run: %+v", first) + } + if len(w.said.of(EventImported)) != 11 { + t.Fatalf("%d imported events", len(w.said.of(EventImported))) + } + // Loses nothing: every file's record, whole. + commit := strings.TrimSpace(readFile(t, "testdata/hq/COMMIT")) + entries, _ := os.ReadDir("testdata/hq/02-DECISIONS") + for _, f := range entries { + if !adrFile.MatchString(f.Name()) { + continue + } + text := readFile(t, filepath.Join("testdata/hq/02-DECISIONS", f.Name())) + p, err := ParseDecision(f.Name(), text) + if err != nil { + t.Fatal(err) + } + e := mustGet(t, s, p.Record.Key()) + r := e.Current + if r.Title != p.Record.Title || r.Status != p.Record.Status || r.Topic != p.Record.Topic || r.Date != p.Record.Date || + len(r.Sections) != len(p.Record.Sections) || len(r.Clauses) != len(p.Record.Clauses) || e.Versions != 1 || + r.Source != "hq@"+commit+":02-DECISIONS/"+f.Name() { + t.Errorf("%s is not held whole: %+v", f.Name(), r) + } + for i, sec := range p.Record.Sections { + if r.Sections[i] != sec { + t.Errorf("%s's section %s differs", f.Name(), sec.Name) + } + } + // Every section of the file, read back in order, is the file's body after its heading. + if body := sectionsText(r); !strings.Contains(text, strings.TrimSpace(r.Sections[len(r.Sections)-1].Text)) || body == "" { + t.Errorf("%s's last section is not the file's", f.Name()) + } + } + // 0276 writes nine numbered clauses with a bold lead: written and in force. 0240 writes prose: none yet, + // waiting to be drafted. 0127 is superseded by 0131. + r276 := mustGet(t, s, Key{KindDecision, 276}).Current + if len(r276.Clauses) != 9 || r276.Clauses[0].Source != SourceWritten || r276.Clauses[0].Status != ClauseInForce || + !strings.HasPrefix(r276.Clauses[0].Text, "**A rolling merge window.**") { + t.Fatalf("0276's clauses: %+v", r276.Clauses) + } + if r := mustGet(t, s, Key{KindDecision, 240}).Current; len(r.Clauses) != 0 || r.Drafting == nil || r.Drafting.State != DraftWaiting { + t.Fatalf("0240: %+v", r.Drafting) + } + if r := mustGet(t, s, Key{KindDecision, 127}).Current; r.Status != StatusSuperseded { + t.Fatalf("0127: %s", r.Status) + } + // 0282 supersedes part of 0276, as hq wrote it: the link is held, its clause waiting to be judged. + r282 := mustGet(t, s, Key{KindDecision, 282}).Current + var changes *Link + for i, l := range r282.Links { + if l.Kind == LinkChanges && l.To.Number == 276 { + changes = &r282.Links[i] + } + } + if changes == nil || changes.Source != LinkPending || changes.To.Clause != 0 { + t.Fatalf("0282's change of 0276: %+v", r282.Links) + } + in, _ := s.Inbound(ctx, Key{KindDecision, 276}) + if len(in) == 0 { + t.Fatal("0276 reads no link from its end") + } + // Research: 045 and 042, every document a section. + r45 := mustGet(t, s, Key{KindResearch, 45}).Current + if r45.Status != StatusGraduated || len(r45.Sections) != 2 || r45.Sections[0].Name != "00-overview.md" { + t.Fatalf("research 045: %s %v", r45.Status, r45.Sections) + } + // A second run changes nothing, and says so. + before := countVersions(t, s) + second, err := w.k.Import("import@test", ImportAsk{}, fixture()) + if err != nil { + t.Fatal(err) + } + if second.Created != 0 || second.Held != 11 || !strings.Contains(second.Said, "Nothing changed") || countVersions(t, s) != before { + t.Fatalf("second run: %+v", second) + } + if len(w.said.of(EventImported)) != 11 { + t.Fatal("the second run said an import") + } + }) +} + +// The import never overwrites: a record changed here keeps its change through a later import. +func TestTheImportNeverOverwrites(t *testing.T) { + w := imported(t, newMemStore()) + ctx := context.Background() + e := mustGet(t, w.k.Store, Key{KindDecision, 11}) + r := e.Current.Clone() + r.Title = "changed here" + if err := w.k.Store.Save(ctx, Version{By: "test", Why: "changed", Record: r}, true, false, e.Revision); err != nil { + t.Fatal(err) + } + if _, err := w.k.Import("import@test", ImportAsk{}, fixture()); err != nil { + t.Fatal(err) + } + if got := mustGet(t, w.k.Store, Key{KindDecision, 11}).Current.Title; got != "changed here" { + t.Fatalf("overwritten: %q", got) + } +} + +func TestTheImportRefusesACommitTheRecordDoesNotStandAt(t *testing.T) { + w := newWorld(t, newMemStore(), fixture()) + if _, err := w.k.Import("import@test", ImportAsk{Commit: "0000000"}, fixture()); err == nil || !strings.Contains(err.Error(), "not 0000000") { + t.Fatalf("%v", err) + } + commit := strings.TrimSpace(readFile(t, "testdata/hq/COMMIT")) + if out, err := w.k.Import("import@test", ImportAsk{Commit: commit[:9]}, fixture()); err != nil || out.Created != 11 { + t.Fatalf("%v %+v", err, out) + } + if _, err := w.k.Import("import@test", ImportAsk{Commit: "not-hex"}, fixture()); err == nil { + t.Fatal("a commit that is not one was taken") + } +} + +// The whole of main at a recorded commit is imported without a refusal, and leaves the whole set consistent. +func TestTheWholeOfMainImports(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := newWorld(t, s, hqMain(t)) + out, err := w.k.Import("import@test", ImportAsk{}, hqMain(t)) + if err != nil { + t.Fatal(err) + } + if out.Refused != 0 || out.Created < 300 { + t.Fatalf("%s; rows %+v", out.Said, out.Rows) + } + if len(out.Findings) != 0 { + t.Fatalf("the whole set of main fails: %+v", out.Findings) + } + t.Log(out.Said) + }) +} + +func readFile(t *testing.T, path string) string { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return string(raw) +} + +func countVersions(t *testing.T, s Store) int { + t.Helper() + all, err := s.All(context.Background()) + if err != nil { + t.Fatal(err) + } + n := 0 + for _, e := range all { + n += e.Versions + } + return n +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/kb.go b/modules/mesh-knowledge/cmd/mesh-knowledge/kb.go new file mode 100644 index 00000000..34b353f6 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/kb.go @@ -0,0 +1,609 @@ +package main + +// The knowledge base's writes (novox/hq ADR 0290 rules 4 and 5, to-be 55 §4, §5 and §6): a number reserved from +// the issue tracker, a record proposed and judged by the 29 rules, reviewed by someone who did not write it, and +// accepted — at once, or on the operator's proven warrant when it needs the operator — with every clause it +// changes or supersedes moved accordingly and the whole record set judged after. + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "regexp" + "strconv" + "strings" + "sync" + "time" +) + +// Unproven is who acted, as recorded, while the runtime does not tell a module who called it (hq issue 365). The +// caller is never taken from an argument. +const Unproven = "unproven" + +// proven says whether a caller is one the runtime proved. +func proven(caller string) bool { return caller != "" && caller != Unproven } + +// The events this module says (ADR 0290 rule 7): every change, with what changed and who changed it. +const ( + EventReserved = "reserved" + EventProposed = "proposed" + EventReviewed = "reviewed" + EventAccepted = "accepted" + EventChanged = "changed" + EventImported = "imported" + EventDrafted = "drafted" + EventConfirmed = "confirmed" + EventReturned = "returned" +) + +// Events are the events, in the order the manifest declares them. +var Events = []string{EventReserved, EventProposed, EventReviewed, EventAccepted, EventChanged, EventImported, + EventDrafted, EventConfirmed, EventReturned} + +// Settings are the operator's values for this module, merged by the mesh into one JSON file. +type Settings struct { + // JudgeModel is the model the drafting and confirming judgements ask; sonnet when empty. + JudgeModel string `json:"judge-model,omitempty"` + // Drafting is "off" to stop drafting and confirming clauses; anything else is on. + Drafting string `json:"drafting,omitempty"` + // ReservedFor is how long a reserved number waits for its proposal before it is withdrawn, as a duration + // such as 168h; DefaultReservedFor when empty. + ReservedFor string `json:"reserved-for,omitempty"` +} + +// DefaultReservedFor is how long a reservation waits for its proposal unless the settings say otherwise. +const DefaultReservedFor = 7 * 24 * time.Hour + +// Reserver asks the issue tracker for the next number of a kind (issue-tracker.reserve). +type Reserver func(kind, title string) (int, error) + +// Judgement asks the agent on the node one question with no tools (claude-code.claude_code_judge), and answers +// the structured answer and who judged. +type Judgement func(question string, schema map[string]any, model string) (json.RawMessage, string, error) + +// Notifier tells the operator's channel something (operator-channel.notify). +type Notifier func(key, summary, headline, explanation string) error + +// KB is the knowledge base. +type KB struct { + Store Store + Records Records + Reserve Reserver + Judge Judgement + Asks AskBus + Notify Notifier + Emit func(event string, body any) error + Settings func() (Settings, error) + Now func() time.Time + Logf func(format string, a ...any) + + // mu serialises the writes: one instance holds the seat, and a write reads, judges and writes as one. + mu sync.Mutex + + refMu sync.Mutex + ref *Reference + refAt time.Time +} + +func (k *KB) now() time.Time { + if k.Now != nil { + return k.Now().UTC() + } + return time.Now().UTC() +} + +func (k *KB) logf(format string, a ...any) { + if k.Logf != nil { + k.Logf(format, a...) + } +} + +func (k *KB) settings() Settings { + if k.Settings == nil { + return Settings{} + } + s, err := k.Settings() + if err != nil { + k.logf("[mesh-knowledge] the settings cannot be read (%v): the defaults hold", err) + } + return s +} + +// emit says a change. A change that could not be said is logged and kept: the store holds it whole, and its +// versions are the record of who changed what. +func (k *KB) emit(event string, r *Record, by, what string, version int) { + if k.Emit == nil { + return + } + body := map[string]any{"kind": r.Kind, "number": r.Number, "record": r.Key().String(), "by": by, "at": k.now(), + "what": what, "version": version, "status": r.Status} + if err := k.Emit(event, body); err != nil { + k.logf("[mesh-knowledge] the event %s for %s could not be said: %v", event, r.Key(), err) + } +} + +// reference is the reading order and the glossary, read again after ten minutes. +func (k *KB) reference() (*Reference, error) { + k.refMu.Lock() + defer k.refMu.Unlock() + if k.ref != nil && k.now().Sub(k.refAt) < 10*time.Minute { + return k.ref, nil + } + if k.Records == nil { + return nil, errors.New("hq cannot be read: the module has no way to the records module") + } + ref, err := LoadReference(k.Records) + if err != nil { + if k.ref != nil { + return k.ref, nil + } + return nil, err + } + k.ref, k.refAt = ref, k.now() + return ref, nil +} + +func (k *KB) lookup(ctx context.Context) Lookup { + return func(key Key) *Record { + e, err := k.Store.Get(ctx, key) + if err != nil || e == nil { + return nil + } + return e.Current + } +} + +var kindNumber = regexp.MustCompile(`^\d{1,4}$`) + +// ---- reserve ----------------------------------------------------------------------------------------------- + +// Reserved is the answer to reserve. +type Reserved struct { + Kind string `json:"kind"` + Number int `json:"number"` + Record string `json:"record"` + Says string `json:"says"` +} + +// ReserveNumber asks the issue tracker for the next number of a kind and holds it as reserved, with who reserved +// it and when (to-be 55 §6). The knowledge base keeps no counter. +func (k *KB) ReserveNumber(caller, kind, title string) (*Reserved, error) { + if !contains(Kinds, kind) { + return nil, fmt.Errorf("kind is decision or research, not %q; nothing was reserved", kind) + } + title = strings.TrimSpace(title) + if title == "" || strings.Contains(title, "\n") || len([]rune(title)) > 200 { + return nil, errors.New("for is the record's title, one line of at most 200 characters; nothing was reserved") + } + if k.Reserve == nil { + return nil, errors.New("the issue tracker cannot be asked for a number here; nothing was reserved") + } + n, err := k.Reserve(kind, title) + if err != nil { + return nil, fmt.Errorf("the issue tracker did not hand out a %s number (%v); nothing was reserved", kind, err) + } + k.mu.Lock() + defer k.mu.Unlock() + ctx := context.Background() + at := k.now() + r := &Record{Kind: kind, Number: n, Title: title, Status: StatusReserved, ReservedBy: caller, ReservedAt: at, ReservedFor: title} + err = k.Store.Create(ctx, Version{By: caller, At: at, Why: "number reserved from the issue tracker", Record: r}) + if errors.Is(err, ErrExists) { + return nil, fmt.Errorf("the issue tracker handed out %s, which the knowledge base already holds: the number is spent, "+ + "and its counter stands below what the knowledge base holds. Reserve again, and tell the operator", r.Key()) + } + if err != nil { + return nil, fmt.Errorf("%s was handed out and could not be kept (%v): the number is spent; reserve again", r.Key(), err) + } + k.emit(EventReserved, r, caller, "reserved for "+title, 1) + return &Reserved{Kind: kind, Number: n, Record: r.Key().String(), + Says: fmt.Sprintf("%s is reserved for %q. Propose the record under this number; a number never proposed is withdrawn, and never handed out again.", r.Key(), title)}, nil +} + +// ---- propose ----------------------------------------------------------------------------------------------- + +// LinkAsk is a link as a proposal gives it: its kind, its target as written ("0276 §4"), the clause it goes out +// from. +type LinkAsk struct { + Kind string `json:"kind"` + To string `json:"to"` + From int `json:"from,omitempty"` +} + +// ProposeAsk is a record proposed whole. +type ProposeAsk struct { + Kind string + Number int + Title string + Topic string + Date string + Deciders []string + Clauses []string + Sections []Section + Links []LinkAsk + Homes []string + NeedsOperator bool + // Status is a research effort's: active, graduated or abandoned. + Status string + // Insight, for an accepted record, is what stood before: the proposal is a progressive insight. + Insight string + Why string +} + +// Proposed is the answer to propose. +type Proposed struct { + Record string `json:"record"` + Version int `json:"version"` + Status string `json:"status"` + Says string `json:"says"` +} + +var numberedClause = regexp.MustCompile(`^(\d+)\.\s+`) + +// MostRecordBytes is the most one record may hold. +const MostRecordBytes = 512 << 10 + +// Propose writes a record whole as proposed, or refuses it naming every rule it breaks. A proposal is revised +// as new versions until it is reviewed; an accepted record is changed only by a progressive insight (Insight), +// which waits for its own review while the accepted version stays the one served. +func (k *KB) Propose(caller string, a ProposeAsk) (*Proposed, error) { + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + if !contains(Kinds, a.Kind) { + return nil, fmt.Errorf("kind is decision or research, not %q; nothing was proposed", a.Kind) + } + key := Key{a.Kind, a.Number} + held, err := k.Store.Get(ctx, key) + if err != nil { + return nil, fmt.Errorf("the knowledge base cannot be read (%v); nothing was proposed", err) + } + ref, err := k.reference() + if err != nil { + return nil, fmt.Errorf("the rules cannot be judged: %v. Nothing was proposed", err) + } + r := &Record{Kind: a.Kind, Number: a.Number, Title: strings.TrimSpace(a.Title), Topic: strings.TrimSpace(a.Topic), + Status: StatusProposed, Date: strings.TrimSpace(a.Date), Deciders: a.Deciders, Effort: a.Status, + NeedsOperator: a.NeedsOperator, Sections: a.Sections, Homes: a.Homes, Author: caller} + if r.Date == "" { + r.Date = k.now().Format("2006-01-02") + } + var refusals []Refusal + for i, text := range a.Clauses { + n := i + 1 + text = strings.TrimSpace(text) + if m := numberedClause.FindStringSubmatch(text); m != nil { + n, _ = strconv.Atoi(m[1]) + text = strings.TrimSpace(text[len(m[0]):]) + } + r.Clauses = append(r.Clauses, Clause{N: n, Text: text, Status: ClauseProposed, Source: SourceWritten}) + } + for _, l := range a.Links { + to, err := ParseRef(l.To) + if err != nil { + refusals = append(refusals, Refusal{Rule: "link-target", Says: err.Error()}) + continue + } + r.Links = append(r.Links, Link{Kind: l.Kind, From: l.From, To: to, Source: SourceWritten}) + } + current := true + why := strings.TrimSpace(a.Why) + if held != nil { + switch w := held.Working(); { + case held.Current.Status == StatusReserved: + r.ReservedBy, r.ReservedAt, r.ReservedFor = held.Current.ReservedBy, held.Current.ReservedAt, held.Current.ReservedFor + if proven(r.ReservedBy) && caller != r.ReservedBy { + return nil, fmt.Errorf("%s was reserved by %s: only who reserved a number proposes under it. Nothing was proposed", key, r.ReservedBy) + } + if why == "" { + why = "proposed" + } + case w.Status == StatusProposed && held.Pending == nil: + // A revision of a proposal: a new version, its earlier reviews kept with the versions they reviewed. + if !proven(caller) || caller != w.Author { + return nil, fmt.Errorf("%s is proposed by %s: only its author, proven by the runtime, revises it (this call is by %s). "+ + "Nothing was proposed", key, w.Author, caller) + } + r.Author, r.Reviews, r.ReservedBy, r.ReservedAt, r.ReservedFor = w.Author, w.Reviews, w.ReservedBy, w.ReservedAt, w.ReservedFor + r.Writers = w.Writers + if a.Insight != "" { + refusals = append(refusals, Refusal{Rule: "reserved", Says: "a proposal not yet accepted is revised, not corrected by an insight"}) + } + if why == "" { + why = "revised" + } + case held.Current.Accepted(): + if strings.TrimSpace(a.Insight) == "" { + return nil, fmt.Errorf("%s is %s: it changes only by a new record, or by a progressive insight, a correction of fact "+ + "that leaves the decision standing (give insight: what stood before). Nothing was proposed", key, held.Current.Status) + } + current = false + r.Author = caller + r.Writers = append([]string{}, held.Current.Writers...) + if held.Current.Author != "" && !contains(r.Writers, held.Current.Author) { + r.Writers = append(r.Writers, held.Current.Author) + } + r.AcceptedAt = held.Current.AcceptedAt + r.Pending = &PendingState{Insight: Insight{Date: k.now().Format("2006-01-02"), Before: strings.TrimSpace(a.Insight), By: caller}} + r.Date = held.Current.Date + r.Effort = a.Status + if why == "" { + why = "a progressive insight proposed" + } + default: + return nil, fmt.Errorf("%s is %s and takes no proposal; nothing was proposed", key, held.Working().Status) + } + } + if current && held != nil && !held.Current.ReservedAt.IsZero() && r.Date < held.Current.ReservedAt.Format("2006-01-02") { + // A new record is of its own day: a date before its reservation would put it under rules older records keep. + refusals = append(refusals, Refusal{Rule: "date", Says: fmt.Sprintf("a new record is dated from the day its number was reserved, %s, not %s", + held.Current.ReservedAt.Format("2006-01-02"), r.Date)}) + } + if !contains(r.Writers, caller) { + r.Writers = append(r.Writers, caller) + } + if !current && a.Kind == KindResearch && r.Effort == "" { + r.Effort = held.Current.Status + } + refusals = append(refusals, JudgeRecord(r, held, k.lookup(ctx), ref)...) + if raw, _ := json.Marshal(r); len(raw) > MostRecordBytes { + refusals = append(refusals, Refusal{Rule: "size", Says: fmt.Sprintf("the record is %d bytes; a record holds at most %d", len(raw), MostRecordBytes)}) + } + if len(refusals) > 0 { + lines := make([]string, len(refusals)) + for i, x := range refusals { + lines[i] = x.String() + } + return nil, fmt.Errorf("%s is refused by %d rule(s), and nothing was written:\n- %s", key, len(refusals), strings.Join(lines, "\n- ")) + } + v := Version{By: caller, At: k.now(), Why: why, Record: r, Insight: !current} + if err := k.Store.Save(ctx, v, current, false, held.Revision); err != nil { + return nil, fmt.Errorf("%s could not be written (%v); nothing was proposed", key, err) + } + n := held.Versions + 1 + k.emit(EventProposed, r, caller, why, n) + says := fmt.Sprintf("%s is proposed as version %d. Nothing of it is in force until a review by someone who did not write it approves it", key, n) + if r.NeedsOperator { + says += ", and then the operator's Approve" + } + if !current { + says = fmt.Sprintf("A progressive insight to %s is proposed as version %d; the accepted version is served until a review approves it", key, n) + } + return &Proposed{Record: key.String(), Version: n, Status: StatusProposed, Says: says + "."}, nil +} + +// ---- review and acceptance --------------------------------------------------------------------------------- + +// ReviewAsk is one review. +type ReviewAsk struct { + Kind string + Number int + Verdict string + Findings string +} + +// Reviewed is the answer to review. +type Reviewed struct { + Record string `json:"record"` + Verdict string `json:"verdict"` + Status string `json:"status"` + Says string `json:"says"` +} + +// Review takes a review of the version proposed, by someone who did not write it. On approve the record is +// accepted, or, when it needs the operator, asked of the operator; on changes it stays proposed, the findings +// kept with it. +func (k *KB) Review(caller string, a ReviewAsk) (*Reviewed, error) { + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + key := Key{a.Kind, a.Number} + if a.Verdict != VerdictApprove && a.Verdict != VerdictChanges { + return nil, fmt.Errorf("verdict is approve or changes, not %q; nothing was reviewed", a.Verdict) + } + if a.Verdict == VerdictChanges && strings.TrimSpace(a.Findings) == "" { + return nil, errors.New("a review that asks for changes says them in findings; nothing was reviewed") + } + held, err := k.Store.Get(ctx, key) + if err != nil { + return nil, fmt.Errorf("the knowledge base cannot be read (%v); nothing was reviewed", err) + } + if held == nil || held.Working().Status != StatusProposed { + status := "not held" + if held != nil { + status = held.Working().Status + } + return nil, fmt.Errorf("%s is %s: only a proposed record is reviewed", key, status) + } + if !proven(caller) { + return nil, fmt.Errorf("the runtime does not prove who calls (hq issue 365), so a review cannot be told from one by %s's "+ + "author, and none is taken until it does. Nothing was reviewed", key) + } + w := held.Working().Clone() + // The record's writers; an imported record's are hq's, reviewed there, and none of them is named here. + var writers []string + for _, by := range append([]string{w.Author}, w.Writers...) { + if by != "" { + writers = append(writers, by) + } + } + for _, by := range writers { + if !proven(by) { + return nil, fmt.Errorf("%s has a version written while the runtime proved no caller, so who wrote it cannot be told from "+ + "who reviews it: its author is to propose it again under a new number. Nothing was reviewed", key) + } + } + if contains(writers, caller) { + return nil, fmt.Errorf("%s has a version %s wrote, who called this review: a record is reviewed by someone who wrote none "+ + "of it. Nothing was reviewed", key, caller) + } + reviewed := held.CurrentVersion + if held.Pending != nil { + reviewed = held.PendingVersion + } + w.Reviews = append(w.Reviews, Review{By: caller, At: k.now(), Verdict: a.Verdict, Findings: strings.TrimSpace(a.Findings), Version: reviewed}) + if a.Verdict == VerdictChanges { + if err := k.Store.Save(ctx, Version{By: caller, At: k.now(), Why: "reviewed: changes asked", Record: w, Insight: held.Pending != nil}, + held.Pending == nil, false, held.Revision); err != nil { + return nil, fmt.Errorf("the review could not be kept (%v)", err) + } + k.emit(EventReviewed, w, caller, "changes asked", held.Versions+1) + return &Reviewed{Record: key.String(), Verdict: a.Verdict, Status: StatusProposed, + Says: fmt.Sprintf("%s stays proposed; the findings are kept with version %d for its author to answer with a revision.", key, reviewed)}, nil + } + if w.NeedsOperator { + return k.askOperator(ctx, held, w, caller) + } + k.emit(EventReviewed, w, caller, "approved", held.Versions+1) + r, err := k.accept(ctx, held, w, caller, "accepted after a review by "+caller, "") + if err != nil { + return nil, err + } + return &Reviewed{Record: key.String(), Verdict: a.Verdict, Status: r.Status, + Says: fmt.Sprintf("%s is %s and its clauses are in force.", key, r.Status)}, nil +} + +// accept makes a reviewed version the one served: accepted, its clauses in force, every clause and record it +// changes or supersedes moved; then the whole record set is judged. by is who acted; warrant, the operator's word +// it rests on. +func (k *KB) accept(ctx context.Context, held *Entry, w *Record, by, why, warrant string) (*Record, error) { + r := w.Clone() + insight := r.Pending != nil + at := k.now() + if insight { + in := r.Pending.Insight + r.Pending = nil + r.Insights = append(r.Insights, in) + r.Status = held.Current.Status + if r.Kind == KindResearch && contains(ResearchStatuses, r.Effort) && held.Current.Status != StatusSuperseded { + r.Status = r.Effort + } + for i, c := range r.Clauses { + r.Clauses[i].Status = ClauseInForce + if old := held.Current.Clause(c.N); old != nil && (old.Status == ClauseChanged || old.Status == ClauseSuperseded) { + r.Clauses[i].Status, r.Clauses[i].ChangedBy = old.Status, old.ChangedBy + } + } + } else { + r.Status = StatusAccepted + if r.Kind == KindResearch { + r.Status = r.Effort + } + r.AcceptedAt = at + for i := range r.Clauses { + r.Clauses[i].Status = ClauseInForce + } + } + r.Effort = "" + if warrant != "" { + why += "; " + warrant + } + if err := k.Store.Save(ctx, Version{By: by, At: at, Why: why, Record: r, Insight: insight}, true, true, held.Revision); err != nil { + return nil, fmt.Errorf("%s could not be accepted (%v); it stays proposed", r.Key(), err) + } + k.emit(EventAccepted, r, by, why, held.Versions+1) + // What it changes and supersedes, each a new version of that record. + for _, l := range r.Links { + to, ok := l.To.Record() + if !ok || (l.Kind != LinkSupersedes && l.Kind != LinkChanges) || (insight && hasLink(held.Current, l)) { + continue + } + t, err := k.Store.Get(ctx, to) + if err != nil || t == nil { + k.logf("[mesh-knowledge] %s %s %s, which cannot be read now (%v): the whole-set rules will say so", r.Key(), l.Kind, to, err) + continue + } + c := t.Current.Clone() + var what string + switch l.Kind { + case LinkSupersedes: + c.Status = StatusSuperseded + c.Extra = addExtra(c.Extra, "superseded-by", r.Key().String()) + for i := range c.Clauses { + c.Clauses[i].Status = ClauseSuperseded + c.Clauses[i].ChangedBy = append(c.Clauses[i].ChangedBy, r.Key().String()) + } + what = "superseded by " + r.Key().String() + case LinkChanges: + cl := c.Clause(l.To.Clause) + if cl == nil || cl.Status == ClauseSuperseded { + continue + } + cl.Status = ClauseChanged + from := r.Key().String() + if l.From > 0 { + from += " §" + strconv.Itoa(l.From) + } + cl.ChangedBy = append(cl.ChangedBy, from) + what = fmt.Sprintf("§%d changed by %s", l.To.Clause, from) + } + if err := k.Store.Save(ctx, Version{By: by, At: at, Why: what, Record: c}, true, false, t.Revision); err != nil { + k.logf("[mesh-knowledge] %s could not be marked %s (%v): the whole-set rules will say so", to, what, err) + continue + } + k.emit(EventChanged, c, by, what, t.Versions+1) + } + k.judgeSet(ctx) + return r, nil +} + +// judgeSet holds the whole record set to its 4 rules and keeps what fails, which the module's health answers. +func (k *KB) judgeSet(ctx context.Context) []Finding { + all, err := k.Store.All(ctx) + if err != nil { + k.logf("[mesh-knowledge] the whole record set cannot be read to judge it: %v", err) + return nil + } + records := make([]*Record, len(all)) + for i, e := range all { + records[i] = e.Current + } + findings := JudgeSet(records) + at := k.now() + for i := range findings { + findings[i].At = at + } + if err := k.Store.SetFindings(ctx, findings); err != nil { + k.logf("[mesh-knowledge] the whole-set findings could not be kept: %v", err) + } + for _, f := range findings { + k.logf("[mesh-knowledge] the whole-set rule %s fails: %s", f.Rule, f.Says) + } + return findings +} + +// ---- reservations that were never proposed ------------------------------------------------------------------- + +// Withdraw withdraws every number reserved longer ago than the settings allow and never proposed: it stays spent, +// with who reserved it and when. +func (k *KB) Withdraw() (int, error) { + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + bound := DefaultReservedFor + if s := strings.TrimSpace(k.settings().ReservedFor); s != "" { + if d, err := time.ParseDuration(s); err == nil && d > 0 { + bound = d + } + } + all, err := k.Store.All(ctx) + if err != nil { + return 0, err + } + n := 0 + for _, e := range all { + if e.Current.Status != StatusReserved || k.now().Sub(e.Current.ReservedAt) < bound { + continue + } + r := e.Current.Clone() + r.Status = StatusWithdrawn + why := fmt.Sprintf("withdrawn: reserved by %s at %s and never proposed", r.ReservedBy, r.ReservedAt.Format(time.RFC3339)) + if err := k.Store.Save(ctx, Version{By: "the knowledge base", At: k.now(), Why: why, Record: r}, true, false, e.Revision); err != nil { + return n, err + } + k.emit(EventChanged, r, "the knowledge base", why, e.Versions+1) + n++ + } + return n, nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/kit_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/kit_test.go new file mode 100644 index 00000000..98cea051 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/kit_test.go @@ -0,0 +1,430 @@ +package main + +// What the tests stand on: the knowledge base over each store — in memory always, and on a real postgres when one +// can be had (MESH_TEST_POSTGRES, or a private server started here from the machine's initdb on a socket of its own, +// listening on no port) — hq as real files copied from it at a commit (testdata/hq, and testdata/hq-main.tar.gz, +// the whole of main), and the issue tracker, the agent's judgement and the router played here. + +import ( + "archive/tar" + "compress/gzip" + "context" + "encoding/json" + "fmt" + "io" + "net/url" + "os" + "os/exec" + "path/filepath" + "regexp" + "strings" + "sync" + "testing" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" + "git.novox.be/novox/mesh-sdk/go/asks" +) + +const bundleVar = "MESH_KNOWLEDGE_TEST_BUNDLE" + +func TestMain(m *testing.M) { + if os.Getenv(bundleVar) == "1" { + // The bundle the bus test starts: the knowledge base as the runtime launches it, its store in memory. + k := newKB(newMemStore()) + serve(k, os.Getenv("MESH_KNOWLEDGE_STATUS")) + os.Exit(0) + } + code := m.Run() + stopPostgres() + os.Exit(code) +} + +// ---- postgres ------------------------------------------------------------------------------------------------ + +var ( + pgOnce sync.Once + pgBase string + pgDir string + pgWhy string + pgCount int + pgMu sync.Mutex +) + +func startPostgres() { + if url := os.Getenv("MESH_TEST_POSTGRES"); url != "" { + pgBase = url + return + } + initdb, err := exec.LookPath("initdb") + if err != nil { + pgWhy = "no postgres: MESH_TEST_POSTGRES is unset and the machine has no initdb" + return + } + pgctl := filepath.Join(filepath.Dir(initdb), "pg_ctl") + dir, err := os.MkdirTemp("", "mkpg") + if err != nil { + pgWhy = err.Error() + return + } + pgDir = dir + if out, err := exec.Command(initdb, "-D", dir+"/data", "-U", "postgres", "-A", "trust", "-E", "UTF8", "--no-locale").CombinedOutput(); err != nil { + pgWhy = fmt.Sprintf("initdb: %v: %s", err, out) + return + } + // A socket in its own directory and no port: nothing on this machine can reach it but these tests. + if out, err := exec.Command(pgctl, "-D", dir+"/data", "-l", dir+"/log", "-w", "-o", + "-k "+dir+" -c listen_addresses='' -F", "start").CombinedOutput(); err != nil { + pgWhy = fmt.Sprintf("pg_ctl start: %v: %s", err, out) + return + } + pgBase = "postgres://postgres@/postgres?host=" + dir +} + +func stopPostgres() { + if pgDir == "" { + return + } + if initdb, err := exec.LookPath("initdb"); err == nil { + _ = exec.Command(filepath.Join(filepath.Dir(initdb), "pg_ctl"), "-D", pgDir+"/data", "-m", "immediate", "stop").Run() + } + _ = os.RemoveAll(pgDir) +} + +// freshDatabase is a database of its own for one test, on the test postgres, or a skip saying why there is none. +func freshDatabase(t *testing.T) string { + t.Helper() + pgOnce.Do(startPostgres) + if pgBase == "" { + t.Skipf("NOT TESTED ON POSTGRES: %s", pgWhy) + } + pgMu.Lock() + pgCount++ + name := fmt.Sprintf("kb_%d_%d", os.Getpid(), pgCount) + pgMu.Unlock() + ctx := context.Background() + admin, err := OpenPgStore(ctx, pgBase) + if err != nil { + t.Fatal(err) + } + defer admin.Close() + if _, err := admin.pool.Exec(ctx, "create database "+name); err != nil { + t.Fatal(err) + } + return replaceDatabase(pgBase, name) +} + +func replaceDatabase(base, name string) string { + u, err := url.Parse(base) + if err != nil { + return base + } + u.Path = "/" + name + return u.String() +} + +// pgStore is a prepared store on a fresh database. +func pgStore(t *testing.T) *PgStore { + t.Helper() + url := freshDatabase(t) + s, err := OpenPgStore(context.Background(), url) + if err != nil { + t.Fatal(err) + } + t.Cleanup(s.Close) + if _, err := s.Prepare(context.Background()); err != nil { + t.Fatal(err) + } + return s +} + +// eachStore runs a test over the store in memory and over postgres. +func eachStore(t *testing.T, run func(t *testing.T, s Store)) { + t.Run("memory", func(t *testing.T) { run(t, newMemStore()) }) + t.Run("postgres", func(t *testing.T) { run(t, pgStore(t)) }) +} + +// ---- hq --------------------------------------------------------------------------------------------------------- + +// fixture is the handful of real hq files copied at one commit. +func fixture() dirRecords { return dirRecords{dir: filepath.Join("testdata", "hq")} } + +var ( + mainOnce sync.Once + mainDir string + mainErr error +) + +// hqMain is the whole of hq's main at the commit testdata/hq-main.commit names, unpacked once. +func hqMain(t *testing.T) dirRecords { + t.Helper() + mainOnce.Do(func() { + dir, err := os.MkdirTemp("", "kb-hq-main") + if err != nil { + mainErr = err + return + } + mainDir = dir + f, err := os.Open(filepath.Join("testdata", "hq-main.tar.gz")) + if err != nil { + mainErr = err + return + } + defer f.Close() + gz, err := gzip.NewReader(f) + if err != nil { + mainErr = err + return + } + tr := tar.NewReader(gz) + for { + h, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + mainErr = err + return + } + path := filepath.Join(dir, filepath.FromSlash(h.Name)) + if h.Typeflag == tar.TypeDir { + _ = os.MkdirAll(path, 0o755) + continue + } + _ = os.MkdirAll(filepath.Dir(path), 0o755) + raw, err := io.ReadAll(tr) + if err != nil { + mainErr = err + return + } + if err := os.WriteFile(path, raw, 0o644); err != nil { + mainErr = err + return + } + } + commit, _ := os.ReadFile(filepath.Join("testdata", "hq-main.commit")) + mainErr = os.WriteFile(filepath.Join(dir, "hq", "COMMIT"), commit, 0o644) + }) + if mainErr != nil { + t.Fatal(mainErr) + } + return dirRecords{dir: filepath.Join(mainDir, "hq")} +} + +// ---- what the knowledge base reaches, played here ------------------------------------------------------------ + +// tracker is the issue tracker's counters, played: the next number of a kind, from where hq's main stands. +type tracker struct { + mu sync.Mutex + next map[string]int + asked []map[string]string +} + +func newTracker() *tracker { + return &tracker{next: map[string]int{KindDecision: 300, KindResearch: 50}} +} + +func (tr *tracker) reserve(kind, title string) (int, error) { + tr.mu.Lock() + defer tr.mu.Unlock() + tr.asked = append(tr.asked, map[string]string{"kind": kind, "for": title}) + n := tr.next[kind] + tr.next[kind]++ + return n, nil +} + +// judge is the agent's judgement, played: it drafts a clause from each sentence of a section that says "is", "are" +// or "decid", confirms every clause but those whose text holds "NOT EXACT", and names clause 1 for a change. +type judge struct { + mu sync.Mutex + questions []string + fail bool +} + +func (j *judge) judge(question string, schema map[string]any, model string) (json.RawMessage, string, error) { + j.mu.Lock() + defer j.mu.Unlock() + j.questions = append(j.questions, question) + if j.fail { + return nil, "", fmt.Errorf("the agent holds no licence") + } + props := schema["properties"].(map[string]any) + switch { + case props["verdicts"] != nil: + var verdicts []map[string]any + for _, line := range strings.Split(question[strings.Index(question, "The drafted clauses:"):], "\n") { + m := regexp.MustCompile(`^(\d+)\. `).FindStringSubmatch(line) + if m == nil { + continue + } + exact := !strings.Contains(line, "NOT EXACT") + v := map[string]any{"clause": atoi(m[1]), "exact": exact} + if !exact { + v["reason"] = "the section says it of the controller only" + } + verdicts = append(verdicts, v) + } + raw, _ := json.Marshal(map[string]any{"verdicts": verdicts}) + return raw, "the agent on test (" + model + ")", nil + case props["clauses"] != nil && props["reason"] != nil: + return json.RawMessage(`{"clauses":[1],"reason":"it replaces the first"}`), "the agent on test (" + model + ")", nil + case props["exact"] != nil: + return json.RawMessage(`{"exact":true,"reason":"it does"}`), "the agent on test (" + model + ")", nil + } + section := question[strings.Index(question, "The Decision section:")+len("The Decision section:"):] + var clauses []map[string]string + for _, s := range regexp.MustCompile(`[^.]+\.`).FindAllString(strings.Join(strings.Fields(section), " "), -1) { + s = strings.TrimSpace(s) + if len(clauses) < 4 && len(s) > 20 && regexp.MustCompile(`(?i)\b(is|are|decid)`).MatchString(s) { + clauses = append(clauses, map[string]string{"lead": "Drafted " + fmt.Sprint(len(clauses)+1), "text": s}) + } + } + if len(clauses) == 0 { + clauses = append(clauses, map[string]string{"lead": "Drafted 1", "text": "what the section decides."}) + } + raw, _ := json.Marshal(map[string]any{"clauses": clauses}) + return raw, "the agent on test (" + model + ")", nil +} + +// router is the operator channel, played: the asks published, and its own record of each, which only it writes. +type router struct { + mu sync.Mutex + asked []asks.Ask + records map[string]routerRecord +} + +func newRouter() *router { return &router{records: map[string]routerRecord{}} } + +func (r *router) Publish(subject string, body any, id string) error { + r.mu.Lock() + defer r.mu.Unlock() + if subject != asks.AskSubject(Module) { + return fmt.Errorf("published on %s", subject) + } + a := body.(asks.Ask) + r.asked = append(r.asked, a) + cp := a + r.records[a.ID] = routerRecord{State: "open", Ask: &cp} + return nil +} + +func (r *router) Record(id string) (*stdio.StateEntry, error) { + r.mu.Lock() + defer r.mu.Unlock() + rec, ok := r.records[id] + if !ok { + return nil, nil + } + raw, _ := json.Marshal(rec) + return &stdio.StateEntry{Key: id, Value: raw, Revision: 1}, nil +} + +// answer closes an ask as the router does when the operator taps an option on a channel that proves who answered. +func (r *router) answer(id, option string, at time.Time) asks.Warrant { + r.mu.Lock() + defer r.mu.Unlock() + rec := r.records[id] + o, _ := rec.Ask.Option(option) + w := asks.Warrant{Ask: id, Asker: Module, About: rec.Ask.About, Outcome: asks.OutcomeChosen, Option: o.ID, Label: o.Label, + Level: o.Level, By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "1", Verified: "user id verified"}, + Channel: "telegram", At: at, AskDigest: rec.Ask.Digest()} + rec.State, rec.Warrant = "closed", &w + r.records[id] = rec + return w +} + +// said is the events said. +type said struct { + mu sync.Mutex + events []map[string]any +} + +func (s *said) emit(event string, body any) error { + s.mu.Lock() + defer s.mu.Unlock() + b := body.(map[string]any) + cp := map[string]any{"event": event} + for k, v := range b { + cp[k] = v + } + s.events = append(s.events, cp) + return nil +} + +func (s *said) of(event string) []map[string]any { + s.mu.Lock() + defer s.mu.Unlock() + var out []map[string]any + for _, e := range s.events { + if e["event"] == event { + out = append(out, e) + } + } + return out +} + +// world is one knowledge base and everything it reaches, played. +type world struct { + k *KB + tracker *tracker + judge *judge + router *router + said *said + told []string + clock time.Time +} + +func newWorld(t *testing.T, s Store, records Records) *world { + t.Helper() + w := &world{tracker: newTracker(), judge: &judge{}, router: newRouter(), said: &said{}, + clock: time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC)} + w.k = &KB{Store: s, Records: records, Reserve: w.tracker.reserve, Judge: w.judge.judge, Asks: w.router, + Emit: w.said.emit, Settings: func() (Settings, error) { return Settings{}, nil }, Now: func() time.Time { return w.clock }, + Logf: func(f string, a ...any) { t.Logf(f, a...) }, + Notify: func(key, summary, headline, explanation string) error { w.told = append(w.told, summary); return nil }} + return w +} + +// valid is a proposal every rule takes, for the record numbered n, against the fixture's records. +func valid(n int) ProposeAsk { + return ProposeAsk{Kind: KindDecision, Number: n, Title: "A merge window closes after ninety seconds of quiet", + Topic: "the mesh", Date: "2026-10-11", Deciders: []string{"jochen"}, + Clauses: []string{"**The window closes after ninety seconds of quiet.** A merge opens or extends it.", + "**A batch is delivered by one walk.** One commit per repository."}, + Sections: []Section{ + {Name: "Context", Text: "The merge window of [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md) waits too long."}, + {Name: "Considered Options", Text: "1. **Keep it.** Rejected."}, + {Name: "Consequences", Text: "Merges land sooner."}, + {Name: "How it is checked", Text: "The controller's merge-window tests."}, + }, + Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}}} +} + +// imported is a world whose knowledge base holds the fixture's records. +func imported(t *testing.T, s Store) *world { + t.Helper() + w := newWorld(t, s, fixture()) + if _, err := w.k.Import("import@test", ImportAsk{}, fixture()); err != nil { + t.Fatal(err) + } + return w +} + +// reserved reserves a decision number and answers it. +func (w *world) reserved(t *testing.T, caller string) int { + t.Helper() + r, err := w.k.ReserveNumber(caller, KindDecision, "a test record") + if err != nil { + t.Fatal(err) + } + return r.Number +} + +func mustGet(t *testing.T, s Store, k Key) *Entry { + t.Helper() + e, err := s.Get(context.Background(), k) + if err != nil || e == nil { + t.Fatalf("%s: %v %v", k, e, err) + } + return e +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/main.go b/modules/mesh-knowledge/cmd/mesh-knowledge/main.go new file mode 100644 index 00000000..9b6783f5 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/main.go @@ -0,0 +1,942 @@ +// mesh-knowledge: the knowledge base, holder of the knowledge-base seat (novox/hq ADR 0290, to-be 55). A Go +// bundle the control-node's runtime launches. It holds every decision record and research effort whole — its +// fields, its clauses each with a status and a source, its typed links read from both ends, its reasoning — with +// every version kept, in a database in the store; answers what holds in one bounded call; takes a record proposed, +// checked when it is written, reviewed by someone who did not write it and accepted, on the operator's proven +// word when it needs it; and imports hq's records at a commit. stdout is the MCP channel; what this module says, +// it says on stderr. +// +// mesh-knowledge the bundle: the seat's verbs and the health tool +// mesh-knowledge prepare the preparation step: the database's schema brought to this version's +// mesh-knowledge keep acts: judges the whole record set every few minutes and keeps the findings +// mesh-knowledge watch judges: reads whether the database serves and what the findings say, every +// minute, and leaves its verdict in the status file the health tool reads +// mesh-knowledge dry-run the import's dry run over a checkout on disk, against no database +// +// The two long-running processes are the act and the judge, as the mounts module's are: keep is judged by its +// unit, watch by the health tool, so the module never judges itself by its own tool alone (ADR 0240 rule 2). +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "math" + "os" + "sort" + "strings" + "sync" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" + "git.novox.be/novox/mesh-sdk/go/asks" +) + +// Seat is the role this module holds, once for the mesh. +const Seat = "knowledge-base" + +// Verbs are the seat's verbs, as the manifest declares and claims them (ADR 0290 rule 6). +var Verbs = []string{"ask", "get", "holds", "links", "history", "reserve", "propose", "review", "import", "return"} + +// HealthTool is the module's own tool the node-engine asks how it is. +const HealthTool = "knowledge_health" + +func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) } + +func readSettings(path string) (Settings, error) { + var s Settings + if path == "" { + return s, errors.New("the module was started without a settings file") + } + raw, err := os.ReadFile(path) + if err != nil { + return s, err + } + if err := json.Unmarshal(raw, &s); err != nil { + return s, fmt.Errorf("the settings file is not the module's settings: %v", err) + } + return s, nil +} + +func main() { + if len(os.Args) > 1 { + switch os.Args[1] { + case "prepare": + os.Exit(prepare()) + case "keep": + os.Exit(keep()) + case "watch": + os.Exit(watch(os.Getenv("MESH_KNOWLEDGE_STATUS"))) + case "dry-run": + os.Exit(dryRunAtTheTerminal(os.Args[2:])) + } + } + st := &lazyStore{open: func(ctx context.Context) (*PgStore, error) { return PgStoreFromEnv(ctx) }} + k := newKB(st) + serve(k, os.Getenv("MESH_KNOWLEDGE_STATUS")) +} + +// newKB is the knowledge base as the runtime launches it: everything it reaches, over the bus. +func newKB(store Store) *KB { + file := os.Getenv("MESH_KNOWLEDGE_SETTINGS") + return &KB{ + Store: store, + Records: busRecords{}, + Reserve: busReserver, + Judge: busJudgement(os.Getenv("MESH_NODE")), + Asks: sdkAskBus{}, + Notify: busNotifier, + Emit: func(event string, body any) error { return stdio.Emit(event, body) }, + Settings: func() (Settings, error) { return readSettings(file) }, + Logf: logf, + } +} + +// serve runs the bundle: the verbs, the operator's answers heard, and the background work. +func serve(k *KB, status string) { + go hear(k) + go background(k) + if err := stdio.Serve("", append(Tools(k, callerOf), healthTool(status, k.now))); err != nil { + logf("%v", err) + os.Exit(1) + } +} + +// callerOf is who called, as the runtime proves it: the one place the caller is read, so the change that brings it +// is this one function. The runtime does not hand a module its caller yet (hq issue 365): the SDK's tool call carries +// a name and arguments only, so the SDK must first pass the runtime's proven caller to a tool. Until it does, every +// caller is Unproven. It is never read from the call's arguments, so no caller can name another. +func callerOf(map[string]any) string { return Unproven } + +// hear subscribes to the router's word on this module's asks, asked again until the runtime takes it. +func hear(k *KB) { + for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) { + time.Sleep(wait) + err := stdio.Subscribe(asks.Seat+".decided."+Module, func(e stdio.Envelope) error { + var w asks.Warrant + if json.Unmarshal(e.Body, &w) == nil { + k.HeardEnd(w) + } + return nil + }) + if err == nil { + logf("[mesh-knowledge] hears the operator's answers") + return + } + logf("[mesh-knowledge] does not hear the operator's answers yet (%v); asking again in %s", err, wait) + } +} + +// background reads the router's record of every waiting ask once a minute, withdraws reservations never +// proposed, and drafts and confirms clauses one step at a time. +func background(k *KB) { + tick := time.NewTicker(time.Minute) + defer tick.Stop() + time.Sleep(5 * time.Second) + for { + if said, err := k.Reconcile(); err != nil { + logf("[mesh-knowledge] reading the operator's answers: %v", err) + } else { + for _, s := range said { + logf("[mesh-knowledge] %s", s) + } + } + if n, err := k.Withdraw(); err != nil { + logf("[mesh-knowledge] withdrawing reservations: %v", err) + } else if n > 0 { + logf("[mesh-knowledge] withdrew %d reservation(s) never proposed", n) + } + for { + said, err := k.DraftStep() + if err != nil { + logf("[mesh-knowledge] %v", err) + break + } + if said == "" { + break + } + logf("[mesh-knowledge] %s", said) + } + <-tick.C + } +} + +// ---- the bus ------------------------------------------------------------------------------------------------- + +// busReserver asks the issue tracker for a number (issue-tracker.reserve, hq issue 424). +func busReserver(kind, title string) (int, error) { + raw, err := stdio.Ask("seat:issue-tracker.reserve", map[string]any{"kind": kind, "for": title}) + if err != nil { + return 0, err + } + var out struct { + Kind string `json:"kind"` + Number int `json:"number"` + } + if err := json.Unmarshal(raw, &out); err != nil || out.Number < 1 || out.Kind != kind { + return 0, fmt.Errorf("the issue tracker's answer is not a %s number: %.200s", kind, raw) + } + return out.Number, nil +} + +// busJudgement asks the agent module on this node for one judgement (ADR 0278). +func busJudgement(node string) Judgement { + return func(question string, schema map[string]any, model string) (json.RawMessage, string, error) { + key := "claude-code.claude_code_judge" + if node != "" { + key += "@" + node + } + raw, err := stdio.Ask(key, map[string]any{"question": question, "schema": schema, "model": model}) + if err != nil { + return nil, "", err + } + var out struct { + Answer json.RawMessage `json:"answer"` + Model string `json:"model"` + Node string `json:"node"` + } + if err := json.Unmarshal(raw, &out); err != nil || len(out.Answer) == 0 { + return nil, "", fmt.Errorf("the agent module's answer is not a judgement: %.200s", raw) + } + return out.Answer, fmt.Sprintf("the agent on %s (%s)", out.Node, out.Model), nil + } +} + +// busNotifier tells the operator's channel, as a warning: it is the one severity below urgent. +func busNotifier(key, summary, headline, explanation string) error { + _, err := stdio.Ask("seat:operator-channel.notify", map[string]any{"key": key, "subject": "the knowledge base", + "severity": "warning", "summary": summary, "headline": headline, "explanation": explanation}) + return err +} + +// ---- the store, opened when it is first needed ------------------------------------------------------------ + +// lazyStore opens the database on the first call, and again after a failure, so a database that arrives late is +// not fatal; and refuses to serve a schema behind this version. +type lazyStore struct { + open func(context.Context) (*PgStore, error) + mu sync.Mutex + pg *PgStore +} + +func (l *lazyStore) store(ctx context.Context) (*PgStore, error) { + l.mu.Lock() + defer l.mu.Unlock() + if l.pg == nil { + pg, err := l.open(ctx) + if err != nil { + return nil, fmt.Errorf("the knowledge base's database cannot be opened: %w", err) + } + l.pg = pg + } + if err := l.pg.Ready(ctx); err != nil { + return nil, err + } + return l.pg, nil +} + +// Ready says why the store cannot serve, or nil. +func (l *lazyStore) Ready(ctx context.Context) error { _, err := l.store(ctx); return err } + +func (l *lazyStore) Get(ctx context.Context, k Key) (*Entry, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.Get(ctx, k) +} +func (l *lazyStore) All(ctx context.Context) ([]*Entry, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.All(ctx) +} +func (l *lazyStore) Create(ctx context.Context, v Version) error { + s, err := l.store(ctx) + if err != nil { + return err + } + return s.Create(ctx, v) +} +func (l *lazyStore) Save(ctx context.Context, v Version, current, clearPending bool, revision int64) error { + s, err := l.store(ctx) + if err != nil { + return err + } + return s.Save(ctx, v, current, clearPending, revision) +} +func (l *lazyStore) Versions(ctx context.Context, k Key) ([]Version, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.Versions(ctx, k) +} +func (l *lazyStore) Search(ctx context.Context, q string, limit int) ([]Hit, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.Search(ctx, q, limit) +} +func (l *lazyStore) Inbound(ctx context.Context, k Key) ([]Inbound, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.Inbound(ctx, k) +} +func (l *lazyStore) SetFindings(ctx context.Context, f []Finding) error { + s, err := l.store(ctx) + if err != nil { + return err + } + return s.SetFindings(ctx, f) +} +func (l *lazyStore) Findings(ctx context.Context) ([]Finding, error) { + s, err := l.store(ctx) + if err != nil { + return nil, err + } + return s.Findings(ctx) +} +func (l *lazyStore) Told(ctx context.Context, topic string) (bool, error) { + s, err := l.store(ctx) + if err != nil { + return false, err + } + return s.Told(ctx, topic) +} + +// ---- prepare, check, dry-run -------------------------------------------------------------------------------- + +func prepare() int { + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + s, err := PgStoreFromEnv(ctx) + if err != nil { + logf("[mesh-knowledge] preparing the database: %v", err) + return 1 + } + defer s.Close() + said, err := s.Prepare(ctx) + for _, line := range said { + logf("[mesh-knowledge] %s", line) + } + if err != nil { + logf("[mesh-knowledge] preparing the database failed: %v", err) + return 1 + } + return 0 +} + +// KeepEvery is how often keep judges the whole record set; WatchEvery how often watch looks. +const ( + KeepEvery = 5 * time.Minute + WatchEvery = time.Minute +) + +// keep judges the whole record set every few minutes and keeps the findings in the database: a change that +// reached the database by any way is judged within minutes, not only one an acceptance made. +func keep() int { + for { + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + s, err := PgStoreFromEnv(ctx) + if err == nil { + if err = s.Ready(ctx); err == nil { + k := &KB{Store: s, Logf: logf} + f := k.judgeSet(ctx) + logf("[mesh-knowledge] the whole record set judged: %d finding(s)", len(f)) + } + s.Close() + } + if err != nil { + logf("[mesh-knowledge] the whole record set cannot be judged now: %v", err) + } + cancel() + time.Sleep(KeepEvery) + } +} + +// watch reads, every minute, whether the database serves this version and what the whole-set findings say, +// and leaves its verdict in the status file. +func watch(status string) int { + if status == "" { + logf("[mesh-knowledge] watch was started without MESH_KNOWLEDGE_STATUS") + return 2 + } + for { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + var answer HealthAnswer + s, err := PgStoreFromEnv(ctx) + if err != nil { + answer = HealthAnswer{State: "unhealthy", Why: "the knowledge base's database cannot be opened: " + err.Error()} + } else { + answer = (&KB{Store: s}).Health(s) + s.Close() + } + cancel() + answer.At = time.Now().UTC() + if err := writeStatus(status, answer); err != nil { + logf("[mesh-knowledge] the status could not be written: %v", err) + } + time.Sleep(WatchEvery) + } +} + +func writeStatus(path string, a HealthAnswer) error { + raw, err := json.Marshal(a) + if err != nil { + return err + } + tmp := path + ".next" + if err := os.WriteFile(tmp, raw, 0o600); err != nil { + return err + } + return os.Rename(tmp, path) +} + +// StatusStale is how old the watch's verdict may be before the health tool says the watch stopped looking. +const StatusStale = 3 * WatchEvery + +// ReadStatus is the watch's last verdict, as the health tool answers it: unhealthy when there is none, or when it +// is older than StatusStale. +func ReadStatus(path string, now time.Time) HealthAnswer { + raw, err := os.ReadFile(path) + if err != nil { + return HealthAnswer{State: "unhealthy", Why: "the watch has left no verdict yet: " + err.Error()} + } + var a HealthAnswer + if err := json.Unmarshal(raw, &a); err != nil || (a.State != "healthy" && a.State != "unhealthy") { + return HealthAnswer{State: "unhealthy", Why: "the watch's verdict cannot be read"} + } + if age := now.Sub(a.At); age > StatusStale { + return HealthAnswer{State: "unhealthy", Why: fmt.Sprintf("the watch has not looked for %s (its last verdict: %s)", age.Round(time.Second), a.Why)} + } + a.Healthy = a.State == "healthy" + return a +} + +// dryRunAtTheTerminal is the import's dry run over a checkout on disk, against an empty knowledge base: what the +// import would create, before the module is rolled out. Only the verb writes. +func dryRunAtTheTerminal(args []string) int { + if len(args) != 1 { + logf("usage: mesh-knowledge dry-run ") + return 2 + } + k := &KB{Store: newMemStore(), Logf: logf} + out, err := k.Import(Unproven, ImportAsk{DryRun: true}, dirRecords{dir: args[0]}) + if err != nil { + logf("%v", err) + return 1 + } + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + _ = enc.Encode(out) + return 0 +} + +// ---- health --------------------------------------------------------------------------------------------------- + +// Healther is a store that can say why it cannot serve. +type Healther interface { + Ready(ctx context.Context) error +} + +// HealthAnswer is the health tool's answer (novox/hq ADR 0240, ADR 0283): healthy or unhealthy, and why; At is +// when the watch looked. +type HealthAnswer struct { + State string `json:"state"` + Healthy bool `json:"healthy"` + Why string `json:"why"` + At time.Time `json:"at,omitempty"` +} + +// Health is how the knowledge base is: unhealthy when its database cannot serve, or when a rule that judges the +// whole record set fails — the mesh raises a condition naming the rule and the records. +func (k *KB) Health(ready Healther) HealthAnswer { + ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second) + defer cancel() + if ready != nil { + if err := ready.Ready(ctx); err != nil { + return HealthAnswer{State: "unhealthy", Why: err.Error()} + } + } + findings, err := k.Store.Findings(ctx) + if err != nil { + return HealthAnswer{State: "unhealthy", Why: "the whole-set findings cannot be read: " + err.Error()} + } + if len(findings) > 0 { + var says []string + for _, f := range findings { + says = append(says, "the rule "+f.Rule+": "+f.Says) + } + sort.Strings(says) + if len(says) > 8 { + says = append(says[:8], fmt.Sprintf("and %d more", len(says)-8)) + } + return HealthAnswer{State: "unhealthy", Why: fmt.Sprintf("%d rule(s) that judge the whole record set fail: %s", len(findings), strings.Join(says, "; "))} + } + return HealthAnswer{State: "healthy", Healthy: true, Why: "the database serves, and every rule that judges the whole record set holds"} +} + +func healthTool(status string, now func() time.Time) stdio.Tool { + return stdio.Tool{Name: HealthTool, + Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version and every " + + "rule that judges the whole record set holds; unhealthy otherwise, naming the rule and the records — a decision accepted " + + "that nothing cites, a supersession or a change said on one end only, a record extending a superseded one.", + Run: func(map[string]any) (any, error) { return ReadStatus(status, now()), nil }} +} + +// ---- the verbs -------------------------------------------------------------------------------------------------- + +// args reads one call's arguments strictly: an argument the verb does not take is refused, never ignored — and no +// argument can say who the caller is. +type args struct { + verb string + m map[string]any +} + +func readArgs(verb string, m map[string]any, takes ...string) (args, error) { + allowed := map[string]bool{} + for _, k := range takes { + allowed[k] = true + } + var unknown []string + for k := range m { + if !allowed[k] { + unknown = append(unknown, k) + } + } + if len(unknown) > 0 { + sort.Strings(unknown) + return args{}, fmt.Errorf("%s takes %s; it does not take %s", verb, strings.Join(takes, ", "), strings.Join(unknown, ", ")) + } + return args{verb: verb, m: m}, nil +} + +func (a args) str(k string) (string, error) { + v, ok := a.m[k] + if !ok || v == nil { + return "", nil + } + s, ok := v.(string) + if !ok { + return "", fmt.Errorf("%s is text", k) + } + return s, nil +} + +func (a args) strs(k string) ([]string, error) { + v, ok := a.m[k] + if !ok || v == nil { + return nil, nil + } + switch t := v.(type) { + case string: + return []string{t}, nil + case []any: + out := make([]string, 0, len(t)) + for _, e := range t { + s, ok := e.(string) + if !ok { + return nil, fmt.Errorf("every entry of %s is text", k) + } + out = append(out, s) + } + return out, nil + } + return nil, fmt.Errorf("%s is a list of text", k) +} + +func (a args) num(k string, required bool) (int, error) { + v, ok := a.m[k] + if !ok || v == nil { + if required { + return 0, fmt.Errorf("%s needs %s", a.verb, k) + } + return 0, nil + } + switch t := v.(type) { + case float64: + if t >= 0 && t <= 1e6 && t == math.Trunc(t) { + return int(t), nil + } + case string: + if kindNumber.MatchString(strings.TrimSpace(t)) { + return atoi(strings.TrimSpace(t)), nil + } + } + return 0, fmt.Errorf("%s is a whole number, such as 276", k) +} + +func (a args) flag(k string) (bool, error) { + v, ok := a.m[k] + if !ok || v == nil { + return false, nil + } + b, ok := v.(bool) + if !ok { + return false, fmt.Errorf("%s is true or false", k) + } + return b, nil +} + +// key reads kind (decision when absent) and number. +func (a args) key() (Key, error) { + kind, err := a.str("kind") + if err != nil { + return Key{}, err + } + if kind == "" { + kind = KindDecision + } + if !contains(Kinds, kind) { + return Key{}, fmt.Errorf("kind is decision or research, not %q", kind) + } + n, err := a.num("number", true) + if err != nil { + return Key{}, err + } + if n < 1 { + return Key{}, errors.New("number is a record's number, from 1") + } + return Key{kind, n}, nil +} + +func text(d string) map[string]any { return map[string]any{"type": "string", "description": d} } +func texts(d string) map[string]any { + return map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": d} +} +func integer(d string) map[string]any { + return map[string]any{"type": "integer", "minimum": 1, "description": d} +} +func flag(d string) map[string]any { return map[string]any{"type": "boolean", "description": d} } +func kindArg() map[string]any { + return map[string]any{"type": "string", "enum": Kinds, "description": "decision (the default) or research"} +} +func boundArg() map[string]any { + return map[string]any{"type": "integer", "minimum": LeastBound, "maximum": MostBound, + "description": "the most bytes the answer may hold (default 4096)"} +} + +func schema(required []string, props map[string]any) map[string]any { + s := map[string]any{"type": "object", "properties": props, "additionalProperties": false} + if len(required) > 0 { + s["required"] = required + } + return s +} + +// Tools are the seat's verbs, each said in plain words: what it does and what it refuses. caller is who called, +// as the runtime proves it. +func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool { + return []stdio.Tool{ + {Name: Seat + ".ask", + Description: "Ask what holds about something, in plain words — \"what holds about the merge window, and why\" — and get the " + + "decision records and research efforts that bear on it, ranked, with their matching clauses: each clause says whether " + + "it is in force, what changed or superseded it, or that it is only drafted and not the record's word; each record says " + + "what changed it since and how to read its reasoning. One call, within a bound it states (4 KiB unless asked for more), " + + "saying what it cut.", + Input: schema([]string{"question"}, map[string]any{"question": map[string]any{"type": "string", "maxLength": MostQuestion, + "description": "the question, in plain words, at most 500 characters"}, "bound": boundArg()}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("ask", m, "question", "bound") + if err != nil { + return nil, err + } + q, err := a.str("question") + if err != nil { + return nil, err + } + b, err := a.num("bound", false) + if err != nil { + return nil, err + } + return k.Ask(q, b) + }}, + {Name: Seat + ".get", + Description: "One record by its number: its title, status, topic, date and deciders; its clauses, each with whether it holds; " + + "its links both ways; what changed it since; its versions; its reviews, and the operator's ask when it waits for one. " + + "Its reasoning (context, options, consequences, how it is checked; a research effort's documents) only with reasoning, " + + "or one section by name; an earlier version with version. Within a bound it states, 4 KiB unless asked for more, " + + "saying what it cut.", + Input: schema([]string{"number"}, map[string]any{"kind": kindArg(), "number": integer("the record's number, such as 276"), + "reasoning": flag("include its reasoning whole"), "section": text("only this section of its reasoning, by name"), + "version": integer("an earlier version"), "bound": boundArg()}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("get", m, "kind", "number", "reasoning", "section", "version", "bound") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + g := GetAsk{Kind: key.Kind, Number: key.Number} + if g.Reasoning, err = a.flag("reasoning"); err != nil { + return nil, err + } + if g.Section, err = a.str("section"); err != nil { + return nil, err + } + if g.Version, err = a.num("version", false); err != nil { + return nil, err + } + if g.Bound, err = a.num("bound", false); err != nil { + return nil, err + } + return k.Get(g) + }}, + {Name: Seat + ".holds", + Description: "The clauses in force now, for a topic of the reading order or for one record, each with the record it comes " + + "from; a clause changed in part says by what. Never a clause only drafted from prose: those are counted and not served. " + + "Within a bound it states, 4 KiB unless asked for more, saying what it cut.", + Input: schema(nil, map[string]any{"topic": text("a topic, such as the mesh"), "kind": kindArg(), + "number": integer("one record's number"), "bound": boundArg()}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("holds", m, "topic", "kind", "number", "bound") + if err != nil { + return nil, err + } + topic, err := a.str("topic") + if err != nil { + return nil, err + } + b, err := a.num("bound", false) + if err != nil { + return nil, err + } + var key *Key + if _, given := m["number"]; given { + kk, err := a.key() + if err != nil { + return nil, err + } + key = &kk + } + if key != nil && topic != "" { + return nil, errors.New("holds takes a topic or a record's number, not both") + } + return k.Holds(topic, key, b) + }}, + {Name: Seat + ".links", + Description: "A record's typed links, both ways: what it supersedes, changes (clause by clause), extends, became or cites, " + + "and every record that points at it, each with how it was found.", + Input: schema([]string{"number"}, map[string]any{"kind": kindArg(), "number": integer("the record's number")}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("links", m, "kind", "number") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + return k.Links(key) + }}, + {Name: Seat + ".history", + Description: "A record's versions, every one kept: who made it, when, why, and whether it was a progressive insight; with " + + "from and to, the difference between those two versions, line by line. Within a bound it states, 4 KiB unless asked for " + + "more, saying what it cut.", + Input: schema([]string{"number"}, map[string]any{"kind": kindArg(), "number": integer("the record's number"), + "from": integer("the earlier version to compare"), "to": integer("the later version to compare"), "bound": boundArg()}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("history", m, "kind", "number", "from", "to", "bound") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + from, err := a.num("from", false) + if err != nil { + return nil, err + } + to, err := a.num("to", false) + if err != nil { + return nil, err + } + b, err := a.num("bound", false) + if err != nil { + return nil, err + } + return k.History(key, from, to, b) + }}, + {Name: Seat + ".reserve", + Description: "Reserve the next decision or research number for a record you are about to propose, from the issue " + + "tracker's counter: never one another caller got. The number is spent whether or not its record is ever accepted; " + + "one never proposed is withdrawn after a week. Refused: a kind other than decision or research, an empty title.", + Input: schema([]string{"kind", "for"}, map[string]any{"kind": map[string]any{"type": "string", "enum": Kinds}, + "for": text("the title of the record the number is for")}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("reserve", m, "kind", "for") + if err != nil { + return nil, err + } + kind, err := a.str("kind") + if err != nil { + return nil, err + } + title, err := a.str("for") + if err != nil { + return nil, err + } + return k.ReserveNumber(caller(m), kind, title) + }}, + {Name: Seat + ".propose", + Description: "Propose a record whole under a reserved number: its title, topic, date and deciders; for a decision its " + + "clauses, each opening with a bold lead (\"**What is decided.** …\"), and its reasoning in sections (Context, " + + "Considered Options, Consequences, How it is checked, References); for a research effort its status and documents; its " + + "links (supersedes, changes \"0276 §4\", extends, became, cites), and whether it needs the operator. Refused, naming " + + "every reason at once and writing nothing, when it breaks one of the 29 rules that judge one record or its prose — a " + + "link to nothing, a clause without a bold lead, a superseded record cited alone, an unknown topic, a retired word. A " + + "proposal is revised by proposing it again; an accepted record changes only by a progressive insight (insight: what " + + "stood before). Nothing proposed is in force.", + Input: schema([]string{"number", "title"}, map[string]any{ + "kind": kindArg(), "number": integer("the number reserve gave"), "title": text("one line"), + "topic": text("for a decision: a topic of the reading order, such as the mesh"), + "date": text("the day, YYYY-MM-DD (today when absent)"), + "deciders": texts("who decided"), + "clauses": texts("a decision's clauses in order, each \"**Lead.** the statement\""), + "sections": map[string]any{"type": "array", "description": "the reasoning, in order: each a name and its markdown", + "items": map[string]any{"type": "object", "properties": map[string]any{"name": text("such as Context"), "text": text("markdown")}, + "required": []string{"name", "text"}}}, + "links": map[string]any{"type": "array", "description": "its links: kind, to (\"0276\", \"0276 §4\", \"research 045\", \"issue 365\", a design's path), from (the clause it goes out from)", + "items": map[string]any{"type": "object", "properties": map[string]any{"kind": map[string]any{"type": "string", "enum": LinkKinds}, + "to": text("the target"), "from": integer("the clause of this record it goes out from")}, "required": []string{"kind", "to"}}}, + "homes": texts("hq documents that cite it, such as a design's path"), + "needs-operator": flag("it holds only once the operator approves it"), + "status": map[string]any{"type": "string", "enum": ResearchStatuses, "description": "a research effort's status"}, + "insight": text("for an accepted record: a progressive insight, saying what stood before"), + "why": text("one line: why this version"), + }), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("propose", m, "kind", "number", "title", "topic", "date", "deciders", "clauses", "sections", + "links", "homes", "needs-operator", "status", "insight", "why") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + p := ProposeAsk{Kind: key.Kind, Number: key.Number} + for _, f := range []struct { + k string + p *string + }{{"title", &p.Title}, {"topic", &p.Topic}, {"date", &p.Date}, {"status", &p.Status}, {"insight", &p.Insight}, {"why", &p.Why}} { + if *f.p, err = a.str(f.k); err != nil { + return nil, err + } + } + for _, f := range []struct { + k string + p *[]string + }{{"deciders", &p.Deciders}, {"clauses", &p.Clauses}, {"homes", &p.Homes}} { + if *f.p, err = a.strs(f.k); err != nil { + return nil, err + } + } + if p.NeedsOperator, err = a.flag("needs-operator"); err != nil { + return nil, err + } + if err := decode(m["sections"], &p.Sections, "sections is a list of {name, text}"); err != nil { + return nil, err + } + if err := decode(m["links"], &p.Links, "links is a list of {kind, to, from}"); err != nil { + return nil, err + } + return k.Propose(caller(m), p) + }}, + {Name: Seat + ".review", + Description: "Review a proposed record you did not write: approve, or ask for changes with your findings (name the section " + + "and clause). Approved, it is accepted and its clauses are in force, and every clause it changes or supersedes moves; a " + + "record that needs the operator is asked of the operator first, on a channel that proves who answered, and accepted only " + + "on their Approve. Refused: a review by the record's author, a record not proposed, changes without findings.", + Input: schema([]string{"number", "verdict"}, map[string]any{"kind": kindArg(), "number": integer("the record's number"), + "verdict": map[string]any{"type": "string", "enum": []string{VerdictApprove, VerdictChanges}}, + "findings": text("what you found, by section and clause")}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("review", m, "kind", "number", "verdict", "findings") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + r := ReviewAsk{Kind: key.Kind, Number: key.Number} + if r.Verdict, err = a.str("verdict"); err != nil { + return nil, err + } + if r.Findings, err = a.str("findings"); err != nil { + return nil, err + } + return k.Review(caller(m), r) + }}, + {Name: Seat + ".import", + Description: "Import hq's decision records and research efforts, read through the records module at the commit its checkout " + + "stands at, each whole at its own number as its first version naming that commit. It only creates a record that is not " + + "held, never overwrites one, never moves a number counter, and run again changes nothing and says so. A decision written " + + "as prose waits for its clauses to be drafted and confirmed. With dry-run it writes nothing and says what it would do; " + + "with commit it refuses a checkout standing elsewhere.", + Input: schema(nil, map[string]any{"dry-run": flag("read everything, write nothing, say what would be done"), + "commit": text("the commit of hq the record must stand at (a prefix is enough)")}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("import", m, "dry-run", "commit") + if err != nil { + return nil, err + } + var ask ImportAsk + if ask.DryRun, err = a.flag("dry-run"); err != nil { + return nil, err + } + if ask.Commit, err = a.str("commit"); err != nil { + return nil, err + } + return k.Import(caller(m), ask, k.Records) + }}, + {Name: Seat + ".return", + Description: "Send the clauses drafted from a prose decision back to be drafted again, with your reason, which the next " + + "drafting reads. Until they are drafted and confirmed again the record serves none. Refused: a record whose clauses its " + + "author wrote, an empty reason.", + Input: schema([]string{"number", "reason"}, map[string]any{"kind": kindArg(), "number": integer("the record's number"), + "reason": text("what is wrong with the drafted clauses")}), + Run: func(m map[string]any) (any, error) { + a, err := readArgs("return", m, "kind", "number", "reason") + if err != nil { + return nil, err + } + key, err := a.key() + if err != nil { + return nil, err + } + reason, err := a.str("reason") + if err != nil { + return nil, err + } + return k.Return(caller(m), key, reason) + }}, + } +} + +// decode reads a structured argument strictly. +func decode(v any, into any, says string) error { + if v == nil { + return nil + } + raw, err := json.Marshal(v) + if err != nil { + return errors.New(says) + } + dec := json.NewDecoder(strings.NewReader(string(raw))) + dec.DisallowUnknownFields() + if err := dec.Decode(into); err != nil { + return fmt.Errorf("%s (%v)", says, err) + } + return nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/manifest_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/manifest_test.go new file mode 100644 index 00000000..bee4ab69 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/manifest_test.go @@ -0,0 +1,198 @@ +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + "time" +) + +// The manifest says what the code does (novox/hq to-be 55 §1): the seat it declares and holds once for the mesh +// with the verbs it serves, the events it says, what it uses and invokes, the database it requires, its two +// long-running processes with their health, and its settings. + +type manifest struct { + Module string `json:"module"` + Requires []string `json:"requires"` + Contributes map[string]map[string]any `json:"contributes"` + Seats []seat `json:"seats"` + Claims []seat `json:"claims"` + Tools []string `json:"tools"` + Emits []string `json:"emits"` + Consumes []string `json:"consumes"` + Uses []string `json:"uses"` + Invokes []string `json:"invokes"` + State []json.RawMessage `json:"state"` + Resources []map[string]any `json:"resources"` + Build struct { + Artifacts []struct { + From string `json:"from"` + Binary string `json:"binary"` + Loads []string `json:"loads"` + Env map[string]string `json:"env"` + } `json:"artifacts"` + } `json:"build"` +} + +type seat struct { + Name string `json:"name"` + Scope string `json:"scope"` + Serves []string `json:"serves"` +} + +func readManifest(t *testing.T) (manifest, string) { + t.Helper() + raw, err := os.ReadFile(filepath.Join("..", "..", "module.json")) + if err != nil { + t.Fatal(err) + } + var m manifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + return m, string(raw) +} + +func TestItDeclaresAndHoldsTheKnowledgeBaseSeatOnceForTheMesh(t *testing.T) { + m, raw := readManifest(t) + want := []seat{{Name: Seat, Scope: "mesh", Serves: Verbs}} + if m.Module != Module || !reflect.DeepEqual(m.Seats, want) || !reflect.DeepEqual(m.Claims, want) { + t.Fatalf("module %q, seats %+v, claims %+v", m.Module, m.Seats, m.Claims) + } + if !reflect.DeepEqual(Verbs, []string{"ask", "get", "holds", "links", "history", "reserve", "propose", "review", "import", "return"}) { + t.Fatalf("the verbs are not ADR 0290 rule 6's: %v", Verbs) + } + if !reflect.DeepEqual(m.Emits, Events) { + t.Fatalf("emits %v, the code says %v", m.Emits, Events) + } + // Its records are in a database in the store (ADR 0290 rule 7), not in buckets. + if !reflect.DeepEqual(m.Requires, []string{"postgres-database"}) || m.Contributes["postgres-database"]["name"] != "knowledge" || len(m.State) != 0 { + t.Fatalf("requires %v, contributes %v, state %v", m.Requires, m.Contributes, m.State) + } + // It asks the operator on the router, hears the router's word, and calls exactly what it calls. + if !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) || len(m.Consumes) != 0 { + t.Fatalf("uses %v, consumes %v", m.Uses, m.Consumes) + } + if !reflect.DeepEqual(m.Invokes, []string{"seat:issue-tracker.reserve", "claude-code.claude_code_judge", "records.records_status", + "records.records_list", "records.records_read", "seat:operator-channel.notify"}) { + t.Fatalf("invokes %v", m.Invokes) + } + src := "" + for _, f := range []string{"main.go", "hq.go"} { + src += readFile(t, f) + } + for _, inv := range m.Invokes { + if !strings.Contains(src, `"`+inv) { + t.Errorf("the manifest invokes %s, which the code never calls", inv) + } + } + if len(m.Build.Artifacts) != 1 || m.Build.Artifacts[0].From != "cmd/mesh-knowledge" || m.Build.Artifacts[0].Binary != Module || + m.Build.Artifacts[0].Env["DATABASE_URL_FILE"] == "" || m.Build.Artifacts[0].Env["MESH_KNOWLEDGE_SETTINGS"] == "" || + m.Build.Artifacts[0].Env["MESH_KNOWLEDGE_STATUS"] == "" || !reflect.DeepEqual(m.Build.Artifacts[0].Loads, []string{Module}) { + t.Fatalf("build %+v", m.Build) + } + for _, never := range []string{"/home/", "jochen", "shanks", "novox.be"} { + if strings.Contains(strings.ToLower(raw), never) { + t.Errorf("module.json names %q", never) + } + } +} + +// Its processes: the preparation step once, keep judged by its unit, watch by the health tool, which the module +// serves; the status file watch writes is the one the bundle's health tool reads. +func TestItsProcessesAreThePreparationTheActAndTheJudge(t *testing.T) { + m, _ := readManifest(t) + byID := map[string]map[string]any{} + for _, r := range m.Resources { + byID[r["id"].(string)] = r + } + prep, keep, watch := byID["prepare"], byID["keep"], byID["watch"] + if prep == nil || prep["run-once"] != true || prep["run"].([]any)[1] != "prepare" { + t.Fatalf("prepare %v", prep) + } + if keep == nil || keep["run"].([]any)[1] != "keep" || keep["health"].(map[string]any)["kind"] != "unit" { + t.Fatalf("keep %v", keep) + } + h, _ := watch["health"].(map[string]any) + if watch == nil || watch["run"].([]any)[1] != "watch" || h["kind"] != "tool" || h["tool"] != HealthTool || + !reflect.DeepEqual(m.Tools, []string{HealthTool}) { + t.Fatalf("watch %v, tools %v", watch, m.Tools) + } + if watch["env"].(map[string]any)["MESH_KNOWLEDGE_STATUS"] != m.Build.Artifacts[0].Env["MESH_KNOWLEDGE_STATUS"] { + t.Fatal("watch writes another status file than the health tool reads") + } + settings := byID["settings"] + var s Settings + if err := json.Unmarshal([]byte(settings["content"].(string)), &s); err != nil || s.JudgeModel == "" || s.ReservedFor != "168h" { + t.Fatalf("the settings the manifest writes are not the module's: %v %+v", err, s) + } + if d, err := time.ParseDuration(s.ReservedFor); err != nil || d != DefaultReservedFor { + t.Fatalf("reserved-for %q is not the default the code holds", s.ReservedFor) + } +} + +// The tools are the seat's verbs and the health tool, each said in plain words with what it refuses. +func TestTheToolsAreTheSeatsVerbsAndSayWhatTheyDo(t *testing.T) { + m, _ := readManifest(t) + var verbs []string + for _, tool := range Tools(&KB{}, callerOf) { + s, verb, ok := strings.Cut(tool.Name, ".") + if !ok || s != Seat { + t.Errorf("%s is not a verb of %s", tool.Name, Seat) + } + verbs = append(verbs, verb) + if len(tool.Description) < 120 { + t.Errorf("%s says too little of what it does", tool.Name) + } + for _, internal := range []string{"ADR", "to-be", "bucket", "postgres", "KV"} { + if strings.Contains(tool.Description, internal) { + t.Errorf("%s's description uses the internal name %q", tool.Name, internal) + } + } + } + if !reflect.DeepEqual(verbs, m.Claims[0].Serves) { + t.Fatalf("verbs %v, claim %v", verbs, m.Claims[0].Serves) + } +} + +// No argument can name the caller: who acted is the runtime's word, unproven until it gives one. +func TestNoArgumentNamesTheCaller(t *testing.T) { + for _, tool := range Tools(&KB{}, callerOf) { + props := tool.Input["properties"].(map[string]any) + for _, who := range []string{"by", "caller", "author", "reviewer", "who"} { + if _, ok := props[who]; ok { + t.Errorf("%s takes %s", tool.Name, who) + } + } + if tool.Input["additionalProperties"] != false { + t.Errorf("%s takes arguments it does not name", tool.Name) + } + } + if callerOf(map[string]any{"by": "someone@else"}) != Unproven { + t.Fatal("the caller was read from an argument") + } +} + +func TestTheHealthToolReadsTheWatchsVerdictAndSaysWhenItIsStale(t *testing.T) { + path := filepath.Join(t.TempDir(), "health.json") + now := time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC) + if a := ReadStatus(path, now); a.Healthy { + t.Fatal("healthy with no verdict") + } + if err := writeStatus(path, HealthAnswer{State: "unhealthy", Why: "the rule cited: ADR 0300 is accepted and nothing cites it", At: now}); err != nil { + t.Fatal(err) + } + if a := ReadStatus(path, now.Add(time.Minute)); a.Healthy || !strings.Contains(a.Why, "ADR 0300") { + t.Fatalf("%+v", a) + } + _ = writeStatus(path, HealthAnswer{State: "healthy", Why: "fine", At: now}) + if a := ReadStatus(path, now.Add(time.Minute)); !a.Healthy { + t.Fatalf("%+v", a) + } + if a := ReadStatus(path, now.Add(StatusStale+time.Second)); a.Healthy || !strings.Contains(a.Why, "has not looked") { + t.Fatalf("a stale verdict: %+v", a) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/migrations/0001-the-knowledge-base-holds-records-their-versions-clauses-and-links.sql b/modules/mesh-knowledge/cmd/mesh-knowledge/migrations/0001-the-knowledge-base-holds-records-their-versions-clauses-and-links.sql new file mode 100644 index 00000000..44b4fba4 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/migrations/0001-the-knowledge-base-holds-records-their-versions-clauses-and-links.sql @@ -0,0 +1,73 @@ +-- The knowledge base (novox/hq ADR 0290, to-be 55): every decision record and research effort, whole, with every +-- version kept; the clauses and links of the version each record serves, for ranked search and links read from +-- both ends; the findings of the rules that judge the whole record set; and the topics the operator was told of. + +create table record ( + kind text not null check (kind in ('decision', 'research')), + number integer not null check (number > 0), + status text not null, + topic text not null default '', + current_version integer not null, + pending_version integer, + revision bigint not null default 1, + search tsvector not null default ''::tsvector, + primary key (kind, number) +); + +create index record_search on record using gin (search); +create index record_topic on record (topic); + +create table version ( + kind text not null, + number integer not null, + n integer not null check (n > 0), + by text not null, + at timestamptz not null, + why text not null, + insight boolean not null default false, + body jsonb not null, + primary key (kind, number, n), + foreign key (kind, number) references record (kind, number) +); + +create table clause ( + kind text not null, + number integer not null, + n integer not null check (n > 0), + text text not null, + status text not null, + source text not null, + search tsvector not null, + primary key (kind, number, n), + foreign key (kind, number) references record (kind, number) +); + +create index clause_search on clause using gin (search); + +create table link ( + from_kind text not null, + from_number integer not null, + from_clause integer not null default 0, + kind text not null, + to_kind text not null, + to_number integer not null default 0, + to_clause integer not null default 0, + to_path text not null default '', + body jsonb not null, + foreign key (from_kind, from_number) references record (kind, number) +); + +create index link_from on link (from_kind, from_number); +create index link_to on link (to_kind, to_number); + +create table finding ( + rule text not null, + records text[] not null, + says text not null, + at timestamptz not null +); + +create table told ( + topic text primary key, + at timestamptz not null default now() +); diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/operator.go b/modules/mesh-knowledge/cmd/mesh-knowledge/operator.go new file mode 100644 index 00000000..43d7fbfd --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/operator.go @@ -0,0 +1,273 @@ +package main + +// A record that needs the operator (novox/hq ADR 0290 rule 4, to-be 55 §5): once an independent review approves +// it, the operator is asked on the router with Approve and Decline, both at the level approve, so only a channel +// that proves who answered can carry either (ADR 0259, as claude-code's proposals ask it, ADR 0274). Each option +// binds the act it authorises: this record, this version's digest. The knowledge base accepts the record only on +// the router's warrant for Approve, read back from the router's own record of the ask (which only the router +// writes) and held to the ask composed again from the record: the same digest, an option it offered, at its +// level, in time, and the act the option bound. A record changed after it was asked finds the digests differ, and +// nothing is accepted. The warrant is kept with the record. + +import ( + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "regexp" + "strconv" + "strings" + "time" + + stdio "git.novox.be/novox/mesh-sdk/go" + "git.novox.be/novox/mesh-sdk/go/asks" +) + +// Module is this module's name: the one the bus grants its asks under. +const Module = "mesh-knowledge" + +// The two answers, both at the level approve. +const ( + OptionApprove = "approve" + OptionDecline = "decline" +) + +// AskLasts is how long the operator has to answer: a day less a margin, under the router's bound for an approval. +const AskLasts = asks.ApproveLasts - 10*time.Minute + +// The states of an operator's ask, as the record keeps them. +const ( + AskWaiting = "waiting for the operator's answer" + AskApproved = "approved by the operator" + AskDeclined = "declined by the operator" +) + +// AskBus is the operator channel as this module reaches it: publish an ask under its own name, and read the +// router's record of one of its asks. +type AskBus interface { + Publish(subject string, body any, id string) error + Record(id string) (*stdio.StateEntry, error) +} + +type sdkAskBus struct{} + +func (sdkAskBus) Publish(subject string, body any, id string) error { + return stdio.SeatPublish(subject, body, id) +} +func (sdkAskBus) Record(id string) (*stdio.StateEntry, error) { return stdio.SeatRecord("", id) } + +// act is what an answer binds: this record's version, by its digest, and the answer. +func act(r *Record, a *OperatorAsk, option string) asks.Act { + return asks.Act{"verb": "knowledge-base.accept", "option": option, "ask": a.ID, "kind": r.Kind, + "number": strconv.Itoa(r.Number), "version": strconv.Itoa(a.Version), "digest": r.Digest()} +} + +var ( + unshownPath = regexp.MustCompile(`[/\\@~$]|\d+\.\d+\.\d+|[A-Za-z0-9+/=_]{24,}|(?i)\b[a-z0-9-]+\.[a-z]{2,12}\b`) +) + +// shown is a line the operator's phone may be shown: no path, address, host-like name or long unbroken word — +// narrower than the router's own rule, so the ask is not refused for its words. +func shown(line string) bool { return !unshownPath.MatchString(line) } + +// fingerprint is a digest as the operator is shown it: its first 24 hexadecimal digits in groups of four. +func fingerprint(digest string) string { + h := strings.TrimPrefix(digest, "sha256:") + if len(h) < 24 { + return h + } + var g []string + for i := 0; i < 24; i += 4 { + g = append(g, h[i:i+4]) + } + return strings.Join(g, " ") +} + +// composeAsk is the ask for a record, composed from the record and its ask alone: asked once, and composed +// again to hold the warrant to it. +func composeAsk(r *Record, a *OperatorAsk) asks.Ask { + name := r.Key().String() + headline := "Accept " + name + "?" + var b strings.Builder + fmt.Fprintf(&b, "%s was proposed by %s and approved by an independent review. Its author says it needs your approval before it holds.", name, r.Author) + if shown(r.Title) { + fmt.Fprintf(&b, "\n\nIts title: %s.", strings.TrimSuffix(r.Title, ".")) + } + if len(r.Clauses) > 0 { + fmt.Fprintf(&b, "\n\nIts %d clause(s):", len(r.Clauses)) + size, hidden := 0, 0 + for _, c := range r.Clauses { + lead := c.Lead() + if !shown(lead) || size+len(lead) > 1200 { + hidden++ + continue + } + size += len(lead) + fmt.Fprintf(&b, "\n%d. %s", c.N, lead) + } + if hidden > 0 { + fmt.Fprintf(&b, "\n(%d lead(s) not shown here)", hidden) + } + } + fmt.Fprintf(&b, "\n\nRead it whole with the knowledge base's get, number %d; its fingerprint is %s.", r.Number, fingerprint(r.Digest())) + out := asks.Ask{ID: a.ID, Headline: headline, Explanation: b.String(), Who: asks.Operator, + Expires: a.At.Add(AskLasts), OnExpiry: "the record stays proposed; nothing of it is in force", + About: "knowledge." + r.Kind + "." + strconv.Itoa(r.Number)} + for _, o := range []struct{ id, label, does string }{ + {OptionApprove, "Approve", "the record is accepted and its clauses are in force"}, + {OptionDecline, "Decline", "the record stays proposed; nothing of it is in force"}, + } { + binds, _ := asks.ActDigest(act(r, a, o.id)) + out.Options = append(out.Options, asks.Option{ID: o.id, Label: o.label, Does: o.does, Level: asks.Approve, Binds: binds}) + } + return out +} + +func newAskID(r *Record) string { + var b [4]byte + _, _ = rand.Read(b[:]) + return fmt.Sprintf("kb-%s-%d-%s", r.Kind, r.Number, hex.EncodeToString(b[:])) +} + +// askOperator asks the operator to approve a reviewed record, and keeps the ask with it. +func (k *KB) askOperator(ctx context.Context, held *Entry, w *Record, by string) (*Reviewed, error) { + if k.Asks == nil { + return nil, fmt.Errorf("%s needs the operator, and the operator cannot be asked from here; it stays proposed", w.Key()) + } + version := held.CurrentVersion + if held.Pending != nil { + version = held.PendingVersion + } + oa := &OperatorAsk{ID: newAskID(w), At: k.now(), Version: version, Said: AskWaiting} + ask := composeAsk(w, oa) + if err := ask.Check(k.now()); err != nil { + return nil, fmt.Errorf("the ask for %s could not be made (%v); it stays proposed", w.Key(), err) + } + w.Ask = oa + if err := k.Store.Save(ctx, Version{By: by, At: k.now(), Why: "reviewed: approved; the operator is asked", Record: w, Insight: held.Pending != nil}, + held.Pending == nil, false, held.Revision); err != nil { + return nil, fmt.Errorf("the review could not be kept (%v); the operator was not asked", err) + } + k.emit(EventReviewed, w, by, "approved; the operator is asked", held.Versions+1) + if err := k.Asks.Publish(asks.AskSubject(Module), ask, ask.ID); err != nil { + k.logf("[mesh-knowledge] the ask for %s could not be published: %v", w.Key(), err) + return nil, fmt.Errorf("the review is kept, and the operator could not be asked (%v): %s stays proposed; review it again to ask again", err, w.Key()) + } + return &Reviewed{Record: w.Key().String(), Verdict: VerdictApprove, Status: StatusProposed, + Says: fmt.Sprintf("%s needs the operator: asked on a channel that proves who answers (%q). It is accepted only on the operator's Approve; until then nothing of it is in force.", w.Key(), ask.Headline)}, nil +} + +// routerRecord is the router's record of one ask, as far as this module reads it. +type routerRecord struct { + State string `json:"state"` + Ask *asks.Ask `json:"ask"` + Warrant *asks.Warrant `json:"warrant"` +} + +// Reconcile decides every record waiting for the operator whose ask the router has closed: accepted on a warrant +// for Approve that holds, left proposed on Decline, an expiry or a warrant that does not hold. It answers a line +// for each decision. +func (k *KB) Reconcile() ([]string, error) { + if k.Asks == nil { + return nil, nil + } + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + all, err := k.Store.All(ctx) + if err != nil { + return nil, err + } + var said []string + for _, e := range all { + w := e.Working() + if w.Status != StatusProposed || w.Ask == nil || w.Ask.Said != AskWaiting { + continue + } + entry, err := k.Asks.Record(w.Ask.ID) + if err != nil || entry == nil { + if err == nil && k.now().After(w.Ask.At.Add(AskLasts+time.Hour)) { + said = append(said, k.endAsk(ctx, e, w, "ended: the router keeps no record of its ask (refused before it was taken, or forgotten)")) + } + continue + } + var rec routerRecord + if json.Unmarshal(entry.Value, &rec) != nil || rec.State == "open" || rec.Warrant == nil { + continue + } + said = append(said, k.decide(ctx, e, w, *rec.Warrant)) + } + return said, nil +} + +// endAsk records an ask ended without the operator's choice: the record stays proposed. +func (k *KB) endAsk(ctx context.Context, e *Entry, w *Record, why string) string { + w = w.Clone() + w.Ask.Said = why + if err := k.Store.Save(ctx, Version{By: "the knowledge base", At: k.now(), Why: "the operator's ask " + why, Record: w, Insight: e.Pending != nil}, + e.Pending == nil, false, e.Revision); err != nil { + return fmt.Sprintf("%s: the end of its ask could not be kept (%v)", w.Key(), err) + } + k.emit(EventReviewed, w, "the knowledge base", "the operator's ask "+why, e.Versions+1) + return fmt.Sprintf("%s: the operator's ask %s; it stays proposed", w.Key(), why) +} + +// decide holds one closed ask's warrant to the record, and accepts the record when it is the operator's Approve. +func (k *KB) decide(ctx context.Context, e *Entry, w *Record, warrant asks.Warrant) string { + if warrant.Outcome != asks.OutcomeChosen { + why := "ended " + string(warrant.Outcome) + if warrant.Words != "" { + why += " (" + warrant.Words + ")" + } + return k.endAsk(ctx, e, w, why) + } + ask := composeAsk(w, w.Ask) + option, err := warrant.For(Module, ask) + if err == nil { + err = option.Performs(act(w, w.Ask, option.ID)) + } + if err != nil { + k.logf("[mesh-knowledge] REFUSED the answer to the ask for %s: %v", w.Key(), err) + return k.endAsk(ctx, e, w, "was answered, and the answer does not hold here: "+err.Error()) + } + switch option.ID { + case OptionApprove: + w = w.Clone() + w.Ask.Said, w.Ask.Warrant = AskApproved, warrant.Says() + r, err := k.accept(ctx, e, w, "the operator", "accepted on the operator's Approve after an independent review", "warrant: "+warrant.Says()) + if err != nil { + return err.Error() + } + return fmt.Sprintf("%s: %s on the operator's warrant (%s)", r.Key(), r.Status, warrant.Says()) + case OptionDecline: + w = w.Clone() + w.Ask.Warrant = warrant.Says() + return k.endAsk(ctx, e, w, "declined by the operator") + } + return k.endAsk(ctx, e, w, "was answered with "+option.ID+", which it does not offer") +} + +// HeardEnd takes the router's word that one of this module's asks ended without a choice: nothing is accepted on +// such a word, so it needs no proof beyond its subject. A choice is only ever taken from the router's record. +func (k *KB) HeardEnd(w asks.Warrant) { + if w.Asker != Module || w.Outcome == asks.OutcomeChosen { + if _, err := k.Reconcile(); err != nil { + k.logf("[mesh-knowledge] reading the router's answers: %v", err) + } + return + } + ctx := context.Background() + k.mu.Lock() + defer k.mu.Unlock() + all, err := k.Store.All(ctx) + if err != nil { + return + } + for _, e := range all { + r := e.Working() + if r.Ask != nil && r.Ask.ID == w.Ask && r.Ask.Said == AskWaiting { + k.logf("[mesh-knowledge] %s", k.decide(ctx, e, r, w)) + } + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/parse.go b/modules/mesh-knowledge/cmd/mesh-knowledge/parse.go new file mode 100644 index 00000000..dc6bfe26 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/parse.go @@ -0,0 +1,365 @@ +package main + +// Reading hq's records as they are written (novox/hq ADR 0290 rule 8, to-be 55 §8): a decision record's +// frontmatter, heading and sections, its clauses where its Decision section writes numbered statements with a +// bold lead, and its links; a research effort's overview and documents. Read the way hq's own checks read them +// (00-META/checks/records.py and cycle.py), so the import takes what those checks judged. + +import ( + "fmt" + "path" + "regexp" + "sort" + "strconv" + "strings" +) + +// frontmatter is the block between the first two --- lines: each key with its scalar or its list. +func frontmatter(text string) (map[string][]string, string, bool) { + if !strings.HasPrefix(text, "---\n") { + return nil, text, false + } + end := strings.Index(text[4:], "\n---") + if end < 0 { + return nil, text, false + } + block := text[4 : 4+end] + rest := text[4+end+4:] + rest = strings.TrimPrefix(rest, "\n") + out := map[string][]string{} + key := "" + for _, line := range strings.Split(block, "\n") { + if m := listItem.FindStringSubmatch(line); m != nil && key != "" { + out[key] = append(out[key], strings.TrimSpace(m[1])) + continue + } + m := pair.FindStringSubmatch(line) + if m == nil { + continue + } + key = m[1] + v := strings.TrimSpace(m[2]) + switch { + case strings.HasPrefix(v, "[") && strings.HasSuffix(v, "]"): + out[key] = nil + for _, x := range strings.Split(v[1:len(v)-1], ",") { + if x = strings.TrimSpace(x); x != "" { + out[key] = append(out[key], x) + } + } + case v == "": + out[key] = nil + default: + out[key] = []string{v} + } + } + return out, rest, true +} + +var ( + listItem = regexp.MustCompile(`^\s+-\s+(.*)$`) + pair = regexp.MustCompile(`^([A-Za-z_-]+):\s*(.*)$`) + heading = regexp.MustCompile(`(?m)^# (\d+)\.\s*(.+)$`) + h1 = regexp.MustCompile(`(?m)^# (.+)$`) + // adrFile is a decision record's file name; it appears in a frontmatter value or a link target. + adrFile = regexp.MustCompile(`(?:^|/)(\d{4})-[0-9a-z-]+\.md`) + // researchFolder is a research effort's folder in a path. + researchFolder = regexp.MustCompile(`01-RESEARCH/(\d{3})-[0-9a-z-]+`) + issueFolder = regexp.MustCompile(`04-ISSUES/(\d{3,5})-[0-9a-z-]+`) + designPath = regexp.MustCompile(`03-DESIGN/0[01]-[a-z-]+/[0-9a-z-]+\.md`) + mdLink = regexp.MustCompile(`\]\(([^)\s]+)\)`) + // clauseStart is a numbered statement with a bold lead, at the start of a line: records.py's CLAUSE. + clauseStart = regexp.MustCompile(`(?m)^(\d+)\. (\*\*.*)$`) +) + +func first(f map[string][]string, k string) string { + if v := f[k]; len(v) > 0 { + return v[0] + } + return "" +} + +// splitSections cuts a body at its `## ` headings: the text before the first (after the `# ` heading) is the +// section "preamble". +func splitSections(body string) []Section { + var out []Section + name := "preamble" + var cur []string + flush := func() { + text := strings.Trim(strings.Join(cur, "\n"), "\n") + if text != "" || name != "preamble" { + out = append(out, Section{Name: name, Text: text}) + } + } + inFence := false + for _, line := range strings.Split(body, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "```") { + inFence = !inFence + } + if !inFence && strings.HasPrefix(line, "## ") { + flush() + name, cur = strings.TrimSpace(line[3:]), nil + continue + } + cur = append(cur, line) + } + flush() + return out +} + +// WrittenClauses are the numbered statements with a bold lead of a Decision section, each with the lines that +// continue it, in order. None when the section is prose. +func WrittenClauses(section string) []Clause { + lines := strings.Split(section, "\n") + var out []Clause + var cur []string + n := 0 + flush := func() { + if n > 0 { + out = append(out, Clause{N: n, Text: strings.TrimSpace(strings.Join(cur, "\n"))}) + } + } + for _, line := range lines { + if m := clauseStart.FindStringSubmatch(line); m != nil { + flush() + n, _ = strconv.Atoi(m[1]) + cur = []string{m[2]} + continue + } + if n == 0 { + continue + } + // A clause goes on over its indented lines and the blank lines between them; a line at the margin that + // is not a clause ends it. + if strings.TrimSpace(line) == "" || strings.HasPrefix(line, " ") || strings.HasPrefix(line, "\t") { + cur = append(cur, strings.TrimSpace(line)) + continue + } + flush() + n, cur = 0, nil + } + flush() + for i := range out { + out[i].Text = collapseBlankRuns(out[i].Text) + } + return out +} + +func collapseBlankRuns(s string) string { + for strings.Contains(s, "\n\n\n") { + s = strings.ReplaceAll(s, "\n\n\n", "\n\n") + } + return s +} + +// ParsedRecord is a record read from hq, and what the reading found odd. +type ParsedRecord struct { + Record *Record + Odd []string +} + +// ParseDecision reads one decision record file of hq. +func ParseDecision(file, text string) (*ParsedRecord, error) { + m := adrFile.FindStringSubmatch(file) + if m == nil { + return nil, fmt.Errorf("%s is not a decision record's file (NNNN-slug.md)", file) + } + number, _ := strconv.Atoi(m[1]) + front, body, ok := frontmatter(text) + if !ok { + return nil, fmt.Errorf("%s has no frontmatter", file) + } + h := heading.FindStringSubmatch(body) + if h == nil { + return nil, fmt.Errorf("%s has no '# N. Title' heading", file) + } + if hn, _ := strconv.Atoi(h[1]); hn != number { + return nil, fmt.Errorf("%s says %d in its file name and %d in its heading", file, number, hn) + } + p := &ParsedRecord{} + r := &Record{Kind: KindDecision, Number: number, Title: strings.TrimSpace(h[2]), Topic: first(front, "topic"), + Status: first(front, "status"), Date: first(front, "date"), Extra: map[string]string{}} + for _, d := range strings.Split(first(front, "deciders"), ",") { + if d = strings.TrimSpace(d); d != "" { + r.Deciders = append(r.Deciders, d) + } + } + after := body[strings.Index(body, h[0])+len(h[0]):] + r.Sections = splitSections(after) + known := map[string]bool{"topic": true, "status": true, "date": true, "deciders": true, "extends": true, + "supersedes": true, "superseded-by": true, "supersedes-in-part": true} + keys := make([]string, 0, len(front)) + for k := range front { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + if !known[k] { + r.Extra[k] = strings.Join(front[k], ", ") + } + } + if len(r.Extra) == 0 { + r.Extra = nil + } + // Links from the frontmatter: each stored once, on the record it goes out from. + add := func(kind string, values []string, field string) { + for _, v := range values { + mm := adrFile.FindStringSubmatch(v) + if mm == nil { + mm = regexp.MustCompile(`^(\d{4})$`).FindStringSubmatch(strings.TrimSpace(v)) + } + if mm == nil { + p.Odd = append(p.Odd, fmt.Sprintf("%s's %s names %q, which is not a decision record", file, field, v)) + continue + } + n, _ := strconv.Atoi(mm[1]) + r.Links = append(r.Links, Link{Kind: kind, To: Ref{Kind: KindDecision, Number: n}, Source: SourceWritten}) + } + } + add(LinkExtends, front["extends"], "extends") + add(LinkSupersedes, front["supersedes"], "supersedes") + add(LinkChanges, front["supersedes-in-part"], "supersedes-in-part") + if by := first(front, "superseded-by"); by != "" { + r.Extra = addExtra(r.Extra, "superseded-by", by) + } + for _, rest := range front["rests-on"] { + add(LinkCites, []string{rest}, "rests-on") + } + r.Links = append(r.Links, citations(r, after)...) + // Clauses: written, where the Decision section writes them. + for _, s := range r.Sections { + if strings.HasPrefix(s.Name, "Decision") { + r.Clauses = WrittenClauses(s.Text) + break + } + } + for i := range r.Clauses { + r.Clauses[i].Source = SourceWritten + r.Clauses[i].Status = clauseStatusOf(r.Status) + if r.Clauses[i].N != i+1 { + p.Odd = append(p.Odd, fmt.Sprintf("%s numbers its clauses out of order (clause %d is the %d.)", file, r.Clauses[i].N, i+1)) + } + } + if len(r.Clauses) == 0 && r.Section("Decision") != "" { + r.Drafting = &Drafting{State: DraftWaiting} + } + p.Record = r + return p, nil +} + +func addExtra(m map[string]string, k, v string) map[string]string { + if m == nil { + m = map[string]string{} + } + m[k] = v + return m +} + +// clauseStatusOf is a clause's status under its record's. +func clauseStatusOf(status string) string { + switch status { + case StatusAccepted: + return ClauseInForce + case StatusSuperseded: + return ClauseSuperseded + } + return ClauseProposed +} + +// citations are the records, issues and designs a record's text links to, once each, as `cites`; a record its +// frontmatter already links is not cited again. +func citations(r *Record, text string) []Link { + linked := map[Ref]bool{{Kind: r.Kind, Number: r.Number}: true} + for _, l := range r.Links { + linked[Ref{Kind: l.To.Kind, Number: l.To.Number}] = true + } + var out []Link + addRef := func(ref Ref) { + if !linked[ref] { + linked[ref] = true + out = append(out, Link{Kind: LinkCites, To: ref, Source: SourceWritten}) + } + } + for _, m := range mdLink.FindAllStringSubmatch(text, -1) { + target := m[1] + switch { + case adrFile.MatchString(target) && !strings.Contains(target, "01-RESEARCH") && !strings.Contains(target, "04-ISSUES"): + n, _ := strconv.Atoi(adrFile.FindStringSubmatch(target)[1]) + addRef(Ref{Kind: KindDecision, Number: n}) + case researchFolder.MatchString(target): + n, _ := strconv.Atoi(researchFolder.FindStringSubmatch(target)[1]) + addRef(Ref{Kind: KindResearch, Number: n}) + case issueFolder.MatchString(target): + n, _ := strconv.Atoi(issueFolder.FindStringSubmatch(target)[1]) + addRef(Ref{Kind: TargetIssue, Number: n}) + case designPath.MatchString(target): + addRef(Ref{Kind: TargetDesign, Path: designPath.FindString(target)}) + } + } + return out +} + +// ParseResearch reads one research effort: its folder's name, its overview, and every document of it. +func ParseResearch(folder string, docs map[string]string) (*ParsedRecord, error) { + m := regexp.MustCompile(`^(\d{3})-[0-9a-z-]+$`).FindStringSubmatch(folder) + if m == nil { + return nil, fmt.Errorf("%s is not a research effort's folder (NNN-slug)", folder) + } + number, _ := strconv.Atoi(m[1]) + overview, ok := docs["00-overview.md"] + if !ok { + return nil, fmt.Errorf("%s has no 00-overview.md", folder) + } + front, body, ok := frontmatter(overview) + if !ok { + return nil, fmt.Errorf("%s/00-overview.md has no frontmatter", folder) + } + p := &ParsedRecord{} + r := &Record{Kind: KindResearch, Number: number, Status: first(front, "status"), Date: first(front, "initiated"), + Topic: "research", Extra: map[string]string{}} + if h := h1.FindStringSubmatch(body); h != nil { + r.Title = strings.TrimSpace(h[1]) + } else { + r.Title = folder + p.Odd = append(p.Odd, folder+"/00-overview.md has no '# ' heading; the folder's name is its title") + } + for k, v := range front { + if k != "status" && k != "initiated" && k != "became" { + r.Extra[k] = strings.Join(v, ", ") + } + } + if len(r.Extra) == 0 { + r.Extra = nil + } + for _, b := range front["became"] { + switch { + case adrFile.MatchString(b) && strings.Contains(b, "02-DECISIONS"): + n, _ := strconv.Atoi(adrFile.FindStringSubmatch(b)[1]) + r.Links = append(r.Links, Link{Kind: LinkBecame, To: Ref{Kind: KindDecision, Number: n}, Source: SourceWritten}) + case designPath.MatchString(b): + r.Links = append(r.Links, Link{Kind: LinkBecame, To: Ref{Kind: TargetDesign, Path: designPath.FindString(b)}, Source: SourceWritten}) + default: + p.Odd = append(p.Odd, fmt.Sprintf("research %03d became %q, which is neither a decision record nor a design", number, b)) + } + } + names := make([]string, 0, len(docs)) + for name := range docs { + names = append(names, name) + } + sort.Strings(names) + var all strings.Builder + for _, name := range names { + text := docs[name] + if name == "00-overview.md" { + text = body + } + r.Sections = append(r.Sections, Section{Name: name, Text: strings.Trim(text, "\n")}) + all.WriteString(text + "\n") + } + r.Links = append(r.Links, citations(r, all.String())...) + p.Record = r + return p, nil +} + +// docName is a research document's name within its folder. +func docName(p string) string { return path.Base(p) } diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore.go b/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore.go new file mode 100644 index 00000000..c5c55204 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore.go @@ -0,0 +1,558 @@ +package main + +// The store on the mesh's postgres (novox/hq ADR 0290 rule 7): the database the mesh provisions for this module +// (the postgres-database provision), its address in the file DATABASE_URL_FILE names. Its schema is numbered +// migrations in migrations/, applied by the preparation step (`mesh-knowledge prepare`, ADR 0135) and recorded +// in the database itself; a fresh database is given the consolidated schema, schema.sql, at once, and every +// migration is recorded as applied. The serving bundle never changes the schema: it refuses to serve one that is +// behind, and says so. + +import ( + "context" + "crypto/sha256" + "embed" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io/fs" + "os" + "path" + "regexp" + "sort" + "strconv" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + "github.com/jackc/pgx/v5/pgxpool" +) + +//go:embed migrations/*.sql +var migrationFiles embed.FS + +//go:embed schema.sql +var consolidated string + +// Migration is one numbered change of the schema. +type Migration struct { + Number int + Name string + SQL string + Checksum string +} + +var migrationFile = regexp.MustCompile(`^(\d{4})-([a-z0-9-]+)\.sql$`) + +// Migrations are the migrations this binary carries, in order. A misnamed file, two of one number or an empty +// one is refused: each would make the schema something nobody wrote down. +func Migrations() ([]Migration, error) { + entries, err := fs.ReadDir(migrationFiles, "migrations") + if err != nil { + return nil, err + } + var out []Migration + seen := map[int]string{} + for _, e := range entries { + m := migrationFile.FindStringSubmatch(e.Name()) + if m == nil { + return nil, fmt.Errorf("%s is not a migration's name (NNNN-what-it-does.sql)", e.Name()) + } + n, _ := strconv.Atoi(m[1]) + if other, twice := seen[n]; twice { + return nil, fmt.Errorf("two migrations are numbered %04d: %s and %s", n, other, e.Name()) + } + seen[n] = e.Name() + body, err := fs.ReadFile(migrationFiles, path.Join("migrations", e.Name())) + if err != nil { + return nil, err + } + if strings.TrimSpace(string(body)) == "" { + return nil, fmt.Errorf("%s is empty", e.Name()) + } + sum := sha256.Sum256(body) + out = append(out, Migration{Number: n, Name: m[2], SQL: string(body), Checksum: hex.EncodeToString(sum[:])}) + } + sort.Slice(out, func(i, j int) bool { return out[i].Number < out[j].Number }) + for i, m := range out { + if m.Number != i+1 { + return nil, fmt.Errorf("the migrations are not numbered 0001 on without a gap: %04d is the %d. one", m.Number, i+1) + } + } + return out, nil +} + +// ledger is the record of what ran, kept in the database it describes. +const ledger = `create table if not exists migration ( + number integer primary key, + name text not null, + checksum text not null, + applied timestamptz not null default now())` + +// lockKey is the advisory lock every migration run takes, so two preparations never interleave. +const lockKey int64 = 4_250_290_425 + +// PgStore is the Store on postgres. +type PgStore struct{ pool *pgxpool.Pool } + +// PgStoreFromEnv opens the store the mesh provisioned, its address in the file DATABASE_URL_FILE names. +func PgStoreFromEnv(ctx context.Context) (*PgStore, error) { + file := os.Getenv("DATABASE_URL_FILE") + if file == "" { + return nil, errors.New("DATABASE_URL_FILE is not set: the knowledge base's database is a requirement the mesh resolves") + } + raw, err := os.ReadFile(file) + if err != nil { + return nil, fmt.Errorf("the database's address cannot be read: %w", err) + } + return OpenPgStore(ctx, strings.TrimSpace(string(raw))) +} + +// OpenPgStore opens a store at an address. +func OpenPgStore(ctx context.Context, url string) (*PgStore, error) { + pool, err := pgxpool.New(ctx, url) + if err != nil { + return nil, err + } + return &PgStore{pool: pool}, nil +} + +// Close closes the store. +func (s *PgStore) Close() { s.pool.Close() } + +// Prepare brings the schema to what this version needs, and says what it did: on a database that holds nothing +// of it, the consolidated schema with every migration recorded; otherwise every migration not yet applied, each +// in its own transaction with its row in the ledger. A migration that ran and whose file changed since, or one +// numbered below the highest applied that never ran, stops it: the database's contents are then not known. +func (s *PgStore) Prepare(ctx context.Context) ([]string, error) { + migrations, err := Migrations() + if err != nil { + return nil, err + } + conn, err := s.pool.Acquire(ctx) + if err != nil { + return nil, err + } + defer conn.Release() + if _, err := conn.Exec(ctx, `select pg_advisory_lock($1)`, lockKey); err != nil { + return nil, fmt.Errorf("the migration lock cannot be taken: %w", err) + } + defer func() { _, _ = conn.Exec(context.WithoutCancel(ctx), `select pg_advisory_unlock($1)`, lockKey) }() + if _, err := conn.Exec(ctx, ledger); err != nil { + return nil, err + } + applied, err := appliedMigrations(ctx, conn.Conn()) + if err != nil { + return nil, err + } + var said []string + if len(applied) == 0 { + var exists bool + if err := conn.QueryRow(ctx, `select to_regclass('record') is not null`).Scan(&exists); err != nil { + return nil, err + } + if exists { + return nil, errors.New("the database holds the knowledge base's tables and no record of a migration: its contents are not known, so nothing is changed") + } + tx, err := conn.Begin(ctx) + if err != nil { + return nil, err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + if _, err := tx.Exec(ctx, consolidated); err != nil { + return nil, fmt.Errorf("the consolidated schema failed, and nothing of it was kept: %w", err) + } + for _, m := range migrations { + if _, err := tx.Exec(ctx, `insert into migration (number, name, checksum) values ($1, $2, $3)`, m.Number, m.Name, m.Checksum); err != nil { + return nil, err + } + } + if err := tx.Commit(ctx); err != nil { + return nil, err + } + return []string{fmt.Sprintf("a fresh database: the consolidated schema, through migration %04d", len(migrations))}, nil + } + highest := 0 + var problems []string + byNumber := map[int]appliedMigration{} + for _, a := range applied { + byNumber[a.number] = a + highest = max(highest, a.number) + } + var pending []Migration + for _, m := range migrations { + a, ran := byNumber[m.Number] + switch { + case ran && a.checksum != m.Checksum: + problems = append(problems, fmt.Sprintf("migration %04d-%s ran, and its file has changed since: write a new one", m.Number, m.Name)) + case !ran && m.Number < highest: + problems = append(problems, fmt.Sprintf("migration %04d-%s never ran, and %04d did: renumber it above %04d", m.Number, m.Name, highest, highest)) + case !ran: + pending = append(pending, m) + } + } + if len(problems) > 0 { + return nil, errors.New("this database and these migrations disagree: " + strings.Join(problems, "; ")) + } + for _, m := range pending { + err := func() error { + tx, err := conn.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + if _, err := tx.Exec(ctx, m.SQL); err != nil { + return fmt.Errorf("migration %04d-%s failed, and nothing it did was kept: %w", m.Number, m.Name, err) + } + if _, err := tx.Exec(ctx, `insert into migration (number, name, checksum) values ($1, $2, $3)`, m.Number, m.Name, m.Checksum); err != nil { + return err + } + return tx.Commit(ctx) + }() + if err != nil { + return said, err + } + said = append(said, fmt.Sprintf("migration %04d-%s applied", m.Number, m.Name)) + } + if len(said) == 0 { + said = []string{fmt.Sprintf("the schema is at migration %04d, as this version needs", len(migrations))} + } + return said, nil +} + +type appliedMigration struct { + number int + checksum string +} + +func appliedMigrations(ctx context.Context, conn *pgx.Conn) ([]appliedMigration, error) { + rows, err := conn.Query(ctx, `select number, checksum from migration order by number`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []appliedMigration + for rows.Next() { + var a appliedMigration + if err := rows.Scan(&a.number, &a.checksum); err != nil { + return nil, err + } + out = append(out, a) + } + return out, rows.Err() +} + +// Ready says why the store cannot serve this version, or nil: the schema must be at the migrations it carries. +func (s *PgStore) Ready(ctx context.Context) error { + migrations, err := Migrations() + if err != nil { + return err + } + var at int + err = s.pool.QueryRow(ctx, `select coalesce(max(number), 0) from migration`).Scan(&at) + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) && pgErr.Code == "42P01" { + at = 0 + } else if err != nil { + return fmt.Errorf("the database cannot be reached: %w", err) + } + if at != len(migrations) { + return fmt.Errorf("the database's schema is at migration %04d and this version needs %04d: the preparation step "+ + "(mesh-knowledge prepare) has not run", at, len(migrations)) + } + return nil +} + +const entryColumns = `r.kind, r.number, r.current_version, coalesce(r.pending_version, 0), r.revision, + (select count(*) from version v where v.kind = r.kind and v.number = r.number), + (select body from version v where v.kind = r.kind and v.number = r.number and v.n = r.current_version), + (select body from version v where v.kind = r.kind and v.number = r.number and v.n = r.pending_version)` + +func scanEntry(row pgx.Row) (*Entry, error) { + var e Entry + var kind string + var number int + var current, pending []byte + if err := row.Scan(&kind, &number, &e.CurrentVersion, &e.PendingVersion, &e.Revision, &e.Versions, ¤t, &pending); err != nil { + return nil, err + } + e.Current = &Record{} + if err := json.Unmarshal(current, e.Current); err != nil { + return nil, fmt.Errorf("%s's version %d cannot be read: %w", Key{kind, number}, e.CurrentVersion, err) + } + if len(pending) > 0 { + e.Pending = &Record{} + if err := json.Unmarshal(pending, e.Pending); err != nil { + return nil, fmt.Errorf("%s's version %d cannot be read: %w", Key{kind, number}, e.PendingVersion, err) + } + } + return &e, nil +} + +func (s *PgStore) Get(ctx context.Context, k Key) (*Entry, error) { + e, err := scanEntry(s.pool.QueryRow(ctx, `select `+entryColumns+` from record r where r.kind = $1 and r.number = $2`, k.Kind, k.Number)) + if errors.Is(err, pgx.ErrNoRows) { + return nil, nil + } + return e, err +} + +func (s *PgStore) All(ctx context.Context) ([]*Entry, error) { + rows, err := s.pool.Query(ctx, `select `+entryColumns+` from record r order by r.kind, r.number`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []*Entry + for rows.Next() { + e, err := scanEntry(rows) + if err != nil { + return nil, err + } + out = append(out, e) + } + return out, rows.Err() +} + +func sectionsText(r *Record) string { + var b strings.Builder + for _, s := range r.Sections { + b.WriteString(s.Text) + b.WriteString("\n") + } + return b.String() +} + +// served writes the rows of the version a record serves: its status, topic and search vector, its clauses and +// its links. +func served(ctx context.Context, tx pgx.Tx, r *Record) error { + if _, err := tx.Exec(ctx, `update record set status = $3, topic = $4, + search = setweight(to_tsvector('english', $5), 'A') || setweight(to_tsvector('english', $4), 'B') || + setweight(to_tsvector('english', $6), 'D') + where kind = $1 and number = $2`, r.Kind, r.Number, r.Status, r.Topic, r.Title, sectionsText(r)); err != nil { + return err + } + if _, err := tx.Exec(ctx, `delete from clause where kind = $1 and number = $2`, r.Kind, r.Number); err != nil { + return err + } + for _, c := range r.Clauses { + if _, err := tx.Exec(ctx, `insert into clause (kind, number, n, text, status, source, search) + values ($1, $2, $3, $4, $5, $6, setweight(to_tsvector('english', $4), 'A'))`, + r.Kind, r.Number, c.N, c.Text, c.Status, c.Source); err != nil { + return err + } + } + if _, err := tx.Exec(ctx, `delete from link where from_kind = $1 and from_number = $2`, r.Kind, r.Number); err != nil { + return err + } + for _, l := range r.Links { + body, _ := json.Marshal(l) + if _, err := tx.Exec(ctx, `insert into link (from_kind, from_number, from_clause, kind, to_kind, to_number, to_clause, to_path, body) + values ($1, $2, $3, $4, $5, $6, $7, $8, $9)`, r.Kind, r.Number, l.From, l.Kind, l.To.Kind, l.To.Number, l.To.Clause, l.To.Path, body); err != nil { + return err + } + } + return nil +} + +func insertVersion(ctx context.Context, tx pgx.Tx, v Version) error { + body, err := json.Marshal(v.Record) + if err != nil { + return err + } + _, err = tx.Exec(ctx, `insert into version (kind, number, n, by, at, why, insight, body) values ($1, $2, $3, $4, $5, $6, $7, $8)`, + v.Record.Kind, v.Record.Number, v.N, v.By, v.At, v.Why, v.Insight, body) + return err +} + +func (s *PgStore) Create(ctx context.Context, v Version) error { + tx, err := s.pool.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + r := v.Record + tag, err := tx.Exec(ctx, `insert into record (kind, number, status, topic, current_version) values ($1, $2, $3, $4, 1) + on conflict do nothing`, r.Kind, r.Number, r.Status, r.Topic) + if err != nil { + return err + } + if tag.RowsAffected() == 0 { + return ErrExists + } + v.N = 1 + if err := insertVersion(ctx, tx, v); err != nil { + return err + } + if err := served(ctx, tx, r); err != nil { + return err + } + return tx.Commit(ctx) +} + +func (s *PgStore) Save(ctx context.Context, v Version, current, clearPending bool, revision int64) error { + tx, err := s.pool.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + r := v.Record + var held int64 + var versions int + err = tx.QueryRow(ctx, `select revision, (select count(*) from version v where v.kind = r.kind and v.number = r.number) + from record r where kind = $1 and number = $2 for update`, r.Kind, r.Number).Scan(&held, &versions) + if errors.Is(err, pgx.ErrNoRows) { + return fmt.Errorf("%s is not held", r.Key()) + } + if err != nil { + return err + } + if held != revision { + return ErrChanged + } + v.N = versions + 1 + if err := insertVersion(ctx, tx, v); err != nil { + return err + } + if current { + if _, err := tx.Exec(ctx, `update record set current_version = $3 where kind = $1 and number = $2`, r.Kind, r.Number, v.N); err != nil { + return err + } + if err := served(ctx, tx, r); err != nil { + return err + } + } else { + if _, err := tx.Exec(ctx, `update record set pending_version = $3 where kind = $1 and number = $2`, r.Kind, r.Number, v.N); err != nil { + return err + } + } + if clearPending { + if _, err := tx.Exec(ctx, `update record set pending_version = null where kind = $1 and number = $2`, r.Kind, r.Number); err != nil { + return err + } + } + if _, err := tx.Exec(ctx, `update record set revision = revision + 1 where kind = $1 and number = $2`, r.Kind, r.Number); err != nil { + return err + } + return tx.Commit(ctx) +} + +func (s *PgStore) Versions(ctx context.Context, k Key) ([]Version, error) { + rows, err := s.pool.Query(ctx, `select n, by, at, why, insight, body from version where kind = $1 and number = $2 order by n`, k.Kind, k.Number) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Version + for rows.Next() { + var v Version + var body []byte + if err := rows.Scan(&v.N, &v.By, &v.At, &v.Why, &v.Insight, &body); err != nil { + return nil, err + } + v.Record = &Record{} + if err := json.Unmarshal(body, v.Record); err != nil { + return nil, err + } + out = append(out, v) + } + return out, rows.Err() +} + +// Search ranks by the database's full-text search: any of the question's words, a clause weighing more than a +// record's sections, a title most. +func (s *PgStore) Search(ctx context.Context, question string, limit int) ([]Hit, error) { + words := questionWords(question) + if len(words) == 0 { + return nil, nil + } + if limit <= 0 { + limit = 200 + } + rows, err := s.pool.Query(ctx, `with q as (select replace(plainto_tsquery('english', $1)::text, ' & ', ' | ')::tsquery as q) + select kind, number, clause, rank from ( + select c.kind, c.number, c.n as clause, ts_rank_cd(c.search, q.q) as rank + from clause c join record r on r.kind = c.kind and r.number = c.number, q + where c.search @@ q.q and r.status not in ('reserved', 'withdrawn') + union all + select r.kind, r.number, 0, ts_rank_cd(r.search, q.q) + from record r, q where r.search @@ q.q and r.status not in ('reserved', 'withdrawn') + ) hits order by rank desc, number desc, clause limit $2`, strings.Join(words, " "), limit) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Hit + for rows.Next() { + var h Hit + if err := rows.Scan(&h.Key.Kind, &h.Key.Number, &h.Clause, &h.Rank); err != nil { + return nil, err + } + out = append(out, h) + } + return out, rows.Err() +} + +func (s *PgStore) Inbound(ctx context.Context, k Key) ([]Inbound, error) { + rows, err := s.pool.Query(ctx, `select from_kind, from_number, body from link where to_kind = $1 and to_number = $2 + order by from_number, kind`, k.Kind, k.Number) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Inbound + for rows.Next() { + var in Inbound + var body []byte + if err := rows.Scan(&in.From.Kind, &in.From.Number, &body); err != nil { + return nil, err + } + if err := json.Unmarshal(body, &in.Link); err != nil { + return nil, err + } + out = append(out, in) + } + return out, rows.Err() +} + +func (s *PgStore) SetFindings(ctx context.Context, f []Finding) error { + tx, err := s.pool.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }() + if _, err := tx.Exec(ctx, `delete from finding`); err != nil { + return err + } + for _, x := range f { + if _, err := tx.Exec(ctx, `insert into finding (rule, records, says, at) values ($1, $2, $3, $4)`, x.Rule, x.Records, x.Says, x.At); err != nil { + return err + } + } + return tx.Commit(ctx) +} + +func (s *PgStore) Findings(ctx context.Context) ([]Finding, error) { + rows, err := s.pool.Query(ctx, `select rule, records, says, at from finding order by rule, says`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Finding + for rows.Next() { + var f Finding + if err := rows.Scan(&f.Rule, &f.Records, &f.Says, &f.At); err != nil { + return nil, err + } + out = append(out, f) + } + return out, rows.Err() +} + +func (s *PgStore) Told(ctx context.Context, topic string) (bool, error) { + tag, err := s.pool.Exec(ctx, `insert into told (topic) values ($1) on conflict do nothing`, topic) + if err != nil { + return false, err + } + return tag.RowsAffected() == 1, nil +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore_test.go new file mode 100644 index 00000000..6817d243 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/pgstore_test.go @@ -0,0 +1,133 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "reflect" + "strings" + "testing" +) + +// The schema is numbered migrations and a consolidated schema for a fresh database (the mesh's rule: both). A +// database given the consolidated schema and one given every migration in order hold the same tables, columns, +// constraints and indexes. +func TestTheConsolidatedSchemaIsTheMigrationsApplied(t *testing.T) { + ctx := context.Background() + fresh := pgStore(t) // Prepare on an empty database: the consolidated schema + stepped, err := OpenPgStore(ctx, freshDatabase(t)) + if err != nil { + t.Fatal(err) + } + defer stepped.Close() + migrations, err := Migrations() + if err != nil { + t.Fatal(err) + } + for _, m := range migrations { + if _, err := stepped.pool.Exec(ctx, m.SQL); err != nil { + t.Fatalf("migration %04d: %v", m.Number, err) + } + } + a, b := shape(t, fresh), shape(t, stepped) + if !reflect.DeepEqual(a, b) { + t.Fatalf("the consolidated schema and the migrations differ:\nconsolidated %v\nmigrations %v", a, b) + } + if len(a) < 20 { + t.Fatalf("the schema read is too small to compare: %v", a) + } + // The consolidated schema says how far it is consolidated. + if !strings.Contains(consolidated, "through migration "+itoa4(len(migrations))) { + t.Fatalf("schema.sql does not say it is consolidated through migration %04d", len(migrations)) + } +} + +func shape(t *testing.T, s *PgStore) []string { + t.Helper() + var out []string + for _, q := range []string{ + `select table_name || '.' || column_name || ' ' || data_type || ' ' || is_nullable || ' ' || coalesce(column_default, '') + from information_schema.columns where table_schema = 'public' and table_name <> 'migration' order by 1`, + `select conrelid::regclass::text || ' ' || pg_get_constraintdef(oid) from pg_constraint + where connamespace = 'public'::regnamespace and conrelid::regclass::text <> 'migration' order by 1`, + `select indexdef from pg_indexes where schemaname = 'public' and tablename <> 'migration' order by 1`, + } { + rows, err := s.pool.Query(context.Background(), q) + if err != nil { + t.Fatal(err) + } + for rows.Next() { + var line string + if err := rows.Scan(&line); err != nil { + t.Fatal(err) + } + out = append(out, line) + } + rows.Close() + } + return out +} + +func TestPreparingTwiceChangesNothingAndAChangedMigrationIsRefused(t *testing.T) { + ctx := context.Background() + s := pgStore(t) + said, err := s.Prepare(ctx) + if err != nil || len(said) != 1 || !strings.Contains(said[0], "as this version needs") { + t.Fatalf("a second preparation: %v %v", said, err) + } + if err := s.Ready(ctx); err != nil { + t.Fatal(err) + } + if _, err := s.pool.Exec(ctx, `update migration set checksum = 'other' where number = 1`); err != nil { + t.Fatal(err) + } + if _, err := s.Prepare(ctx); err == nil || !strings.Contains(err.Error(), "has changed since") { + t.Fatalf("a changed migration: %v", err) + } +} + +func TestAStoreBehindItsMigrationsRefusesToServe(t *testing.T) { + ctx := context.Background() + s, err := OpenPgStore(ctx, freshDatabase(t)) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if err := s.Ready(ctx); err == nil || !strings.Contains(err.Error(), "preparation step") { + t.Fatalf("an unprepared database: %v", err) + } + l := &lazyStore{open: func(context.Context) (*PgStore, error) { return s, nil }} + k := &KB{Store: l} + if _, err := k.Get(GetAsk{Number: 276}); err == nil || !strings.Contains(err.Error(), "preparation step") { + t.Fatalf("a verb over an unprepared database: %v", err) + } + if h := k.Health(l); h.Healthy || !strings.Contains(h.Why, "preparation step") { + t.Fatalf("health over an unprepared database: %+v", h) + } +} + +// The migrations carried are numbered from 0001 without a gap, and every file in migrations/ is one. +func TestTheMigrationsAreNumberedWithoutAGap(t *testing.T) { + ms, err := Migrations() + if err != nil { + t.Fatal(err) + } + files, _ := os.ReadDir("migrations") + if len(ms) == 0 || len(ms) != len(files) { + t.Fatalf("%d migrations, %d files", len(ms), len(files)) + } + for _, f := range files { + if filepath.Ext(f.Name()) != ".sql" { + t.Fatalf("%s is not a migration", f.Name()) + } + } +} + +// The daemons' lines the rules refuse are the catalogue's own copy. +func TestTheDaemonLinesAreTheCataloguesCopy(t *testing.T) { + theirs := DaemonLines(readFile(t, filepath.Join("..", "..", "..", "..", "retired-lines"))) + ours := DaemonLines(daemonLinesFile) + if len(ours) < 50 || !reflect.DeepEqual(theirs, ours) { + t.Fatalf("daemon-lines differs from the catalogue's retired-lines (%d against %d)", len(ours), len(theirs)) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/record.go b/modules/mesh-knowledge/cmd/mesh-knowledge/record.go new file mode 100644 index 00000000..2569f65c --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/record.go @@ -0,0 +1,444 @@ +package main + +// What one record holds (novox/hq ADR 0290 rules 2 and 3, to-be 55 §2 and §3): a decision record or a research +// effort, whole — its fields, its clauses each with a status and a source, its typed links, its reasoning in +// sections — and every version of it. + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "regexp" + "sort" + "strconv" + "strings" + "time" +) + +// The kinds this slice holds. +const ( + KindDecision = "decision" + KindResearch = "research" +) + +// Kinds are the kinds a record may be. +var Kinds = []string{KindDecision, KindResearch} + +// A record's status. A decision is proposed, accepted, then perhaps superseded; a research effort is proposed, +// then active, graduated or abandoned. A number reserved and never proposed is reserved, and withdrawn once its +// time has passed: a number is spent whether or not its record is accepted (rule 5). +const ( + StatusReserved = "reserved" + StatusProposed = "proposed" + StatusAccepted = "accepted" + StatusSuperseded = "superseded" + StatusWithdrawn = "withdrawn" + StatusActive = "active" + StatusGraduated = "graduated" + StatusAbandoned = "abandoned" +) + +// ResearchStatuses are the statuses an accepted research effort may have. +var ResearchStatuses = []string{StatusActive, StatusGraduated, StatusAbandoned} + +// A clause's status (to-be 55 §3). +const ( + ClauseProposed = "proposed" + ClauseInForce = "in force" + ClauseChanged = "changed" + ClauseSuperseded = "superseded" +) + +// A clause's source (to-be 55 §3). +const ( + SourceWritten = "written" + SourceDrafted = "drafted" + SourceConfirmed = "confirmed" +) + +// The kinds of a link (to-be 55 §2). +const ( + LinkSupersedes = "supersedes" + LinkChanges = "changes" + LinkExtends = "extends" + LinkBecame = "became" + LinkCites = "cites" +) + +// LinkKinds are the kinds a link may be. +var LinkKinds = []string{LinkSupersedes, LinkChanges, LinkExtends, LinkBecame, LinkCites} + +// What a link may point at besides a record of this knowledge base: an issue in the tracker, a design in hq. +const ( + TargetIssue = "issue" + TargetDesign = "design" +) + +// Key is a record's key: its kind and number together. +type Key struct { + Kind string `json:"kind"` + Number int `json:"number"` +} + +func (k Key) String() string { + if k.Kind == KindResearch { + return fmt.Sprintf("research %03d", k.Number) + } + return fmt.Sprintf("ADR %04d", k.Number) +} + +// Clause is one numbered statement of what a decision decided, with its own status and source. +type Clause struct { + N int `json:"n"` + // Text is the clause as written: its bold lead, then the rest. + Text string `json:"text"` + Status string `json:"status"` + Source string `json:"source"` + // ChangedBy names the clauses or records of later records that changed or superseded it: "ADR 0282 §2". + ChangedBy []string `json:"changed-by,omitempty"` + // Unconfirmed is why the second judgement did not confirm a drafted clause. + Unconfirmed string `json:"unconfirmed,omitempty"` +} + +// Lead is the clause's bold lead, or "" when it has none. +func (c Clause) Lead() string { return boldLead(c.Text) } + +var leadPattern = regexp.MustCompile(`^\*\*([^*\n]+?)\*\*`) + +func boldLead(text string) string { + m := leadPattern.FindStringSubmatch(strings.TrimSpace(text)) + if m == nil { + return "" + } + return strings.TrimSpace(m[1]) +} + +// Ref is what a link points at: a record (and a clause of it, when Clause is set), an issue, or a design. +type Ref struct { + Kind string `json:"kind"` + Number int `json:"number,omitempty"` + Clause int `json:"clause,omitempty"` + // Path is a design's path in hq, for a link to a design. + Path string `json:"path,omitempty"` +} + +// Record is the record this ref names, when it names one. +func (r Ref) Record() (Key, bool) { + if r.Kind == KindDecision || r.Kind == KindResearch { + return Key{r.Kind, r.Number}, true + } + return Key{}, false +} + +func (r Ref) String() string { + switch r.Kind { + case KindDecision, KindResearch: + s := Key{r.Kind, r.Number}.String() + if r.Clause > 0 { + s += " §" + strconv.Itoa(r.Clause) + } + return s + case TargetIssue: + return fmt.Sprintf("issue %d", r.Number) + case TargetDesign: + return "design " + r.Path + } + return r.Kind +} + +var ( + refRecord = regexp.MustCompile(`^(?i)(?:(adr|decision|research)\s*)?(\d{1,4})(?:\s*§\s*(\d{1,3}))?$`) + refIssue = regexp.MustCompile(`^(?i)issue\s*(\d{1,5})$`) + refDesign = regexp.MustCompile(`^(?i)(?:design\s+)?(03-DESIGN/[0-9a-z/-]+\.md)$`) +) + +// ParseRef reads a link's target as an agent writes it: "0276", "ADR 0276 §4", "research 045", "issue 365", +// "03-DESIGN/01-to-be/55-the-knowledge-base.md". A bare number is a decision. +func ParseRef(s string) (Ref, error) { + s = strings.TrimSpace(s) + if m := refIssue.FindStringSubmatch(s); m != nil { + n, _ := strconv.Atoi(m[1]) + return Ref{Kind: TargetIssue, Number: n}, nil + } + if m := refDesign.FindStringSubmatch(s); m != nil { + return Ref{Kind: TargetDesign, Path: m[1]}, nil + } + if m := refRecord.FindStringSubmatch(s); m != nil { + kind := KindDecision + if strings.EqualFold(m[1], "research") { + kind = KindResearch + } + n, _ := strconv.Atoi(m[2]) + c := 0 + if m[3] != "" { + c, _ = strconv.Atoi(m[3]) + } + if n < 1 { + return Ref{}, fmt.Errorf("%q names no number from 1", s) + } + return Ref{Kind: kind, Number: n, Clause: c}, nil + } + return Ref{}, fmt.Errorf("%q is not a record (0276, ADR 0276 §4, research 045), an issue (issue 365) or a design "+ + "(03-DESIGN/01-to-be/55-the-knowledge-base.md)", s) +} + +// Link is one typed link, held on the record it goes out from and read from both ends. From is the clause it +// goes out from, 0 for the record as a whole. +type Link struct { + Kind string `json:"kind"` + From int `json:"from,omitempty"` + To Ref `json:"to"` + // Source says how the link's clause was found: written by the author, or drafted and confirmed by the + // judgement at the import (to-be 55 §7) for a change whose clause hq never named. + Source string `json:"source,omitempty"` + // Unresolved says the import found a target this knowledge base does not hold. + Unresolved bool `json:"unresolved,omitempty"` +} + +func (l Link) String() string { + from := "" + if l.From > 0 { + from = fmt.Sprintf("§%d ", l.From) + } + return from + l.Kind + " " + l.To.String() +} + +// Section is one part of a record's reasoning, markdown as written: for a decision, context, the options weighed, +// the decision as written, consequences, how it is checked, references; for a research effort, its documents by +// name. +type Section struct { + Name string `json:"name"` + Text string `json:"text"` +} + +// Review is one review of one version. +type Review struct { + By string `json:"by"` + At time.Time `json:"at"` + Verdict string `json:"verdict"` + Findings string `json:"findings,omitempty"` + Version int `json:"version"` +} + +// The verdicts of a review. +const ( + VerdictApprove = "approve" + VerdictChanges = "changes" +) + +// OperatorAsk is the ask a record that needs the operator waits on, and its answer once proven. +type OperatorAsk struct { + ID string `json:"id"` + At time.Time `json:"at"` + Version int `json:"version"` + // Said is what came of it: waiting, approved, declined, expired, refused (and why). + Said string `json:"said"` + Warrant string `json:"warrant,omitempty"` +} + +// Drafting is where a prose record stands in the drafting and confirming of its clauses (to-be 55 §7). +type Drafting struct { + // State is waiting (to be drafted), drafted (waiting for the second judgement), done, or failed. + State string `json:"state"` + Reason string `json:"reason,omitempty"` + Tries int `json:"tries,omitempty"` + Drafter string `json:"drafter,omitempty"` + Checker string `json:"checker,omitempty"` + Returned string `json:"returned,omitempty"` + At time.Time `json:"at,omitempty"` +} + +// The states of Drafting. +const ( + DraftWaiting = "waiting" + DraftDrafted = "drafted" + DraftDone = "done" + DraftFailed = "failed" +) + +// Record is one record, whole, as one version holds it. +type Record struct { + Kind string `json:"kind"` + Number int `json:"number"` + Title string `json:"title"` + Topic string `json:"topic,omitempty"` + Status string `json:"status"` + Date string `json:"date,omitempty"` + Deciders []string `json:"deciders,omitempty"` + // Effort is the status a proposed research effort takes when it is accepted. + Effort string `json:"effort,omitempty"` + NeedsOperator bool `json:"needs-operator,omitempty"` + Clauses []Clause `json:"clauses,omitempty"` + Links []Link `json:"links,omitempty"` + Sections []Section `json:"sections,omitempty"` + // Homes are the documents of hq that cite the record (a design, a playbook), while those live in hq. + Homes []string `json:"homes,omitempty"` + // Extra holds the fields of an imported record this slice has no field of its own for, as written. + Extra map[string]string `json:"extra,omitempty"` + // Author is who proposed it, as the runtime proves the caller; Source names the commit and path of hq an + // imported record was read from. + Author string `json:"author,omitempty"` + // Writers are everyone who wrote a version of it, as the runtime proved them: none of them reviews it. + Writers []string `json:"writers,omitempty"` + Source string `json:"source,omitempty"` + // Reserved is who reserved the number, when, and for what title. + ReservedBy string `json:"reserved-by,omitempty"` + ReservedAt time.Time `json:"reserved-at,omitempty"` + ReservedFor string `json:"reserved-for,omitempty"` + // AcceptedAt is when it was accepted here; zero for one imported accepted. + AcceptedAt time.Time `json:"accepted-at,omitempty"` + Reviews []Review `json:"reviews,omitempty"` + Ask *OperatorAsk `json:"ask,omitempty"` + Drafting *Drafting `json:"drafting,omitempty"` + Insights []Insight `json:"insights,omitempty"` + Pending *PendingState `json:"pending,omitempty"` +} + +// Insight is a progressive insight: a correction of fact, dated, saying what stood before. +type Insight struct { + Date string `json:"date"` + Before string `json:"before"` + By string `json:"by,omitempty"` +} + +// PendingState marks a version that is a proposal over an accepted record: a progressive insight waiting for +// its review. The accepted version stays the one served until it is accepted. +type PendingState struct { + Insight Insight `json:"insight"` +} + +// Key is the record's key. +func (r *Record) Key() Key { return Key{r.Kind, r.Number} } + +// Clause is the clause numbered n, or nil. +func (r *Record) Clause(n int) *Clause { + for i := range r.Clauses { + if r.Clauses[i].N == n { + return &r.Clauses[i] + } + } + return nil +} + +// Section is the section of this name, or "". +func (r *Record) Section(name string) string { + for _, s := range r.Sections { + if strings.EqualFold(s.Name, name) { + return s.Text + } + } + return "" +} + +// Accepted says the record holds: an accepted decision, or a research effort accepted at any of its statuses. +func (r *Record) Accepted() bool { + switch r.Status { + case StatusAccepted, StatusActive, StatusGraduated, StatusAbandoned: + return true + } + return false +} + +// Clone is a deep copy, so a change to the copy never reaches the version it came from. +func (r *Record) Clone() *Record { + raw, _ := json.Marshal(r) + var c Record + _ = json.Unmarshal(raw, &c) + return &c +} + +// Digest is the SHA-256 of what the record says — every field but its reviews and its ask — which an operator's +// approval binds: an approval authorises this version and nothing written after it. +func (r *Record) Digest() string { + c := r.Clone() + c.Reviews, c.Ask, c.Drafting = nil, nil, nil + raw, _ := json.Marshal(c) + sum := sha256.Sum256(raw) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +// Version is one version of a record: who made it, when, why in one line, and the record whole as it stood. +type Version struct { + N int `json:"n"` + By string `json:"by"` + At time.Time `json:"at"` + Why string `json:"why"` + Record *Record `json:"record"` + // Insight marks a version that is a progressive insight. + Insight bool `json:"insight,omitempty"` +} + +// Entry is a record as the store keeps it: the version served and, while one waits for its review, the version +// proposed over it. +type Entry struct { + Current *Record + // CurrentVersion is the number of the version served, PendingVersion of the version proposed over it (0 for + // none), and Versions how many there are. + CurrentVersion int + PendingVersion int + Pending *Record + Versions int + // Revision is the store's own count of writes to this entry: a write names the one it read. + Revision int64 +} + +// Working is the version a writer works on: the pending one when there is one, else the current one. +func (e *Entry) Working() *Record { + if e.Pending != nil { + return e.Pending + } + return e.Current +} + +// Render is the record as markdown, the way its history compares two versions line by line. +func Render(r *Record) string { + var b strings.Builder + fmt.Fprintf(&b, "# %s. %s\n\n", strings.TrimPrefix(strings.TrimPrefix(r.Key().String(), "ADR "), "research "), r.Title) + field := func(k, v string) { + if v != "" { + fmt.Fprintf(&b, "%s: %s\n", k, v) + } + } + field("kind", r.Kind) + field("status", r.Status) + field("topic", r.Topic) + field("date", r.Date) + field("deciders", strings.Join(r.Deciders, ", ")) + field("effort", r.Effort) + if r.NeedsOperator { + field("needs operator", "yes") + } + field("homes", strings.Join(r.Homes, ", ")) + keys := make([]string, 0, len(r.Extra)) + for k := range r.Extra { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + field(k, r.Extra[k]) + } + for _, in := range r.Insights { + fmt.Fprintf(&b, "insight %s: %s\n", in.Date, in.Before) + } + if len(r.Clauses) > 0 { + b.WriteString("\n## Clauses\n\n") + for _, c := range r.Clauses { + fmt.Fprintf(&b, "%d. %s\n [%s, %s", c.N, c.Text, c.Status, c.Source) + if len(c.ChangedBy) > 0 { + b.WriteString("; by " + strings.Join(c.ChangedBy, ", ")) + } + b.WriteString("]\n") + } + } + if len(r.Links) > 0 { + b.WriteString("\n## Links\n\n") + for _, l := range r.Links { + fmt.Fprintf(&b, "- %s\n", l) + } + } + for _, s := range r.Sections { + fmt.Fprintf(&b, "\n## %s\n\n%s\n", s.Name, strings.TrimRight(s.Text, "\n")) + } + return b.String() +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/reference.go b/modules/mesh-knowledge/cmd/mesh-knowledge/reference.go new file mode 100644 index 00000000..47dc473b --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/reference.go @@ -0,0 +1,222 @@ +package main + +// What the rules judge a record against and this module does not hold: the reading order's topics +// (02-DECISIONS/README.md) and the glossary's retired words (00-META/glossary.md), read from hq through the +// records module, and the daemons' lines, a copy kept beside this code (daemon-lines, compared with the +// catalogue's own copy by a test). + +import ( + _ "embed" + "fmt" + "regexp" + "strings" + "unicode" +) + +//go:embed daemon-lines +var daemonLinesFile string + +// Reference is what the rules read besides the records. +type Reference struct { + Commit string + // Topics are the reading order's topics. + Topics []string + // Retired are the glossary's words retired in hq's prose (no scope, or hq). + Retired []string + // Lines are the daemons' lines: a daemon and a command, and whether it is a line only with an argument. + Lines []DaemonLine +} + +// DaemonLine is one daemon's command a person never types. +type DaemonLine struct { + Daemon, Command string + WithArgument bool +} + +var ( + topicLine = regexp.MustCompile(`(?m)^\d+\.\s+\*\*([^*]+)\*\*\s+—\s+` + "`" + `topic:\s*([^` + "`" + `]+)` + "`") + notLine = regexp.MustCompile(`(?m)^\s*(?:[-*]\s+)?\*Not:\*(.*)$`) + struck = regexp.MustCompile(`~~([^~]+)~~(?:\s*\((hq|tools)\))?`) +) + +// ReadingOrder is the topics the reading order writes, in order. +func ReadingOrder(readme string) []string { + var out []string + for _, m := range topicLine.FindAllStringSubmatch(readme, -1) { + out = append(out, strings.TrimSpace(m[2])) + } + return out +} + +// RetiredWords are the glossary's words retired in hq's prose: a struck word on a *Not:* line with no scope or +// the scope hq; "a / b" is two words. +func RetiredWords(glossary string) []string { + var out []string + for _, line := range notLine.FindAllStringSubmatch(glossary, -1) { + for _, m := range struck.FindAllStringSubmatch(line[1], -1) { + if m[2] == "tools" { + continue + } + for _, w := range strings.Split(m[1], " / ") { + if w = strings.TrimSpace(w); w != "" { + out = append(out, w) + } + } + } + } + return out +} + +// DaemonLines reads the daemons' lines in the catalogue's format: ` `, ` …` after a command +// that is a line only with an argument after it. +func DaemonLines(file string) []DaemonLine { + var out []DaemonLine + for _, line := range strings.Split(file, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + with := strings.HasSuffix(line, " …") + line = strings.TrimSuffix(line, " …") + d, c, ok := strings.Cut(line, " ") + if ok { + out = append(out, DaemonLine{Daemon: d, Command: c, WithArgument: with}) + } + } + return out +} + +// LoadReference reads the reading order and the glossary from hq through the records module. +func LoadReference(records Records) (*Reference, error) { + commit, err := records.Standing() + if err != nil { + return nil, fmt.Errorf("hq cannot be read through the records module (%v)", err) + } + readme, _, err := records.Read("02-DECISIONS/README.md") + if err != nil { + return nil, fmt.Errorf("the reading order cannot be read (%v)", err) + } + glossary, _, err := records.Read("00-META/glossary.md") + if err != nil { + return nil, fmt.Errorf("the glossary cannot be read (%v)", err) + } + ref := &Reference{Commit: commit, Topics: ReadingOrder(readme), Retired: RetiredWords(glossary), Lines: DaemonLines(daemonLinesFile)} + if len(ref.Topics) == 0 { + return nil, fmt.Errorf("hq's 02-DECISIONS/README.md at %s writes no reading order", short(commit)) + } + if len(ref.Retired) == 0 { + return nil, fmt.Errorf("hq's glossary at %s names no retired word: the words rule would pass on anything", short(commit)) + } + return ref, nil +} + +func short(commit string) string { + if len(commit) > 12 { + return commit[:12] + } + return commit +} + +// ---- running prose --------------------------------------------------------------------------------------- + +var ( + codeBlock = regexp.MustCompile("(?ms)^\\s*```.*?^\\s*```") + comment = regexp.MustCompile(`(?s)`) + blockQuote = regexp.MustCompile(`(?m)^\s*>.*$`) + codeSpan = regexp.MustCompile("`[^`\n]+`") + linkTarget = regexp.MustCompile(`\]\([^)\s]*\)`) + strike = regexp.MustCompile(`~~[^~\n]+~~`) + quoted = regexp.MustCompile(`"[^"\n]*"`) + curly = regexp.MustCompile(`“[^”]*”`) + single = regexp.MustCompile(`‘[^’\n]*’`) +) + +func blankOut(text string, res ...*regexp.Regexp) string { + for _, re := range res { + text = re.ReplaceAllStringFunc(text, func(m string) string { + return strings.Map(func(r rune) rune { + if r == '\n' { + return r + } + return ' ' + }, m) + }) + } + return text +} + +// Prose is the running prose of a text, as hq's words check reads it: code, comments, block quotes, quotations, +// struck-through words and link targets blanked. +func Prose(text string) string { + return blankOut(text, codeBlock, comment, blockQuote, codeSpan, linkTarget, strike, quoted, curly, single) +} + +// OwnWords is a text with what is not its own words blanked; code is kept, since a line to type is written there. +func OwnWords(text string) string { + return blankOut(text, comment, blockQuote, linkTarget, strike, quoted, curly, single) +} + +// wordPattern is a retired word matched whole and in any case, a space matching any run of space or hyphen, its +// plural matched as the word. +func wordPattern(word string) *regexp.Regexp { + parts := strings.Fields(word) + for i, p := range parts { + parts[i] = regexp.QuoteMeta(p) + } + return regexp.MustCompile(`(?i)` + strings.Join(parts, `[\s-]+`) + `(?:e?s)?`) +} + +func isWordRune(r rune) bool { + return r == '-' || r == '_' || unicode.IsLetter(r) || unicode.IsDigit(r) +} + +// wholeMatches are the matches of re in text that stand alone: no letter, digit, - or _ on either side. +func wholeMatches(re *regexp.Regexp, text string) []string { + var out []string + for _, loc := range re.FindAllStringIndex(text, -1) { + before, after := rune(' '), rune(' ') + if loc[0] > 0 { + before = []rune(text[:loc[0]])[len([]rune(text[:loc[0]]))-1] + } + if loc[1] < len(text) { + after = []rune(text[loc[1]:])[0] + } + if !isWordRune(before) && !isWordRune(after) { + out = append(out, text[loc[0]:loc[1]]) + } + } + return out +} + +// RetiredIn are the retired words a text uses in its running prose. +func (ref *Reference) RetiredIn(text string) []string { + prose := Prose(text) + var out []string + for _, w := range ref.Retired { + for _, m := range wholeMatches(wordPattern(w), prose) { + out = append(out, strings.Join(strings.Fields(m), " ")) + } + } + return out +} + +// LinesIn are the daemons' lines a text writes. +func (ref *Reference) LinesIn(text string) []string { + own := OwnWords(text) + var out []string + for _, l := range ref.Lines { + d, c := regexp.QuoteMeta(l.Daemon), regexp.QuoteMeta(l.Command) + var re *regexp.Regexp + if l.WithArgument { + re = regexp.MustCompile(`(?m)(?:^|` + "`" + `)[ \t]*(?:\$[ \t]*)?` + d + `\s+` + c + `\s+[^\s` + "`" + `]`) + if re.MatchString(own) { + out = append(out, l.Daemon+" "+l.Command) + } + continue + } + if len(wholeMatches(regexp.MustCompile(d+`\s+`+c), own)) > 0 { + out = append(out, l.Daemon+" "+l.Command) + } + } + return out +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/review_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/review_test.go new file mode 100644 index 00000000..bbb81cf8 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/review_test.go @@ -0,0 +1,276 @@ +package main + +import ( + "context" + "strings" + "testing" + "time" + + "git.novox.be/novox/mesh-sdk/go/asks" +) + +// to-be 55 §10: "A record is accepted only after a review by someone else, and, when it needs the operator, only +// on the warrant — the module's tests: a review by the author refused; a needs operator record unaccepted without +// the warrant, accepted with it." + +func proposed(t *testing.T, w *world, author string, needsOperator bool) int { + t.Helper() + n := w.reserved(t, author) + p := valid(n) + p.NeedsOperator = needsOperator + if _, err := w.k.Propose(author, p); err != nil { + t.Fatal(err) + } + return n +} + +func TestAReviewByTheAuthorIsRefused(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + n := proposed(t, w, "author@g14", false) + _, err := w.k.Review("author@g14", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}) + if err == nil || !strings.Contains(err.Error(), "someone who wrote none") { + t.Fatalf("a review by the author: %v", err) + } + if e := mustGet(t, s, Key{KindDecision, n}); e.Current.Status != StatusProposed || len(e.Current.Reviews) != 0 { + t.Fatalf("the author's review changed the record: %+v", e.Current) + } + }) +} + +// While the runtime proves no caller, a review cannot be told from one by the author, and none is taken: the +// caller is never read from the arguments. +func TestAReviewByAnUnprovenCallerIsRefused(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, Unproven, false) + _, err := w.k.Review(Unproven, ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}) + if err == nil || !strings.Contains(err.Error(), "issue 365") { + t.Fatalf("an unproven review: %v", err) + } + if mustGet(t, w.k.Store, Key{KindDecision, n}).Current.Status != StatusProposed { + t.Fatal("accepted on an unproven review") + } +} + +func TestAReviewBySomeoneElseAcceptsTheRecordAndMovesWhatItChanges(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + n := proposed(t, w, "author@g14", false) + // Changes asked first: it stays proposed, the findings kept. + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictChanges}); err == nil { + t.Fatal("changes without findings taken") + } + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictChanges, Findings: "§2: name the bound"}); err != nil { + t.Fatal(err) + } + e := mustGet(t, s, Key{KindDecision, n}) + if e.Current.Status != StatusProposed || len(e.Current.Reviews) != 1 || e.Current.Reviews[0].Findings != "§2: name the bound" { + t.Fatalf("after changes: %+v", e.Current) + } + got, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}) + if err != nil { + t.Fatal(err) + } + e = mustGet(t, s, Key{KindDecision, n}) + if got.Status != StatusAccepted || e.Current.Status != StatusAccepted || e.Current.Clauses[0].Status != ClauseInForce || + e.Current.AcceptedAt.IsZero() { + t.Fatalf("not accepted: %+v", e.Current) + } + // 0276 §1 is changed by the new record's clause 1, as its changes link says. + old := mustGet(t, s, Key{KindDecision, 276}) + c := old.Current.Clause(1) + if c.Status != ClauseChanged || len(c.ChangedBy) != 1 || c.ChangedBy[0] != itoaKey(n)+" §1" { + t.Fatalf("0276 §1: %+v", c) + } + if len(w.said.of(EventAccepted)) != 1 || len(w.said.of(EventChanged)) != 1 { + t.Fatalf("events: %v", w.said.events) + } + // holds of 0276 says what changed §1. + h, err := w.k.Holds("", &Key{KindDecision, 276}, 0) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(h.Clauses[0].Holds, "changed by "+itoaKey(n)) { + t.Fatalf("holds: %+v", h.Clauses[0]) + } + }) +} + +func itoaKey(n int) string { return Key{KindDecision, n}.String() } + +// A record that supersedes another moves it and every clause of it to superseded. +func TestASupersedingRecordSupersedesTheOlderOnAcceptance(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@g14") + p := valid(n) + p.Links = []LinkAsk{{Kind: LinkSupersedes, To: "0276"}} + p.Sections[0].Text = "The merge window is replaced." + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatal(err) + } + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + old := mustGet(t, w.k.Store, Key{KindDecision, 276}).Current + if old.Status != StatusSuperseded || old.Clause(4).Status != ClauseSuperseded || old.Extra["superseded-by"] != itoaKey(n) { + t.Fatalf("0276: %s, §4 %+v", old.Status, old.Clause(4)) + } + h, _ := w.k.Holds("", &Key{KindDecision, 276}, 0) + if len(h.Clauses) != 0 { + t.Fatalf("a superseded record holds %v", h.Clauses) + } +} + +func TestARecordThatNeedsTheOperatorIsAcceptedOnlyOnTheWarrant(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + n := proposed(t, w, "author@g14", true) + got, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}) + if err != nil { + t.Fatal(err) + } + if got.Status != StatusProposed || len(w.router.asked) != 1 { + t.Fatalf("after the review: %+v, asks %d", got, len(w.router.asked)) + } + ask := w.router.asked[0] + if err := ask.Check(w.clock); err != nil || ask.Options[0].Level != asks.Approve || ask.Options[1].Level != asks.Approve { + t.Fatalf("the ask: %v %+v", err, ask) + } + // Without the warrant: nothing is accepted, however often the router's record is read. + for i := 0; i < 2; i++ { + if _, err := w.k.Reconcile(); err != nil { + t.Fatal(err) + } + } + if e := mustGet(t, s, Key{KindDecision, n}); e.Current.Status != StatusProposed { + t.Fatalf("accepted with no warrant: %s", e.Current.Status) + } + h, _ := w.k.Holds("", &Key{KindDecision, n}, 0) + if len(h.Clauses) != 0 { + t.Fatalf("a record waiting for the operator holds %v", h.Clauses) + } + // The operator's Approve, on a channel that proves who answered: accepted, the warrant kept. + w.clock = w.clock.Add(10 * time.Minute) + warrant := w.router.answer(ask.ID, OptionApprove, w.clock) + said, err := w.k.Reconcile() + if err != nil { + t.Fatal(err) + } + e := mustGet(t, s, Key{KindDecision, n}) + if e.Current.Status != StatusAccepted || e.Current.Ask == nil || e.Current.Ask.Warrant != warrant.Says() || + e.Current.Ask.Said != AskApproved || len(said) != 1 { + t.Fatalf("on the warrant: %+v, ask %+v, said %v", e.Current.Status, e.Current.Ask, said) + } + vs, _ := s.Versions(context.Background(), Key{KindDecision, n}) + if last := vs[len(vs)-1]; !strings.Contains(last.Why, "warrant") || last.By != "the operator" { + t.Fatalf("the accepting version: %+v", last) + } + // Read again: decided once. + if said, _ := w.k.Reconcile(); len(said) != 0 { + t.Fatalf("decided twice: %v", said) + } + }) +} + +// A warrant for an ask whose record changed after it was asked does not hold: the act it bound is another version. +func TestAWarrantForAnotherVersionDoesNotAccept(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, "author@g14", true) + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + // The record is changed behind the ask, keeping the ask: what the operator was shown is not what is held. + ctx := context.Background() + e := mustGet(t, w.k.Store, Key{KindDecision, n}) + r := e.Current.Clone() + r.Clauses[1].Text = "**A batch is delivered by many walks.** One commit per repository." + if err := w.k.Store.Save(ctx, Version{By: "someone", Why: "changed behind the ask", Record: r}, true, false, e.Revision); err != nil { + t.Fatal(err) + } + w.router.answer(w.router.asked[0].ID, OptionApprove, w.clock.Add(time.Minute)) + if _, err := w.k.Reconcile(); err != nil { + t.Fatal(err) + } + e = mustGet(t, w.k.Store, Key{KindDecision, n}) + if e.Current.Status != StatusProposed || !strings.Contains(e.Current.Ask.Said, "does not hold") { + t.Fatalf("accepted on a warrant for another version: %s, %+v", e.Current.Status, e.Current.Ask) + } +} + +func TestTheOperatorsDeclineLeavesTheRecordProposed(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, "author@g14", true) + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + w.router.answer(w.router.asked[0].ID, OptionDecline, w.clock.Add(time.Minute)) + if _, err := w.k.Reconcile(); err != nil { + t.Fatal(err) + } + e := mustGet(t, w.k.Store, Key{KindDecision, n}) + if e.Current.Status != StatusProposed || e.Current.Ask.Said != "declined by the operator" { + t.Fatalf("%s %+v", e.Current.Status, e.Current.Ask) + } +} + +// An accepted record changes only by a progressive insight, which waits for its review while the accepted version +// is served. +func TestAnAcceptedRecordChangesOnlyByAnInsightReviewedLikeAProposal(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, "author@g14", false) + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + p := valid(n) + p.Sections[2].Text = "Merges land sooner, by about a minute." + if _, err := w.k.Propose("author@g14", p); err == nil || !strings.Contains(err.Error(), "progressive insight") { + t.Fatalf("an accepted record rewritten: %v", err) + } + p.Insight = "it said only that merges land sooner" + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatal(err) + } + e := mustGet(t, w.k.Store, Key{KindDecision, n}) + if e.Pending == nil || e.Current.Section("Consequences") != "Merges land sooner." { + t.Fatalf("the insight was served before its review: %+v", e.Current.Sections) + } + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + e = mustGet(t, w.k.Store, Key{KindDecision, n}) + if e.Pending != nil || e.Current.Status != StatusAccepted || len(e.Current.Insights) != 1 || + e.Current.Section("Consequences") != "Merges land sooner, by about a minute." { + t.Fatalf("after the insight's review: %+v", e.Current) + } + // 0276 §1 was changed once, by the acceptance; the insight keeps the link and changes it no further. + if c := mustGet(t, w.k.Store, Key{KindDecision, 276}).Current.Clause(1); len(c.ChangedBy) != 1 { + t.Fatalf("0276 §1 changed again by an insight: %v", c.ChangedBy) + } +} + +func TestReserveAsksTheTrackerAndANumberNeverProposedIsWithdrawn(t *testing.T) { + w := newWorld(t, newMemStore(), fixture()) + r, err := w.k.ReserveNumber("author@g14", KindResearch, "how a knowledge base is asked") + if err != nil { + t.Fatal(err) + } + if r.Number != 50 || len(w.tracker.asked) != 1 || w.tracker.asked[0]["kind"] != KindResearch || + w.tracker.asked[0]["for"] != "how a knowledge base is asked" { + t.Fatalf("%+v %v", r, w.tracker.asked) + } + if _, err := w.k.ReserveNumber("author@g14", "design", "x"); err == nil { + t.Fatal("a design number reserved") + } + w.clock = w.clock.Add(DefaultReservedFor + time.Hour) + if n, err := w.k.Withdraw(); err != nil || n != 1 { + t.Fatalf("withdrew %d (%v)", n, err) + } + e := mustGet(t, w.k.Store, Key{KindResearch, 50}) + if e.Current.Status != StatusWithdrawn || e.Current.ReservedBy != "author@g14" { + t.Fatalf("%+v", e.Current) + } + if _, err := w.k.Propose("author@g14", ProposeAsk{Kind: KindResearch, Number: 50, Title: "late", Status: StatusActive, + Sections: []Section{{Name: "00-overview.md", Text: "x"}}}); err == nil { + t.Fatal("a withdrawn number took a proposal") + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/rules.go b/modules/mesh-knowledge/cmd/mesh-knowledge/rules.go new file mode 100644 index 00000000..e3b62980 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/rules.go @@ -0,0 +1,577 @@ +package main + +// The rules a record is held to (novox/hq ADR 0290 rule 4, to-be 55 §4 and §5). +// +// **29 rules judge one record or its prose**, and `propose` refuses a record that breaks any of them, naming every +// one it breaks at once. They are hq's checks (00-META/checks/records.py, cycle.py, index.py, words.py) as they +// judge one record, moved to the write: 25 on its structure, 4 on its prose. Each has an id, said in the refusal, +// and one case in the table test (TestEveryPerRecordRuleRefusesAProposal). +// +// **4 rules judge the whole record set** (wholeSetRules) and run after every acceptance, and in the check +// process every few minutes: a failure is kept as a finding, which the module's health answers, so the mesh +// raises a condition naming the rule and the records. It does not undo the acceptance. + +import ( + "fmt" + "regexp" + "strings" +) + +// Rule is one rule that judges one record: its id and what it holds. +type Rule struct { + ID string + Holds string + // From is the hq check it comes from. + From string +} + +// PerRecordRules are the 29, in the order they are judged. +var PerRecordRules = []Rule{ + {"kind", "a record is a decision or a research effort", "cycle.py"}, + {"reserved", "its number was reserved for its kind and holds no other record", "cycle.py (duplicate numbers)"}, + {"title", "it has a title, one line of at most 200 characters", "records.py numbering"}, + {"topic", "a decision's topic is one of the reading order's", "records.py topics, index.py"}, + {"date", "its date is a day, YYYY-MM-DD", "records.py insights, cycle.py deferrals"}, + {"deciders", "a decision names who decided it", "records.py (deciders on every record)"}, + {"research-status", "a research effort is active, graduated or abandoned", "cycle.py research"}, + {"has-clause", "a decision has at least one clause", "records.py clauses"}, + {"clause-numbering", "its clauses are numbered from 1, in order, each once", "records.py clauses"}, + {"clause-lead", "every clause opens with a bold lead", "records.py clauses"}, + {"clause-size", "every clause says something, in at most 2000 bytes", "records.py clauses"}, + {"context", "a decision has a Context section", "the decision record's form"}, + {"consequences", "a decision has a Consequences section", "the decision record's form"}, + {"checked", "a decision says how it is checked", "AGENTS.md: a rule states how it is checked"}, + {"overview", "a research effort has its overview", "cycle.py research"}, + {"link-kind", "every link is of a known kind and goes out from the record or one of its clauses", "records.py links"}, + {"link-target", "every link names a record, and a clause, that exist", "records.py links, rests-on"}, + {"link-fits", "a link's kind fits its ends", "cycle.py research became, records.py supersession"}, + {"self-link", "no link points at the record itself", "records.py supersession"}, + {"rests-on", "a record it extends is accepted or proposed, never superseded or withdrawn", "records.py rests-on"}, + {"supersedes-accepted", "a record it supersedes or changes is accepted", "records.py supersession, partial-supersession"}, + {"changes-in-force", "a clause it changes is in force or changed, never superseded, proposed or drafted", "records.py partial-supersession"}, + {"live-citation", "a superseded record it cites is cited beside the record that replaced it", "records.py live-citation"}, + {"became", "a graduated research effort names what it became", "cycle.py research"}, + {"text-links", "every link its text writes to a decision record or research effort names one the knowledge base holds", "records.py links"}, + {"retired-word", "its prose uses no word the glossary retired", "words.py retired"}, + {"daemon-line", "it writes no daemon's line", "words.py lines"}, + {"deferral", "a paragraph that puts work off names the issue that tracks it", "cycle.py deferrals"}, + {"insight-form", "a progressive insight its text mentions is in the dated, marked form", "records.py insights"}, +} + +// Refusal is one rule a record breaks, and how. +type Refusal struct { + Rule string `json:"rule"` + Says string `json:"says"` +} + +func (r Refusal) String() string { return r.Rule + ": " + r.Says } + +// Lookup is the served version of a record the knowledge base holds, or nil. +type Lookup func(Key) *Record + +// MostClauseBytes is the most one clause may hold. +const MostClauseBytes = 2000 + +var ( + isDay = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`) + deferral = regexp.MustCompile(`(?i)until (?:it is |they are |the code (?:is )?)?renamed|until the code rename|` + + `\ba later rename|not renamed here|identifier to rename|left (?:for|to) (?:a )?later|not decided here|` + + `\bpostponed\b|future work|\bdeferred\b|later work|not built here|\ba later change`) + issueRef = regexp.MustCompile(`(?i)tracked (?:by|in|as)\s+\[?issue (\d{3,5})|\[?issue (\d{3,5})\]?(?:\([^)]*\))?\s+tracks\b`) + insightPhrase = regexp.MustCompile(`(?i)progressive insights?`) + insightMarker = regexp.MustCompile(`\*\*Progressive insights?[ \t]*[—–-][ \t]*(\d{4}-\d{2}-\d{2})[^*\n]*\*\*`) +) + +// decisionSection is the text of a record's Decision section. +func decisionSection(r *Record) (string, bool) { + for _, s := range r.Sections { + if s.Name == "Decision" || strings.HasPrefix(s.Name, "Decision ") || strings.HasPrefix(s.Name, "Decision:") { + return s.Text, true + } + } + return "", false +} + +func hasSection(r *Record, names ...string) bool { + for _, s := range r.Sections { + for _, n := range names { + if strings.EqualFold(strings.TrimSpace(s.Name), n) && strings.TrimSpace(s.Text) != "" { + return true + } + } + } + return false +} + +// allText is every word of a record an agent reads: its title, clauses and sections. +func allText(r *Record) string { + var b strings.Builder + b.WriteString(r.Title + "\n\n") + for _, c := range r.Clauses { + b.WriteString(c.Text + "\n\n") + } + for _, s := range r.Sections { + b.WriteString(s.Text + "\n\n") + } + return b.String() +} + +// JudgeRecord holds a proposed record to the 29 rules. held is the entry its number holds (a reservation, the +// proposal being revised, or the accepted record an insight corrects); lookup reads every other record. +func JudgeRecord(r *Record, held *Entry, lookup Lookup, ref *Reference) []Refusal { + var out []Refusal + refuse := func(rule, format string, a ...any) { + if Holds(rule, r.Kind, r.Date) { + out = append(out, Refusal{Rule: rule, Says: fmt.Sprintf(format, a...)}) + } + } + decision := r.Kind == KindDecision + if r.Kind != KindDecision && r.Kind != KindResearch { + refuse("kind", "%q is neither decision nor research", r.Kind) + } + switch { + case held == nil: + refuse("reserved", "%s was never reserved: reserve gives the number", r.Key()) + case held.Current.Status == StatusWithdrawn: + refuse("reserved", "%s was reserved and withdrawn: reserve a new number", r.Key()) + } + title := strings.TrimSpace(r.Title) + if title == "" || strings.Contains(title, "\n") || len([]rune(title)) > 200 { + refuse("title", "the title is empty, more than one line, or longer than 200 characters") + } + if decision && !contains(ref.Topics, r.Topic) { + refuse("topic", "%q is not a topic of the reading order: %s", r.Topic, strings.Join(ref.Topics, ", ")) + } + if !isDay.MatchString(r.Date) { + refuse("date", "%q is not a day written YYYY-MM-DD", r.Date) + } + if decision && len(r.Deciders) == 0 { + refuse("deciders", "a decision names who decided it") + } + if !decision && !contains(ResearchStatuses, r.Effort) { + refuse("research-status", "%q is not active, graduated or abandoned", r.Effort) + } + if decision && len(r.Clauses) == 0 { + refuse("has-clause", "a decision is made of clauses, and this one has none") + } + for i, c := range r.Clauses { + if c.N != i+1 { + refuse("clause-numbering", "clause %d stands %d. in order: clauses are numbered 1, 2, 3 … each once", c.N, i+1) + } + if c.Lead() == "" { + refuse("clause-lead", "clause %d does not open with a bold lead (**What is decided.** …)", c.N) + } + if strings.TrimSpace(c.Text) == "" || len(c.Text) > MostClauseBytes { + refuse("clause-size", "clause %d is empty or longer than %d bytes", c.N, MostClauseBytes) + } + } + if decision && !hasSection(r, "Context") { + refuse("context", "a decision has a Context section: why it was decided") + } + if decision && !hasSection(r, "Consequences") { + refuse("consequences", "a decision has a Consequences section: what follows from it, and what got harder") + } + if decision && !hasSection(r, "How it is checked") { + refuse("checked", "a decision has a section \"How it is checked\": a rule says how it is verified") + } + if !decision && !hasSection(r, "00-overview.md", "overview") { + refuse("overview", "a research effort has its overview, a section named 00-overview.md") + } + self := r.Key() + superseders := map[Key]bool{} + for _, l := range r.Links { + if to, ok := l.To.Record(); ok && l.Kind == LinkSupersedes { + superseders[to] = true + } + } + cited := map[Key]bool{} + for _, l := range r.Links { + if to, ok := l.To.Record(); ok { + cited[to] = true + } + } + for _, l := range r.Links { + if !contains(LinkKinds, l.Kind) { + refuse("link-kind", "%q is not a kind of link: %s", l.Kind, strings.Join(LinkKinds, ", ")) + continue + } + if l.From > 0 && r.Clause(l.From) == nil { + refuse("link-kind", "the link %s goes out from clause %d, which the record does not have", l, l.From) + } + to, isRecord := l.To.Record() + if isRecord && to == self { + refuse("self-link", "the link %s points at the record itself", l) + continue + } + var target *Record + if isRecord { + target = lookup(to) + if target == nil || target.Status == StatusReserved || target.Status == StatusWithdrawn { + refuse("link-target", "the link %s names %s, which the knowledge base does not hold", l, to) + continue + } + if l.To.Clause > 0 && target.Clause(l.To.Clause) == nil { + refuse("link-target", "the link %s names clause %d, which %s does not have", l, l.To.Clause, to) + continue + } + } + kept := held != nil && held.Current.Accepted() && hasLink(held.Current, l) + switch { + case kept: + // An insight keeps what the accepted record linked, as hq wrote it: a change hq named by record waits + // for its clause to be judged. + case l.Kind == LinkBecame: + if r.Kind != KindResearch || !(l.To.Kind == KindDecision || l.To.Kind == TargetDesign) || l.To.Clause > 0 { + refuse("link-fits", "became goes from a research effort to a decision record or a design: not %s", l) + } + case l.Kind == LinkChanges: + if !decision || l.To.Kind != KindDecision || l.To.Clause == 0 { + refuse("link-fits", "changes goes from a decision (or a clause of it) to a clause of another decision, named as 0276 §4: not %s", l) + } + case l.Kind == LinkSupersedes || l.Kind == LinkExtends: + if r.Kind != l.To.Kind || !isRecord || l.To.Clause > 0 { + refuse("link-fits", "%s goes from a record to a record of its kind: not %s", l.Kind, l) + } + } + if target == nil { + continue + } + if held != nil && held.Current.Accepted() && hasLink(held.Current, l) { + // An insight keeps what the accepted record linked: judged when it was accepted. + continue + } + switch l.Kind { + case LinkExtends: + if target.Status == StatusSuperseded && !superseders[to] { + refuse("rests-on", "%s extends %s, which is superseded", self, to) + } + case LinkSupersedes: + if !target.Accepted() || target.Status == StatusSuperseded { + refuse("supersedes-accepted", "%s supersedes %s, which is %s, not accepted", self, to, target.Status) + } + case LinkChanges: + if !target.Accepted() || target.Status == StatusSuperseded { + refuse("supersedes-accepted", "%s changes %s, which is %s, not accepted", self, to, target.Status) + } else if c := target.Clause(l.To.Clause); c != nil { + if (c.Status != ClauseInForce && c.Status != ClauseChanged) || c.Source == SourceDrafted { + refuse("changes-in-force", "%s changes %s §%d, which is %s (%s)", self, to, l.To.Clause, c.Status, c.Source) + } + } + case LinkCites: + if target.Status == StatusSuperseded { + named := false + for _, by := range supersedersOf(target) { + named = named || cited[by] + } + if !named { + refuse("live-citation", "%s cites %s, which is superseded, and not the record that replaced it", self, to) + } + } + } + } + if r.Kind == KindResearch && r.Effort == StatusGraduated { + became := false + for _, l := range r.Links { + became = became || l.Kind == LinkBecame + } + if !became { + refuse("became", "a graduated research effort names what it became, as a became link") + } + } + for _, s := range r.Sections { + for _, m := range mdLink.FindAllStringSubmatch(s.Text, -1) { + target := m[1] + var k Key + switch { + case researchFolder.MatchString(target): + n := atoi(researchFolder.FindStringSubmatch(target)[1]) + k = Key{KindResearch, n} + case adrFile.MatchString(target) && !strings.Contains(target, "04-ISSUES") && !strings.Contains(target, "03-DESIGN"): + k = Key{KindDecision, atoi(adrFile.FindStringSubmatch(target)[1])} + default: + continue + } + if k == self { + continue + } + if t := lookup(k); t == nil || t.Status == StatusReserved || t.Status == StatusWithdrawn { + refuse("text-links", "the section %s links %s, which the knowledge base does not hold", s.Name, k) + } + } + } + text := allText(r) + if words := ref.RetiredIn(text); len(words) > 0 { + refuse("retired-word", "it uses %s, retired by the glossary", quoteAll(unique(words))) + } + if lines := ref.LinesIn(text); len(lines) > 0 { + refuse("daemon-line", "it writes %s, a daemon's line: the operator types nox", quoteAll(unique(lines))) + } + for _, s := range r.Sections { + if strings.Contains(strings.ToLower(s.Name), "option") || strings.Contains(strings.ToLower(s.Name), "rejected") { + continue + } + for _, p := range paragraphsOf(s.Text) { + if m := deferral.FindString(p); m != "" && !issueRef.MatchString(p) { + refuse("deferral", "the section %s puts work off (%q) and names no issue that tracks it (\"tracked by issue N\")", s.Name, m) + } + } + } + for _, c := range r.Clauses { + if m := deferral.FindString(c.Text); m != "" && !issueRef.MatchString(c.Text) { + refuse("deferral", "clause %d puts work off (%q) and names no issue that tracks it (\"tracked by issue N\")", c.N, m) + } + } + marked := insightMarker.FindAllStringSubmatchIndex(text, -1) + for _, loc := range insightPhrase.FindAllStringIndex(text, -1) { + inside := false + for _, mk := range marked { + inside = inside || (mk[0] <= loc[0] && loc[1] <= mk[1]) + } + line := lineAround(text, loc[0]) + if inside || strings.Contains(line, "](") || strings.HasPrefix(strings.TrimSpace(line), "#") { + continue + } + refuse("insight-form", "it mentions a progressive insight outside the dated marked form **Progressive insight — YYYY-MM-DD.**") + break + } + for _, mk := range insightMarker.FindAllStringSubmatch(text, -1) { + if isDay.MatchString(r.Date) && mk[1] < r.Date { + refuse("insight-form", "a progressive insight dated %s predates the record (%s)", mk[1], r.Date) + } + } + return out +} + +func hasLink(r *Record, l Link) bool { + for _, x := range r.Links { + if x.Kind == l.Kind && x.From == l.From && x.To == l.To { + return true + } + } + return false +} + +func lineAround(text string, at int) string { + start := strings.LastIndex(text[:at], "\n") + 1 + end := strings.Index(text[at:], "\n") + if end < 0 { + return text[start:] + } + return text[start : at+end] +} + +// paragraphsOf cuts a text at blank lines and a list's items. +func paragraphsOf(text string) []string { + var out []string + for _, chunk := range strings.Split(text, "\n\n") { + for _, item := range regexp.MustCompile(`\n(?:[-*] |\d+\. )`).Split(chunk, -1) { + if strings.TrimSpace(item) != "" { + out = append(out, item) + } + } + } + return out +} + +// supersedersOf are the records a record says superseded it: its superseded-by, a file name as hq wrote it or +// "ADR NNNN" as an acceptance here writes it. +func supersedersOf(r *Record) []Key { + var out []Key + for _, by := range strings.Split(r.Extra["superseded-by"], ",") { + by = strings.TrimSpace(by) + if m := adrFile.FindStringSubmatch(by); m != nil { + out = append(out, Key{KindDecision, atoi(m[1])}) + } else if ref, err := ParseRef(by); err == nil { + if k, ok := ref.Record(); ok { + out = append(out, k) + } + } + } + return out +} + +func atoi(s string) int { + n := 0 + for _, c := range s { + n = n*10 + int(c-'0') + } + return n +} + +func contains(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} + +func unique(list []string) []string { + seen := map[string]bool{} + var out []string + for _, s := range list { + if !seen[strings.ToLower(s)] { + seen[strings.ToLower(s)] = true + out = append(out, s) + } + } + return out +} + +func quoteAll(list []string) string { + q := make([]string, len(list)) + for i, s := range list { + q[i] = fmt.Sprintf("%q", s) + } + return strings.Join(q, ", ") +} + +// ---- the whole record set ---------------------------------------------------------------------------------- + +// WholeSetRules are the 4 rules that judge the whole record set. +var WholeSetRules = []Rule{ + {"cited", "a decision accepted here is cited by another record or names a home in hq", "cycle.py decisions"}, + {"supersession", "a superseded record is named by the record that superseded it, and every record an accepted record supersedes is superseded", "records.py supersession"}, + {"partial", "a changed clause is changed by an accepted record's changes link, and every clause an accepted record changes is changed or superseded", "records.py partial-supersession"}, + {"rests-on-live", "no accepted record extends a superseded one, but the record that superseded it", "records.py rests-on"}, +} + +// JudgeSet holds the served versions of every record to the 4 whole-set rules. An imported record was held to +// hq's own checks at the commit it was read at, so the first rule judges only a record accepted here. +func JudgeSet(records []*Record) []Finding { + byKey := map[Key]*Record{} + for _, r := range records { + byKey[r.Key()] = r + } + inbound := map[Key][]struct { + from *Record + l Link + }{} + for _, r := range records { + if !r.Accepted() && r.Status != StatusSuperseded { + continue + } + for _, l := range r.Links { + if to, ok := l.To.Record(); ok { + inbound[to] = append(inbound[to], struct { + from *Record + l Link + }{r, l}) + } + } + } + var out []Finding + add := func(rule, says string, keys ...Key) { + names := make([]string, len(keys)) + for i, k := range keys { + names[i] = k.String() + } + out = append(out, Finding{Rule: rule, Records: names, Says: says}) + } + for _, r := range records { + k := r.Key() + if r.Kind == KindDecision && r.Status == StatusAccepted && !r.AcceptedAt.IsZero() && len(r.Homes) == 0 { + cited := false + for _, in := range inbound[k] { + cited = cited || in.from.Key() != k + } + if !cited { + add("cited", fmt.Sprintf("%s is accepted and nothing cites it: no record links it and it names no home", k), k) + } + } + if r.Status == StatusSuperseded { + by := false + for _, in := range inbound[k] { + // The record that superseded it may since have been superseded itself, as hq's 0125 by 0127 by 0131. + by = by || in.l.Kind == LinkSupersedes + } + if !by { + add("supersession", fmt.Sprintf("%s is superseded and no record says it supersedes it", k), k) + } + } + if r.Accepted() { + for _, l := range r.Links { + to, ok := l.To.Record() + target := byKey[to] + if !ok || target == nil { + continue + } + switch l.Kind { + case LinkSupersedes: + if target.Status != StatusSuperseded { + add("supersession", fmt.Sprintf("%s supersedes %s, which is %s", k, to, target.Status), k, to) + } + case LinkChanges: + if l.To.Clause == 0 { + continue + } + if c := target.Clause(l.To.Clause); c != nil && c.Status != ClauseChanged && c.Status != ClauseSuperseded { + add("partial", fmt.Sprintf("%s changes %s §%d, which is %s", k, to, l.To.Clause, c.Status), k, to) + } + case LinkExtends: + if target.Status == StatusSuperseded { + superseder := false + for _, in := range inbound[to] { + superseder = superseder || (in.l.Kind == LinkSupersedes && in.from.Key() == k) + } + if !superseder { + add("rests-on-live", fmt.Sprintf("%s extends %s, which is superseded", k, to), k, to) + } + } + } + } + } + for _, c := range r.Clauses { + if c.Status != ClauseChanged { + continue + } + by := false + for _, in := range inbound[k] { + by = by || (in.l.Kind == LinkChanges && in.l.To.Clause == c.N && in.from.Accepted()) + } + if !by { + add("partial", fmt.Sprintf("%s §%d is changed and no accepted record changes it", k, c.N), k) + } + } + } + return out +} + +// RuleFrom is the day each dated rule began to hold a record, as hq's checks hold it: a record dated earlier was +// never held to it, and a dated correction of it (a progressive insight, which keeps the record's date) is not. +// words.py held the retired words from 2026-10-07 and the daemons' lines from 2026-10-09; cycle.py the deferrals +// from 2026-10-09; records.py the clauses from 2026-10-11 (ADR 0290 rule 3). The decision record's sections, and +// a superseded record cited alone, were held by no check of hq's (words.py and live-citation never read decision +// records): they hold from the day the knowledge base takes the write, 2026-10-11. Every other rule holds every +// record, as hq's checks held it. +var RuleFrom = map[string]string{ + "retired-word": "2026-10-07", + "daemon-line": "2026-10-09", + "deferral": "2026-10-09", + "has-clause": "2026-10-11", + "clause-numbering": "2026-10-11", + "clause-lead": "2026-10-11", + "clause-size": "2026-10-11", + "context": "2026-10-11", + "consequences": "2026-10-11", + "checked": "2026-10-11", + "live-citation": "2026-10-11", +} + +// DecisionFrom overrides RuleFrom for a decision record: hq's words check never read 02-DECISIONS ("a decision +// record keeps the words it was written with"), so its retired words and daemons' lines hold a decision only from +// the day the knowledge base takes the write. +var DecisionFrom = map[string]string{ + "retired-word": "2026-10-11", + "daemon-line": "2026-10-11", +} + +// Holds says whether a rule holds a record of this kind and date. A record with no day is held to every rule. +func Holds(rule, kind, date string) bool { + from, dated := RuleFrom[rule] + if d, ok := DecisionFrom[rule]; ok && kind == KindDecision { + from = d + } + return !dated || !isDay.MatchString(date) || date >= from +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/rules_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/rules_test.go new file mode 100644 index 00000000..6bdbb564 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/rules_test.go @@ -0,0 +1,221 @@ +package main + +import ( + "context" + "fmt" + "strings" + "testing" +) + +// to-be 55 §10: "A proposal that breaks a per-record rule is refused, naming every reason — the module's table +// test, one case per rule of §4." Each case takes a proposal every rule takes and breaks one rule; the proposal is +// refused, the refusal names that rule, and nothing is written. A rule with no case fails the test. +func TestEveryPerRecordRuleRefusesAProposal(t *testing.T) { + cases := []struct { + rule string + break_ func(p *ProposeAsk, w *world, t *testing.T) + }{ + {"kind", func(p *ProposeAsk, w *world, t *testing.T) { + // A kind the verb refuses before the rules: judged by the rules when a record of another kind is held. + p.Kind = "design" + }}, + {"reserved", func(p *ProposeAsk, w *world, t *testing.T) { p.Number = 9999 }}, + {"title", func(p *ProposeAsk, w *world, t *testing.T) { p.Title = "two\nlines" }}, + {"topic", func(p *ProposeAsk, w *world, t *testing.T) { p.Topic = "gardening" }}, + {"date", func(p *ProposeAsk, w *world, t *testing.T) { p.Date = "11 October" }}, + {"deciders", func(p *ProposeAsk, w *world, t *testing.T) { p.Deciders = nil }}, + {"research-status", func(p *ProposeAsk, w *world, t *testing.T) { + p.Kind, p.Number = KindResearch, w.reservedResearch(t) + p.Status, p.Links = "wondering", nil + p.Sections = []Section{{Name: "00-overview.md", Text: "An effort."}} + p.Clauses = nil + }}, + {"has-clause", func(p *ProposeAsk, w *world, t *testing.T) { p.Clauses = nil; p.Links = p.Links[:1] }}, + {"clause-numbering", func(p *ProposeAsk, w *world, t *testing.T) { + p.Clauses = []string{"1. **One.** A.", "3. **Three.** C."} + }}, + {"clause-lead", func(p *ProposeAsk, w *world, t *testing.T) { p.Clauses[1] = "A batch is delivered by one walk." }}, + {"clause-size", func(p *ProposeAsk, w *world, t *testing.T) { + p.Clauses[1] = "**Long.** " + strings.Repeat("word ", 500) + }}, + {"context", func(p *ProposeAsk, w *world, t *testing.T) { p.Sections = p.Sections[1:] }}, + {"consequences", func(p *ProposeAsk, w *world, t *testing.T) { p.Sections = append(p.Sections[:2], p.Sections[3]) }}, + {"checked", func(p *ProposeAsk, w *world, t *testing.T) { p.Sections = p.Sections[:3] }}, + {"overview", func(p *ProposeAsk, w *world, t *testing.T) { + p.Kind, p.Number, p.Status, p.Clauses, p.Links = KindResearch, w.reservedResearch(t), StatusActive, nil, nil + p.Sections = []Section{{Name: "01-findings.md", Text: "Found."}} + }}, + {"link-kind", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: "admires", To: "0276"}) + }}, + {"link-target", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: "0007"}) + }}, + {"link-fits", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkBecame, To: "0276"}) + }}, + {"self-link", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: itoa4(p.Number)}) + }}, + {"rests-on", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkExtends, To: "0127"}) + }}, + {"supersedes-accepted", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkSupersedes, To: "0127"}) + }}, + {"changes-in-force", func(p *ProposeAsk, w *world, t *testing.T) { + // 0131 superseded 0127, so 0127's clauses are superseded; a changes link to one names a clause no longer in force. + w.supersededClause(t) + p.Links = append(p.Links, LinkAsk{Kind: LinkChanges, To: "0011 §1", From: 2}) + }}, + {"live-citation", func(p *ProposeAsk, w *world, t *testing.T) { + p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: "0127"}) + }}, + {"became", func(p *ProposeAsk, w *world, t *testing.T) { + p.Kind, p.Number, p.Status, p.Clauses, p.Links = KindResearch, w.reservedResearch(t), StatusGraduated, nil, nil + p.Sections = []Section{{Name: "00-overview.md", Text: "An effort that graduated."}} + }}, + {"text-links", func(p *ProposeAsk, w *world, t *testing.T) { + p.Sections[0].Text += " See [ADR 0007](0007-a-record-nobody-holds.md)." + }}, + {"retired-word", func(p *ProposeAsk, w *world, t *testing.T) { + p.Sections[2].Text = "The control plane is quicker." + }}, + {"daemon-line", func(p *ProposeAsk, w *world, t *testing.T) { + p.Sections[3].Text = "Run `mesh-controller doctor` after the merge." + }}, + {"deferral", func(p *ProposeAsk, w *world, t *testing.T) { + p.Sections[2].Text = "The bound itself is left for later." + }}, + {"insight-form", func(p *ProposeAsk, w *world, t *testing.T) { + p.Sections[2].Text = "A progressive insight corrected the count." + }}, + } + covered := map[string]bool{} + for _, c := range cases { + covered[c.rule] = true + } + for _, r := range PerRecordRules { + if !covered[r.ID] { + t.Errorf("the rule %s has no case", r.ID) + } + } + if len(PerRecordRules) != 29 { + t.Errorf("%d per-record rules; ADR 0290 counts 29", len(PerRecordRules)) + } + for _, c := range cases { + t.Run(c.rule, func(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@test") + p := valid(n) + c.break_(&p, w, t) + if c.rule == "kind" { + // The verb refuses a kind it does not hold before the rules; the rule itself is held by JudgeRecord. + ref, _ := w.k.reference() + r := &Record{Kind: "design", Number: n, Title: "x", Date: "2026-10-11"} + if !names(JudgeRecord(r, mustGet(t, w.k.Store, Key{KindDecision, n}), w.k.lookup(context.Background()), ref), "kind") { + t.Fatal("a record of an unknown kind was not refused by the kind rule") + } + if _, err := w.k.Propose("author@test", p); err == nil { + t.Fatal("propose took a record of an unknown kind") + } + return + } + before := versionsOf(t, w, Key{p.Kind, p.Number}) + _, err := w.k.Propose("author@test", p) + if err == nil { + t.Fatalf("a proposal breaking %s was taken", c.rule) + } + if !strings.Contains(err.Error(), "- "+c.rule+":") { + t.Fatalf("the refusal does not name %s: %v", c.rule, err) + } + if after := versionsOf(t, w, Key{p.Kind, p.Number}); after != before { + t.Fatalf("a refused proposal wrote a version (%d, was %d)", after, before) + } + }) + } +} + +// One proposal breaking three rules is refused naming all three, at once. +func TestARefusalNamesEveryReasonAtOnce(t *testing.T) { + w := imported(t, newMemStore()) + p := valid(w.reserved(t, "author@test")) + p.Topic = "gardening" + p.Clauses[1] = "no lead here." + p.Sections[2].Text = "The control plane is quicker." + _, err := w.k.Propose("author@test", p) + if err == nil { + t.Fatal("taken") + } + for _, rule := range []string{"topic", "clause-lead", "retired-word"} { + if !strings.Contains(err.Error(), "- "+rule+":") { + t.Errorf("the refusal does not name %s: %v", rule, err) + } + } +} + +// The proposal every case breaks is taken as it is: each case fails for its own rule, not for the base's. +func TestTheValidProposalIsTaken(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + n := w.reserved(t, "author@test") + got, err := w.k.Propose("author@test", valid(n)) + if err != nil { + t.Fatal(err) + } + e := mustGet(t, s, Key{KindDecision, n}) + if got.Status != StatusProposed || e.Current.Status != StatusProposed || len(e.Current.Clauses) != 2 || + e.Current.Clauses[0].Status != ClauseProposed || e.Current.Author != "author@test" { + t.Fatalf("%+v / %+v", got, e.Current) + } + if len(w.said.of(EventProposed)) != 1 || len(w.tracker.asked) != 1 || w.tracker.asked[0]["kind"] != KindDecision { + t.Fatalf("events %v, tracker %v", w.said.events, w.tracker.asked) + } + }) +} + +func names(refusals []Refusal, rule string) bool { + for _, r := range refusals { + if r.Rule == rule { + return true + } + } + return false +} + +func versionsOf(t *testing.T, w *world, k Key) int { + t.Helper() + e, err := w.k.Store.Get(context.Background(), k) + if err != nil { + t.Fatal(err) + } + if e == nil { + return 0 + } + return e.Versions +} + +func itoa4(n int) string { return fmt.Sprintf("%04d", n) } + +func (w *world) reservedResearch(t *testing.T) int { + t.Helper() + r, err := w.k.ReserveNumber("author@test", KindResearch, "a test effort") + if err != nil { + t.Fatal(err) + } + return r.Number +} + +// supersededClause gives 0011, accepted, one clause a later record superseded, so a change to that clause names +// one no longer in force. +func (w *world) supersededClause(t *testing.T) { + t.Helper() + ctx := context.Background() + e := mustGet(t, w.k.Store, Key{KindDecision, 11}) + r := e.Current.Clone() + r.Clauses = []Clause{{N: 1, Text: "**Generated.** Managed files are generated.", Status: ClauseSuperseded, Source: SourceWritten, + ChangedBy: []string{"ADR 0012"}}} + if err := w.k.Store.Save(ctx, Version{By: "test", Why: "superseded for the test", Record: r}, true, false, e.Revision); err != nil { + t.Fatal(err) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/schema.sql b/modules/mesh-knowledge/cmd/mesh-knowledge/schema.sql new file mode 100644 index 00000000..3a2f6366 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/schema.sql @@ -0,0 +1,73 @@ +-- The consolidated schema of the knowledge base: what a fresh database is given at once, and what every +-- migration in migrations/ applied in order comes to (the test TestTheConsolidatedSchemaIsTheMigrationsApplied +-- holds the two equal). Consolidated through migration 0001. + +create table record ( + kind text not null check (kind in ('decision', 'research')), + number integer not null check (number > 0), + status text not null, + topic text not null default '', + current_version integer not null, + pending_version integer, + revision bigint not null default 1, + search tsvector not null default ''::tsvector, + primary key (kind, number) +); + +create index record_search on record using gin (search); +create index record_topic on record (topic); + +create table version ( + kind text not null, + number integer not null, + n integer not null check (n > 0), + by text not null, + at timestamptz not null, + why text not null, + insight boolean not null default false, + body jsonb not null, + primary key (kind, number, n), + foreign key (kind, number) references record (kind, number) +); + +create table clause ( + kind text not null, + number integer not null, + n integer not null check (n > 0), + text text not null, + status text not null, + source text not null, + search tsvector not null, + primary key (kind, number, n), + foreign key (kind, number) references record (kind, number) +); + +create index clause_search on clause using gin (search); + +create table link ( + from_kind text not null, + from_number integer not null, + from_clause integer not null default 0, + kind text not null, + to_kind text not null, + to_number integer not null default 0, + to_clause integer not null default 0, + to_path text not null default '', + body jsonb not null, + foreign key (from_kind, from_number) references record (kind, number) +); + +create index link_from on link (from_kind, from_number); +create index link_to on link (to_kind, to_number); + +create table finding ( + rule text not null, + records text[] not null, + says text not null, + at timestamptz not null +); + +create table told ( + topic text primary key, + at timestamptz not null default now() +); diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/store.go b/modules/mesh-knowledge/cmd/mesh-knowledge/store.go new file mode 100644 index 00000000..bbba2dbe --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/store.go @@ -0,0 +1,319 @@ +package main + +// Where the records are kept (novox/hq ADR 0290 rule 7): a database in the store, provided to the module +// (pgstore.go). The rules of the knowledge base are written once, against this interface; memStore is the same +// interface in memory, for the tests of those rules and for the terminal's dry run. + +import ( + "context" + "errors" + "fmt" + "sort" + "strings" + "sync" + "time" + "unicode" +) + +// ErrExists is a create refused because the record is already held. +var ErrExists = errors.New("the record is already held") + +// ErrChanged is a write refused because the entry changed since it was read. +var ErrChanged = errors.New("the record was changed by another write since it was read") + +// Hit is one place a question matched: a clause of a record (Clause > 0) or a record as a whole. +type Hit struct { + Key Key + Clause int + Rank float64 +} + +// Inbound is a link held on another record that points at this one. +type Inbound struct { + From Key + Link Link +} + +// Finding is one failure of a rule that judges the whole record set. +type Finding struct { + Rule string `json:"rule"` + Records []string `json:"records"` + Says string `json:"says"` + At time.Time `json:"at"` +} + +// Store keeps the records, every version of each, and the findings of the whole-set rules. +type Store interface { + // Get is one record's entry, or nil. + Get(ctx context.Context, k Key) (*Entry, error) + // All is every record's entry, in kind then number order. + All(ctx context.Context) ([]*Entry, error) + // Create writes a record that is not held, with its first version; ErrExists when it is. + Create(ctx context.Context, v Version) error + // Save writes a new version of a held record. current says whether it becomes the version served (else it + // is the one pending over it); clearPending drops the pending one. revision is the entry's revision as read: + // ErrChanged when it moved. + Save(ctx context.Context, v Version, current, clearPending bool, revision int64) error + // Versions is every version of a record, oldest first. + Versions(ctx context.Context, k Key) ([]Version, error) + // Search ranks the clauses and records of the served versions against a question. + Search(ctx context.Context, question string, limit int) ([]Hit, error) + // Inbound is every link of another served record that points at this one. + Inbound(ctx context.Context, k Key) ([]Inbound, error) + // SetFindings replaces the whole-set findings; Findings reads them. + SetFindings(ctx context.Context, f []Finding) error + Findings(ctx context.Context) ([]Finding, error) + // Told records that the operator was told about a topic's clauses, once; false when it was already. + Told(ctx context.Context, topic string) (bool, error) +} + +// memStore is the Store in memory. +type memStore struct { + mu sync.Mutex + entries map[Key]*memEntry + findings []Finding + told map[string]bool +} + +type memEntry struct { + versions []Version + current int + pending int + revision int64 +} + +func newMemStore() *memStore { + return &memStore{entries: map[Key]*memEntry{}, told: map[string]bool{}} +} + +func (e *memEntry) entry() *Entry { + out := &Entry{CurrentVersion: e.current, PendingVersion: e.pending, Versions: len(e.versions), Revision: e.revision} + out.Current = e.versions[e.current-1].Record.Clone() + if e.pending > 0 { + out.Pending = e.versions[e.pending-1].Record.Clone() + } + return out +} + +func (s *memStore) Get(_ context.Context, k Key) (*Entry, error) { + s.mu.Lock() + defer s.mu.Unlock() + e := s.entries[k] + if e == nil { + return nil, nil + } + return e.entry(), nil +} + +func (s *memStore) All(context.Context) ([]*Entry, error) { + s.mu.Lock() + defer s.mu.Unlock() + keys := make([]Key, 0, len(s.entries)) + for k := range s.entries { + keys = append(keys, k) + } + sortKeys(keys) + out := make([]*Entry, 0, len(keys)) + for _, k := range keys { + out = append(out, s.entries[k].entry()) + } + return out, nil +} + +func sortKeys(keys []Key) { + sort.Slice(keys, func(i, j int) bool { + if keys[i].Kind != keys[j].Kind { + return keys[i].Kind < keys[j].Kind + } + return keys[i].Number < keys[j].Number + }) +} + +func (s *memStore) Create(_ context.Context, v Version) error { + s.mu.Lock() + defer s.mu.Unlock() + k := v.Record.Key() + if s.entries[k] != nil { + return ErrExists + } + v.N = 1 + v.Record = v.Record.Clone() + s.entries[k] = &memEntry{versions: []Version{v}, current: 1, revision: 1} + return nil +} + +func (s *memStore) Save(_ context.Context, v Version, current, clearPending bool, revision int64) error { + s.mu.Lock() + defer s.mu.Unlock() + e := s.entries[v.Record.Key()] + if e == nil { + return fmt.Errorf("%s is not held", v.Record.Key()) + } + if e.revision != revision { + return ErrChanged + } + v.N = len(e.versions) + 1 + v.Record = v.Record.Clone() + e.versions = append(e.versions, v) + if current { + e.current = v.N + } else { + e.pending = v.N + } + if clearPending { + e.pending = 0 + } + e.revision++ + return nil +} + +func (s *memStore) Versions(_ context.Context, k Key) ([]Version, error) { + s.mu.Lock() + defer s.mu.Unlock() + e := s.entries[k] + if e == nil { + return nil, nil + } + out := make([]Version, len(e.versions)) + for i, v := range e.versions { + v.Record = v.Record.Clone() + out[i] = v + } + return out, nil +} + +// Search in memory: the question's words, stemmed crudely, counted in each clause and each record's title, topic +// and sections; the title and a clause weigh more. Enough to test the answers' shape; the store's ranking is the +// database's full-text search. +func (s *memStore) Search(_ context.Context, question string, limit int) ([]Hit, error) { + words := questionWords(question) + if len(words) == 0 { + return nil, nil + } + s.mu.Lock() + defer s.mu.Unlock() + var hits []Hit + score := func(text string) float64 { + n := 0.0 + for _, w := range wordsOf(text) { + for _, q := range words { + if stem(w) == q { + n++ + } + } + } + return n + } + for k, e := range s.entries { + r := e.versions[e.current-1].Record + if r.Status == StatusReserved || r.Status == StatusWithdrawn { + continue + } + for _, c := range r.Clauses { + if n := score(c.Text); n > 0 { + hits = append(hits, Hit{Key: k, Clause: c.N, Rank: n}) + } + } + body := "" + for _, sec := range r.Sections { + body += " " + sec.Text + } + if n := 2*score(r.Title) + score(r.Topic) + score(body)/20; n > 0 { + hits = append(hits, Hit{Key: k, Rank: n}) + } + } + sort.Slice(hits, func(i, j int) bool { + if hits[i].Rank != hits[j].Rank { + return hits[i].Rank > hits[j].Rank + } + if hits[i].Key != hits[j].Key { + return hits[i].Key.Number > hits[j].Key.Number + } + return hits[i].Clause < hits[j].Clause + }) + if limit > 0 && len(hits) > limit { + hits = hits[:limit] + } + return hits, nil +} + +func (s *memStore) Inbound(_ context.Context, k Key) ([]Inbound, error) { + s.mu.Lock() + defer s.mu.Unlock() + var out []Inbound + for from, e := range s.entries { + r := e.versions[e.current-1].Record + for _, l := range r.Links { + if to, ok := l.To.Record(); ok && to == k { + out = append(out, Inbound{From: from, Link: l}) + } + } + } + sort.Slice(out, func(i, j int) bool { + if out[i].From != out[j].From { + return out[i].From.Number < out[j].From.Number + } + return out[i].Link.String() < out[j].Link.String() + }) + return out, nil +} + +func (s *memStore) SetFindings(_ context.Context, f []Finding) error { + s.mu.Lock() + defer s.mu.Unlock() + s.findings = append([]Finding(nil), f...) + return nil +} + +func (s *memStore) Findings(context.Context) ([]Finding, error) { + s.mu.Lock() + defer s.mu.Unlock() + return append([]Finding(nil), s.findings...), nil +} + +func (s *memStore) Told(_ context.Context, topic string) (bool, error) { + s.mu.Lock() + defer s.mu.Unlock() + if s.told[topic] { + return false, nil + } + s.told[topic] = true + return true, nil +} + +// questionWords are the words of a question that say what it is about: the words of asking itself ("what +// holds about … and why") are left out, and each is stemmed. +func questionWords(q string) []string { + var out []string + seen := map[string]bool{} + for _, w := range wordsOf(q) { + if askingWords[w] || len(w) < 3 { + continue + } + st := stem(w) + if !seen[st] { + seen[st] = true + out = append(out, st) + } + } + return out +} + +// askingWords say that something is asked, not what about. +var askingWords = map[string]bool{"what": true, "holds": true, "hold": true, "about": true, "and": true, "why": true, + "how": true, "does": true, "the": true, "decided": true, "decide": true, "say": true, "says": true, "which": true, + "when": true, "who": true, "for": true, "with": true, "this": true, "that": true, "are": true, "was": true} + +func wordsOf(s string) []string { + return strings.FieldsFunc(strings.ToLower(s), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) }) +} + +// stem is a crude stem: a plural's s and a few endings off. +func stem(w string) string { + for _, suf := range []string{"ing", "es", "ed", "s"} { + if len(w) > len(suf)+3 && strings.HasSuffix(w, suf) { + return strings.TrimSuffix(w, suf) + } + } + return w +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.commit b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.commit new file mode 100644 index 00000000..2c279512 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.commit @@ -0,0 +1 @@ +331e8d0bbef401cee83c28aef27db9e823a901c7 diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.tar.gz b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.tar.gz new file mode 100644 index 00000000..2be1a134 Binary files /dev/null and b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq-main.tar.gz differ diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/00-META/glossary.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/00-META/glossary.md new file mode 100644 index 00000000..d2ed9f0d --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/00-META/glossary.md @@ -0,0 +1,680 @@ +# Glossary — the words of the mesh, by domain, and the ones it stopped using + +One name per thing. This page is the authority; where an older record says something else, that +record keeps its words and this page says how to read them. It exists because the words kept drifting +in conversation — "control plane", "controller" and "master" for one thing, "substrate" and +"foundation" for another — and a mesh you cannot name precisely is a mesh two people describe differently. The rule and +its check are [ADR 0244](../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md); +the domains are drawn in [to-be 49](../03-DESIGN/01-to-be/49-the-mesh-in-domains.md). + +## How to read this page + +A **domain** is an area of the mesh that owns a set of concepts: inside it each concept has one word, +and the domain decides what that word means. Other domains use the word as it is defined here and never +change its meaning. Every word is defined once, in the domain that owns it; a section lists the words +it uses from other domains under *Uses*, with no second definition. + +Two fixed lines follow an entry where they apply, and `00-META/checks/words.py` reads them: + +- *Not:* followed by struck-through words — the words this entry replaced. A struck word with no scope + is retired everywhere; *(hq)* retires it in this repository's prose only; *(tools)* in the + descriptions of the mesh's tools and seat verbs only. A retired word may be quoted, never used. +- *Identifier until renamed:* followed by a name in a code span — code that still carries an old name. + It may stand in a code span and nowhere else until the code is renamed, and then the line goes. + +Nothing else on this page is struck through. A word that means different things in different domains +is listed under [Homonyms](#homonyms), with the qualified form each domain uses. + +## Words every domain uses + +- **mesh** — the whole: the nodes, the modules assigned to them, the controller that decides and the + records it keeps. One mesh per operator; a second mesh is a second everything. +- **domain** — an area of the mesh that owns a set of concepts and the one word for each, as described + in the section above ([ADR 0244](../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md)). + ADR 0006 called seven of them the controller's *contexts*; they carry over as domains under the same + names, except *observability*, which is now **Health and repair**. Where a domain's records live in + the controller, the domain owns that store alone ([ADR 0008](../02-DECISIONS/0008-a-context-owns-its-store.md)). + *Context* in that sense is not used in new writing; the word stays ordinary English, so this is + checked by review, not by `words.py`. +- **machine** — any computer: hardware, a virtual machine, before, during or outside its membership of + the mesh. Prose about the computer itself — its disks, its kernel, the network it sits on, what was on + it before the mesh — says machine. +- **node** — a machine the mesh has adopted and owns. A machine becomes a node when it joins + ([ADR 0004](../02-DECISIONS/0004-a-node-and-how-it-joins.md)), and from then on it runs the node-engine + and is either **adopted** (the mesh holds what it found there as found) or **converged** (the mesh made + it what it is) — [ADR 0100](../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md). + Prose about a member of the mesh says node. A node is just a machine that has joined; being one + implies nothing about what it runs. +- **module** — the unit the mesh assigns: one named thing, defined by its manifest, that a node runs — + a service, a container, files, packages, its own code — and the tools it serves + ([ADR 0040](../02-DECISIONS/0040-what-a-module-is.md)). Everything configurable on a node is a module. +- **seat** — a named role at a scope (node / site / mesh), held by a module assignment, from a **closed + set** the mesh defines: a claim naming a seat outside the set is refused. A seat may **deliver a + provision**, and its holder is then the mesh's answer for it when several modules provide it + ([ADR 0126](../02-DECISIONS/0126-a-module-declares-its-own-seats.md)). The set, with who holds each + seat, is the overview of what a mesh has ([26 — The seats](../03-DESIGN/01-to-be/26-the-seats.md)). + A seat carries **verbs** — the tools every holder must serve. A verb added to a held seat is first + promised as optional, then served, then required ([ADR 0246](../02-DECISIONS/0246-a-seats-new-verb-is-promised-before-it-is-required.md)). + A node seat may also name **the paths its holder owns**, such as `/etc/hosts`: a claim whose module does + not declare them is refused, and only a claimant declares them ([ADR 0287](../02-DECISIONS/0287-a-seat-names-the-paths-its-holder-owns-and-a-claim-without-them-is-refused.md)). +- **operator** — the person who runs the mesh, and the one the mesh talks to. + *Not:* ~~master~~ (hq) +- **person** — any human, as against the mesh acting unattended: *a person's word* releases a held + delivery, *a person* deletes a consumer's data. The operator is one; the word says that a human, not + the mesh, acted. + +## Module — what a module is and declares + +**Purpose.** To say what one module is, in the one document every other domain reads. +**Recorded in** the catalogue, one manifest per module. **Upstream of** every other domain; its words +are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0040, 0126, 0174, 0188, +0207, 0236, 0262. +**Uses:** module, seat, provision (Provisioning), data class (Data), health (Health and repair). + +- **manifest** — the file a module is defined by (`module.json`): what it is, what it provides and + requires, the seats it claims, the resources it declares, its tools, its data and its health. Say + manifest in prose, not the file's name. + *Not:* ~~module definition~~ (hq) +- **resource** — one thing a manifest declares a node must have: a file, a directory, a service, a + container, a package, a bundle, an archive. A module **declares** resources; the verb *declares* is + this domain's and means what a manifest states. +- **claim** — a module taking a spot on a seat: `claims: [{name, scope}]` in a manifest. A mesh-scoped + exclusive claim is how the mesh says "there is one of me". A foundation seat is named after the role + it guards: the `mesh-controller`, `postgres` and `nats` modules claim the `mesh-controller`, + `mesh-store` and `mesh-broker` seats ([ADR 0079](../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md), + [ADR 0116](../02-DECISIONS/0116-the-bus-is-built-in-five-steps.md)). +- **setting** — a value a manifest declares and an assignment gives, one of the two ways a node varies + a module ([ADR 0174](../02-DECISIONS/0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)). + What a "flavor" once varied is a setting or a separate module. + *Not:* ~~flavor~~ +- **preference** — a setting whose manifest gives it a default, because its value is true of the + software and only tuned per node: a font size, a width + ([ADR 0262](../02-DECISIONS/0262-a-preference-has-a-default-in-its-manifest-and-the-operators-own-value-has-none.md)). + A setting with no default is the operator's own value: a domain, an identity. + *Not:* ~~tunable~~ (hq) +- **kept region** — a marked block in a managed file the mesh writes *into*, where the operator's own + lines survive every send and are given back when the module goes (ADR 0174). The other way a node + varies a module. +- **bundle** — the artifact a module's own code is built into — its tools, a seat's implementation, a + daemon — in any language the mesh has a toolchain for; never an image. A **tools bundle** speaks MCP to + the tool runner. One module may declare several + ([ADR 0188](../02-DECISIONS/0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)). +- **tool** — one capability a module serves through the tool runner, addressed `.`. A + seat's **verb** is the same thing carried by a seat rather than a module. +- **mesh-sdk** — the library a module's own code is written against, including the harness that serves a + tools bundle to the tool runner. The harness is part of it, not a library of its own. + *Not:* ~~tools-sdk~~ +- **invokes** — the manifest word for the tools a module calls, `.` each or `*` for every + one. A grant on the publish side and nothing else; a module that declares none calls nothing. +- **upgrade policy** — how a module's new builds reach its nodes: `roll` (one node first, judged, then + the rest), `together`, or `record` (register the build, send it nowhere) + ([ADR 0236](../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)). + The controller's verb `upgrade` shows it. + +## Core — the mesh's own machinery + +**Purpose.** To keep the controller, the bus, the store and every node's engine running and agreed, so +every other domain has something to run on. **Recorded by** the controller and the store. +**Upstream of** every domain but Module; the others use the bus and the store as they are. +**Decided by** ADR 0005, 0006, 0067, 0106, 0141, 0175, 0227, 0229, 0289. +**Uses:** node, module, seat. + +- **controller** — the component that decides what each node should be, holds the mesh's records, and + tells nodes over the bus. The relationship is *controller and nodes*, and no node is subordinate: a + node applies its declaration on its own and survives the control-node dying. + *Not:* ~~control plane~~, ~~slave~~ (hq), ~~mesh-control~~ (hq) +- **mesh-controller** — the module that runs the controller. It claims the `mesh-controller` seat at + mesh scope, which is what makes it singular. +- **control-node** — the one node that also holds the `mesh-controller` seat. There is exactly one per + mesh. It is not a separate kind of node — it is a node that additionally runs the controller (and, + today, the foundation). Lose it and the other nodes keep running what they were last told; they simply + cannot be told anything new. +- **node-engine** — the program on every node that applies what the controller declares: it receives + the node's declaration, writes the files, runs the services and containers, and reports what it did. + It is the engine, not a module: it owns no file's content, and every file it writes belongs to the + module that declared it. *Agent* is avoided because that word means a coding agent here. A record + written before the rename keeps the old name. "The host" is retired in this repository's prose only: + in the tools' descriptions it is also an SSH server's or a container runtime's own word, so a + description that means the node-engine by it is found by review. + *Not:* ~~host agent~~, ~~the host~~ (hq), ~~node host~~ + *Identifier until renamed:* `mesh-host` — the repository, its binary and its service unit; the rename is + [issue 354](../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md) +- **tool runner** — the one program on each node that loads every assigned module's tools bundle and + serves every tool and every held seat's verb on the subjects the memberships issue; the node-engine + supervises it as a process beside the services it runs, never a container + ([ADR 0175](../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md), + which called it "node tools"). Its mode on loopback is the mesh MCP server. + *Not:* ~~node tools~~, ~~tool runtime~~ (hq) + *Identifier until renamed:* `node-tools` — the module, its unit and its bus account; the rename is + [issue 354](../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md) +- **foundation** — the store and the bus, raised at genesis before any module system exists. The + foundation is not a third thing beside the store and the bus — it *is* those two, named together. + *Not:* ~~substrate~~ +- **genesis** — raising the foundation and the controller on an empty mesh, by the one installation + done by hand ([ADR 0067](../02-DECISIONS/0067-genesis-is-a-pivot.md)). +- **store** — the one postgres server. It holds the controller's own databases (`inventory`, + `identity`, `licences`, each owned by one domain, ADR 0008) and every module's own database. One server, + many databases — never one shared mesh database. Bare *store* means this and nothing else + (see [Homonyms](#homonyms)). +- **bus** — the mesh's own nervous system: NATS, one per mesh, carrying every link the mesh has — + control, declarations, builds, events, tool calls ([ADR 0106](../02-DECISIONS/0106-the-bus-is-nats.md)). + A module reaches it by requiring `mesh-bus` ([ADR 0128](../02-DECISIONS/0128-the-mesh-bus-is-required-not-ambient.md)); + one that does not require it has no account on it. Held by the `mesh-broker` seat, which is named + after the role rather than the server, so the server can change without the seat doing so. The `nats` + module holds it. +- **the deprecated broker** — the lavinmq module. It was the mesh's bus and is not any more. It keeps + running as an **ordinary provider** of the `amqp` provision, for modules that need a message broker + of their own the way something needs a database + ([ADR 0131](../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) — it claims no + seat, is not foundation, is never raised at genesis, and a mesh that never installs it is complete. + Say *the deprecated broker*, not *the compatibility broker* (it serves the mesh's own modules, not only + the predecessor's) and not *the AMQP broker* (naming it after a protocol invites describing the bus by + contrast with it, which is backwards). +- **layer** — one of the four levels the mesh is built in, from the bottom: the node-engine, the + foundation, the controller, the surfaces. The records' `topic: the tiers` and `repos.md` say *tier* for + this; new prose says layer (see [Homonyms](#homonyms)). +- **lease** and **epoch** — which controller may send (the lease, held in the bus and renewed), and the + order of what it sent (the epoch, which a node reads before it accepts a declaration) + ([ADR 0229](../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)). + +- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its + records (nodes, assignments, walks, builds, settings), after each change commits in its store and only + when the value differs, with one event per write. The store stays the record; the self-check compares + the two and raises `mirror-drifted` on any difference + ([ADR 0289](../02-DECISIONS/0289-the-controller-mirrors-its-records-into-buckets-written-only-on-a-change-and-audited-against-its-store.md)). Shortened to *mirror* where the + text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is + a different thing, a copy of an outside image in the artifact store. + +## Placement — what runs on which node + +**Purpose.** To decide, send and apply what each node runs, and to say why. +**Recorded by** the controller's `inventory` database. **Upstream of** Provisioning, Connectivity, Data +and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261. +**Uses:** node, machine, module, seat, setting (Module), manifest (Module). + +- **assignment** — a module put on a node, with the settings that node gives it. +- **pending assignment** — an assignment of a module whose build is still running, kept, and made by the + controller when that build registers the module, or ended with why when it does not + ([ADR 0261](../02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md)). +- **scope** — where a seat or a claim holds: `node`, `site` or `mesh`. `node` is also the scope's + value in a manifest. +- **capacity** — how many holders a seat takes. A capacity-1 seat is exclusive. A higher-capacity seat + is a **bench**: several holders coexist. A **replicated** bench has one holder per node, each on record + and each answering the same, like `mesh-dns-resolver` + ([ADR 0223](../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)). A + **kinded** bench has holders that are different modules, each claiming one **kind**, and a verb's + subject carries the kind; `channel` and `intake` are the only ones (ADR 0234). +- **installed / holding** — a module may be assigned (its package installed, its files placed) without + holding the seat its family declares; *holding* is being the one — the login shell, the display session + — on that node ([ADR 0176](../02-DECISIONS/0176-the-login-shell-is-a-node-seat-and-execute-is-its-contract.md)). + The module that holds a seat is its **holder**. +- **depends on a seat** — a module needing a seat held on its node by some module, without holding it. + Derived from the resources it declares, never stated: a `service` depends on `node-service-manager`, a + `package` on `node-package-manager`, a `container` on `node-container-runtime` + ([ADR 0207](../02-DECISIONS/0207-a-module-depends-on-the-node-seats-that-apply-its-resources.md)). + Not a claim: a module claims a seat it holds and declares resources. +- **declaration** — what the controller sends one node: every resource of every module assigned there, + composed. The noun is this domain's; what a manifest says is that it *declares* (Module), and a manifest + is never called a declaration. The controller's verb `plan` previews a node's declaration. +- **send** — the controller giving a node its declaration. The controller's verb `push` asks for a + send (of one node, or of every node); prose says send. A git push and a phone's push notification are + other things (see [Homonyms](#homonyms)). +- **apply** — the node-engine making its node match its declaration; **reconcile** is doing so again + until nothing differs. +- **operator account** — the login name of the person who works on a node, stated on the node record; + empty for a node nobody logs into. Everything the mesh places under a person's home is resolved + against this account's home and owned by it + ([ADR 0181](../02-DECISIONS/0181-the-operator-account-is-a-node-fact-and-a-home-is-a-placement-root.md)). + Not a name a manifest carries. +- **agent account** — the login every coding agent on a node runs as, when the node names one of its own; + stated on the node record at the controller's terminal, never by a verb or a setting. Empty means agents + run as the operator account. It cannot become root without a person, and the node-engine judges that on + every look ([ADR 0266](../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)). +- **account group** — a group of the machine's group database that a module puts an account in, by + declaring the account with that group and nothing else of it. Several modules may each add one to the + same account; the mesh takes back only an account group it added + ([ADR 0252](../02-DECISIONS/0252-a-module-puts-an-account-in-a-group-and-the-mesh-says-when-a-new-login-is-needed.md)). + Never *membership*, which is the bus's word. + +## Provisioning — one module serving another + +**Purpose.** To resolve which module serves a provision for which consumer, and to wire the two. +**Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity. +**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263. +**Uses:** module, seat, credential (Identity and access). + +- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider + and wires the two with an endpoint and a credential. A provision is a service you offer, a seat is a + role you occupy, and the two meet where a seat delivers a provision: occupying the seat is what makes + a module *the* provider of it. +- **provider** and **consumer** — the module that provides a provision, and the module that requires + it. Which of several providers serves a consumer is resolved + ([ADR 0084](../02-DECISIONS/0084-which-provider-serves-a-consumer.md)). +- **pin** — a person's choice of provider for a consumer, which resolution then respects. +- **endpoint** — where a consumer reaches its provider; an assignment binds it and says how far it + reaches ([ADR 0138](../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)). +- **retire** — a provider stops serving a consumer the mesh no longer asks for; what it kept is deleted + only by a person ([ADR 0230](../02-DECISIONS/0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md)). +- **share** — a folder one node offers to the mesh's other nodes over the private network: the provision + `nfs-share`, provided by the holder of the node seat `node-nfs-server`, one share per folder + ([ADR 0263](../02-DECISIONS/0263-a-share-is-a-provision-a-node-mounts-it-on-demand-and-only-nodes-that-ask-get-it.md)). +- **mount** — a share, or an occasional source, made visible at a path on one node by the holder of the + node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263). +- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk; + its absence is a state, *not present*, never a fault (ADR 0263). + +## Identity and access — who may do what + +**Purpose.** To issue, hold, rotate and check the credentials and grants by which modules, nodes, agents +and the operator act. **Recorded by** the controller's `identity` and `licences` databases and the vault. +**Upstream of** Provisioning, Change and delivery, and Operator and conversation. +**Decided by** ADR 0085, 0113, 0183, 0225, 0234. +**Uses:** module, node, operator, consumer (Provisioning). + +- **credential** — anything that proves an identity: a password, a key, a token. A **pair credential** + is the two ends of one credential, the consumer's and the provider's. +- **secret** — a credential or other value the vault keeps sealed and the node-engine unseals into a + file at apply ([ADR 0085](../02-DECISIONS/0085-a-secret-is-a-provision.md)). +- **secret family** — a module's own secrets declared as one name ending in `-*`; each member is the prefix + and a name (`smb-password-`, one per source the module's settings name), given by its full name + and never made by the mesh + ([ADR 0283](../02-DECISIONS/0283-a-part-that-waits-for-the-operators-secret-or-setting-is-said-as-waiting-passes-the-first-node-gate-and-names-the-act.md)). +- **vault** — the module that owns every secret (`mesh-vault`, + [24 — The secrets vault](../03-DESIGN/01-to-be/24-the-secrets-vault.md)). +- **grant** — what an identity may call or reach, bounded by the provisions it requires + ([ADR 0225](../02-DECISIONS/0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md)). +- **bus account** and **membership** — a module's or a node's account on the bus, and the subjects it + is issued. The bus server calls its accounts *users*; the mesh says bus account. +- **rotate** — replacing a credential without a consumer holding one the provider does not know. +- **licence** — a model-access account the licence manager holds and **binds** to a consumer, the + coding agent on a node ([ADR 0183](../02-DECISIONS/0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md)). + Taking a consumer off a licence is **unbinding** it; the licence manager's verb for it is still + `release`, an identifier to rename. +- **proof** — what an authorising answer carries: a verified sender, a one-time code, a security key's + touch (ADR 0234). How much proof a kind of ask needs is its **assurance level**. +- **controller's terminal** — a command line the controller runs as typed by its operator, not asked through a + verb, so a change that is the terminal's alone is taken (issue 339, issue 340) — or, since + [ADR 0277](../02-DECISIONS/0277-a-trusted-setting-is-proposed-through-a-verb-and-set-on-the-operators-warrant-and-a-modules-own-secret-is-asked-for-and-typed-at-the-desk.md), + proposed through a verb and set on the operator's warrant. In the controller's code it is a + line run without `MESH_VERB`. The operator reaches it with `mesh-cli` from their own account on a node where no + agent can run as that account: the control-node, and a node whose agents run under an account of their own + ([ADR 0272](../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md)). + +## Change and delivery — a commit on its way to the nodes + +**Purpose.** To take one commit from its pull request to every node that should run it, and to put it +back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the +planner, the walk). **Downstream of** Module, Core, Health and repair and Data. +**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276. +**Uses:** module, node, declaration and send (Placement), health (Health and repair), person. + +- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate** + (`mesh/merge-gate`, the build seat's composed check of the modules the change touches) and the + **repository check** (`mesh/repo-check`, the repository's own `merge-check.sh`) + ([ADR 0238](../02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md)). +- **build seat** — the seat whose holder builds modules, `node-build-agent`; its holder on a node is the + **builder**. Any node may hold the seat, so there is no build node by nature + ([ADR 0237](../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md)). + *Not:* ~~build machine~~ +- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**. + The queue's verbs still say *ask* for an entry; the conversation's ask is another thing + (see [Homonyms](#homonyms)). +- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**. + Served by the **package-registry** (gitea). Only a builder talks to it. +- **artifact** — anything a build produces and the mesh delivers to a node by **digest**: an `image`, a + mirrored `upstream` image, a `bundle` of the module's own code, an `archive`. Served by the + **artifact store**, an OCI registry that holds every kind as content-addressed blobs + ([ADR 0156](../02-DECISIONS/0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md)). + An image is one kind of artifact, and a module is not an image. A package and an artifact are two + protocols, not one store being weak ([ADR 0075](../02-DECISIONS/0075-two-stores-and-which-provides-what.md)). +- **build source** — the files one artifact is built from, in one or more repositories: for a Go program + its import closure (the directories of its packages, its Go module files) and its recipe; for any other + artifact the paths its manifest names, by default its module's own directory. Each artifact of a build + has its own, and a change rebuilds the artifacts whose build source it touches + ([ADR 0267](../02-DECISIONS/0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md)). + Older records (ADR 0015, ADR 0111) and designs say *build source* for the repository a module + is built from; this entry narrows it to the files within, and that use reads as *the repository holding + the build source*. Sharing a repository is not a dependency, and no build source is a whole repository + holding another module's files. +- **catalogue** — what the mesh holds of every module, at which commit; and the repository the + modules live in. `mesh-catalog` and `catalog_modules` are identifiers and keep their spelling. + *Not:* ~~catalog~~ (hq) +- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head + being announced to one of its final states: one pull request, one status, one note + ([ADR 0239](../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)). + From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk + ([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). + Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`, + `held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`; + [ADR 0250](../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md), + [ADR 0270](../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)). Not *change* (a diff). *Pipeline* is the predecessor's build-and-deploy + mechanism, which the as-is designs describe; it is never a word for a delivery. + *Not:* ~~deployment~~ (hq) +- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was + sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own + passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery + whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan + ends in ([ADR 0270](../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)). +- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's + repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One + level: a group holds deliveries, never groups. Its state is derived from its members, never set. +- **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and + their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a + person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a + diffset; the controller's verb `delivery-plan` shows it. *Deploy* is used in this one place. + *Not:* ~~change plan~~, ~~release plan~~ +- **merge window** — the rolling wait in which the controller collects merges before it plans them: each + merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest + 10 minutes after its first merge ([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). +- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more + than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*, + which is a delivery group. +- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat + of the same name. It records and decides; the controller sends, judges and rolls back when it is asked. +- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per + repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236, + [ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). One walk is open at a time; it names every merge it answers. A primitive the delivering stage + asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules + built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own: + a module *rolls out* by its `roll` policy. + *Not:* ~~rollout~~ (hq) +- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it + back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)). +- **core module** and **leaf module** — a core module is one on the controller's own path (the controller, + the node-engine, the tool runner, the bus, mesh-delivery) or the mesh's resolver, whose judgement protects + the mesh's ability to reach a node at all; a leaf module is every other module. Decided by + [ADR 0282](../02-DECISIONS/0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md). + Not a *core component* (ADR 0236's controller, node-engine, tool runner and bus, each with a health probe): + the core modules add mesh-delivery and the resolver. +- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its + modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: five minutes for + a leaf module, ten for a core module (ADR 0282). Not the walk's time, which ends at the send. +- **release** — a person's word that a held delivery goes on; the verb `mesh-delivery.release`. Used in + this sense only. +- **rollback** — putting a node back on the build it ran before, by something other than the build being + judged (a witness, ADR 0236). +- **the lab**, **scenario**, **replay** — a real mesh raised to run a change against before it reaches + nodes; what a lab run declares; and a recorded failure run again to prove a fix (ADR 0016, ADR 0237). + +## Health and repair — what is wrong, and putting it right + +**Purpose.** To notice what is wrong with the mesh, say it once, repair what may be repaired unattended, +and record what was done by hand. **Recorded by** the controller's condition store. +**Upstream of** Change and delivery (the first-node gate) and Operator and conversation (a condition +becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283. +**Uses:** node, module, node-engine (Core). + +- **health** — a module's or a core component's statement of how it is alive and ready, which the + node-engine judges ([ADR 0240](../02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md)). +- **relogin needed** — an account's health when the group database lists it in an account group and its + running session, started before, does not have that group yet. Said by the node-engine as the module's + resource of kind `account`; it clears at the first look after a new login (ADR 0252). +- **waiting** — a resource's health when its module's tool check says it waits for the operator: a named own + secret or setting of the module, not given yet, without which a part cannot work. Not a fault: the + controller checks it and says it as the condition `needs-operator`, naming the act, and the first-node gate + reads it as *waits for a person* + ([ADR 0283](../02-DECISIONS/0283-a-part-that-waits-for-the-operators-secret-or-setting-is-said-as-waiting-passes-the-first-node-gate-and-names-the-act.md)). +- **probe** — one look at one thing's health, run by whoever owns the verdict; what a probe returns is a + **finding**, before it becomes a condition. +- **signal** and **watchdog** — something that must keep happening (a heartbeat, a report after a send), + and the watcher that raises a condition when it stops (to-be 45 §3). +- **condition** — one open fact about something the mesh owns that is wrong, with a key and a severity; + raised and cleared only by observation + ([ADR 0231](../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)). + What a wrapped dashboard raises is its own; the mesh's notion is a condition. + *Not:* ~~alert~~ (hq) +- **self-check** — the controller's own examination of the mesh, run on demand and on a schedule + (to-be 45 §4). Its verb is `doctor`, an identifier; prose says self-check. + *Not:* ~~doctor~~ (hq) +- **healer** — a registered response to one condition kind: its **repair** (the ordinary path again), + its **budget**, its back-off and its brake. A healer may not withdraw, delete or recreate data. +- **drill** — something broken on purpose to see the mesh raise and clear its condition; a drill is + never counted as a repair. +- **rehearsal** — a question the operator starts at the controller's terminal (`rehearse`) and answers on a + channel, to see an answer travel the whole way: its approval performs nothing and is recorded as the + operator's decision ([ADR 0259](../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md)). + Not a drill: nothing is broken. +- **hand-act** — a repair a person made by hand, recorded so the mesh knows it happened. +- **witness** — what rolls a core component back when its new build does not become healthy: never the + component itself (to-be 45 §8). + +## Data — what the mesh keeps, and getting it back + +**Purpose.** To know every item of data a module holds, how precious it is, and to keep it recoverable. +**Recorded by** the manifests' `data` declarations and every node's `node-backup` seat. +**Upstream of** Change and delivery and Health and repair. **Decided by** ADR 0232, 0233, 0235. +**Uses:** module, node, provider and consumer (Provisioning), person. + +- **data class** — how precious an item of data is, ranked by the operator: `irreplaceable`, + `valuable`, `rebuildable`, `cache`; and `none` for a provision that keeps nothing of anybody's + ([ADR 0233](../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)). +- **backup** — a copy kept elsewhere by the node's `node-backup` holder; **restore point** — one backup + at one moment; **restore** — bringing it back beside the live data, never over it. +- **stream snapshot** — the bus's own backup of each stream, taken by the module holding `mesh-broker` + ([ADR 0235](../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md)). +- **sticky binding** — a consumer's tie to the data a provider keeps for it, which moves only by a + person ([ADR 0232](../02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)). + +## Connectivity — how nodes and modules reach one another + +**Purpose.** To make every node and module reachable by name where it should be, and unreachable where +it should not. **Recorded by** the controller. **Upstream of** Provisioning and Health and repair. +**Decided by** ADR 0007, 0117, 0138, 0223, 0226, 0247. +**Uses:** node, machine, assignment and endpoint. + +- **private network** — the mesh's own encrypted network between its nodes, on which every node has an + address and a mesh name ([ADR 0226](../02-DECISIONS/0226-the-private-network-is-assigned-by-its-own-name-and-the-proxy-names-its-public-issuer.md)). + The network the machine sits on without it is the **underlay**. + *Not:* ~~overlay~~ (hq) +- **anchor** and **hub** — the roles a node plays for the private network: the anchor is reachable from + outside and every node reaches it; a hub relays for nodes that cannot reach each other directly. +- **resolver** — a seat holder that answers the mesh's names; a node lists only the mesh's resolvers + (ADR 0223), or, where it holds one, its own resolver, which asks them. **uplink** — a node's + connection to the outside network, a seat ([ADR 0117](../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)). **hostname** — a node's own name, + a seat. +- **split DNS** — resolving names by domain on one machine: a VPN's domains through the VPN's servers + over its link, every other name through the mesh's resolvers. The provision `split-dns`, provided by + the holder of the node seat `node-resolver`, the machine's **own resolver**, which exists only where + something requires it ([ADR 0247](../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md)). +- **proxy** and **public name** — the module that answers a public name and forwards it to an + endpoint on the private network. +- **packet filter** — what the mesh enforces on a node about which packets pass, the + `node-packet-filter` seat and its verb `rules`. A **found firewall** is a program the mesh found on a + machine and keeps retired or in force; *firewall* says that, and a wrapped program's firewall is its own. +- **intrusion prevention** and **ban** — refusing a source that misbehaved, and the refusal itself. +- **reach** — how far an assignment's endpoint may be reached from: the node, the private network, or + the outside (ADR 0138). + +## Operator and conversation — the person the mesh works for + +**Purpose.** To let the mesh and its operator talk: tell, ask, answer, and act only on an answer it can +trust. **Recorded by** the router (asks, warrants and the operator's identities). +**Downstream of** Health and repair, Change and delivery, and Identity and access. An authorising +answer acts in whichever domain asked, through that domain's own verb; the conversation owns the +asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272. +**Uses:** operator, person, proof (Identity and access), release (Change and delivery). + +- **mesh MCP server** — the endpoint on a node's loopback through which every agent and person on that + node reaches the mesh: the MCP server named `mesh`, with its five tools `mesh_overview`, + `mesh_machine`, `mesh_search`, `mesh_describe` and `mesh_call`. It is the tool runner's loopback mode, + not a module of its own. "Console" suggested a terminal or a shell, and the module `mesh-console` it + once named no longer exists; "the tool bridge" and "the brain" named the predecessor's program. + *Not:* ~~console~~ (hq), ~~mesh-console~~, ~~tool bridge~~ +- **mesh-cli** — the module on every node that places the operator's command, and the command itself: + `/usr/local/bin/mesh-cli ` asks the controller to run one of its command lines. It asks the + node-engine on its own machine, which says to the controller which account is asking; the controller runs the + line as the **controller's terminal** or as an ordinary call + ([ADR 0272](../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md)). +- **nox** — `mesh-cli`'s alias, in interactive zsh only. A script, `sudo` and a document meant to work everywhere + say `mesh-cli`. The operator also calls the mesh itself *nox mesh*; on this page and in the records *nox* is the + alias alone. It is the one command the operator types: every line the mesh addresses to the operator — a + condition's words, a verb's answer, a README, a design — names a `nox` line, never a daemon's (the controller's + or the node-engine's binary followed by a command), which `words.py` and the catalogue's check refuse + ([ADR 0272](../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md), + its insight of 2026-10-09). +- **channel / intake** — the two kinded benches the mesh talks to its operator through: `channel` + sends, `intake` turns what arrives into one envelope. A holder of either declares **capabilities** from + the fixed vocabulary `channel-capabilities/1`. Not *notifier* (that is the desktop's node seat, one + holder of kind `desktop`) and not *bot* (that is one service's account). +- **router** — the module that holds the `operator-channel` seat, orders the channels and checks every + sender ([ADR 0234](../02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md)). +- **headline / verdict / explanation / resolved line** — what a condition says to the operator in plain + words: a few words naming the thing and what is wrong; "Nothing for you to do." or "Needs you:" and the + one thing the operator can do themselves (its **needs**); one or two sentences on what happened and what + it means; the line said when it clears. Its **actions** are the answers a notification offers. Carried by + the condition beside its **summary**, which keeps the plans, commits and verbs for whoever looks closer + ([ADR 0253](../02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md)). + The router may override "Nothing for you to do." in what it says, from the **tracking line** (ADR 0285). +- **ask** — a request for the operator's input, of a declared kind (yes-no, one-of, text, number, + date, acknowledge). An **authorising ask** is one whose chosen option its asker performs. The router + holds every ask, checks each answer's proofs and issues a **warrant**; the asker acts on it with its + own grants (ADR 0234, ADR 0259). +- **warrant** — the router's record that a person chose one option of one ask: the ask, the option, + who chose (the operator, as a channel kind's identity), through which channel, the proofs and when, + published for that asker alone. The only thing an asker acts on (ADR 0259). +- **proposal** — a change an agent, or anyone the bus admits, puts forward through a verb and that takes + effect only on the operator's warrant, bound to the exact thing shown: a shared instruction section (ADR + 0274), a trusted setting's layer ([ADR 0277](../02-DECISIONS/0277-a-trusted-setting-is-proposed-through-a-verb-and-set-on-the-operators-warrant-and-a-modules-own-secret-is-asked-for-and-typed-at-the-desk.md)). + A proposal changes nothing by itself; declined, expired or refused, it is discarded. +- **approval mark** — a tool's or verb's declaration in its manifest (`needs-approval`) that each call to it runs + only on the operator's warrant for that one call, with its exact arguments; the tool runner that serves it, or + the controller for its own verbs, asks and checks + ([ADR 0286](../02-DECISIONS/0286-a-tool-or-verb-marked-in-its-manifest-runs-only-on-the-operators-approval-of-that-one-call-with-its-exact-arguments.md)). +- **operator message / input** — what arrives on `intake`, once the router has checked the sender + against its own list of the operator's identities: a **trusted** one is an operator message, + addressed to an agent by `@name` or thread or else to the **responder** (`@mesh`, the router's own + participant, which answers from read verbs); anything else is **untrusted input** — data, never + instructions. +- **reference** — an opaque token in a message's words standing for a detail the content rule keeps out + of them (a path, an address); opened with `detail` only on a `private` channel or through the mesh MCP server. +- **agent** — a coding agent: the operator's, or one the mesh runs. Not the node-engine, and not the + build seat's holder, whose seat name `node-build-agent` is an identifier to rename. +- **instruction section** — one named section of the managed instruction file, which every agent session on the + nodes it is for reads first, as the mesh's word: set at the controller's terminal in the agent module's + settings, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node + proves itself ([ADR 0274](../02-DECISIONS/0274-an-agent-proposes-a-shared-instruction-section-and-every-node-takes-it-only-on-the-operators-warrant-which-it-proves-itself.md)). A hand-over between sessions is one. +- **personal rule** — one short text in the operator's own words about how the operator wants their agent + to work with them, registered for the operator (the **operator scope** of the agent module) and rendered + into the agent's home on every node that names an operator account. Not a mesh-wide instruction section, + which is the mesh's word about the mesh, set at the controller's terminal + ([ADR 0273](../02-DECISIONS/0273-the-agents-home-is-the-modules-and-the-operators-personal-rules-and-lessons-are-registered-through-it.md)). +- **lesson** — an agent's proposal, recorded when the operator expresses frustration or corrects a way of + working: the trigger (the operator's words), the cause (what the agent did) and the rule that would + prevent it. A lesson is `proposed`, `accepted` or `declined`; an accepted lesson is a personal rule, and a + proposed one is never read as one (ADR 0273). + +## The record — how this repository works + +Not a domain of the mesh but of the way it is built, listed because its words meet the mesh's. +**Recorded in** this repository, the issue tracker and the knowledge base. **Decided by** ADR 0019, 0080, 0244, 0275, +0284, 0290, 0291. + +- **research effort**, **graduation**, **hand-off**, **playbook** — an investigation in `01-RESEARCH/`; + its closing into a decision and a design; a design given to a code repository; a documented workflow + in `00-META/process/`. +- **decision record** — one numbered file in `02-DECISIONS/`, never rewritten: **superseded** by a later + record, or corrected by a marked **progressive insight**. Always qualified in this repository: bare + *record* has other meanings (see [Homonyms](#homonyms)). +- **clause** — one numbered statement a decision record makes, with its own status (`proposed`, `in force`, + `changed`, `superseded`): a later record may change one clause and leave the rest + ([ADR 0290](../02-DECISIONS/0290-a-decision-record-lives-whole-in-the-knowledge-base-made-of-clauses.md)). Not a *rule*, which is a line of + `00-META/how-we-build.md`. +- **knowledge base** — the role that holds the decision records and research efforts whole (number, status, topic, + typed links, clauses and reasoning, with every version), takes them as proposals, has them reviewed and accepted, + and answers what holds: the seat `knowledge-base`, held by the module `mesh-knowledge` + ([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader + of this repository. +- **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built). +- **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which + holds it whole: its report, diagnosis and resolution included ([ADR 0291](../02-DECISIONS/0291-an-issue-lives-whole-in-the-issue-tracker.md)); + the folders in `04-ISSUES/` are imported there and removed. + An **incident** is a past event that taught a rule. +- **issue tracker** — the role that holds the mesh's issues and hands out their numbers: the seat `issue-tracker`, + held by the module `mesh-issues` ([53 — The issue tracker](../03-DESIGN/01-to-be/53-the-issue-tracker.md)). + Not hits's *item*; bare *ticket* is a homonym (see [Homonyms](#homonyms)). + *Not:* ~~ticketing system~~ (hq), ~~ticket system~~ (hq) +- **duplicate check** — the issue tracker's judgement, when an issue is opened, of whether it asks a question an + older issue not resolved already asks; a certain duplicate is closed `wontfix` with its report kept on the older + issue ([ADR 0278](../02-DECISIONS/0278-the-issue-tracker-checks-every-new-issue-for-a-duplicate-through-a-judgement-the-agent-makes-without-tools.md)). + Its pass over the backlog is the verb `groom`. +- **triage** — the issue tracker's judgement of an opened issue, made by the agent with no tools: whether it is a + duplicate (the **duplicate check**), its **severity** and what it **needs**, a suggested owner, and the issues it + depends on or relates to; it fills only what nobody gave ([ADR 0279](../02-DECISIONS/0279-the-issue-tracker-triages-every-new-issue-and-an-issue-is-ready-when-what-it-depends-on-is-delivered.md)). +- **ready** — of an issue: every issue it depends on is wontfix, or resolved with its fix delivered. +- **worker** — of an issue: the agent or person, with its node (who@node), that moved it to `diagnosing` or `fixing` + and works on it; another is refused while the worker is current, unless it gives a reason to **take it over**. A + worker with nothing done on the issue for a day is **stale**. **Fixing** is the status an issue is at while its fix + is worked on; an issue is resolved only from it ([ADR 0284](../02-DECISIONS/0284-an-agent-moves-an-issue-to-fixing-under-its-own-name-and-an-issue-is-resolved-only-from-fixing.md)). + Not a *claim* or a *holder*, which are a module's on a seat. +- **tracks** — of an issue and a condition: the issue links the condition's key with a link of kind `condition`. The + **tracking line** is the line the router says in every message about a condition: the issues that track it and + where each stands, or that none does ([ADR 0285](../02-DECISIONS/0285-an-issue-tracks-a-condition-by-its-key-and-every-message-about-the-condition-says-the-issue-and-where-it-stands.md)). +- **hq check** — one of `00-META/checks/`, run by `merge-check.sh` as this repository's check. + +## Homonyms + +A word below means different things in different domains. In a governing document it is never bare; +it is always the qualified form. **Checked by review, not by `words.py`:** a word list cannot tell one +sense from another, so the reviewer looks for the bare word in the diff. Where a homonym is settled by +renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. + +| Word | Sense | Say | Domain | +|---|---|---|---| +| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement | +| | what a delivery would do to the mesh | **delivery plan** | Change and delivery | +| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery | +| | a step a person starts, like the bus's | **planned step** | Core | +| membership | the subjects a bus account is issued | **membership** | Provisioning | +| | an account in a group of the machine | **account group** | Placement | +| push | the controller giving nodes their declarations | **send** (verb `push`) | Placement | +| | a git push | **git push** | The record | +| | a phone notification | **push notification** | Operator and conversation | +| release | a person letting a held delivery go on | **release** | Change and delivery | +| | taking a consumer off a licence | **unbind** (verb `release`, to rename) | Identity and access | +| gate | the judgement on a delivery's first node | **first-node gate** | Change and delivery | +| | the pull request status | **merge gate** | Change and delivery | +| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement | +| tier | a level of the mesh | **layer** | Core | +| | a step of a walk | **tier** | Change and delivery | +| | how much proof an ask needs | **assurance level** | Identity and access | +| ask | a request for the operator's input | **ask** | Operator and conversation | +| | an entry in the build queue | **build request** | Change and delivery | +| store | the one database server | **store** | Core | +| | the OCI registry | **artifact store** | Change and delivery | +| | a node's own copy of its last declaration | **last declaration** | Placement | +| record | a numbered decision | **decision record** | The record | +| | the literal reader of this repository, searched first for a symptom | **the record** (the `records` module) | Operator and conversation | +| | the upgrade policy that sends nowhere | `record` | Module | +| knowledge base | the module holding the decision records and research efforts whole | **knowledge base** | The record | +| | the predecessor's operational memory, in older playbooks (issue 430) | none: it no longer exists | — | +| check | a pull request's two statuses | **merge check**, **merge gate**, **repository check** | Change and delivery | +| | a delivery group's verdict | **composed check** | Change and delivery | +| | the controller's own examination | **self-check** | Health and repair | +| | a file in `00-META/checks/` | **hq check** | The record | +| agent | a coding agent | **agent** | Operator and conversation | +| | the build seat's holder | **builder** | Change and delivery | +| the user | an account of a wrapped program, or of the bus server | **its account**, **bus account** | Identity and access | +| | a human | **operator** or **person** | words every domain uses | +| firewall | what the mesh enforces | **packet filter** | Connectivity | +| | what was found on a machine | **found firewall** | Connectivity | +| ticket | an answer's one-time token (ADR 0258) | **answer ticket** | Operator and conversation | +| | a question someone must answer | **issue** | The record | +| deploy | the per-node part of a delivery plan | **deploy plan** | Change and delivery | +| | a wrapped program's own deploy | its own word | — | + +## How this page is kept + +A new word for an existing thing lands here first, in the domain that owns it, in the same change that +introduces it in code. A word moves to another domain only with a decision record. A retired word goes +on a *Not:* line and nowhere else on this page, and `words.py` then fails on it in running prose. A +record under `02-DECISIONS/` keeps whatever word it was written with — those are immutable — so a word +retired here may still appear there, and the *Not:* lines are how to read it. + +**How it is checked** ([ADR 0244](../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md)): +`python3 00-META/checks/words.py`, run by `merge-check.sh` on every pull request, fails on a retired +word or a bare identifier in running prose in `00-META/`, `03-DESIGN/`, `AGENTS.md`, `README.md`, and +research and issues dated from 2026-10-07 — code spans, quotations and link targets excepted — and on a +head word that heads two entries or is also retired. The words retired with no scope or with *(tools)* +are copied into the catalogue as `retired-words`, whose own repository check holds the descriptions of +the mesh's tools to them; a change here that retires such a word changes that copy in the same delivery. +Homonyms are checked by review. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/00-overview.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/00-overview.md new file mode 100644 index 00000000..b4272350 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/00-overview.md @@ -0,0 +1,86 @@ +--- +status: graduated +initiated: 2026-10-10 +touches: + - 02-DECISIONS/0019-how-this-repository-works.md + - 02-DECISIONS/0080-the-development-cycle-is-checked.md + - 00-META/process/03-issues.md + - 00-META/glossary.md + - the runtime's state verbs and the SDK's state interface +became: + - 03-DESIGN/01-to-be/53-the-issue-tracker.md + - 02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md +--- + +# 042 — The mesh keeps its own issues + +## What + +Whether the mesh should keep its issues in a module of its own, on the bus, instead of only as +numbered folders in this repository's `04-ISSUES/`, and what the smallest useful first version of +that module is. The operator's ask of 2026-10-10: work on a ticketing system on the bus, consolidate +every current issue into it, start small and extend the module along the way. + +The effort answers six questions, in [01-findings.md](01-findings.md): + +1. The smallest useful first slice: what the module holds, which tools it serves, which events it + emits, and what it explicitly leaves out. +2. Where the truth lives during and after the move: the folders here, the module, or a split + between them, and what each choice does to the public-repository rule, playbook 03 and the hq + checks. +3. How the 357 issue folders are imported and consolidated (duplicates, still-open ones, grouping). +4. How the controller's conditions, the router's asks and deliveries relate to issues. +5. How the operator sees issues: through the mesh's tools first, through a view onto the bus later. +6. The open questions for the operator, and the slices after the first. + +The operator also named an existing project as inspiration: *hits*, a headless issue tracker built +as NATS micro services on JetStream, with an MCP server for agents. Its model is studied in the +findings, along with whether to reuse its code or build on its model. + +## Why + +Numbers collide. Playbook 03 tells an author to take the next free number across `main` and every +open pull request, and [issue 155](../../04-ISSUES/155-two-records-may-share-a-number-and-nothing-says-so/00-report.md) +records the second collision reaching `main` with every check passing. `cycle.py` now refuses two +records sharing a number, which catches a collision after it happens and does not prevent one. On +`main` at the time of writing, 31 commits mention a renumbering. On 2026-10-07 three decision numbers +collided in one evening. On 2026-10-10 an open pull request still carries a branch named for issue +349 and a title naming issue 350, because 349 was taken on `main` first. The operator's answer on +2026-10-07 was that numbering belongs to the mesh's own issue tracker. + +Status is also hard to see. Of 357 issue folders on `main`, 73 are not closed (27 `open`, 45 +`located`, 1 `diagnosing`). Finding them means reading frontmatter across the repository. +Nothing connects a condition the controller raises to the issue that would fix it: on 2026-10-10 a +condition was silenced with the words "hq issue to follow", and nothing records whether one followed. + +## What it touches + +- [ADR 0019](../../02-DECISIONS/0019-how-this-repository-works.md) (statuses live in frontmatter) and + [ADR 0080](../../02-DECISIONS/0080-the-development-cycle-is-checked.md) (the cycle is checked), with + playbook [03](../../00-META/process/03-issues.md) and `00-META/checks/cycle.py`. +- [ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md) + (a module keeps its state in key-value buckets it declares), + [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md) + (tools announce themselves on the bus) and + [ADR 0041](../../02-DECISIONS/0041-events-are-a-relationship.md) (events are declared). +- The conditions of [ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md) + and the asks of [ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md). +- The records module, which indexes this repository for agents. + +**Graduated 2026-10-10** into [ADR 0275](../../02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md) +and [to-be 53](../../03-DESIGN/01-to-be/53-the-issue-tracker.md). The operator delegated the five open questions; each +was decided as recommended. `touches:` names what the decision changes. + +## Words + +The glossary's word for one of these is **issue** (*The record*: "a report in `04-ISSUES/`"). This +effort keeps that word for the thing the module holds, and proposes **issue tracker** for the module +that holds them, with the module named `issues`. It does not use *ticket*, the operator's word in +the ask, or *item*, the word hits uses. If the glossary entry is widened (an issue is then held by +the issue tracker, and its narrative may live in `04-ISSUES/`), *ticket* would go on its *Not:* line. +That change belongs to the decision this effort graduates into, not to the effort itself. + +## Open questions for the operator + +Listed in [01-findings.md § 7](01-findings.md#7-open-questions-for-the-operator), each with a +recommendation. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/01-findings.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/01-findings.md new file mode 100644 index 00000000..e14351a7 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/042-the-mesh-keeps-its-own-issues/01-findings.md @@ -0,0 +1,259 @@ +# 01 — Findings + +Measured on 2026-10-10 against `main` of this repository, the open pull requests on the forge, the +controller's open conditions, the module SDK's source, and a clone of the hits repository at its +release v0.7.0. Each section ends with what this effort recommends; section 7 lists what needs the +operator. + +## 0. What exists today, measured + +| Fact | Measured | +|---|---| +| Issue folders on `main` | 357, numbered up to 362; numbers 283 and 284 were never used, and 338, 341 and 350 sit on open pull requests (363 too) | +| Status in frontmatter | 283 `resolved`, 1 `wontfix`, 45 `located`, 27 `open`, 1 `diagnosing` | +| Frontmatter keys | `status`, `opened`, `located-in` on all 357; `fixed-by` 356; `amended-design` 344; `replay` 22; `replay-none` 44 | +| Size | 1.9 MiB of markdown in all; median folder 4.6 KiB, largest 25 KiB; 244 folders hold only a report, 111 also a diagnosis | +| Links between issues | 168 folders link another issue's folder (265 links); 193 name another issue in prose | +| `located-in` | free text: the controller is written `mesh-controller`, `cmd/mesh-controller`, `mesh-control` and as file paths; 17 issues name nothing | +| Renumberings | 31 commits on `main` mention one; one open pull request still carries the old number in its branch name | +| The forge's own issues | none in this repository or the controller's; [issue 178](../../04-ISSUES/178-a-routed-name-resolves-to-a-provider-merely-told-it/00-report.md) was filed there once and moved here, because "its numbers collide with hq's pull request numbers in conversation" | +| Conditions | 10 open; 5 of kind `healer-wanted` (a hand repair repeated, "an automatic repair is wanted"); one silenced with "hq issue to follow" | + +Two things follow. The collisions come from allocating a number by reading a repository that many +branches write at once; nothing short of one allocator removes them. And the frontmatter is a small, +regular data set (six keys, 1.9 MiB) that fits easily in the bus's key-value store under the limits +[ADR 0201](../../02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md) +§6 sets (256 KiB a value, 64 MiB a bucket). + +## 1. The first slice + +**A Go module `issues`, one instance on the control-node, serving one seat held once for the mesh.** +Its state is in buckets it declares (ADR 0201), its tools announce themselves on the bus +([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)), +and every change is an event it declares under `emits` +([ADR 0041](../../02-DECISIONS/0041-events-are-a-relationship.md)). The module that delivers pull +requests already has this shape: Go, one mesh seat, two state buckets, declared events. + +**What one issue holds in v1**, the frontmatter of today plus what the folders carry in prose: + +- `number` (the key), `title`, `status` (`open`, `diagnosing`, `located`, `resolved`, `wontfix`), + `opened`, `closed`; +- `located-in`, from a registered list of repositories and modules rather than free text; +- `fixed-by` (pull requests or commits), `amended-design`, `replay` or `replay-none`; +- `notes`, appended and never edited: who wrote, when, and the text; +- `links`, typed: to another issue (`duplicates`, `relates-to`), to a pull request, a decision + record, a design, a condition key or a delivery; +- `record`: the path of the issue's folder in this repository, when it has one (section 2). + +**Storage.** One bucket `issues` with one key per issue, keeping a short history per key, and a +`counter` key for the next number. The events (`opened`, `changed`, `noted`, `linked`) carry what +changed and who changed it, so they are the audit trail, and a view can watch the bucket. Section 6 +compares this with a stream as the source of record. + +**Tools** (verbs of the seat, found with `mesh_search`): + +| Tool | Does | +|---|---| +| `open` | takes the next number atomically and writes the issue; refuses a title or report that is empty | +| `get` | one issue, whole | +| `list` | by status, by `located-in`, by a phrase in title or notes; compact rows, newest first | +| `move` | changes status, enforcing today's cycle rules at the write: `located` needs `located-in`; `resolved` needs `fixed-by`; a core issue needs `replay` or `replay-none`; `wontfix` needs a reason | +| `note` | appends to the trail | +| `link` / `unlink` | adds or removes a typed link | +| `import` | reads this repository's issue folders and writes them as issues with their own numbers (section 3); idempotent, reporting what it would change before it does | + +**The caller is the bus credential, not an argument.** The module records who acted from the +identity the runtime gives it with each call, so an agent cannot write as someone else. + +**Atomic numbering needs compare-and-set.** The SDK's state interface (v0.1.9) offers `get`, `put`, +`delete`, `keys` and `watch`, and no write that is refused when the key changed since it was read. +Whether the runtime's `mesh/state.put` takes an expected revision was not checked; the tool runner's +code was not read for this effort. Two ways out: add `create` and an `update` that takes the revision +read to the state interface (small, and every module with a counter gains it), or run exactly one +instance and number inside it. **Recommended: the first.** One instance is also right for v1, but the +counter should not depend on it. + +> **Correction — 2026-10-10.** The finding above is wrong about the SDK. Version v0.1.9, the one read, had no +> compare-and-set, but the SDK's Go state interface already had it from go/v0.1.10: `Create` (writes a key only if it +> does not exist) and `Update` (writes only at the revision the caller read), with the runtime's matching state +> verbs, both from [ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md). +> What the tracker's counter needed and slice 1 added is `Increment`, in go/v0.1.12, which reads a counter and writes +> the next number with `Create` or `Update` at the revision it read, retrying a bounded number of times. The +> recommendation stands; the work it named was smaller than stated. ADR 0275 carries the same correction. + +**Explicitly left out of v1:** decision and research numbers (slice 2), any change to playbook 03 or +the hq checks (slice 2), duplicate finding beyond links (slice 3), opening an issue from a condition +(slice 4), a web view (slice 5), full-text or semantic search, priorities, claims, notifications, and +the forge's own issues. + +## 2. Where the truth lives + +| | (a) Split: the tracker holds the issue, hq holds the narrative | (b) Everything moves | (c) The tracker mirrors hq | +|---|---|---|---| +| Number | tracker | tracker | hq, so collisions stay | +| Status, `located-in`, `fixed-by`, links | tracker | tracker | hq frontmatter, copied | +| Report and diagnosis | hq folder for a design or governance question; notes in the tracker for the rest | tracker notes | hq | +| Public-repository rule | holds: hq keeps only scrubbed narrative, the tracker is private and may name nodes | the public record of design questions is lost | holds | +| Playbook 03 | changes: open in the tracker first, write a folder only for a design or governance question | replaced | unchanged | +| `cycle.py` | its issue rules move into the tracker's `move`, refused at the write instead of at merge | same | unchanged, and a second copy to keep in step | +| Records module | still finds the narrative; the tracker's `list` and `get` serve status | finds no issues | unchanged | + +Option (c) keeps the collisions and adds a second copy that can drift, so it answers neither of the +operator's problems. Option (b) loses the public record that hq exists to be +([ADR 0019](../../02-DECISIONS/0019-how-this-repository-works.md)), and with it the +symptom-to-component memory that playbook 03 says is never deleted. + +**Recommended: (a).** Every issue gets a number from the tracker, including ones that never need a +document here (a node's quirk, an operational follow-up, which playbook 03's table already sends +elsewhere). A design or governance issue also gets a folder `04-ISSUES/NNN-slug/` with the tracker's +number, whose frontmatter shrinks to a pointer (`issue: NNN`) and whose prose stays reviewed in pull +requests. Status lives in one place, the tracker. This reverses ADR 0019's "statuses live only in +frontmatter" for issues and moves part of ADR 0080's check, so it needs a decision record that +supersedes those parts, written when this effort graduates. + +**Numbering continues.** The tracker's counter starts above the highest number on `main` and on open +branches at the moment of the switch (today 363, so the first new issue is 364). Numbers 283 and 284 +stay unused, and a number never moves again. + +**During the move**, hq frontmatter stays authoritative until the import has run and been checked +against it, and one pull request then removes the status keys from every report and changes playbook +03 and `cycle.py` together. There is no period in which both are written. + +## 3. Importing and consolidating the 357 issues + +**Import.** `import` reads the folders through the records module (`records_list`, `records_read`), +over the bus, rather than from a checkout. Each folder becomes an issue with its own number, its +frontmatter as fields, its title from the heading, the commit it was read at, and its path as +`record`. Folder-to-folder links (265) become `relates-to` links. The four open pull requests' +issues (338, 341, 350, 363) are reserved as numbers and imported when they merge. + +**Consolidating** means four concrete passes, each producing a list for the operator or an agent to +confirm, never applying itself: + +1. **Close what is fixed.** A `located` or `open` issue whose `located-in` repository has a merged + pull request naming its number is a candidate for `resolved`. hits's auditor (section 6) checks + exactly this against git and is the model. +2. **Merge duplicates.** Candidates are pairs that link each other and share a `located-in`, and + pairs whose titles share most words. A confirmed duplicate is closed `wontfix` with a + `duplicates` link and the reason, keeping its number and folder. +3. **Normalise `located-in`** onto the registered list (`mesh-controller`, `mesh-catalog`, the + node-engine's repository, a module's name), keeping the file paths as a note. +4. **Group the 73 not closed** by domain (the glossary's domains, ADR 0244) and by `located-in`, so + the operator sees, for example, how many open issues the delivery domain carries. + +## 4. Conditions, asks and deliveries + +A condition is one open fact, raised and cleared only by observation (ADR 0231); an issue is a +question someone must answer. They are different things, and a condition must not be closed by +closing an issue. But a condition can say that a fix is owed: five of today's ten are `healer-wanted`, +and one silence says "hq issue to follow". + +Recommended for a later slice (slice 4): an issue links a condition by its key; the controller shows +the link on the condition; and for `healer-wanted` the controller opens one issue per condition key +when it first raises it, linked both ways, instead of an agent remembering to. A silence whose reason +names an owed issue should carry its number. Asks +([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md)) +stay with the router: the tracker asks nothing in v1. Deliveries relate through pull requests: when +the forge says a pull request merged, the tracker notes it on every issue the pull request names, and +offers `resolved` (slice 4). Nothing in v1 depends on these. + +## 5. How the operator sees issues + +First through the mesh's tools: `list` returns rows short enough to read in a chat, `get` one issue +whole, and every agent session reaches them through the mesh MCP server. The responder (`@mesh`, +ADR 0234) answers from read verbs, so "what is open in delivery" works on a channel once the seat +exists. Later a view onto the bus: the `issues` bucket watched over the bus (a browser reaches it through a WebSocket listener, which this effort did not check for), +read-only, which the operator named as a future direction for every module's state. Nothing in v1 +blocks it, because the state is already a bucket and every change an event. + +## 6. hits: what it is, and what to take from it + +**What it is.** A headless issue tracker in Go (module `github.com/impire-io/hits`), with no web +interface and no HTTP interface: NATS micro services on JetStream, a command line for people and a +stdio MCP server for agents. One author, 81 commits between 2026-09-02 and 2026-09-14, release +v0.7.0, about 9,700 lines of Go outside tests and 21 test files, tested against an embedded NATS +server. Its model descends from a method like this repository's: a sibling `hits-hq` with numbered +playbooks and decisions. + +**Data model.** An *item* has a type (`bug`, `task`, `improvement`), a status (`open`, `diagnosing`, +`located`, `blocked`, `resolved`, `wontfix`), a priority, `located-in` (from a registered list of +*projects*), `fixed-by` (pull request, commit or action), `amended-design`, append-only notes, typed +links (`duplicates`, `relates-to`), a claim, a block, and a tombstone for a filing mistake. Its +statuses and fields are this repository's frontmatter almost key for key. Projects, *initiatives* +(groups of projects that also scope the numbering) and *releases* are registered lists that are +never reused once retired. + +**How it stores.** The source of record is a stream `hits-ops` with subjects +`hits.ops.item.` (and `project`, `initiative`, `release`). Every write appends an operation, +published with the per-subject expected last sequence, so two writers to one item cannot both win. +A bucket `hits-state` holds the folded snapshot of each item, the registered lists and one counter +per initiative. The counter is advanced by compare-and-set and an item is numbered only after it is +validated, so a refused create spends no number. On start it rebuilds the bucket from the stream. + +**Services and subjects.** One micro service `hits` with 23 endpoints under `hits.api.*` (`create`, +`get`, `edit`, `transition`, `claim`, `release`, `block`, `unblock`, `link`, `unlink`, `note`, +`tombstone`, and register, list and retire for projects, initiatives and releases); three index +services, each a projection of the stream: full text (`hits.search.query`, a local index on disk), +a graph (`hits.graph.neighbors`, `hits.graph.walk`) and an optional semantic search that needs an +outside embedding provider. + +**MCP tools.** 27, one per endpoint: `create_item`, `get_item`, `edit_item`, `transition_item`, +`claim_item`, `release_item`, `block_item`, `unblock_item`, `link_items`, `unlink_items`, +`note_item`, `tombstone_item`, the project, initiative and release tools, and `search_items`, +`semantic_search`, `graph_neighbors`, `graph_walk`. One server serves one agent under one actor. +Its command line also has an auditor that checks every item's `fixed-by` against git and finds +merged pull requests whose items are still open. + +| | Adopt as it is | Adapt to the mesh | +|---|---|---| +| Statuses, fields, append-only notes, typed links, tombstone | yes: they are already ours | — | +| Stream of operations as the record, bucket as its projection | the pattern | the stream and bucket declared in the manifest and created by the controller (ADR 0201 §3), not created by the service on start | +| Expected-sequence writes; compare-and-set counter; number only after validation | yes | needs compare-and-set in the state interface (section 1) | +| The auditor | the idea, as a tool | reads the forge through the gitea module, not a local clone | +| Subjects `hits.api.*`, its own MCP server | no | the seat's verbs, served by the tool runner and found with `mesh_search` | +| Actor named by the caller | no | the caller is the bus credential the mesh issued | +| Contexts, OAuth device flow, semantic search | no | the mesh issues credentials; no outside provider | +| *item*, *project*, *initiative*, *release* | no | *issue*; a repository or module; no initiatives in v1; *release* is already a homonym here (glossary) | + +**Reuse its code, or build on its model?** Its licence is the Sustainable Use License 1.0: use and +modification are allowed for internal business, non-commercial or personal use, and distribution only +free of charge and for non-commercial purposes. It is not an open-source licence. Running it on a +personal mesh is allowed; vendoring it into the catalogue binds the catalogue to that licence for as +long as the code is there. Its services create their own stream and bucket on start, which ADR 0201 +§3 forbids a module to do; its subjects carry no prefix and assume one tracker per bus account; its +words clash with the glossary. Changing those means a fork of a two-week-old, single-author project. +v1 here is the item store without the indexes: roughly the size of the module that delivers pull +requests. + +**Recommended: build our own module on hits's model,** and say so in the module's README. Do not +depend on or vendor its code. Revisit only if its licence changes, or if its author agrees to a +licence for this use. + +## 7. Open questions for the operator + +1. **Should the tracker hold every issue's number and status, while hq keeps a folder only for design + and governance issues, pointing at the tracker by number (option (a) in section 2)?** + Recommended: yes. It ends the collisions, keeps the public record, and moves the cycle rules into + the write, where they refuse instead of report. +2. **Should we build our own `issues` module on hits's model rather than run or vendor hits?** + Recommended: build our own. hits's licence limits distribution, its services create their own bus + resources (against ADR 0201), and its words clash with ours. Its model, its compare-and-set + numbering and its auditor are worth copying. +3. **Should the tracker also hand out decision and research numbers?** Recommended: yes, in slice 2. + Decision numbers collided three times on 2026-10-07; the same counter mechanism serves any + sequence, and the tracker then reserves the number when a branch starts. +4. **Should the word stay *issue*, with the module named `issues` and *ticket* retired?** + Recommended: yes. *Issue* is the glossary's word for this, and one word per thing (ADR 0244). +5. **Should the controller open an issue by itself for a `healer-wanted` condition, or only link + one an agent or the operator opened?** Recommended: link only, until slice 4 has run for a week; + then open by itself for `healer-wanted`, one issue per condition key. + +## 8. Slices after the first + +| Slice | Delivers | Done when the operator sees | +|---|---|---| +| 1 | The `issues` module: open, get, list, move, note, link, import; compare-and-set in the state interface | the next issue number comes from the tracker and the 73 not closed are listed by one call | +| 2 | Decision and research numbers from the tracker; playbook 03 and `cycle.py` changed; status keys removed from every report | two agents opening at once get two numbers, and no hq pull request is renumbered | +| 3 | Consolidation: the four passes of section 3, and an audit tool for `fixed-by` against the forge | the not-closed list shrinks by what was already fixed or duplicated, each with its reason | +| 4 | Conditions and pull requests linked; `healer-wanted` opens an issue | a condition shows its issue, and a merge notes the issues it names | +| 5 | A read-only view onto the `issues` bucket | the operator reads issues in a browser without an agent | diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md new file mode 100644 index 00000000..8025d394 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md @@ -0,0 +1,112 @@ +--- +status: graduated +initiated: 2026-10-10 +touches: + - 02-DECISIONS/0019-how-this-repository-works.md + - 02-DECISIONS/0080-the-development-cycle-is-checked.md + - 02-DECISIONS/0153-the-record-is-read-by-a-module-and-the-console-lists-it.md + - 02-DECISIONS/0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md + - 02-DECISIONS/0274-an-agent-proposes-a-shared-instruction-section-and-every-node-takes-it-only-on-the-operators-warrant-which-it-proves-itself.md + - 02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md + - 03-DESIGN/01-to-be/53-the-issue-tracker.md + - 00-META/process/ + - 00-META/checks/ + - 00-META/glossary.md + - the records module's tools +became: + - 02-DECISIONS/0290-a-decision-record-lives-whole-in-the-knowledge-base-made-of-clauses.md + - 02-DECISIONS/0291-an-issue-lives-whole-in-the-issue-tracker.md + - 03-DESIGN/01-to-be/55-the-knowledge-base.md +--- + +# 045 — A knowledge base the mesh writes: decisions, guidelines, policies + +## What + +Whether the mesh should **write** its own knowledge (decision records, guidelines, policies, designs and +playbooks) through a module's verbs, instead of only **reading** hand-written markdown, and whether it +should hold light content beside them: meeting notes, architectural drawings and blog posts. + +The operator's purpose, given on 2026-10-11: the record becomes **shared knowledge that agents reach +through tools**, so that an agent gets what was decided as an answer instead of reading markdown files, +and a session no longer has to start in this repository's folder to know it. A session still starts here +for other work, such as writing a record. The operator later put the light content last, and decided that +an issue's content belongs in the issue tracker rather than in this repository. + +Today the record is read-only. The `records` module reads this repository and serves it to sessions +([ADR 0153](../../02-DECISIONS/0153-the-record-is-read-by-a-module-and-the-console-lists-it.md)). +Everything in it is written by hand as markdown and reaches `main` through a pull request; the hq checks +in `00-META/checks/` judge it only after it is written. The issue tracker has already taken issues out +of this path: it numbers and tracks them in a module of its own, and hq keeps only the narrative +([ADR 0275](../../02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md)). + +## Why + +Numbers collide, and the checks find a collision only after it happens. On 2026-10-10, research number +043 was taken by one open pull request while a second session held an unpushed branch claiming the same +number for a different effort. On 2026-10-07 three decision numbers collided in one evening. A +record written by a verb could take its number from one counter, the way the issue tracker does. + +Reading is costly too. Answering one question about what holds today means reading several records +whole, and the generated decision index is larger than one tool answer may be. + +Some knowledge also has no home. This repository is public, so it may not name nodes or describe +operations; the mesh's knowledge about its own installation lives in agents' private notes, in no +shared and searchable place. + +## The two options to weigh + +- **A. Git stays the store.** The `records` module gains write verbs (propose, amend as a progressive + insight, supersede, retire), each of which makes a branch and a pull request through the forge. Numbers + come from the issue tracker's counters, as a further sequence beside `issue` + ([to-be 53](../../03-DESIGN/01-to-be/53-the-issue-tracker.md) §4). +- **B. A store is the truth, and hq is its rendered public export**, as the issue tracker already is for + issues (ADR 0275). + +## The questions + +1. **Types and lifecycles.** Which kinds of knowledge exist: decision, guideline, policy, design, + playbook, and any others? What lifecycle does each have (proposed, accepted, superseded, retired; + to-be and as-is for a design), and which transitions does each kind allow? +2. **Numbering from one counter.** One sequence per kind, or one for all? How is a number reserved for + a draft that is later abandoned, given that a number is never reused? How are today's collisions + (043 is the latest example) resolved on import? +3. **Typed links.** Which links between records are needed: supersedes, amends, implements, cites, + decided-by, and others? Which of them does frontmatter express today (`became:`, `decisions:`, + `fixed-by:`), and which exist only as prose links that no check reads? +4. **Approval.** A decision or a policy changes how the mesh is run. Should accepting one require the + operator's warrant on a channel that proves who answered, as a shared instruction section already + does ([ADR 0274](../../02-DECISIONS/0274-an-agent-proposes-a-shared-instruction-section-and-every-node-takes-it-only-on-the-operators-warrant-which-it-proves-itself.md))? + Which kinds need it, and which may an agent write on its own? +5. **Public and private knowledge.** Node names and operational knowledge have no home today. Does the + knowledge base hold private records beside public ones, with a rule for which may be exported? How is + a record checked so that nothing private reaches the public repository? +6. **Checks as refusals at write.** `records.py`, `cycle.py`, `words.py` and `index.py` judge a pull + request after it is written. Which of their rules can become refusals of the write verb itself, and + which only make sense across the whole record? +7. **Import.** How are the decision records (274 on `main` on 2026-10-11, numbered up to 0289), the designs and the playbooks imported without + losing their history, their numbers or their links? Is the import repeatable, as the issue tracker's + is? +8. **What git gives today.** Review on a pull request, line comments, a history of every change with + its author, a diff, a revert, and the forge's required checks. What of that would a store have to + rebuild under option B, and what would option A lose by making every write a pull request? + +## Words + +The glossary has words for **decision record**, **design**, **playbook** and **the record** (the +`records` module), but none for **guideline** or **policy** in this sense, and *policy* is already used +for a module's **upgrade policy**. This effort uses "guideline" and "policy" as plain descriptions, and +"knowledge base" for the whole, until a decision settles the words through the glossary's own rules. + +## What it touches + +How this repository works (ADR 0019) and how the development cycle is checked (ADR 0080), the +`records` module (ADR 0153) and the generated decision index (ADR 0248), the issue tracker and its +counters (ADR 0275, to-be 53), approval on a proving channel (ADR 0274), the playbooks in +`00-META/process/`, the hq checks in `00-META/checks/`, and the glossary. + +## Documents + +- [01 — Findings](01-findings.md): what exists, measured on 2026-10-11; where the truth lives (the store + for each record's structure and rules, this repository for its reasoning); issues whole in the issue + tracker; the order of work; and what is still open. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md new file mode 100644 index 00000000..dacb0bfd --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md @@ -0,0 +1,295 @@ +# 01 — Findings + +Measured on 2026-10-11 against `main` of this repository, the records module and the issue tracker as they +answer through the mesh MCP server, and the hq checks in `00-META/checks/`. Section 0 is what exists; section 1 is where the truth lives, which +decides the rest; sections 2 to 9 answer the overview's questions and the operator's additions; section 10 is +the order of work, and section 11 what is still open. + +## The purpose, in the operator's words + +The operator gave the purpose on 2026-10-11: the decisions and the rest of the record should become **shared +knowledge that agents reach through tools**, instead of reading markdown files, and a session should no longer +have to start in this repository's folder to know what was decided. A session still starts here for other work, +such as writing a record. Later that day the operator widened the scope to **light records** +(meeting notes, architectural drawings, blog posts) and then put them last; asked for a structured store of +the decisions and research efforts; and decided that an issue's content belongs in the issue tracker, not in +this repository. + +That purpose moves the weight of the overview's questions. Where the truth is stored (option A or B) matters +less than **what an answer looks like**: an agent needs the rule that holds now, with the records it rests on, +in one call and a few kilobytes. + +An acceptance sentence for the whole effort: *a session started in any folder on any node asks "what holds +about the merge window, and why" and gets, in one call of under 4 KiB, the rules in force, the records they +come from, what changed them since, and the commit it was read at.* + +## 0. What exists today, measured + +| Fact | Measured | +|---|---| +| Records on `main` | 274 decision records, 44 research efforts (119 files), 55 to-be and 16 as-is designs, 8 playbooks, 361 issue narratives (479 files): 951 files, and the glossary | +| Size | 6.9 MB of markdown, about 1.7 million tokens at four bytes a token; decision records alone 2.6 MB (about 650,000 tokens) | +| One question, today | "what holds about the merge window": `records_search` gives 26 places in 11 documents; reading the five records and the design they point to is 85 KB, about 21,000 tokens | +| The decision index | `records_decisions` answers 77 KB in one piece; the agent's tool-answer limit refused it inline, and it had to be read from a file in parts | +| What `records_search` does | a literal, case-insensitive phrase match giving document, line and heading; no ranking, no notion of which record is still in force | +| Frontmatter of a decision | `topic`, `status`, `date`, `deciders` on all 274; `extends` on 229 (182 with the folder prefix, 47 without); `supersedes-in-part` on 15 (21 links), `supersedes` on 8, `superseded-by` on 9, `rests-on` on 2, `amended` on 1 | +| Status of a decision | 265 `accepted`, 9 `superseded`; no record is marked as changed in part, though 21 links say so | +| Supersession said only in prose | 6 records say they supersede, amend or replace another and carry no field for it; 7 records carry a "superseded" marker in the body, of which only 1 has the field or the status | +| Progressive insights | 39 records carry the dated marker, 55 markers in all; 2 more use an older form | +| Change after writing | 189 of 271 traceable decision records were changed after the commit that created them; 34 paths were changed five times or more | +| Pace | commits touching decision records: 87 in the last ten days of August, 160 in September, 274 in the first eleven days of October | +| Number collisions | 16 decision numbers were taken twice by sessions working at once (2026-09-25 to 10-10), each repaired by a renumbering commit; 15 numbers on `main` are gaps; no research number collided on `main` (the 043 collision of the overview never reached it) | +| A decision's rules | 263 records have a `Decision` section; only 37 of them write it as numbered rules with a bold lead (198 rules in all); the rest write prose | +| Drawings | no file but markdown in the repository; no diagram source; 22 designs carry diagrams drawn in text characters | +| Knowledge with no home | the repository is public, and its README says what only makes sense to someone who knows this installation "is either a note in the wrong place or a disclosure"; one agent's private notes hold 89 files for one repository on one node, and issue 367 (a session's memory stays on the node where it was learned) is open | +| The hq checks | 33 rules: 25 judge one record, 4 the whole record, 4 the prose (section 7) | + +Three things follow. + +1. **The cost of an answer is the problem, not where the files are.** 85 KB for one question, and an index + too large for one tool answer, mean a session reads in parts or reads less than it should. Moving the + markdown into a store without changing the shape of the answer would keep that cost. +2. **The data that makes an answer cheap is missing or loose.** What a record decided is prose in 226 of 263 + records. Which records changed which is half in fields, half in prose: 6 records say it only in prose, and + 6 more carry a body marker that no field or status agrees with. + Status says `accepted` for records that 21 later links change in part. +3. **The records change after they are written, and that is reviewed.** 189 of 271 decision records were + edited after creation, every edit through a pull request with a human approval. Whatever is built must keep + that review for what governs the mesh. + +## 1. Where the truth lives: what a clean start would choose + +The operator asked on 2026-10-11 which option this effort would choose with no prior setup. The answer +decides the rest, so it comes first. + +**The unit the mesh reasons about is a rule, not a file.** A decision is a handful of rules. A later decision +changes one of them: ADR 0282 replaces one rule of ADR 0276 and leaves the others standing. Git and markdown +know files only, so that change can be written only as prose. The earlier record keeps `accepted`, and every +reader must read both records to learn what holds. Section 0's loose links all have this one cause: 21 partial +supersessions with no reverse, 6 said only in prose, 6 markers no field agrees with. + +**A store can hold what a repository cannot.** In a store, each rule is an entry with its own status and links +("in force", "changed by rule 2 of 0282"). A write that leaves the record inconsistent is refused when it is +made, not found by a check afterwards. Numbers come from one counter. "What holds now" is one query. + +**What git does best is review of prose.** The reasoning behind a decision (its context, the options weighed, +its consequences) is read and commented on by people. Agents rarely ask for it. They ask what holds. + +So a clean start divides **by layer, not by kind**: + +- **the store is the truth for the structure** of every governing record: its number, title, kind, topic, + status, its rules one by one, and its typed links, at the level of the rule where a link is about a rule; +- **this repository keeps the reasoning** as markdown under the store's number, reviewed in pull requests. + +That is the division the issue tracker already made for issues +([ADR 0275](../../02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md)): +the tracker holds number, status and links, and hq held the narrative. Section 9 takes issues one step further, +on the operator's word: their narrative leaves this repository too. + +Option A (git stays the truth, and a module's write verbs open pull requests, with numbers from a counter) was weighed and is not chosen. It keeps review, but it +keeps the rule-level links in prose and the refusals after the fact, which is the cause of section 0's mess. +Option B in full (the reasoning moves into the store too) is not chosen either: it would rebuild review, line +comments, history and diff for prose, which git already gives. + +## 2. Kinds and their lifecycles + +| Kind | Lifecycle | Notes | +|---|---|---| +| decision | proposed → accepted → superseded; a rule can be changed or superseded on its own; a dated progressive insight corrects a fact | 274 records | +| research | active → graduated or abandoned | 44 efforts | +| design | proposed → designed → in-progress → implemented, in one of two layers (to-be, as-is) | 71 documents; a later slice (section 10) | +| playbook | in force, or retired | 8, with no frontmatter today; a later slice | +| glossary entry | in force, or retired with its replacement | one file today; a later slice | + +The first slice holds decisions and research, the two the operator named. + +A **guideline** or a **policy**, which the overview named, has no record of its own today. What the operator +calls a policy is written as a decision; what an agent should do is a shared instruction section (ADR 0274) or +a personal rule ([ADR 0273](../../02-DECISIONS/0273-the-agents-home-is-the-modules-and-the-operators-personal-rules-and-lessons-are-registered-through-it.md)). +No new kind is proposed for either. + +**Light records** (meeting notes, drawings, blog posts) were added to the scope by the operator on 2026-10-11 +and put last by the operator the same day. They govern nothing, need no review to exist, and are private by +default; a drawing's file belongs in an object-store bucket +([ADR 0288](../../02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md)) +with a text description an agent reads. They are a later slice (section 10) and shape nothing in the first. + +## 3. Numbering from one counter + +The 16 concurrent collisions all come from reading the next number off a repository that many branches write +at once. The issue tracker hands out numbers by compare-and-set on a counter key +([to-be 53](../../03-DESIGN/01-to-be/53-the-issue-tracker.md) §4); the knowledge base asks it for every number. + +- **One sequence per kind**: decision, research, and later design. Numbers keep their meaning in conversation + ("ADR 0276"). +- **A reserved number is spent.** A proposal that is abandoned keeps its number with status `withdrawn`, who + reserved it and when. Gaps already exist (15 on `main`) and harm nothing; a reused number would. +- **The counter starts above the highest number on `main` and on every open branch**, as the tracker's did. +- **The counters stay where ADR 0275 put them.** Its rule 8 decided that the issue tracker hands out decision + and research numbers too, in its second slice; to-be 53 §4 so far names only the issue counter. The knowledge + base's `reserve` asks the tracker for the number, so one counter holds every sequence and no accepted record + is superseded for it. + +## 4. Rules and typed links + +**A rule** is one short statement a decision makes, numbered within its record ("0276 rule 1"), with its own +status: `in force`, `changed by` a rule of a later record, or `superseded`. 37 of 263 decision sections already +write numbered rules with a bold lead (198 rules); the other 226 write prose, from which the import drafts the +rules (section 7). + +The links the first slice holds, each stored in both directions: + +| Link | From → to | Today | +|---|---|---| +| supersedes | record → record | `supersedes` / `superseded-by`, 9 of 9 mirrored | +| changes | rule or record → rule | `supersedes-in-part` on 15 records, one direction; 6 more only in prose | +| extends | record → record | `extends` on 229, two spellings | +| became | research → decision or design | `became` | +| decided by | design → decision | `decisions` | +| corrects | progressive insight → record | the dated marker, 55 in 39 records | +| cites | any → any | prose links: 1,785 decision to decision, 469 decision to issue | + +"What holds about X" is then: the rules matching X whose status is `in force`, each with its record and the +changes that brought it to its present wording. + +## 5. Approval + +A decision governs how the mesh is built and run, so it needs a human approval before it holds. + +- **Proposing** reserves a number and writes the structure with status `proposed`. Any agent may propose. +- **Accepting** happens when the pull request carrying the record's reasoning merges with its human approval + and both required checks. The knowledge base sees the merge on the bus and moves the record and its rules to + `accepted` and `in force`, and every rule it changes to `changed by`. The merge is the warrant; a second + approval beside it would split the evidence. +- **A progressive insight** is a correction of fact and goes the same way: a pull request on the reasoning, + and the structure recorded on merge. +- A **structural write without a pull request** (a link added, a rule's wording fixed) is refused for an + accepted record. Changing what an accepted record says is a new decision, as it is today. + +## 6. Public and private + +This repository is public and stays so. The reasoning of a decision or research effort stays here, public. +The structure in the store is derived from public records and may be exported here as generated pages, like +the decision index of [ADR 0248](../../02-DECISIONS/0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md), +generated when read and never stored. + +What changes is the issues' narrative (section 9): it moves into the tracker, which is the mesh's own and not +public. The public repository loses those narratives. ADR 0275 kept them public on purpose, citing *the mesh +remembers*; the tracker remembers them as well, but only for the mesh. Whether a resolved issue's narrative is +ever exported publicly is a question for the decision that supersedes that part of ADR 0275. + +Knowledge about this installation (one agent's 89 private notes, issue 367) stays out of scope. Agents' lessons +and rules belong to the agent module's register (ADR 0273). + +## 7. Checks as refusals at write, and the import + +Of the 33 rules the hq checks enforce, **25 judge one record** and become refusals of the knowledge base's +write verbs: a number taken twice, a supersession not mirrored, a cited record not accepted, a record without a +topic, a research effort graduated without `became`. Two need a look beyond the record: a write that +supersedes must check who cites what it supersedes. **4 judge the whole record** and **4 judge the prose**; +those stay checks on this repository, run on every pull request. As frontmatter leaves this repository +(section 8), the 25 leave `records.py` and `cycle.py` with it, and a check is added that a record's reasoning +names a number the knowledge base holds. + +**The import** reads this repository at a commit through the records module and writes each decision and +research effort at its own number: status, topic, date, deciders, links, and its rules. Like the tracker's, it +creates what is absent, never overwrites, never advances a counter, and a second run changes nothing and says +so. Two things a reading cannot recover alone: + +- **Loose links.** The 6 supersessions said only in prose, the 6 markers no field agrees with, the 47 bare + `extends` targets and the 21 one-way partial supersessions are repaired in this repository first, by one + pull request, so the import reads clean input. +- **The rules of 226 decisions written as prose.** The import drafts them through the agent's judgement + ([ADR 0278](../../02-DECISIONS/0278-the-issue-tracker-checks-every-new-issue-for-a-duplicate-through-a-judgement-the-agent-makes-without-tools.md)) + and stores them as `drafted`, not `in force`. A topic's drafts are confirmed by the operator or a reviewer in + one pass. An answer marks a drafted rule as drafted, and never presents it as the record's word. + +## 8. What leaves this repository, and what stays + +| | Today, in this repository | After | +|---|---|---| +| a decision's number, status, topic, links | frontmatter | the knowledge base; the file keeps the number in its name and heading | +| a decision's rules | prose in the `Decision` section | the knowledge base; the section stays as prose for the reasoning | +| a decision's context, options, consequences | markdown | unchanged | +| a research effort's status and `became` | frontmatter | the knowledge base | +| a research effort's documents | markdown | unchanged | +| the decision index | generated from frontmatter when read | generated from the knowledge base when read | +| an issue's report, diagnosis, resolution | `04-ISSUES/NNN-slug/` for 361 issues | the issue tracker (section 9) | + +Frontmatter leaves a file only after the import has proven it holds the same values, one kind at a time, as +ADR 0275 did for issues' status. + +## 9. Issues live in the issue tracker, whole + +The operator decided on 2026-10-11 that an issue's content belongs in the issue tracker, not in this +repository. Measured on `main`: 361 issue folders, 1.9 MB; a folder's median is 4.4 KB and the largest +25.9 KB, well under a tracker value's 256 KiB. Every folder has `00-report.md`; 107 have a diagnosis, 9 a +resolution, 1 a progress file. Since ADR 0275, new issues live in the tracker already; the highest issue folder +is 364. + +What the tracker needs to hold it: + +- **an issue's documents**: the report it already holds, plus a diagnosis and a resolution as named documents + of the issue, each with its author and date, revised in place with history (notes stay append-only); +- **an import** of the 361 folders, repeatable, keeping each issue's number and dates; +- **links** from records: 250 files in this repository mention a path into `04-ISSUES/` 765 times, counted + as every `04-ISSUES/NNN` path in the text (markdown links alone are fewer, about 660). Those mentions become + plain "issue N" references that the knowledge base and the tracker resolve, rewritten in one pull request + once the import is proven; the link check in `records.py` then refuses a new link into `04-ISSUES/`. + +This supersedes part of ADR 0275 (hq keeps the narrative of a design or governance issue), so it needs its own +decision record. + +## 10. Slices + +1. **Clean the input** (this repository, one pull request): repair the loose links of section 7, so the import + reads clean records. In review as hq pull request 305, which declares every partial supersession in both + records and adds the `partial-supersession` check. Beside it, a shared instruction section for every + session: take a decision or research number above every one on `main` and on open branches until the + counter exists, and name a partial supersession in a field, never only in prose. +2. **The knowledge base, decisions and research**: a Go module serving one seat held once for the mesh, on + the control-node beside the issue tracker. The import, the rules and links, and the answers; numbers come from the issue tracker (section 3): + + | Verb | Answers | + |---|---| + | `ask` | the records and rules that bear on a question, ranked: number, kind, status, title, and whether each rule still holds or what changed it | + | `get` | one record: its rules, its links both ways, what changed it since; its reasoning's text only when asked | + | `holds` | for a topic or a record, the rules in force now, each with the record it comes from | + | `reserve` | the next number of a kind, from the issue tracker's counter (section 3), spent whether or not the record lands | + | `propose` | a new record's structure, status `proposed` | + | `import` | this repository's decisions and research at a commit, repeatable | + + Storage is a database in the store, provided to the module: the structure is small, but `ask` needs ranked + full-text search, which the store's server does and a bucket does not. Every answer is bounded in size. +3. **Frontmatter leaves this repository**, kind by kind, once the import is proven equal. +4. **Issues whole in the tracker** (section 9): documents, import, links rewritten, folders removed. +5. **Designs, playbooks and glossary entries** into the knowledge base the same way. +6. **Light records**: notes, posts and drawings. + +How a session reaches it: the knowledge base's tools reach every session through the mesh MCP server, and one +shared instruction section, approved by ADR 0274's route, tells every session to ask the knowledge base what +holds before reading files. A session needs this repository's folder only to write a record's reasoning. + +Proved by the acceptance sentence at the top: the merge window question, asked from a session outside this +repository, answered in one call under 4 KiB, naming ADR 0276's rules in force and the one ADR 0282 changed. + +## 11. What needs the operator + +Decided by the operator on 2026-10-11: the store is the truth for structure and this repository for reasoning +(section 1); light records come last; issues live whole in the tracker (section 9). + +Still open: + +1. **Drafted rules (section 7)**: the 226 prose decisions get their rules drafted by the agent's judgement and + confirmed a topic at a time. Who confirms: the operator, or an independent reviewing agent with the operator + seeing a summary? +2. **Issues' public narrative (section 6)**: once issues leave this repository, is a resolved design or + governance issue ever exported here publicly, or does it stay the mesh's alone? + +## Words + +The glossary has **decision record**, **design**, **playbook** and **the record** (the records module). This +effort proposes, for a decision to settle through the glossary's own rules: **knowledge base** for the module +and its seat; **rule** for one statement a decision makes, with its own status; **governing record** for a kind +reviewed before it holds; and **note**, **post** and **drawing** for the light kinds. *Policy* stays the word +for a module's upgrade policy, and *guideline* is not proposed, since section 2 finds no record of that kind. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0011-managed-files-are-generated-never-edited.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0011-managed-files-are-generated-never-edited.md new file mode 100644 index 00000000..6bcb1cd6 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0011-managed-files-are-generated-never-edited.md @@ -0,0 +1,64 @@ +--- +topic: building it +status: accepted +date: 2026-04-03 +deciders: jochen +reconstructed: true +--- + +# 11. Managed files are generated onto nodes and never edited there + +> Reconstructed after the fact from the evidence cited below. + +## Context + +[ADR 0006](0006-the-substrate-and-the-control-plane.md) put every binding in the mesh +database. But the things that consume those bindings — environment files, service +definitions, daemon configuration, firewall rules — are files on a node's disk, because that +is what the software reading them requires. + +So the same value exists twice: authoritatively in the database, and materialised in a file. +Any edit to the file is a change to a copy. Before this decision, environment values could be +pushed from a node back into the database, which made the direction ambiguous in both +directions at once. + +## Considered options + +1. **Bidirectional sync** — a node's edits flow back to the database. Rejected, and removed. + Two writers and no arbiter: whichever synced last wins, and neither is authority. +2. **Files are authoritative; the database is a cache of them.** Rejected — it inverts + ADR 0013 and returns to state that cannot be reconciled across nodes. +3. **Strictly one-directional: the database is written, files are generated.** Chosen. + +## Decision + +Every managed file is **derived**. A synchroniser regenerates it from the mesh database +whenever the underlying values change. The write path is the mesh tool that owns the value; +the file is an output. + +This applies to generated environment files, service definitions, managed configuration, and +anything else a synchroniser lists as its own. + +An edit to a managed file survives until the next synchronisation and is then overwritten, +without a warning, taking whatever it was fixing with it. + +## Consequences + +- **A file edited on a node is a bug with a delay on it.** This is now one of the mesh's core + values, and it is a consequence of this decision rather than a stance taken independently. +- To change a value you must know which tool owns it. That is a real cost, paid every time, + and the reason the mesh provides a way to ask whether a given file is managed. +- Debugging by editing a file no longer works, and fails in the most confusing way available: + it works, and then stops working later for no locally visible reason. +- Recovery is cheap. A node's entire managed surface can be regenerated from the database. +- Values resolve by precedence — database override, then existing file value, then generated, + then manifest default — which means an unset override does not clobber a generated + password. The subtlety is real and has caused its own confusion. + +## References + +- `Extract hal/env-sync module, remove syncEnvToDb`, 2026-04-03 — the commit that removed the + node-to-database direction. +- Knowledge base: `conventions/no-direct-file-mutation`, `provisioning` (resolution priority). +- Regeneration gaps: `troubleshooting/changed-manifest-default-not-rerendered`, + `troubleshooting/config-removed-from-manifest-not-pruned`. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md new file mode 100644 index 00000000..e417e6e5 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0127-amqp-is-a-provision-not-the-bus.md @@ -0,0 +1,104 @@ +--- +topic: the mesh +status: superseded +superseded-by: 0131-everything-on-the-mesh-speaks-to-the-broker-seat.md +date: 2026-09-26 +deciders: jochen +reconstructed: false +supersedes: 02-DECISIONS/0125-the-bus-is-the-only-broker.md +--- + +# 127. AMQP is a provision, not the bus + +## Context + +[ADR 0125](0125-the-bus-is-the-only-broker.md) decided that the bus is the only broker, and went +one step further than it had grounds for: it also decided that the `amqp` **interface** — a module +requiring a message broker of its own — "is not carried forward" and "retires with the +compatibility broker rather than gaining a successor", with the two modules declaring it converted +to the bus in step 4. + +The operator's correction: **AMQP is deprecated as the mesh's transport, not abolished as a +service.** The broker module keeps running and keeps answering `amqp` requirements. It is no +longer a core part of the mesh — *"it's just a module like mssql now."* + +**What 0117 conflated** is two different reasons a module might ask for a broker, which look +identical in a manifest: + +1. **To talk to other modules.** Wrong under one bus, and the thing 0117 was right to refuse: a + private broker used as inter-module transport is a second bus, with every guarantee crossing a + seam and no scoping the mesh can see. +2. **Because it genuinely needs an AMQP broker**, the way something needs a database — a queue for + its own internals, or interop with software that speaks AMQP and nothing else. That is a + backing service, and the mesh has a word for backing services already. + +0117 saw the first and legislated against both. The second is ordinary, and forbidding it would +make the mesh unable to run a large class of perfectly normal software while claiming that as +architecture. + +## Considered Options + +1. **Keep 0117 as written** — retire the interface, convert the two modules. Rejected by the + operator, and wrongly reasoned besides: it treats "needs an AMQP broker" as always a mistake. +2. **Keep the broker as the predecessor's compatibility module**, as ADR 0106 framed it, with a + retirement condition. Rejected: it is not single-purpose and its clients are not only the + predecessor's, so the retirement condition describes a day that will not come. +3. **The broker is an ordinary provider module of an ordinary provision.** Adopted. + +## Decision + +**The mesh's bus is NATS and only NATS.** Everything 0117 decided about *the bus* stands: one bus, +a module's messaging is subjects on it scoped by what it declares, no module is handed a bus of +its own, and the `mesh-broker` seat is the NATS server's. + +**`amqp` remains a provision a module may require**, answered by the broker module the way +`postgres-database` is answered by the store module or a database is answered by mssql. It is not +deprecated as an interface; the software behind it is simply no longer the mesh's nervous system. + +**The broker module stops being foundation.** It claims no seat — `mesh-broker` is the NATS +server's — it is not raised at genesis, nothing in the mesh requires it, and a mesh that never +installs it is a complete mesh. It is installed when something wants it, like any other provider. + +**The rule that survives, stated so it can be applied:** *inter-module communication goes over the +bus.* A module may hold a broker, a database or a cache as a backing service; it may not use one +as a channel to another module. The line is not which software is involved, it is whether a second +module is on the other end. + +**Neither `amqp-ping` nor `amqp-email-forwarder` needs converting.** 0117 put that work in step 4; +it is removed. They require a backing service and a provider answers. + +## Consequences + +- **The "compatibility broker" framing is wrong and goes.** There is no `lavinmq-compat`, no + single purpose and no retirement condition. Design 25 §5 is corrected. +- **[ADR 0106](0106-the-bus-is-nats.md)'s progressive insight was itself wrong** and is corrected + by a second one there. It said 0117 would make 0106's "one purpose — the predecessor's clients" + sentence true by moving the mesh's modules off. Nothing moves off; the sentence is simply not + what the broker is. +- **The seat change stands**, for a better reason than 0117 gave: not because a broker cannot be + provisioned, but because *this* broker is not the mesh's bus. The broker module drops its + `mesh-broker` claim and the `nats` module takes it. +- **Step 4 loses two conversions**; step 1 and the WBS are otherwise unaffected. +- **What got harder:** the rule is now a judgement rather than a prohibition. "Is this a backing + service or a channel to another module?" has to be asked in review, where 0117 could have + answered it with a parser. That is the honest cost of allowing the legitimate case. + +## How it is checked + +- **A module's own messaging needs no `requires`.** The check from 0117, unchanged: a module + declaring only `emits` and `consumes` reaches its subjects and is refused every other. +- **The broker holds no seat.** A manifest test: the broker module claims nothing, and a mesh + raised without it is complete — genesis names it nowhere. +- **`amqp` resolves like any provision.** A resolution test: a module requiring it is answered by + the provider, refused when none is assigned, and neither case touches the bus. +- **What cannot be checked mechanically**, and is said rather than implied: that a module holding + a broker is not using it to reach another module. Review, not a parser. + +## References + +- [ADR 0125](0125-the-bus-is-the-only-broker.md) — superseded; its ruling on the bus is kept + whole and only its ruling on the interface is reversed. +- [ADR 0106](0106-the-bus-is-nats.md) — the bus is NATS; its compatibility-broker framing is + corrected here. +- [ADR 0126](0126-a-module-declares-its-own-seats.md) — seats, including the one the NATS server + now holds alone. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md new file mode 100644 index 00000000..5b1155b7 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md @@ -0,0 +1,94 @@ +--- +topic: the mesh +status: accepted +date: 2026-09-27 +deciders: jochen +reconstructed: false +supersedes: 0127-amqp-is-a-provision-not-the-bus.md +--- + +# 131. Everything on the mesh speaks to the broker seat, and AMQP is not a provision + +## Context + +[ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) settled the old broker as an ordinary provider +of an ordinary provision, `amqp`, kept for whatever wanted a message broker of its own. The day the +bus moved was the day that framing was tested, and it failed in a way that took the control plane +down for an evening. + +Three things came out of the wreckage. **The protocol had leaked into the seat's contract**: for a +module to hold `mesh-broker`, it had to provide what the seat delivers, and what it delivered was +`amqp` — so the module that will carry the bus on NATS could not hold the seat that names the bus, +while the module the mesh was leaving could. **A consumer of `amqp` is not asking for AMQP.** The two +modules requiring it wanted the mesh's messaging — to emit an event, to hear a topic — and named the +wire protocol only because that was the word available. **And AMQP and NATS are not interchangeable +at the wire.** A provision named after a protocol can only ever be answered by that protocol, so once +the bus is NATS an `amqp` provision has one possible provider, and it is the thing being retired. + +The operator's position, stated during the outage: modules depend on the broker *seat*, not on a +protocol; AMQP is obsolete as anything the mesh's core knows about; a module that depends on `amqp` +is wrong; and everything should reach the mesh's bus and be able to emit events and consume topics +through it. + +## Decision + +**A module that needs messaging uses the mesh's bus, and the mesh's bus is whatever holds +`mesh-broker`.** Emitting an event and consuming a topic go through the sdk, which is handed the +bus by the mesh with the module's own credential. No manifest names a wire protocol to get it. + +**`amqp` is neither a provision nor a requirement.** Registration refuses a manifest that provides +it or requires it. The `mesh-broker` seat delivers `mesh-bus`, and its holder is the module that +provides `mesh-bus` — today the nats module, and only it. + +**The old broker's module and the two modules that required it leave the catalogue.** They are +removed, not converted: one was a proof that a grant worked end to end, the other forwards mail off a +queue, and both are re-done against the bus if wanted, as new modules under this record. + +**The controller's AMQP transport is deleted once every node reports on the new bus**, and the +switch that selects a transport goes with it — one bus, so nothing to select. + +The predecessor's own broker is outside the mesh and not this record's concern +([ADR 0130](0130-the-predecessor-is-ending-and-its-broker-goes-with-it.md)): what the predecessor's +tooling loses when it stops is accepted there. + +## Options considered + +1. **Keep 0127: AMQP stays an ordinary provision with the old broker as its provider.** Rejected. It + is what put the protocol into the seat's contract, it is why the seat could be left with no valid + holder mid-change, and it keeps two transports in the control plane indefinitely for the benefit of + two modules that did not want AMQP in the first place. +2. **Bridge it: the old broker's module also provides `mesh-bus`, so both can hold the seat during the + change.** Rejected. It makes the retiring broker a legitimate mesh bus for exactly as long as + nobody removes the line, which in practice is forever, and it leaves `amqp` as a thing the core + still knows the name of. +3. **The seat is the dependency; the protocol is nobody's business but the holder's.** Adopted. + +## Consequences + +- **The change of holder is a handover, and it needs a command.** Nothing today moves a seat from + one assignment to another as one act, and a seat the control plane dereferences cannot be empty + in between — that emptiness is the outage this record comes from. The command takes a seat and the + assignment taking it over. Designed and built before the cutover, under + [28 — Building the bus](../03-DESIGN/01-to-be/28-building-the-bus.md). +- **The seat's row moves to `mesh-bus` before the new holder registers, and that is safe.** The + control plane composes its own bus address through the seat *by name* + (`${seat:mesh-broker:…}`), and the overview derives holders by name; only registration and the + provision-to-seat resolution read what a seat delivers. So the row can change under the current + holder without unseating it, the new holder can then register its claim, and the handover happens + when both are running. Verified in the code during the outage, not assumed. +- **Registration gains two refusals**: a manifest providing `amqp`, and one requiring it. +- **The `rollout check` stops saying the old broker stays.** It said so under 0127; it now lists + unassigning it as the last step of the move. +- **What got harder**: a third party that genuinely wants an AMQP broker on a mesh node runs one as + any application module, with no provision and no seat, and nothing on the mesh routes to it. That + is the cost of the mesh not knowing the word. + +## How this is checked + +| Rule | Checked by | +|---|---| +| No manifest provides or requires `amqp` | a registration test refusing each, naming this record; and a whole-catalogue test asserting no registered manifest names it | +| `mesh-broker` delivers `mesh-bus`, and only a `mesh-bus` provider may hold it | the existing registration test for a delivering seat, with the row's value read from the store (mesh-controller#89) | +| The seat's row can change without unseating the holder | a test composing the control plane's own address and the overview under a row that the current holder does not satisfy | +| The rollout does not leave the old broker running | `rollout check` output, asserted in its test | +| The AMQP transport is gone | the package does not compile with it referenced; the switch variable is refused as unknown at start | diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md new file mode 100644 index 00000000..4e8f6cfc --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md @@ -0,0 +1,142 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-01 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md +--- + +# 162. A merge produces a tiered plan the mesh keeps, and a module's dependencies are one relation in the catalogue + +> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).** +> "A merge produces a tiered plan" no longer holds: merges are collected in a rolling merge window, and each +> batch is planned once, as one walk with at most one commit per repository (issue 362). The plan the mesh +> keeps, its tiers and the one dependency relation stand as decided here. + +> **Progressive insight — 2026-10-02.** The context below says a dependent is *built by whichever build machine is running — the only one there could be*. That was a fact of the day, not of the decision: since [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) a tier's asks are taken by every machine holding the build seat. The plan and its tiers are unchanged. + +> **Superseded in part — 2026-10-08, by [ADR 0267](0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md).** +> In decision 1, the `packages` kind ("the module's build reads the other's repository") and the *code* +> dependency ("B packages A's source … rebuilt whenever A changes, in the same tier") no longer hold: a file +> two modules build from moves both through their build sources, and no edge is drawn for sharing a +> repository. `stands-on` and `declared` name an artifact at each end. The build and runtime dependencies, +> `built-by`, and decisions 2 to 4 stand as decided here. + +## Context + +A merge on the forge reaches the controller as an event, and the controller asks the build +machine for what that merge changed. Until today that meant the modules whose recorded source is +that repository; since this afternoon it also means everything standing on what moved +([issue 186](../04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md)). +Both are done inside the handler that received the event: it asks one build, waits for it, asks the +next, and returns when the last is done. Three things followed from that shape on 2026-10-01: + +- The controller hears nothing else for the length of the work — twenty-five minutes for the runtime + image and its forty-three dependents ([issue 184](../04-ISSUES/184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md)). +- A controller replaced mid-merge loses the rest of the merge: the redelivered announcement reads as + history, and the dependents are asked by hand. +- Nothing is deployed between builds. A merge that changes the build machine and something the build + machine builds asks for both in order, but the second is built by whichever build machine is running + — the old one, unless somebody pushed in between. The order the dependents are sorted in exists for + the artifacts; it says nothing about what must be *running*. + +And the knowledge the order is computed from is scattered: a manifest's `build.on`, the artifacts a +build was made against, the repositories a build read, and the fact that every source-built module is +built by the build machine, each read by a different function in the merge handler. + +## Decision + +**1. A module's dependencies are one relation in the catalogue.** `depends-on` edges, each with the +kind of dependency on it: `stands-on` (the module's artifact is built on the other's), `packages` (the +module's build reads the other's repository), `built-by` (the module is built by the holder of the +build-machine seat), and `declared` (a manifest's `build.on`). The relation is answered by one query +of the catalogue — the controller's inventory today, the catalogue seat's tool when something outside +the controller needs it — and nothing else computes an edge. The edges are derived from facts +recorded at two moments and written by nobody: registration records the manifest (`declared`, and +`built-by` for anything with a source), a build's take-in records what the image was built on and +which repositories it read (`stands-on`, `packages`). A module's first build places it by its +declared edges alone; from its second it is placed by what was true. + +**The kinds are three dependencies, not one.** A *code* dependency — B packages A's source — means +B is rebuilt whenever A changes, in the same tier: B's build needs nothing of A's first. A *build* +dependency — B stands on A's artifact, or declares it — means B is rebuilt after A is *built*, the +next tier, and nothing need be deployed in between. A *runtime* dependency — B is built by A — means +B is rebuilt only after A is built *and running*, the next tier with a gate on the machines' reports. +One cycle is real and resolved by the kinds themselves: the runtime image is built by the build +machine, and the build machine stands on the runtime image; the image comes first, built by the +build machine that is running, which is the only one there could be — a `built-by` edge never orders +a module after a build machine that stands on it. A provision is not a dependency of this relation: +a consumer binds to its provider through what the push renders, and a change to the provider's image +changes nothing in the consumer's; a consumer whose build does read a provider's source declares it. +"A was deployed, so restart B" is the push's domain — B is replaced when what it reads changed — and +not the plan's. + +**2. A merge produces a plan, and the plan is a record.** The controller takes the modules the merge +changed and everything reachable from them along `depends-on` edges, and sorts that set into tiers: +tier 0 depends on nothing else in the set, tier 1 only on tier 0, and so on. The plan — the merge it +answers, the tiers, and each module's state — is written to the store before any build is asked. The +handler asks tier 0 and returns. Every build's outcome, taken in by the same handler that takes every +outcome in, advances the plan it belongs to; a controller replaced mid-plan resumes it from the store. + +**3. A tier is done when it is built, and when what the next tier needs from it is running.** A +module whose roll-out policy says *roll out* is sent to its machines when it moves, as today. The next +tier is asked only once every module in this tier is built and every rolled-out module of this tier +that a later tier is `built-by` has been applied by the machines running it — the machines' reports +say so. A module whose policy says *record* is built and not waited for. So a merge +touching the build machine and the controller builds the build machine, waits until it is the build +machine that is running, and only then asks for the controller's build. + +**4. A plan is read where the mesh is read.** `status` lists every open plan: the merge, the tier it +is at of how many, what it is waiting for and since when; `builds` lists the asked beside the built. +A plan that has waited past a bound is named red there, which is the first fact of +[issue 187](../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md)'s list. + +## Consequences + +- The merge handler returns in milliseconds; the receive loop is never held by a build again. Issue + 184's remaining cause — a handler that waits for its own work — is removed rather than worked + around; the bus's heartbeats stop being dropped under a merge. +- A controller roll in the middle of a plan costs nothing: the plan is in the store and the asks are + in the queue (mesh-controller 194). +- A release across repositories is a plan whose edges cross repositories; the order a person kept in + a work-order file is the order the tiers give. Issue 186's third fault is answered by the plan, + not by a separate release record. +- The explicit `build --on ` stays as the way to ask for the same plan by hand. +- A module's `build.on` remains the one place a manifest states a dependency the store cannot see. + +## How this is checked + +| Rule | Checked by | +|---|---| +| Dependencies are one relation, each edge with its kind | an inventory test over a fixture catalogue: a runtime image, a module on it, a module packaging the controller's source, and the build machine; the four kinds come back from one call | +| A merge's set is sorted into tiers along the three kinds: a code dependency in the same tier, a build dependency after its base is built, a runtime dependency after the build machine; the build machine's own base first | a unit test on the tiering over the mesh's real shape: the runtime image, the build machine on it, modules built by it, a plugin declared on one, the proxy packaging the controller, an unrelated module left out; a cycle is one last tier and said | +| Only a runtime dependency gates on deployment, and only for a module that rolls out | the same test's gate cases | +| The plan is written before any build is asked, and the handler returns | a controller test: a merge announcement produces a plan row with its tiers and one asked build per tier-0 module, and the handler is back before any outcome | +| An outcome advances its plan; a complete tier asks the next; a tier with a rolled-out base waits for the machines' reports | store-backed tests over a two-tier plan: the first outcome marks built; the tier's roll-out gate holds until the report; the next tier is asked after | +| A controller restarted mid-plan resumes it | a test that opens a plan, drops the handler, and advances from the store alone | +| `status` lists open plans and names one that waits past the bound | the status JSON test with a fixture plan | +| Live | a catalogue merge touching a base and a dependent: the plan's tiers in `status`, the base rolled before the dependent is asked | + +## Built and proven live, 2026-10-01 + +> **Progressive insight — 2026-10-01.** The decision stands; these are the facts of its building. + +Built in mesh-controller 197 (the relation, the plan record, the driver, `status`), 198 (`plans`), +199 (a `built-by` edge orders and gates but never widens — the first live plan had taken the whole +catalogue along for a controller change; `plans stop`), 200. The first merge handled by the finished +machinery, at 18:56Z, was a controller change and produced the plan this record describes: tier 0 +the build machine; tier 1 the controller and the proxy that packages its source. The handler +returned at once; the build machine was built, rolled, and the plan read *tier 0 built; waiting for +builder on novox to be applied* until the machine reported; then tier 1 was asked, both built, and +the plan read done — three minutes, read through the console with `plans`, the receive loop taking +reports throughout. What the day between decision and proof taught is in issues +[184](../04-ISSUES/184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md), +[186](../04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md) and +[188](../04-ISSUES/188-a-refusal-inside-on-the-network-drops-a-machine-silently/00-report.md). + +## References + +- [ADR 0157](0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md), [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) +- [Design 30 — The mesh updates itself on a push](../03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md) +- Issues [184](../04-ISSUES/184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md), [186](../04-ISSUES/186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md), [187](../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md) diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md new file mode 100644 index 00000000..1eba9012 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md @@ -0,0 +1,275 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-07 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md +--- + +# 240. A module says how it is healthy, and the node-engine judges it + +## Context + +The release gate ([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md) +§2) judges a catalogue module on its first machine by what the mesh sees from outside: the build reported +applied, no witness put it back, no condition raised since the send about the machine or the module there, +and its tools served. None of that looks at what the module *runs*. To-be 45 names the gap in its Phase 4: +*container state in the node-engine's report, without which a container that crash-loops after its compose +applied is seen only through what it breaks.* + +Measured on the live mesh for [research 032](../01-RESEARCH/032-a-module-says-how-it-is-healthy/00-overview.md) +([evidence](../01-RESEARCH/032-a-module-says-how-it-is-healthy/01-evidence.md)), 2026-10-07: + +- **125 catalogue modules; 68 run something long-lived** — 49 a container that stays up, 19 a service + unit stated `running` and no such container. 50 run only their bundle in the node tools, 7 only files, + directories and packages. +- **No manifest can declare a health check, and nothing reads one.** The node-engine's report carries no + container or unit state; no probe of the self-check reads one. +- **19 of 73 long-running catalogue containers have an image that ships a check** (7 of 45 container + modules). The mesh never reads it. **Two of those were wrong in the mesh's hands**: the studio and the + flow editor read *unhealthy* while working, because their checks ask an address the program does not + bind in the mesh's configuration. Read without proof, they would have rolled back two good builds. +- **Every restart count is 0**, because a recreate loses it; the runtime's event history on the home + server is about a minute long, pushed out by its own check executions. +- **Of seven recent incidents**, liveness alone would have caught the agent server's crash loop (about a + hundred restarts, found by a person reading its log for something else); an HTTP readiness check the + web application that accepted TCP and answered nothing for eleven hours + ([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)); + only the module's own check the identity provider's refused administrator + ([issue 179](../04-ISSUES/179-an-adopted-identity-providers-admin-never-took-the-minted-secret/00-report.md)). + A provisioner runtime that restarted until the overlay was up and then worked + ([issue 058](../04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/00-report.md)) + is the warning: a restart count without a start period reads churn that stops as a crash loop. +- **One sample is not a finding.** A single unanswered question raised an urgent alert nobody could read + ([issue 277](../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)); + the self-check now raises such a finding on the second look (to-be 45 §4). + +The mesh now rolls a module out on its own and puts the previous build back when the first machine is not +healthy. That promise is only as good as "healthy" is, and for a module it is judged today from the outside. + +**Checked against GENESIS.** *Failure must be loud* — a crash loop behind every passing check is work that +reported success and did nothing. *Anything requiring a human to notice it will be noticed late* — eleven +hours, and a hundred restarts, each found by a person. *Evidence over assertion* — "applied" is an +assertion about a declaration; "it answers" is a measurement. *The mesh notices when something is wrong +before you do* ([effect](../00-META/effect.md)). *Long-lived user services rather than an orchestrator* +([context](../00-META/context.md)) is why this record judges and reports, and restarts nothing on health: +an orchestrator's liveness restart is the part it does not take. *The mesh is a guest on a personal node* +bounds the cost: the checks' floors below. Nothing here conflicts with GENESIS. + +## Considered Options + +**Who runs the checks.** + +1. *The container runtime's own check, read by the node-engine.* Rejected as the whole answer: containers + only — nothing for the 19 service-only modules or anything only a module's tool knows; every look is an + execution inside the container, which on the home server already pushes every lifecycle event out of + the runtime's history; an image check the module never stated is a check nobody owns, and two of 19 were + wrong. Kept as one *kind*, adopted by name and proved. +2. *The module's own tool answers "healthy", and that is the judgement.* Rejected as the judge: a bundle is + hosted by the node tools, so a tool that answers proves the bundle up, not the server; and a component + that judges itself is what [ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) + rule 8 refuses for the core. Kept as one kind, for function no endpoint shows, and only beside a check + the module does not run itself. +3. *The node-engine runs everything itself, including commands, on its own schedule.* Rejected in part: a + command that only makes sense inside the container is better timed by the runtime's own retries and + start period than by an execution per look from outside. +4. **The node-engine owns every check and every verdict, and runs each kind where it is cheapest.** + Chosen: HTTP, TCP and unit checks it makes itself; a command it hands to the runtime as that + container's check and reads; a tool it asks through the node tools. One runner and one reader per + machine, for every hosting form, and it keeps what the runtime forgets. + +**Where the result goes.** + +1. *Only a field of the report.* Rejected alone: a report follows an apply, so a container that goes bad at + 03:00 waits for the next one. +2. *Only an event on each change.* Rejected alone: events are lost or replayed; an event is a sample, not + a state. +3. *A verb the controller calls per module when it judges.* Rejected: a pull per module per judging, and a + machine slow to answer reads as unhealthy. +4. **State in the report, its change on the bus, and a condition the controller raises on the second + look.** Chosen. It is how the core already says its own state; the gate, the self-check, the healers + ([ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)) and + the operator's conversation ([ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md)) all + already act on conditions. + +**What is judged.** + +1. *Liveness only.* Rejected: no declarations needed, but it misses issue 145 (the port was open and the + program running) and issue 179. +2. *Readiness only, where declared.* Rejected: nothing is judged until every module has declared, and + liveness alone would have caught the worst incident of the window. +3. **Liveness for every long-running resource at once, readiness where declared, the declaration required + over a migration.** Chosen: the gate means something from the first build. + +**What is done with an unhealthy module.** + +1. *Restart it, as an orchestrator's liveness probe does.* Rejected for this record: the evidence holds no + case where a restart would have fixed anything — the crash loop was restarting already — and a restart + hides the failure the gate is meant to see. Whether a healer restarts what stays unhealthy is left to a + record of its own under ADR 0231. +2. **Nothing restarts on health; the condition reaches a person or a healer.** Chosen. + +**A provider down.** With 12 consumers of the database provision and 36 of a route: + +1. *Ignore it.* Rejected: twelve conditions for one fault, and twelve gates failed for something none of + them did. +2. *Judge providers first, consumers only once their providers are healthy.* Rejected: a consumer broken on + its own is not said while its provider is down, which is when it is most needed. +3. **A consumer's check names the provision it exercises; while that provision's provider is unhealthy on + the record, the consumer's finding is held under the provider's condition and its gate waits.** Chosen. + It is how [issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md) + already treats a machine-level fault: what is the machine's is never pinned on a module. + +**Where the declaration sits.** + +1. *One per module.* Rejected: a module of eleven containers (the mail module) could not say which is wrong. +2. **On each long-running resource**, beside the resource's other fields. Chosen. + +**The migration.** + +1. *Required at once.* Rejected: 68 modules to change before the next merge, and nothing judged until all + are. +2. *Optional for ever.* Rejected: 38 of 45 container modules would stay at liveness, and a field that is + optional for ever is a field half the catalogue never gets. +3. **Liveness at once; the declaration required by a date, counted down.** Chosen. + +## Decision + +**1. Liveness is judged for every long-running resource, with no declaration.** A container that stays up, +a process that stays up and a service stated `running` are *alive* when running and not restarted more than +once within the settle window after their grace period. The node-engine observes this itself on each tick +and keeps the restarts it counted across recreates and across its own restarts; it never relies on the +runtime's restart count or event history. A resource held still by an open maintenance step +([ADR 0189](0189-the-store-keeps-what-the-records-name.md)) is neither alive nor dead: it is said as held, +and judged again when the step ends. + +**2. A module declares how each long-running resource is ready**, in a field named `health` on that +resource: one kind — the image's own check adopted by name, an HTTP request to a declared endpoint and the +status expected, a TCP connect to a declared endpoint, a command in the container, the unit's own +readiness, or one of the module's tools — with an interval (default 30 s, **not under 10 s**), a timeout +under the interval, a number of failing looks in a row before it is unhealthy (default 3, **not under 2**), +and a grace period after a start (default 60 s), in which failure does not count. The grace and the failing +looks together are at most five minutes, so a resource broken from its start is said within the gate's +bound. An endpoint is named by its `listens` name, never by a port or an address, so a check follows the +machine's ports as the endpoint does. A check of function that no endpoint shows is a module's own tool, and +only in addition to a check the module does not run itself. + +**3. The node-engine runs every check and owns every verdict.** HTTP, TCP and unit checks it makes itself; +a command in the container it hands to the runtime as that container's check and reads the state; an +adopted image check it reads the same way; a tool it asks through the node tools. Nothing else on the machine +judges a module, and nothing else sets a container's check. + +**4. The state goes in the report, its change on the bus, and a condition is the controller's.** Each +report carries, per module and long-running resource, a state — healthy, unhealthy, starting, held, +unknown — since when, the failing streak and the restarts counted; each change is emitted as an event, and a +state that is not healthy is said again while it lasts. The controller keeps the last state per machine, +raises `module...unhealthy` when two consecutive statements say so, and clears it on the +first that does not. **The gate needs no new rule:** ADR 0236 §2's *a module's own health holds* now reads +the module's stated health — a judging is healthy only when every long-running resource of the module on that +machine is healthy, so a resource still starting is not yet a pass — and its *no condition raised since the +send* holds the module on this condition. Every start begins in `starting`, so a condition from before the +send clears at the new build's start and anything after it is the new build's. At the gate's bound the +build is put back, as ADR 0236 §3 says. + +> **The mechanism changed — 2026-10-10, by [ADR 0283](0283-a-part-that-waits-for-the-operators-secret-or-setting-is-said-as-waiting-passes-the-first-node-gate-and-names-the-act.md).** The four states, the +> condition and the gate's reading stand. A fifth state, `waiting`, now sits beside them: a tool check +> (rule 2) may answer *waits* as well as healthy or not, naming an own secret or a setting the operator has +> yet to give, and one look that says so is enough, inside the grace too. The controller checks the wait, +> says it as `needs-operator` rather than `unhealthy`, and the gate reads it as ADR 0254's *waits for a +> person*: a pass. + +**5. A provider down is said once, at the provider.** A check names the provision it exercises. While +that provision's provider for this consumer is unhealthy on the record, the consumer's finding is held under +the provider's condition — listed there as waiting on it, raised as nothing of its own — and the consumer's +gate waits rather than fails. What a consumer finds while its provider is healthy is its own. + +> **The mechanism changed — 2026-10-11, by [ADR 0292](0292-a-consumer-of-the-part-a-provider-waits-on-is-held-under-the-wait-and-passes-its-gate-as-a-wait-for-a-person.md).** The hold under an unhealthy provider, and its consumer's gate +> waiting, stand. A provider that only waits for the operator (ADR 0283) now holds the consumers of a provision one +> of its waits covers, listed at its `needs-operator` condition, which stays a warning; their gate passes as a wait +> for a person, carrying the wait. A consumer no wait covers is said on its own. + +**6. Nothing is restarted for being unhealthy.** The runtime restarts what exits, as now. A condition +reaches a person through the operator's conversation, or a healer; a healer that restarts on health is its +own decision. + +**7. A declaration is proved before it is trusted.** `module check` refuses a `health` field that names an +endpoint the module does not declare, an interval, timeout or count outside its bounds, or a tool the module +does not serve. A bed from mesh-lab, run by the catalogue's check on the build seat, starts every resource +whose declaration or image changed and requires its check to say healthy within its grace; an image check +adopted by name is proved the same way, because two of nineteen were wrong. This proves the check's +wiring — that it can see the program working — not the change, which the live mesh and the first machine's +gate still judge ([ADR 0149](0149-the-live-mesh-is-the-test-bed.md) stands). + +**8. Every catalogue module that runs something long-lived declares one.** A catalogue-wide count of the +long-running resources without `health` may only go down. `module check` warns from this decision, and +refuses a long-running resource without `health` once the count reaches zero, or six weeks after liveness is +first judged live, whichever is first. A module running nothing long-lived — its bundle only, or files and +packages — declares none: the node tools serving its tools is its liveness, as the gate judges today. + +## Consequences + +- **Liveness alone, from the first build, would have caught the crash loop inside the gate's ten minutes.** + Readiness would have caught the silent web application in about a minute instead of eleven hours. The + identity provider's refused administrator needs the module's own tool, which it already has (ADR 0224 §5). +- **The node-engine grows a small scheduler, a field of its report and an event; the controller a condition + kind and its two-look rule; the gate a reading of state it already had a place for.** A machine of 45 + containers spends tens of milliseconds a look reading state, and about 3 s a minute on in-container + commands at the default interval. +- **A module whose first machine was unhealthy before the send no longer passes the gate for being + unchanged in its unhealthiness** — the start resets it to `starting`, and the new build is judged on its + own. +- **What a check names becomes load-bearing.** A check naming the wrong provision hides a consumer's own + fault under its provider; the bed and the first machine's gate are what catch that. +- **Harder:** a slow starter — a module that needs more than five minutes to be ready — cannot say so + within these bounds and fails its gate. That is accepted until one exists; it would be a change to the + bound, recorded. +- **A provider's health and its consumers' failures meet in two places**: this record's condition (the + provider's resource is unhealthy) and ADR 0224's standing (the provider keeps failing a consumer). They + are different facts, both kept; the hold of rule 5 reads only this record's. +- **Data is not health.** Whether a module's data is there, measured and backed up stays + [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)'s, + watched by the self-check's D13. + +## What this does not decide + +- Whether a healer restarts what stays unhealthy (rule 6 leaves it to its own record, under ADR 0231). +- Health for scheduled work — whether the last scheduled run succeeded belongs to the record of the + scheduled step. +- Health of what a module's events do ([issue 276](../04-ISSUES/276-a-handler-that-did-its-work-was-offered-it-five-times/00-report.md)) + — the event contract's, and the bus watchdog's. +- The core's own definitions (ADR 0236 §1, to-be 45 §8), which stand; a core component may later declare its + own through the same field. + +## How it is checked + +| Rule | Checked by | +|---|---| +| 1 Liveness without declaration | node-engine tests over a fake runtime and service manager: a container recreated keeps its counted restarts, and so does the engine restarted; a restart inside grace is not counted; two restarts within the settle window after grace make it unhealthy; a resource under a maintenance step is held. A mesh-lab replay of the crash loop (a container whose program exits at start) fails its gate within the bound | +| 2 The field and its bounds | `module check` refuses each out-of-range part and each endpoint named by port or address, a test per refusal; a catalogue-wide test parses every `health` field | +| 3 The engine owns the verdict | a node-engine test that a declared command becomes the container's check and nothing else sets one; a test that an HTTP check dials the endpoint's current port after a port change | +| 4 Report, event, condition, gate | controller tests: one unhealthy statement raises nothing and is listed unconfirmed; two raise; a healthy one clears; the gate fails a judging while a resource is starting or unhealthy and holds a module on this condition (the existing gate test, extended); a condition from before the send clears at the new build's start. A replay of issue 145 — a database made unreachable — raises the consumer's condition within two looks | +| 5 Said once at the provider | a controller test with one unhealthy database provider and three consumers failing: one condition, at the provider, the consumers listed as waiting; their gates wait, not fail; a consumer failing while its provider is healthy is raised on its own | +| 6 No restart on health | a node-engine test that an unhealthy container is not restarted, recreated or stopped | +| 7 Proved before trusted | the bed's run of every changed declaration in the catalogue's check; the replay of the studio's false *unhealthy* (a check that asks an address the program does not bind) fails the bed, not a machine | +| 8 Every long-running module declares | the catalogue-wide count of long-running resources without `health`, compared with the number kept in the catalogue: a merge may lower it and never raise it; after the date, `module check` refuses | +| live | every machine's report carries a state for every long-running resource; `conditions` raises `module...unhealthy` for a module stopped on purpose and clears it when it runs | + +## References + +- [Research 032](../01-RESEARCH/032-a-module-says-how-it-is-healthy/00-overview.md) — the evidence, the + options and the recommendation this record takes. +- [To-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md), the design. +- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md) + §2, whose module health this gives a content; + [ADR 0227](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) rules 5, 6 + and 8 and [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §2, §4 and §8 — the + condition, the second look, the witness that is never the component; + [ADR 0224](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md), + [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md), + [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md), + [ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md), + [ADR 0189](0189-the-store-keeps-what-the-records-name.md), + [ADR 0149](0149-the-live-mesh-is-the-test-bed.md), + [ADR 0237](0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md). +- Issues 058, 145, 179, 181, 276, 277, 281. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md new file mode 100644 index 00000000..acd7c2b9 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md @@ -0,0 +1,99 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +supersedes-in-part: + - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +extends: 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +--- + +# 249. A delivery group's order rules have a precedence, and a declared order outranks them all + +## Context + +[ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) +decision 4 orders a delivery group's members by a line `after: ` and by three inferred rules: +*built by* (a member that moves a module goes before a member whose modules are built by it or stand on +it), *version skew* (the controller before a manifest change) and *engine before controller* (the +node-engine before the controller, from ADR 0236's order for its walks). It says nothing about two rules +ordering one pair in opposite directions. The planner treated every such pair as a cycle, and a declared +order against an inferred one as a cycle too. + +On 2026-10-08 the group `feat/a-machine-joins-through-the-tunnel` held three pull requests, each `ready` +on its own: the controller's, the node-engine's and the lab's +([issue 309](../04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/00-report.md)). The controller's +repository also holds the build agent, and its pull request moved it. Every source-built module is built +by the build agent, so *built by* put the controller's member first. *Engine before controller* put the +node-engine's member first. The group was rejected: "its order contradicts itself", naming the two +members and neither rule. The person merged it by hand, the controller first, since that member only added +a verb the new node-engine calls. That is the order *built by* gave, and the order +[ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) asks for: a verb is promised before +it is required. + +The rules are not of one kind. *Built by* and *version skew* are read from the graph and from what the +change does: which module builds which, and whether a manifest changes. *Engine before controller* is +read from two module names alone. It holds when the controller starts sending something the node-engine +must first accept. It is wrong when the node-engine starts asking for something the controller must first +serve, as here. The rule cannot tell the two apart. + +## Considered Options + +1. **A precedence among the rules, and a declared order above them all.** Chosen. +2. **Narrow *built by* so it never applies through the build agent to the node-engine.** It would end + this case. But a builder that changes how the node-engine is built does have to go first, and the next + pair of rules that disagree would be refused the same way. Rejected. +3. **Drop *engine before controller*.** ADR 0236's reason still holds when the node-engine is what must + accept something new. Rejected. +4. **Keep refusing, and name the two rules.** Then every group that moves the controller's repository and + the node-engine together is refused, and the person must still merge by hand. A declared order could not + help, because a declared order against an inferred one was also a cycle. Rejected. + +## Decision + +1. **When rules order one pair of members in opposite directions, the rule of higher precedence decides.** + From highest to lowest: + - *declared*: an `after:` line a person wrote; + - *built by* and *version skew*: what the graph and the change say; + - *engine before controller*: the default for walks, read from module names alone. +2. **A declared order outranks every inferred rule.** This replaces ADR 0239 decision 4's sentence "a + declared order that contradicts an inferred one is a cycle". A person who knows the order the rules + cannot see writes one line, and the group follows it. +3. **The order says which rule won.** Each pair carries its rule and the rules it won over. The delivery's + `show`, `groups` and plan note print them, for example "the controller's member before the node-engine's: + built by, over engine before controller". +4. **Rules of one rank in opposite directions are still refused.** Two `after:` lines naming each other, + or *built by* both ways, have no precedence between them. The group is rejected. The reason names both + members, both rules and the direction each gives, and says how to declare the order. A cycle through + three members or more is refused naming the pairs in it. + +## Consequences + +- The group of issue 309 is ordered without a person: the controller's member first. +- A group can no longer be refused because a person's `after:` disagrees with an inferred rule. The person + can now overrule *version skew*, and a manifest change merged before the controller that parses it fails + at the gate, not silently. The rule the person overruled is printed on the pair. +- *Engine before controller* now decides only where nothing in the graph or the pull requests does. +- The ranks are a list in the controller's planner. A new inferred rule must be given a rank when it is + added. + +**How it is checked.** The controller's tests `TestAnOrderRuleYieldsToAStrongerOne` and +`TestAGroupIsOrderedByTheGraphAndWhatItsPullRequestsSay` check that *built by* wins over *engine before +controller*, that a declared line wins over both and over *version skew*, that each pair names what it won +over, and that equal-rank rules both ways are refused with both rules named and the `after:` lines to +write. mesh-delivery's `TestAGroupSaysWhichOrderRuleWonOrWhyNoneDid` and +`TestACycleWithoutAContradictionNamesItsPairs` check that the refusal and the won-over rules reach +`groups` and the group's state. The replay R309 in mesh-lab's register fails before the fix and passes on it. + +## References + +- [Issue 309](../04-ISSUES/309-two-order-rules-ordered-one-pair-both-ways/00-report.md), its report and + diagnosis. +- [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) + decision 4, the sentence this replaces. +- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md), + *Rollout order*, where *engine before controller* comes from. +- [ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md), a verb promised before it is + required. +- [Design 47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md), *A delivery group*. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md new file mode 100644 index 00000000..38248b47 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md @@ -0,0 +1,249 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-10 +deciders: jochen +reconstructed: false +supersedes-in-part: + - 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md + - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md + - 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md + - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +extends: 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md +--- + +# 276. Merges are assembled in a rolling window, and each batch is delivered by one walk with one commit per repository + +> **Superseded in part — 2026-10-10, by [ADR 0282](0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md).** +> Decision 4's "one walk at a time", and decision 1's window for every merge, no longer hold: a lone merge is +> cut at once, and walks that share no repository, module or edge of the module graph run at once, one +> first-node judgement per node at a time. Two walks of one repository never at once, no walk superseded once +> it started, and the window for a merge that waits for its delivery group or arrives while a batch assembles +> stand as decided here. + +## Context + +Shortly after midnight on 2026-10-10 two pull requests of the catalogue were merged 18 seconds apart, the +claude-code change at 00:20:52 and the dunst change at 00:21:09 +([issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md)). +The controller opened one walk for each merge, as [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) +decided ("a merge produces a tiered plan"). The second walk superseded the first, as +[ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) +decision 3 decided, and both were planned at the dunst commit, as the fix for +[issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md) +decided. mesh-delivery then tied the surviving walk to the dunst delivery, which the supersession had already +ended, and tied no walk to the claude-code delivery. Nothing gave the walk its word. The change waited 58 +minutes, until the operator started the walk by hand with `plans go`. + +The diagnosis found two parts that each behave as written and disagree on what a walk's commit means. That +is the narrow cause. The wide one is that the mesh plans one walk per merge, while merges arrive in bursts: + +- **A burst is the normal case.** The operator merges a batch of reviewed pull requests at once, often + across repositories (a catalogue change with the controller change it needs). Every merge opened a walk, + and every walk after the first superseded the one before. Three supersession paths now exist (ADR 0218, + issue 254, issue 349), each with its own bookkeeping, and issue 362 is a fourth way for them to disagree. +- **A walk superseded mid-gate wastes the gate and confuses its record.** A first machine judged healthy + for a build that the next walk replaces a minute later proves nothing about the build that stays. On + 2026-10-09 a controller merge landing while another walk was at its gate rolled everything back; the rule + "one controller rollout at a time" was kept since then by the person merging, not by the mesh. +- **ADR 0239 made a delivery "one commit … from commit to delivered" and walked it alone**, while its own + decision 4 delivers a delivery group across repositories as one unit. The walk the commit gets is + decided by when the controller heard its merge, which is not something the person merging controls. + +On 2026-10-10 the operator decided the model, in their words: + +> "a delivery doesn't really 'run from commit to delivered', we've already discussed that a delivery plan +> might start with a set of commits spanning multiple repo's, however, only 1 commit per repo is allowed +> … we assemble all commits that are coming in with a 1 minute rolling window, so if a new commit comes +> in, less then 1 or 2 minutes after the last one, the builder again waits 1 or 2 minutes, until no new +> commit came in, then it looks at the 'group of commits' and decides if they all fit into 1 delivery +> plan … This is a hard problem to tackle, but it will improve our build setup I believe." + +> "the 'actively assembling plan' is very interesting, the controller can say 'Still waiting for 13 +> seconds for new commits, these are the currently grouped commits, plan is yet to be calculated'." + +On two commits of one repository in one window, the operator chose the later commit over splitting: +"i like your idea better, I just thought it would be more complex, but if it's reasonably doable, go for +it". + +**Checked against GENESIS.** *Failure must be loud*: an assembling batch and a waiting walk each have a +bound and a condition. *The mesh owns the truth*: the window, the batch and the walk are kept by the +controller and shown by its `plans`, not inferred from timing by a person. Nothing here conflicts with +GENESIS. + +**Words.** The glossary had no word for the merges one window collects, nor for the window. This record +adds **merge window** and **batch** to it (Change and delivery), with this record. *Group* is not used for +either, because **delivery group** already names deliveries that share a head branch name. + +## Considered Options + +**What a walk answers.** + +1. **One walk per merge, as today, with the bookkeeping repaired.** The walk names the merges it answers, + and mesh-delivery ties a superseding walk to every live delivery it contains. Rejected as the model, + kept as a repair (decision 8): it makes the 362 shape deliver, but every burst still opens and + supersedes walks, a walk can still be superseded at its gate, and a cross-repository change is still + walked as separate walks in whatever order the merges were heard. +2. **A rolling merge window, and one walk per batch** (chosen). Merges are collected while they keep + arriving; the batch they form is planned once, as one walk with at most one commit per repository. +3. **Every walk started by hand.** The operator named it as the alternative ("this does mean we need to + manually start all delivery plans, or, even better, …"). Rejected: a person in every delivery is the + wait ADR 0239 exists to remove, and a merge that passed its checks must reach its machines without a + person. + +**Two commits of one repository in one batch.** + +1. **Split into two walks in merge order** (the operator's first proposal). The second walk waits for the + first. Rejected: every machine running the module builds, is sent and is judged on the earlier commit, + and then again on the later one minutes after; the gate judges a build that is about to be replaced; + and the batch's walk stops being one unit. +2. **One walk with the later commit** (chosen). On a linear trunk the later merge commit contains the + earlier one, so building it delivers both. One build, one gate, one rollout. The earlier merge is + answered by its successor and ends `covered` when the walk is delivered. Only when that walk fails are + the earlier merges walked on their own, to find which one brought the failure. + +**Who keeps the window.** + +1. **mesh-delivery**, which owns deliveries (ADR 0239). Rejected: a walk on the controller's own path (the + controller, the node-engine, the node tools, the bus, mesh-delivery itself) must not wait for + mesh-delivery, and with no holder of the seat on record the controller plans alone. The window would + then exist twice, once in each. +2. **The controller** (chosen). It hears every merge (the bus, and the catch-up that reads back what the + bus missed), it owns the planner and the walk, and it already plans alone for its own path. mesh-delivery + keeps what it owns: each commit's delivery record, the order of a delivery group, and the word that lets + a waiting walk start. + +## Decision + +1. **A rolling merge window.** The controller collects every merge into a branch a module follows into the + open **batch**. The window is rolling: each merge restarts it. It closes when no merge arrived for the + window's length, 90 seconds by default (the controller's setting `merge-window`), and at the latest 10 + minutes after the batch's first merge (`merge-window-at-most`), so a busy period never starves delivery. + The batch, its merges and its times are kept in the controller's store, so a restarted controller + resumes the window where it stood rather than losing or doubling a merge. +2. **A batch is planned once, as one walk with at most one commit per repository.** When the window closes, + the planner computes one delivery plan for the batch: every module its commits move, their dependents, + in tiers along the module graph, across repositories. For a repository with two or more merges in the + batch the walk takes the latest commit of its branch, which contains the earlier ones. **Every merge + belongs to exactly one walk**: the walk names every merge it answers, its own commits and the earlier + ones they contain. A merge is never dropped and never walked twice. +3. **The earlier merge of a repository is answered by its successor.** Its delivery follows the walk and + ends `covered` when the walk is delivered, naming the merge that carried it (the glossary's word, + [ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md); + here the containment is the trunk's order inside one batch, so the forge is not asked). **When that walk + fails**, the walk's own commits fail, and each earlier merge it answered is walked on its own commit, the + newest first, one walk each, in the next batches: the first that is delivered ends `delivered`, and the + search stops there. A walk that failed never marks a merge delivered. +4. **One walk at a time, and nothing supersedes a walk that started.** A batch becomes a walk only when no + walk is open. Merges that arrive while a walk runs join the next batch, whatever its window says. A walk + that has not started, because it waits for mesh-delivery's word, is folded into the next batch when that + batch is planned, and the new walk answers every merge the folded one answered. This is the only + supersession left; ADR 0218 decision 3 and the issue 349 rule of planning a late merge at a later commit + are replaced by it. A merge heard after a later merge of its repository and branch was put in a walk is + answered by that walk: by the batch still assembling, by the open walk, or, when that walk was delivered, + it ends `covered` at once. + + > **The mechanism changed — 2026-10-10, by [ADR 0282](0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md).** + > What stands: the window for a merge whose delivery group has a member not yet merged or that arrives while a + > batch assembles (decision 1), two walks of one repository never at once, and no walk superseded once it + > started. What moved: a lone merge is cut at once without the window, and walks that share no repository, + > module or module-graph edge run at once, one first-node judgement per machine at a time. Until ADR 0282's + > second and fourth slices are built, the window for every merge and one walk at a time hold. +5. **The order of a delivery group holds inside a walk** (ADR 0249 unchanged). The pairs its order rules + give (declared, built by, version skew, engine before controller, each with the rule that won) become + edges of the walk's tiers beside the module graph: the modules of a member ordered after another are in + a later tier than that member's. A contradiction still rejects the group before anything merges. + mesh-delivery gives a walk its word only when every delivery it carries is at its turn. +6. **A delivery is one commit's record, and its walk is the batch's.** A delivery keeps its pull request, + its status, its note and its check, as ADR 0239 decided. From its merge on it is delivered by the walk + that answers it, and every delivery a walk carries ends with that walk's result: `delivered` (its own + commit) or `covered` (an earlier commit of the same repository) when the walk is done, `failed` when it + fails, `stopped` when it is stopped. The walk's machine steps are recorded on each delivery it carries. +7. **The assembling state is shown.** The controller's `plans` lists the open batch first, as the operator + asked: "assembling: 13 s left (at the latest 00:31:09); grouped: novox/mesh-catalog@48bda475 (answers + 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated". It says the batch on the bus as + `plan-moved` with the state `assembling`, so mesh-delivery's `deliveries` and `show` say of a merged + delivery that its batch is assembling and how long is left, and of one whose batch waits that it waits + behind which walk. +8. **The bookkeeping is repaired as well** (the narrow cause of issue 362), because decision 4 keeps one + supersession path. A walk names every merge it answers, and a superseded walk names the walk that took it + over. mesh-delivery ties a walk to every live delivery whose merge it answers, follows a walk to the one + that took it over, and **never ties a walk to a delivery that has ended**. A walk waiting for the word + that no live delivery holds is either adopted as a delivery of its own or listed at once by `stalled`. +9. **Every wait has a bound and a condition.** A batch still assembling a minute past + `merge-window-at-most` while no walk is open raises a condition: the controller failed to cut it. A + batch waiting behind an open walk longer than that walk's bound raises one naming the walk. A walk + waiting for its word past 30 minutes is S16's, as before, and its words name the merges it answers. A + merged delivery no walk answers within ten minutes is held with the true reason (which open walks of its + repository exist and that none names its merge, or that the controller holds none) and listed by + `stalled` at once, so probe D14 raises it for the operator. + +> **Decided during the build, 2026-10-10** (on the operator's delegation). Decision 2 batches every merge of the +> window together, and decision 4 starts a walk on the controller's own path without mesh-delivery's word. A +> batch holding both a merge on that path and a catalogue merge would therefore start the catalogue's deliveries +> without their word: their turn in a delivery group, their holds, and the order of decision 5 would be skipped. +> So **a merge on the controller's own path never shares a batch with a merge that waits for the delivery's +> word.** Such a merge goes into a batch of its own, with its own window; the open catalogue batch closes and is +> cut by the normal rules, and queues behind the own-path walk like any other. Catalogue merges heard during the +> own-path batch's window join the next catalogue batch. An own-path walk being cut folds no catalogue walk that +> waits for its word: that walk's merges are batched again behind it, and the walk names that batch as the one +> that took it over, so no catalogue delivery ever skips its turn. Checked by the controller's test +> `TestAMergeOnTheControllersPathNeverSharesABatch`. + +## Consequences + +- **A merge waits up to 90 seconds before it is planned**, and up to 10 minutes in a busy period. A walk + takes minutes to build and longer to judge, so the wait is small beside it, and it removes the builds, + sends and gates that a superseded walk wasted. +- **A cross-repository change merged within a window is walked as one unit**, tiered by the graph and its + group's order. A person no longer has to merge in order and wait in between. +- **One walk at a time is mechanical.** A controller merge can no longer land in the middle of another + walk's gate; it waits for the next batch. +- **A failing walk fails every delivery it carried**, including one whose own modules passed before the + failure. Each delivery's machine steps say how far it got, and decision 3's search finds the culprit only + among the merges of one repository. A failure in a walk that carried several repositories is the + operator's to read. This is the price of one unit, and the operator chose it. +- **What got harder**: the controller's merge handling holds state across merges (the open batch) and is + restart-safe; the walk's record names several repositories and commits instead of one, and every reader of + it (`plans`, S3, S16, `plan-moved`, mesh-delivery) changes with it. The store gains the batch and the + merges a walk answers, as a numbered migration and in the consolidated schema. + +> **Progressive insight — 2026-10-10.** The controller has no consolidated schema file: a fresh install runs its +> numbered migrations from the first, embedded in the build, so the numbered migration (0089) is the whole of +> the schema change. The bullet above said "as a numbered migration and in the consolidated schema"; that second +> half describes mesh-delivery's kind of store, not the controller's. The decision stands. +- **The replay of issue 362** in the lab raises a mesh, merges two commits of one repository 18 seconds + apart, and passes when one walk delivers both without a person; it fails on the commit before the change. + +## How it is checked + +- The controller's tests: two merges of one repository 18 seconds apart make one batch and one walk at the + later commit, answering both; merges in three repositories within the window make one walk; a merge at + second 80 restarts the window; the window closes at `merge-window-at-most` however busy; a merge arriving + while a walk runs joins the next batch; a walk that has not started is folded into the next one; `plans` + shows the assembling state; the batch survives a restart; the two conditions of decision 9 are raised past + their bounds and not before. +- mesh-delivery's tests: both deliveries of one walk end `delivered` and `covered` with its done record; a + failed walk marks none delivered and walks the earlier merge alone; a walk is never tied to an ended + delivery; a walk is given its word only when every delivery it carries is at its turn; a group's order is + unchanged; a held delivery with no walk names the true reason and is listed by `stalled`. +- Each of those tests fails with its part of the change removed. +- mesh-lab's replay R362 (decision's consequence above), proved to fail before the change and pass after it. + +## References + +- [Issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md), + its diagnosis, and the replay R362 in mesh-lab. +- [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) (a merge produces a plan), + [ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) + decision 3 (a newer merge takes over an older plan), [ADR 0238](0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md) + decision 2 (one commit, one change plan), [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) + (a delivery walked by its own commit's walk), each superseded in the part this record names. +- [ADR 0249](0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md), + whose order rules this record keeps and uses as tier edges. +- [ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md), + whose `covered` ends an earlier merge of a repository. +- [Issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md), + whose rule decision 4 replaces. +- To-be [47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md) and + [10](../03-DESIGN/01-to-be/10-delivery.md), amended with this record. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md new file mode 100644 index 00000000..29ad016f --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0282-a-fix-to-a-leaf-module-runs-healthy-on-every-node-within-five-minutes-of-its-merge-and-a-core-module-within-ten.md @@ -0,0 +1,236 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-10 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md +supersedes-in-part: + - 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md +--- + +# 282. A fix to a leaf module runs healthy on every node within five minutes of its merge, and a core module within ten + +## Context + +On 2026-10-10 the operator asked, in their words: "we definitely need to speed up our bug fixing process so a +bugfix can land deployed and rolled out in less than 5 minutes". Issue 382 in the issue tracker (a small fix took +45 minutes to reach a node) names the causes it saw: a 90-second merge window, one walk at a time, about four to +five minutes per tier, and three judgement passes for every module. Issue 338 (a module built from a shared +repository moves on every merge to it) names a fourth: every merge of the controller walks two catalogue modules +in two more tiers. + +### What was measured + +Every walk the controller keeps (`delivery-walks`) records, per module, when its build was asked and finished, +when its first node was sent it, when the first-node gate judged it and when the rest were sent. Read on +2026-10-10 for the 51 walks that finished between 2026-10-09 13:55 and 2026-10-10 17:32 (UTC), each walk's time +from its merge to its builds being sent to every node it targets splits into these phases. A walk is a **core** +walk when it moves a module of the controller's own path (the controller, the node-engine, the tool runner, the +bus, mesh-delivery) or the mesh's resolver, and a **leaf** walk otherwise. Nine walks that failed or were folded +into another are not counted. + +| Phase, in minutes | Leaf: median | Leaf: worst | Leaf: share | Core: median | Core: worst | Core: share | +|---|---|---|---|---|---|---| +| the merge window: merge to walk opened (a wait over 2.2 counted as 1.8 of window, the rest as waiting) | 0.5 | 2.1 | 9 % | 0.4 | 2.1 | 7 % | +| waiting behind another walk | 0.3 | 59.1 | 47 % | 0.0 | 1.8 | 1 % | +| build | 0.4 | 1.6 | 5 % | 1.2 | 4.1 | 14 % | +| built to sent to the first node | 0.3 | 0.6 | 3 % | 0.5 | 3.0 | 5 % | +| first-node judgement, the longest of each tier | 2.6 | 6.3 | 27 % | 7.2 | 14.9 | 63 % | +| first nodes of one tier judged one after another | 0.0 | 4.2 | 8 % | 0.0 | 0.0 | 0 % | +| judged to sent to the rest | 0.0 | 0.9 | 2 % | 0.5 | 5.1 | 6 % | +| between tiers | 0.0 | 0.6 | 0 % | 0.0 | 1.0 | 3 % | +| **merge to sent to every node** | **6.0** | **66.0** | | **9.7** | **20.5** | | + +- **Leaf walks** (20): median 6.0 minutes, 90th percentile 14.8, worst 66.0; **7 of 20 within five minutes**. + The worst is the incident of issue 362 (58 minutes waiting for a person's word), which ADR 0276 already + repaired; without it the worst wait behind another walk is 13.3 minutes. Since the merge window went live on + 2026-10-10 at about 12:40, a lone merge waits 1.6 to 2.1 minutes for its window (median 1.8), and **4 of the 9 + leaf walks since then waited behind another walk**, for 1.8, 4.9, 9.5 and 13.3 minutes, each behind a walk of + the controller's own path. +- **Core walks** (31): median 9.7, 90th percentile 17.0, worst 20.5; 4 of 31 within five minutes. They are two + populations. **13 core walks had one tier**: median 5.6, 90th percentile 7.2, worst 8.8. **18 walks + were controller merges that also walked the build agent and the route proxy** in tiers of their own (issue + 338): median 14.2, worst 20.5, none within ten minutes. +- **The first-node judgement** is three healthy readings, at least forty seconds apart and two minutes after + the send (ADR 0236 decision 2). It took a median of 2.4 minutes for a core module other than the controller (10 + judgements), 3.6 minutes for the controller (19 judgements, worst 4.3, its readiness includes its + self-check), and 2.6 minutes per tier of a leaf walk. **Five of the 20 leaf walks judged two first nodes one + after the other** within one tier (a module first on one node, another first on a second), adding 3.0 to 4.2 + minutes each. +- **The build is small**: a median of 0.4 minutes for a leaf walk and 0.6 minutes for a core walk of one tier. +- **The rest are sent at once already**: a leaf walk sends the rest a median of 0.0 minutes after its judgement. + The walk ends when the rest are *sent*, not when they run the build. Applying a send takes a median of 10 to + 17 seconds per node (the controller's `durations`, seven days, every node), so the measured times are short of + "healthy on every node" by about a quarter of a minute. +- **The merge gate does not build the module.** Its record for a controller pull request on 2026-10-10 shows it + clones the repositories, composes every node's declaration with the change, compiles the controller only to + judge the change with itself (3.2 seconds), and runs the repository's `merge-check.sh` (6 minutes 13 seconds, + before the merge). No artifact is built that the merge could reuse; the merge's build is the first. + +Three limits of the measurement: a walk opens only when the walk before it ends, so time beyond the window's +own 1.6 to 2.1 minutes is counted as waiting; the walk keeps only the last judgement pass, so the time of a +first pass is estimated below, not measured; and the check before the merge is not counted, because the +operator's target starts at the merge. + +**Checked against GENESIS.** *Anything requiring a human to notice it will be noticed late*: a fix that takes +fifteen minutes to land is noticed by the operator waiting for it. *Failure must be loud*: a delivery over its +budget raises a condition. *Evidence over assertion*: every lever below is weighed by what it would have saved +in the walks above. Nothing here conflicts with GENESIS. + +**Words.** The glossary has no word for the time from a merge to its modules running healthy everywhere, nor for +the time it is held to, nor for the two kinds of module. This record proposes **delivery time**, **delivery +budget**, **core module** and **leaf module** for the Change and delivery domain, with this record. + +## Considered Options + +Each lever is weighed by what it saves in the 20 leaf walks measured, against what it costs. + +1. **No merge window for a lone merge** (chosen). Saves the window's 1.8 minutes on every lone merge. The window + exists to put the parts of one change, merged together across repositories, into one walk (ADR 0276). A + merge whose delivery group has no member still unmerged has no part to wait for. +2. **Independent walks in parallel** (chosen). Saves the wait behind another walk: 1.8 to 13.3 minutes in 4 of + the 9 leaf walks since the window went live, 34.7 minutes over the 19 leaf walks outside the issue 362 + incident. Its cost is two builds judged on one node at once, which the rules in decision 3 keep apart. +3. **Judgement passes by the module's class** (chosen). One healthy reading takes an estimated minute after the + send (a send applies in 10 to 17 seconds, and the controller acted on a build in a median of 0.3 minutes), + against 2.6 minutes for three. Saves about 1.5 minutes per tier of a leaf walk. The cost is a crash that shows + only after the first reading, which a second reading forty seconds later catches for a module with a + long-running resource. A module whose every resource is a tools bundle has no process of its own to crash: the + tool runner serves its tools or does not. +4. **The first nodes of one tier judged at once** (chosen; not in issue 382, found by the measurement). Saves + 3.0 to 4.2 minutes in 5 of the 20 leaf walks. +5. **Reuse the merge gate's build of the pull request's head** when the merged tree equals the checked one. + Rejected: the merge gate builds no artifact today (see Context), so there is nothing to reuse. Building every + head's artifacts at its check, to save the merge's build, would save at most the build's median 0.4 minutes + for a leaf walk, cost a build on every push to a pull request, and need ADR 0238 decision 1 ("anything built to + check it is never registrable") reopened. +6. **A build cache.** Rejected as a lever of this record: the build is 5 % of a leaf walk's time, and the + artifact that did not change is already not rebuilt (ADR 0267's build source). +7. **Sending the rest at once after the first passes.** Already so: the rest are sent a median of 0.0 minutes + after the judgement. Nothing to gain. +8. **The plan says when it is expected to land** (chosen). Saves no time. It tells the operator, while waiting, + when to look, from the same measurements. +9. **One budget for every module, five minutes.** Rejected: a core module's judgement is what protects the + mesh's ability to reach a node at all. A broken node-engine, tool runner, bus or resolver cannot be put back + from outside the node, and the controller judges every other build. The controller's judgement alone takes 3.6 + minutes; its three readings are kept. + +With options 1 to 4 applied to the 20 measured leaf walks (no window, no wait, one judgement of about a minute +per tier with its first nodes at once, plus a quarter of a minute for the rest to apply), the median would be 2.1 +minutes and the worst 5.3; **19 of 20 within five minutes**, against 7 of 20 measured. Keeping three readings +and applying only options 1, 2 and 4 gives a median of 3.9 and a worst of 9.6 (18 of 20). + +## Decision + +1. **The target, and how it is measured.** A delivery's **delivery time** runs from its merge to the moment + every node of its deploy plan that is heard from runs its modules' builds and reports them healthy. A node not + heard from (a laptop asleep) is listed and not waited for, as ADR 0236 leaves it. **A leaf module's delivery + budget is five minutes. A core module's is ten.** A delivery held for a person (a module that records, a hold, + a bus step) is shown and not held to a budget: its wait is the person's, and the controller's + condition for a walk left waiting (S16) and its probe of stalled deliveries (D14) already speak for it. + - A **core module** is one on the controller's own path (the controller, the node-engine, the tool runner, the + bus, mesh-delivery) or the mesh's resolver. A **leaf module** is every other module. A delivery's class is + that of the slowest-budgeted module it moves. + - The ten minutes for a core module are the measurement's: the 13 core walks of one tier took a median of + 5.6 minutes and at worst 8.8. A controller merge that also walks the build agent and the route proxy (median + 14.2) meets it once issue 338's path precision is built: that work is tracked by issue 338 and is the + planner's, not this record's. +2. **No merge window for a lone merge.** When a merge arrives, no batch is assembling, and its delivery belongs to + no delivery group with a member not yet merged, the controller cuts its batch at once. A merge of a group still + waiting for members, or a merge arriving while a batch assembles, waits for the window as ADR 0276 decided. +3. **Independent walks run at once.** A batch becomes a walk without waiting for an open walk when it shares with + every open walk no repository, no module, and no edge of the module graph between their modules. Where they + meet, they wait for each other: + - **one first-node judgement per node at a time**: a walk's first node is chosen among its module's nodes + where no other walk is judging; a send to a node where another walk's build is being judged waits for that + verdict, so ADR 0236 decision 4a (no build reaches a node without a judgement) holds as written; + - **a core walk keeps its ground**: no leaf build is sent to a node where a core build is being judged, and + while a new controller is judged, no leaf judgement counts a reading; those resume when it passes or is put + back. A core send never waits for a leaf judgement: a leaf judgement open on that node is set aside without + a verdict and starts again, from its first reading, after the core build's verdict, so a core delivery is + not held past its budget by a leaf one; + - two walks of one repository never run at once (a repository's later merge contains its earlier ones, ADR 0276 decision 2), and a merge into a repository with + an open walk joins that repository's next batch. + + This replaces ADR 0276 decision 4's "one walk at a time"; its rule that a walk which started is never + superseded stands. +4. **Judgement by class.** The first-node gate's readings, each at least forty seconds after the one before: + - a **core module**: three, the first at least two minutes after the send, as ADR 0236 decision 2 decided; + - a **leaf module with a long-running resource** (a service, a container, a process the node-engine runs): + two, the first after the node-engine states every such resource healthy (ADR 0240); + - a **leaf module whose every resource is a tools bundle**: one, once the tool runner serves its tools. + + Everything else ADR 0236 decided of the first-node gate stands: no witness put the build back, no condition + raised since the send, ten minutes at the most, and a rollback when it fails. +5. **One tier's first nodes are judged at once**, each by its own readings, and the tier passes when every one + has passed. +6. **Every walk keeps its phases, and every delivery its delivery time.** + - The controller records in the walk, and says in `plan-moved`, each phase of the table above with its start + and end, each judgement reading with its time, and the first healthy report of each node of the rest. Its + `durations` gains the kind `walk-phase`, per class. + - mesh-delivery records each delivery's delivery time, its class and its phases from the walk. A new verb, + `times`, answers each class's budget, the last seven days' median and worst delivery time, and every + delivery over its budget with where its time went; `deliveries` heads its list with one line of the same, + and `show` gives a delivery's phases. + - The controller's `status` shows that line, read from `times` by the self-check, beside what is wrong. +7. **A delivery over its budget is loud.** The self-check's probe reads `times` and raises the warning + `delivery..over-budget` when a delivery's time exceeds its class's budget, naming the delivery and its + longest phase. It clears when the next delivery of that class lands within its budget. +8. **The plan says when it will land.** `plans` and mesh-delivery's `show` give an open batch and an open walk an + expected landing time: now plus the remaining phases at the last seven days' median for its class, and once a + walk is judging, plus the readings still owed. + +## Consequences + +- **A leaf fix lands within five minutes when nothing else is in its way**, by the measurement's own estimate: + build 0.4, send 0.3, one or two readings at about a minute each, the rest a quarter of a minute. +- **A burst of unrelated merges no longer queues**; it walks in parallel, and one node judges one walk at a time. + The controller holds several open walks, and every reader of a walk (`plans`, the walk's conditions, `plan-moved`, + mesh-delivery) reads a list where it read one. +- **A leaf module is judged on less evidence.** A build that fails after its last reading reaches the rest. That + is bounded: the rest's first healthy report is recorded, a failure there is a condition on that node, and the + build that broke it is put back the ordinary way. The trade is the operator's stated choice of speed for leaf + modules; core modules keep every reading. +- **A core walk still holds its ground.** A leaf delivery whose node is judging a core build waits up to that + judgement (3.6 minutes for the controller) and may exceed its budget; the condition says so, and the week's + numbers show how often. +- **The core budget depends on issue 338.** Until the planner moves only the modules whose build source a merge + touched, a controller merge exceeds ten minutes and raises the condition each time. That is intended: the + condition counts what issue 338 costs. +- **What got harder:** the walk's record grows by the phases and readings, kept in the controller's store as a + numbered migration; the scheduler of walks gains the independence test and the per-node judgement rule, each a + place where two walks can disagree, and each tested. +- **On acceptance**, ADR 0236 (decision 2, three readings for every module) and ADR 0276 (decision 4, one walk at + a time; decision 1, the window for every merge) each gain a note saying the mechanism moved here, and to-be 47 + lists this record among its decisions. + +## How it is checked + +| Rule | Checked by | +|---|---| +| a lone merge is cut at once; a group's member and a merge during assembly wait | the controller's tests: a lone merge asks its build with no window; a merge whose group has an unmerged member waits for the window; a second merge within the window joins the batch | +| independent walks run at once, dependent ones wait | the controller's tests: two catalogue merges of unrelated modules walk at once; two of one repository, two sharing a module, and two joined by a graph edge do not; two walks never judge on one node at once; a send to a node under another walk's judgement waits for its verdict; no leaf reading counts while a new controller is judged | +| judgement by class | the controller's tests: a tools-only leaf passes on one reading, a leaf with a service on two forty seconds apart, a core module on three two minutes after the send; each fails and is put back as before | +| one tier's first nodes at once | the controller's test: a tier whose modules are first on two nodes judges both at once and passes when both pass | +| every phase and the delivery time are recorded | the controller's test of the walk record and `plan-moved`; mesh-delivery's tests of the delivery time, its class, `times` and `show` | +| a delivery over budget is loud | the probe's test: a leaf delivery of 5 minutes 10 seconds raises `delivery.leaf.over-budget` naming its longest phase; the next within budget clears it | +| the target holds on the live mesh | `mesh-delivery.times`: the week's leaf median within five minutes and the condition raised by no more than one leaf delivery in ten; the same measurement as this record's Context, repeated a week after the last part is built | + +## References + +- Issue 382 (a small fix takes 45 minutes to reach a node) and issue 338 (a module built from a shared repository + moves on every merge to it), in the issue tracker. +- [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md): + the first-node gate's three readings, kept for core modules. +- [ADR 0238](0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md): + a check's build is never registrable, which option 5 would have reopened. +- [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md): + mesh-delivery owns the delivery, and so its delivery time. +- [ADR 0267](0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md): + an unchanged artifact is not rebuilt. +- [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md): + the merge window and the walk this record changes in part. +- [Research 044](../01-RESEARCH/044-what-leaves-the-controller-next/00-overview.md), whose question 6 asks for + the time from merge to a node running the fix as a baseline; the table above is that baseline. +- To-be [47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md), amended with this record. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0290-a-decision-record-lives-whole-in-the-knowledge-base-made-of-clauses.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0290-a-decision-record-lives-whole-in-the-knowledge-base-made-of-clauses.md new file mode 100644 index 00000000..11393a8b --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/0290-a-decision-record-lives-whole-in-the-knowledge-base-made-of-clauses.md @@ -0,0 +1,187 @@ +--- +topic: how we work +status: accepted +date: 2026-10-11 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0153-the-record-is-read-by-a-module-and-the-console-lists-it.md +--- + +# 290. A decision record lives whole in the knowledge base, made of clauses + +> **Rewritten — 2026-10-11, on the operator's word.** This record was merged an hour earlier as *The knowledge base +> holds each record's structure and clauses, and hq keeps its reasoning*: the store held the structure, and this +> repository the reasoning, reviewed in pull requests. The operator asked why one record should live in two places, +> and asked for this record to be updated in place rather than superseded. Nothing had been built on the earlier +> version. It is in this repository's history at commit `08b1077b`. + +> **Updated — 2026-10-11, on the operator's word.** Rule 9 left open whether this repository keeps a generated +> public copy of accepted decision records and research efforts (issue 435). The operator answered: no copy is needed. Rule 9 and the consequences +> say so. + +## Context + +[Research 045](../01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md) measured, on 2026-10-11, how an +agent learns what the mesh decided ([findings](../01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md)). +The operator gave the purpose the same day: the decisions and research should become shared knowledge that agents +reach through tools, instead of reading markdown files, and a session should no longer have to start in this +repository's folder to know what was decided. + +**An answer costs too much.** "What holds about the merge window" took 26 search hits in 11 documents and about 85 KB +of reading, some 21,000 tokens. The decision index, generated by the records module +([ADR 0248](0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md)), is 77 KB, more than one +tool answer may be. The records module matches a phrase literally and knows nothing of which record still holds +([ADR 0153](0153-the-record-is-read-by-a-module-and-the-console-lists-it.md)). + +**What a decision decided is not data.** 226 of 263 decision records write their `Decision` section as prose; 37 +write it as numbered statements with a bold lead (198 in all). A later record that changes one statement of an older +one could say so only in prose or by naming the whole record: 21 `supersedes-in-part` links had no reverse, and 6 +more records said it in prose alone, until hq pull request 305 declared all of them in both records and added a check. + +**The checks run after the fact.** Of the 33 rules the hq checks enforce, 25 judge one record and 4 judge its prose; +all 29 could refuse a bad record when it is written. Only 4 need the whole record set. + +**Numbers collide.** 16 decision numbers were taken twice by sessions working at once between 2026-09-25 and +2026-10-10. [ADR 0275](0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md) +rule 8 already decided that the issue tracker hands out decision and research numbers; issue 424 tracks building it. + +### Against GENESIS + +- **The mesh owns the truth** ([mission](../00-META/mission.md)): a record lives in one place, written through one + module, not in a file every branch may edit. +- **The mesh brokers everything the work needs, knowledge included**: an agent asks the mesh what holds, by + capability, instead of knowing which folder to read. +- **The mesh remembers** ([effect](../00-META/effect.md)): what was decided, and why, is available to whoever needs it + next, on any node, in any folder, with every earlier version kept. +- **Failure must be loud**: a write that would leave the records inconsistent is refused when it is made. + +## Considered Options + +### Where a record lives + +1. **hq stays the truth; a module indexes it** (the research's option A). Rejected: a link about one statement of a + decision stays prose, and every inconsistency is found after it is written. +2. **Divided: the knowledge base holds a record's structure and clauses; hq keeps its reasoning.** Rejected by the + operator on 2026-10-11: one record in two places, an agent sent to a file for the *why*, and the structure written + twice until the frontmatter left hq. +3. **The knowledge base holds a record whole: structure, clauses and reasoning, with every version.** Adopted. It is + the division the operator made for issues ([ADR 0291](0291-an-issue-lives-whole-in-the-issue-tracker.md)), applied + to decisions and research. + +### The unit + +1. **The record.** Rejected: a later record changes one statement of an older one far more often than the whole. +2. **The clause: one numbered statement a decision makes, with its own status.** Adopted. The research called it a + *rule*; *rule* already names a line of [how-we-build](../00-META/how-we-build.md), and one word names one thing + ([ADR 0244](0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md)). + +### How a record is reviewed, once it is not a pull request + +1. **The operator approves every record.** Rejected: records are reviewed today by an independent agent, and the + operator is asked only for real decisions; making every record wait for one person slows what works. +2. **An independent agent reviews it, the write-time checks pass, and a record that needs the operator waits for the + operator's Approve on a channel that proves who answered.** Adopted. It is how a pull request here is merged + today, and the operator's Approve already reaches the mesh that way + ([ADR 0274](0274-an-agent-proposes-a-shared-instruction-section-and-every-node-takes-it-only-on-the-operators-warrant-which-it-proves-itself.md)). + +### Who confirms the clauses drafted for the 226 prose decisions (the research's open question 1) + +1. **The operator, topic by topic.** Rejected: 226 records is days of reading for one person, for statements already + decided and reviewed once. +2. **Nobody: drafted clauses are served as they are.** Rejected: a summary nobody checked would speak as the record's + word, which is the work-that-reports-success-and-did-nothing fault. +3. **A second judgement, independent of the one that drafted them, confirms each record's clauses against its text; + the operator is told per topic and may send any record back.** Adopted, and confirmed by the operator on + 2026-10-11. + +## Decision + +1. **The knowledge base is a Go module, `mesh-knowledge`, with one instance on the control-node.** It declares and + claims one seat held once for the mesh, `knowledge-base` + ([ADR 0126](0126-a-module-declares-its-own-seats.md)). Its first kinds are the **decision record** and the + **research effort**. +2. **The knowledge base is the truth for a record whole**: its number, kind, title, topic, status, date, deciders, its + typed links (held in both directions), its clauses, and its reasoning (context, options weighed, consequences, + how it is checked; a research effort's documents). **Every version is kept**, with who wrote it and when, and any + two versions can be compared. Once the import has run and been checked, no record is written in this repository. +3. **A decision record is made of clauses.** A clause is one numbered statement of what was decided, with a status: + `proposed`, `in force`, `changed` by a clause of a later record, or `superseded`. A link that is about a clause + names the clause, not only the record. A record is written as clauses from the start. Until the knowledge base + takes writes, a record written in this repository from 2026-10-11 on writes its `Decision` section as numbered + clauses with a bold lead, and `records.py` refuses an accepted one that does not. +4. **A record is proposed, reviewed and accepted in the knowledge base.** `propose` writes it with a number from + `reserve`, and refuses it outright when one of the 29 rules that judge one record or its prose fails. An + independent agent reviews it (`review`), and its approval moves the record and its clauses to `accepted` and + `in force`, and every clause it changes or supersedes accordingly. A record that says it needs the operator waits + for the operator's Approve, asked on a channel that proves who answered, and the warrant is kept with it + ([ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md)). + An accepted record changes only by a new record, or by a dated correction of a fact, written as a new version and marked as one; + the 4 rules that judge the whole record set run over it after every acceptance and raise a condition when one + fails. +5. **Numbers come from the issue tracker's counters**, as ADR 0275 rule 8 decided: `reserve` asks the tracker for the + next decision or research number, and a reserved number is spent whether or not its record is accepted. Until the + tracker hands them out, tracked by issue 424, a session takes the highest number on `main` and on every open + branch, plus one, as the shared instruction section *writing-decision-records* says. +6. **An answer is the shape agents need, bounded.** The verbs are `ask` (records and clauses bearing on a question, + ranked, each saying whether it holds or what changed it), `get` (one record: clauses, links both ways, what changed + it since; its reasoning only when asked), `holds` (the clauses in force for a topic or a record), `links`, + `history` (a record's versions and the difference between two), `reserve`, `propose`, `review`, `import` and + `return` (send drafted clauses back to be drafted again). `ask`, `get` and `holds` stay under a size bound they + state; the default is 4 KiB. +7. **State is in a database in the store**, provided to the module, not in buckets: `ask` needs ranked full-text + search, and a record's versions are a record set to search, not current state to watch + ([ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md) + is not narrowed). **Every change is an event the module declares** + ([ADR 0041](0041-events-are-a-relationship.md)), carrying what changed and who changed it. +8. **The import reads this repository at a commit and can run again.** It writes every decision record and research + effort whole, at its own number, with the commit as its first version's source; it creates what is absent, never + overwrites, never advances a counter, and a second run changes nothing and says so. A record whose clauses are + written as numbered statements gets them as `written`. A prose record gets them `drafted` by the agent's judgement + ([ADR 0278](0278-the-issue-tracker-checks-every-new-issue-for-a-duplicate-through-a-judgement-the-agent-makes-without-tools.md)), + then `confirmed` by a second judgement that sees only the record and the drafts; the operator is told per topic and + may send a record back. An answer marks a drafted or confirmed clause as such, and never serves a drafted clause as + in force. +9. **This repository stops holding decision records and research efforts** once the import is checked against it, + record by record: one pull request removes them, rewrites the mentions of their paths to "ADR NNNN" and "research + NNN", and changes playbooks 01 and 02, tracked by issue 426. This repository keeps no copy of them, generated or + otherwise; the knowledge base is the only place a decision record or research effort is read. The records module + stays the literal reader of what this repository still holds. +10. **It is built in slices, each tracked by an issue.** The first is the module for decisions and research: import, + clauses, answers, and proposing and reviewing a record, tracked by issue 425. The records leave this repository + under issue 426. Designs, playbooks and glossary entries follow the same way, tracked by issue 428. Meeting notes, + drawings and blog posts come last and are not decided here, tracked by issue 429. The design is + [to-be 55](../03-DESIGN/01-to-be/55-the-knowledge-base.md). + +## Consequences + +- An agent on any node asks what holds and why, and gets the clauses in force, the records they come from and what + changed them, in one call, instead of reading several records whole. No session needs this repository's folder to + read or write a decision. +- A later record that changes one statement of an older one says which, and the older clause's status changes when + the later record is accepted. +- Review moves from a pull request to the knowledge base's `review`, and the forge's line comments and diff are + replaced by the record's versions and `history`. A record is checked when it is written, not when it is merged. +- Until the first slice takes writes, records are still written here as pull requests, with numbered clauses, and the + import takes them in with the rest. +- What got harder: one more module the mesh's own process depends on. Reading or writing a decision needs the mesh to + be reachable; a node that cannot reach the control-node says so rather than guessing. Once the records leave it + (issue 426), the public repository no longer shows the mesh's decisions or research. + +## How it is checked + +- **Clauses in a record written here** (rule 3): `records.py`, check `clauses`, on every pull request of this + repository, until the records leave it. +- **What the module does** (rules 2, 4, 6, 7, 8): the module's tests and its live acceptance, listed in + [to-be 55](../03-DESIGN/01-to-be/55-the-knowledge-base.md) §9, built under issue 425. +- **Nothing lost when the records leave** (rule 9): every record compared with its file at the import's commit before + the files are removed, the result written on issue 426. +- **Numbers** (rule 5): the issue tracker's tests of its counters, built under issue 424; until then the + `numbering` and duplicate checks of `records.py` and `cycle.py` find a collision on `main`. + +## References + +- [Research 045](../01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md) and its + [findings](../01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md). +- [To-be 55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md). +- hq pull request 305: every partial supersession declared in both records, and the `partial-supersession` check. +- Issues 424, 425, 426, 428, 429 and 435 (answered: no copy in this repository). diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/README.md b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/README.md new file mode 100644 index 00000000..a9080ed5 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/02-DECISIONS/README.md @@ -0,0 +1,172 @@ +# 02-DECISIONS + +Architecture decision records — the "why" trail behind the rules in +[`00-META`](../00-META/) and the specifications in [`03-DESIGN`](../03-DESIGN/). + +**Numbered `02` because a decision precedes the design it authorises.** Research concludes, +the decision is recorded here, and only then is the design written. Following the folder +numbers walks the process in the order it happens. + +One file per decision, numbered, never deleted. A superseded record has its `status:` changed +and gains a pointer to what replaced it — **its reasoning is never rewritten**. The reasoning that +was rejected is the expensive half to rediscover. + +## Progressive insight + +A record is a decision, not a snapshot of everything that was true the day it was written, and +those two fail differently. **A fact a record asserted can turn out to be wrong while the decision +it supports stays right** — a count taken before anyone measured, a file named that does not +exist, a proof attributed to a step that cannot run it. Superseding a record for that buries a +correct decision under a second one, and teaches every reader to first work out which of two +records is live. Done a few times, the reading order stops being one. + +So: **a correction of fact that leaves the decision standing is made in the record, in place, +marked and dated.** + +> **Progressive insight — YYYY-MM-DD.** What was found, what the record said before, and what it +> says instead. + +Three conditions, all of which hold: + +- **It corrects a fact, not a judgement.** That a suite does not exist is a fact. That building it + is the wrong order is a judgement, and judgements supersede. +- **It adds; it never quietly replaces.** Where body text changes, the note says what stood there + before, so a reader who followed a citation to the old wording can find out what happened to it. + A correction nobody can see is indistinguishable from a record that was always right, which is + the failure the immutability rule exists to prevent. +- **The decision, the options weighed and the consequences stand untouched.** If the correction + changes what was decided, which alternatives were rejected, or a consequence another record + relies on, it is not an insight — write the superseding record. + +**What still supersedes**, without exception: reversing a decision, changing its scope, rejecting +an option it accepted, or making a consequence false that a later record cites. The test is not +how large the edit looks in a diff; it is whether a reader who acted on the old text would now be +wrong about *what was decided* rather than about *a detail the decision did not rest on*. + +**How this is checked.** `00-META/checks/records.py` requires every insight to be marked in the +form above and dated no earlier than the record's own `date:` — an unmarked edit is a rule +violation the reviewer looks for in the diff, and a marked one is legible in the record itself. +The git history is the backstop, not the record of intent; the note is the record of intent. + +## A pointer back from what a record changes + +A new record naming an old one is not enough. **Where a record changes a mechanism an older record +states — without reversing the decision, so no supersession — the older record gets a dated note +saying where its mechanism now lives.** A reader arrives at the old record by following a citation, +and finds text that is still the decision and no longer the method; nothing in it says a later record +moved the method, and the new record is not in their hands. + +> **The mechanism changed — YYYY-MM-DD, by ADR NNNN.** What still stands, what moved, +> and why. + +Three examples of the shape, all found by being missed: ADR 0066 still described a routed name being +written into every container after 0148 replaced that with resolution; ADR 0047 still said a module's +code runs in a container after 0150 made it a supervised process; and ADR 0016 still read as though the +lab were the test bed after 0149 said the live mesh is. Each was a citation leading to the wrong +answer, in a record that was not wrong about anything it decided. + +**This is not machine-checked, and it cannot be from `extends:` alone.** 102 records extend another and +87 name a parent that does not mention them, which is correct: extending usually means building on a +context, and a one-directional pointer is the right shape for that. What needs a note is the narrower +case where the parent's own text has gone stale, and which case that is, is a judgement — so it is a +rule for the author and the reviewer, and the diff is where it is caught. Making it mechanical would +mean a record declaring the relationship in its frontmatter, which is a change to the record schema and +has not been decided. + +**A supersession in part is the declared case, and it is machine-checked.** A record that replaces part of +an older one names it under `supersedes-in-part:`, and the older record carries, directly under its +heading and newest first, a note in the form + +> **Superseded in part — YYYY-MM-DD, by ADR NNNN** (a link to it). What no longer holds, and what stands +> as decided here. + +dated by the newer record's `date:`. `00-META/checks/records.py` (`partial-supersession`) fails when +either half is missing, when the note's date is not the newer record's, and on a note in the older form +*Narrowed, not replaced*. Where the note sits is the reviewer's to see in the diff. Saying "this narrows" or "this replaces" an older record in prose alone is not +enough: on 2026-10-11, 6 records did only that, and 11 of 21 declared links had no note on the record +they changed. + +The records run in the order the decisions were taken, oldest first. + +**Every decision is a record.** There is no ledger and no index file — if a decision is worth +recording it is worth a record, and if it is not worth a record it is not recorded +([ADR 0019](0019-how-this-repository-works.md)). A "decision" small enough to be one line is +almost always a **rule**, and a rule belongs in +[`00-META/how-we-build.md`](../00-META/how-we-build.md), where it is enforced and keeps the +incident that earned it. + +## Frontmatter + +```yaml +--- +status: proposed | accepted | superseded +date: YYYY-MM-DD # when the decision was taken, not when it was written down +deciders: name +reconstructed: true|false # true when the record was written after the fact from evidence +superseded-by: # 02-DECISIONS/NNNN-....md, when status is superseded +extends: # 02-DECISIONS/NNNN-....md, when this record widens an earlier one +supersedes: # a list of 02-DECISIONS/NNNN-....md this record replaces whole +supersedes-in-part: # a list of 02-DECISIONS/NNNN-....md of which this record replaces a part +--- +``` + +## Body + +``` +# N. Title in plain language + +## Context what was true, with evidence +## Considered Options numbered, each with why it was rejected +## Decision what was decided +## Consequences what follows, including what got harder +## References commits, pull requests, knowledge-base entries, prior art +``` + +State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"* +outranks *"the dependency rule is not followed"*. + +## Reconstructed records + +Records 0001–0014 were written on 2026-08-23, after the decisions they describe. Records 0015 onward were taken as records. Those +decisions were taken in implementation rather than in a document; the records state what was +decided and the evidence it was decided from, and each carries `reconstructed: true` and says +so in its first lines. + +A reconstructed record is not a transcript. Where the deliberation is not recoverable, the +options section states what the alternatives were and why the chosen one won on the evidence +available — not a discussion that did not happen. Where a date is not establishable it says so +rather than guessing. + +## Reading order + +**A number identifies a record and never changes.** Records are referenced from outside this +repository — code comments, commit messages — so a number that moves invalidates them silently. +Renumbering once cost 96 references across two code repositories, and that is why the numbers +are now fixed. + +So the folder is in creation order, and **the reading order lives here**: six topics, in the +order somebody would learn the system. Each record names its own in `topic:`. + +1. **What the mesh is** — `topic: the mesh`. What it is for, and the parts it is made of. +2. **Its tiers, from the bottom up** — `topic: the tiers`. The foundation, the node and the controller, and + what each one provides to the one above. +3. **What runs on them, and how it gets there** — `topic: what runs on it`. Modules, seats and + provisions, and how they reach a node. +4. **How it is built** — `topic: building it`. The code, its repositories, and what is made from + it for a node. +5. **How it is checked** — `topic: checking it`. What judges a change, and what proves a decision holds. +6. **How we work** — `topic: how we work`. This repository, its playbooks, and how decisions and + designs are made. + +**The list of records under each topic is generated when it is read, and never stored here** +([ADR 0248](0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md)). A stored list +was one more line in this file for every decision, so two decisions open at once always conflicted +([issue 297](../04-ISSUES/297-two-open-decisions-always-conflict/00-report.md)). Read it with: + +``` +python3 00-META/checks/index.py --print the records, by topic, in reading order +``` + +or the `hq-status` skill, or the records module's `records_decisions` from anywhere on the mesh. +`index.py` checks that every record's topic is one of the six above, and fails if this file stores the +list again. diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/COMMIT b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/COMMIT new file mode 100644 index 00000000..8b826a45 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/testdata/hq/COMMIT @@ -0,0 +1 @@ +d1edcf544b384502fded36c1605571ae74b07ed5 diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/wholeset_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/wholeset_test.go new file mode 100644 index 00000000..7cbd0776 --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/wholeset_test.go @@ -0,0 +1,92 @@ +package main + +import ( + "strings" + "testing" +) + +// to-be 55 §10: "A whole-set rule that fails is loud — the module's test: an acceptance that leaves a decision +// cited by nothing raises the condition." The condition is the mesh's, raised when the module's health answers +// unhealthy; the test reads the health answer the node-engine reads. +func TestAnAcceptanceThatLeavesADecisionCitedByNothingMakesTheModuleUnhealthy(t *testing.T) { + eachStore(t, func(t *testing.T, s Store) { + w := imported(t, s) + if h := w.k.Health(nil); !h.Healthy { + t.Fatalf("unhealthy after the import of real records: %s", h.Why) + } + n := proposed(t, w, "author@g14", false) + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + h := w.k.Health(nil) + if h.Healthy || h.State != "unhealthy" || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) { + t.Fatalf("an accepted decision nothing cites left the module %+v", h) + } + // The acceptance stands: the condition says it, and undoes nothing. + if mustGet(t, s, Key{KindDecision, n}).Current.Status != StatusAccepted { + t.Fatal("the acceptance was undone") + } + // A record that cites it heals it. + m := w.reserved(t, "author@g14") + p := valid(m) + p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)}) + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatal(err) + } + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: m, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + h = w.k.Health(nil) + if strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") { + t.Fatalf("still said after a record cited it: %s", h.Why) + } + }) +} + +// Each of the 4 whole-set rules, failing on a record set made to break it. +func TestEachWholeSetRuleFails(t *testing.T) { + acc := func(n int, links ...Link) *Record { + return &Record{Kind: KindDecision, Number: n, Status: StatusAccepted, Links: links, + Clauses: []Clause{{N: 1, Text: "**One.** x", Status: ClauseInForce, Source: SourceWritten}}} + } + to := func(kind string, n, c int) Link { + return Link{Kind: kind, To: Ref{Kind: KindDecision, Number: n, Clause: c}} + } + cases := []struct { + rule string + set []*Record + }{ + {"cited", func() []*Record { r := acc(1); r.AcceptedAt = r.AcceptedAt.AddDate(2026, 0, 0); return []*Record{r} }()}, + {"supersession", []*Record{func() *Record { r := acc(1); r.Status = StatusSuperseded; return r }()}}, + {"supersession", []*Record{acc(1), acc(2, to(LinkSupersedes, 1, 0))}}, + {"partial", []*Record{acc(1), acc(2, to(LinkChanges, 1, 1))}}, + {"partial", []*Record{func() *Record { r := acc(1); r.Clauses[0].Status = ClauseChanged; return r }()}}, + {"rests-on-live", []*Record{func() *Record { r := acc(1); r.Status = StatusSuperseded; return r }(), + acc(3, to(LinkSupersedes, 1, 0)), acc(2, to(LinkExtends, 1, 0))}}, + } + seen := map[string]bool{} + for _, c := range cases { + found := false + for _, f := range JudgeSet(c.set) { + found = found || f.Rule == c.rule + } + if !found { + t.Errorf("the rule %s did not fail on its case: %+v", c.rule, JudgeSet(c.set)) + } + seen[c.rule] = true + } + for _, r := range WholeSetRules { + if !seen[r.ID] { + t.Errorf("the whole-set rule %s has no case", r.ID) + } + } + if len(WholeSetRules) != 4 { + t.Errorf("%d whole-set rules; ADR 0290 counts 4", len(WholeSetRules)) + } + // A consistent set: the superseder names it, the extender is the superseder, the change is applied. + ok := []*Record{func() *Record { r := acc(1); r.Status = StatusSuperseded; return r }(), + acc(2, to(LinkSupersedes, 1, 0), to(LinkExtends, 1, 0))} + if f := JudgeSet(ok); len(f) != 0 { + t.Errorf("a consistent set fails: %+v", f) + } +} diff --git a/modules/mesh-knowledge/cmd/mesh-knowledge/writers_test.go b/modules/mesh-knowledge/cmd/mesh-knowledge/writers_test.go new file mode 100644 index 00000000..2cb123dd --- /dev/null +++ b/modules/mesh-knowledge/cmd/mesh-knowledge/writers_test.go @@ -0,0 +1,82 @@ +package main + +import ( + "strings" + "testing" +) + +// Who may write and review (the review of mesh-catalog#206): a proposal is revised only by its author, proven; a +// number reserved by a proven caller is proposed only by that caller; a review is refused from anyone who wrote +// any version of the record, and for a record whose author was never proven. + +func TestARevisionIsTakenOnlyFromTheRecordsAuthor(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, "author@g14", false) + p := valid(n) + p.Sections[2].Text = "Merges land a little sooner." + if _, err := w.k.Propose("other@novox", p); err == nil || !strings.Contains(err.Error(), "only its author") { + t.Fatalf("a revision by someone else: %v", err) + } + if _, err := w.k.Propose(Unproven, p); err == nil || !strings.Contains(err.Error(), "only its author") { + t.Fatalf("a revision by an unproven caller: %v", err) + } + if mustGet(t, w.k.Store, Key{KindDecision, n}).Versions != 2 { + t.Fatal("a refused revision wrote a version") + } + if _, err := w.k.Propose("author@g14", p); err != nil { + t.Fatalf("the author's revision: %v", err) + } +} + +func TestAReservedNumberIsProposedOnlyByWhoReservedIt(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@g14") + if _, err := w.k.Propose("other@novox", valid(n)); err == nil || !strings.Contains(err.Error(), "reserved by author@g14") { + t.Fatalf("a proposal under another's number: %v", err) + } + if _, err := w.k.Propose("author@g14", valid(n)); err != nil { + t.Fatal(err) + } +} + +// A writer of any version, not only the first, does not review it. +func TestAReviewByAnyoneWhoWroteAVersionIsRefused(t *testing.T) { + w := imported(t, newMemStore()) + n := w.reserved(t, "author@g14") + if _, err := w.k.Propose("author@g14", valid(n)); err != nil { + t.Fatal(err) + } + // An insight to an accepted record: its writers are the record's and the insight's. + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } + p := valid(n) + p.Sections[2].Text = "Merges land sooner, by a minute." + p.Insight = "it said only that merges land sooner" + if _, err := w.k.Propose("corrector@ace", p); err != nil { + t.Fatal(err) + } + for _, writer := range []string{"author@g14", "corrector@ace"} { + if _, err := w.k.Review(writer, ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err == nil || + !strings.Contains(err.Error(), "wrote") { + t.Fatalf("a review by %s, who wrote a version: %v", writer, err) + } + } + if _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}); err != nil { + t.Fatal(err) + } +} + +// A record proposed while callers were unproven is never reviewable, even once callers are proven: its real +// author cannot be told from its reviewer. It is proposed again. +func TestARecordProposedUnprovenIsNeverReviewed(t *testing.T) { + w := imported(t, newMemStore()) + n := proposed(t, w, Unproven, false) + _, err := w.k.Review("reviewer@novox", ReviewAsk{Kind: KindDecision, Number: n, Verdict: VerdictApprove}) + if err == nil || !strings.Contains(err.Error(), "propose it again") { + t.Fatalf("a review of a record whose author is unproven: %v", err) + } + if mustGet(t, w.k.Store, Key{KindDecision, n}).Current.Status != StatusProposed { + t.Fatal("accepted") + } +} diff --git a/modules/mesh-knowledge/go.mod b/modules/mesh-knowledge/go.mod new file mode 100644 index 00000000..0478dd45 --- /dev/null +++ b/modules/mesh-knowledge/go.mod @@ -0,0 +1,28 @@ +module mesh-knowledge + +go 1.25.0 + +require ( + git.novox.be/novox/mesh-sdk/go v0.1.12 + github.com/jackc/pgx/v5 v5.11.0 + github.com/nats-io/nats-server/v2 v2.11.17 + github.com/nats-io/nats.go v1.51.0 +) + +require ( + github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect + github.com/google/go-tpm v0.9.8 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/klauspost/compress v1.18.5 // indirect + github.com/minio/highwayhash v1.0.4 // indirect + github.com/nats-io/jwt/v2 v2.8.1 // indirect + github.com/nats-io/nkeys v0.4.15 // indirect + github.com/nats-io/nuid v1.0.1 // indirect + golang.org/x/crypto v0.50.0 // indirect + golang.org/x/sync v0.20.0 // indirect + golang.org/x/sys v0.43.0 // indirect + golang.org/x/text v0.36.0 // indirect + golang.org/x/time v0.15.0 // indirect +) diff --git a/modules/mesh-knowledge/go.sum b/modules/mesh-knowledge/go.sum new file mode 100644 index 00000000..4cfd952c --- /dev/null +++ b/modules/mesh-knowledge/go.sum @@ -0,0 +1,53 @@ +git.novox.be/novox/mesh-sdk/go v0.1.12 h1:QvmLlT88F+OYt8MOytMGM5ZEOo0fKmNZvNVeJe6gnTk= +git.novox.be/novox/mesh-sdk/go v0.1.12/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY= +github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= +github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= +github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg= +github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE= +github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4= +github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ= +github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU= +github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg= +github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I= +github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24= +github.com/nats-io/nats.go v1.51.0 h1:ByW84XTz6W03GSSsygsZcA+xgKK8vPGaa/FCAAEHnAI= +github.com/nats-io/nats.go v1.51.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno= +github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4= +github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs= +github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw= +github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI= +golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= +golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= +golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= +golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= +golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/modules/mesh-knowledge/module.json b/modules/mesh-knowledge/module.json new file mode 100644 index 00000000..df9f51ef --- /dev/null +++ b/modules/mesh-knowledge/module.json @@ -0,0 +1,185 @@ +{ + "module": "mesh-knowledge", + "version": "1", + "slug": "knowledge", + "requires": [ + "postgres-database" + ], + "contributes": { + "postgres-database": { + "name": "knowledge" + } + }, + "binds": { + "postgres-database": "${dir:state}/database.json" + }, + "secrets": { + "postgres-database": "${dir:state}/database.secret" + }, + "seats": [ + { + "name": "knowledge-base", + "scope": "mesh", + "serves": [ + "ask", + "get", + "holds", + "links", + "history", + "reserve", + "propose", + "review", + "import", + "return" + ] + } + ], + "claims": [ + { + "name": "knowledge-base", + "scope": "mesh", + "serves": [ + "ask", + "get", + "holds", + "links", + "history", + "reserve", + "propose", + "review", + "import", + "return" + ] + } + ], + "tools": [ + "knowledge_health" + ], + "emits": [ + "reserved", + "proposed", + "reviewed", + "accepted", + "changed", + "imported", + "drafted", + "confirmed", + "returned" + ], + "uses": [ + "operator-channel" + ], + "invokes": [ + "seat:issue-tracker.reserve", + "claude-code.claude_code_judge", + "records.records_status", + "records.records_list", + "records.records_read", + "seat:operator-channel.notify" + ], + "resources": [ + { + "id": "state", + "type": "directory", + "mode": "0700", + "place": "." + }, + { + "id": "database-url", + "type": "file", + "path": "${dir:state}/database.url", + "mode": "0600", + "content": "postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}\n" + }, + { + "id": "settings", + "trusted": false, + "type": "file", + "path": "${dir:state}/settings.json", + "mode": "0600", + "merge": "json", + "content": "{\n \"judge-model\": \"sonnet\",\n \"drafting\": \"on\",\n \"reserved-for\": \"168h\"\n}\n" + }, + { + "id": "prepare", + "type": "process", + "name": "mesh-knowledge-prepare", + "artifact": "tools", + "run": [ + "./mesh-knowledge", + "prepare" + ], + "run-once": true, + "env": { + "DATABASE_URL_FILE": "${dir:state}/database.url" + }, + "restart-on": [ + "database-url" + ] + }, + { + "id": "keep", + "type": "process", + "name": "mesh-knowledge-keep", + "artifact": "tools", + "run": [ + "./mesh-knowledge", + "keep" + ], + "env": { + "DATABASE_URL_FILE": "${dir:state}/database.url" + }, + "restart-on": [ + "database-url" + ], + "health": { + "kind": "unit" + } + }, + { + "id": "watch", + "type": "process", + "name": "mesh-knowledge-watch", + "artifact": "tools", + "run": [ + "./mesh-knowledge", + "watch" + ], + "env": { + "DATABASE_URL_FILE": "${dir:state}/database.url", + "MESH_KNOWLEDGE_STATUS": "${dir:state}/health.json" + }, + "restart-on": [ + "database-url" + ], + "health": { + "kind": "tool", + "tool": "knowledge_health", + "interval": "60s", + "timeout": "10s", + "looks": 2, + "grace": "180s" + } + } + ], + "build": { + "artifacts": [ + { + "name": "tools", + "kind": "bundle", + "language": "go", + "system": "arch", + "from": "cmd/mesh-knowledge", + "binary": "mesh-knowledge", + "loads": [ + "mesh-knowledge" + ], + "env": { + "DATABASE_URL_FILE": "${dir:state}/database.url", + "MESH_KNOWLEDGE_SETTINGS": "${dir:state}/settings.json", + "MESH_KNOWLEDGE_STATUS": "${dir:state}/health.json" + } + } + ] + } +}