diff --git a/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md b/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md index dd6f6a96..f08e9a1e 100644 --- a/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md +++ b/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/00-overview.md @@ -21,8 +21,14 @@ became: [] ## What Whether the mesh should **write** its own knowledge (decision records, guidelines, policies, designs and -playbooks) through a module's verbs, instead of only **reading** hand-written markdown. The effort -frames the questions below and answers none of them yet; the research itself is done separately. +playbooks) through a module's verbs, instead of only **reading** hand-written markdown, and whether it +should hold light content beside them: meeting notes, architectural drawings and blog posts. + +The operator's purpose, given on 2026-10-11: the record becomes **shared knowledge that agents reach +through tools**, so that an agent gets what was decided as an answer instead of reading markdown files, +and a session no longer has to start in this repository's folder to know it. A session still starts here +for other work, such as writing a record. The operator later put the light content last, and decided that +an issue's content belongs in the issue tracker rather than in this repository. Today the record is read-only. The `records` module reads this repository and serves it to sessions ([ADR 0153](../../02-DECISIONS/0153-the-record-is-read-by-a-module-and-the-console-lists-it.md)). @@ -38,6 +44,9 @@ Numbers collide, and the checks find a collision only after it happens. On 2026- number for a different effort. On 2026-10-07 three decision numbers collided in one evening. A record written by a verb could take its number from one counter, the way the issue tracker does. +Reading is costly too. Answering one question about what holds today means reading several records +whole, and the generated decision index is larger than one tool answer may be. + Some knowledge also has no home. This repository is public, so it may not name nodes or describe operations; the mesh's knowledge about its own installation lives in agents' private notes, in no shared and searchable place. @@ -72,7 +81,7 @@ shared and searchable place. 6. **Checks as refusals at write.** `records.py`, `cycle.py`, `words.py` and `index.py` judge a pull request after it is written. Which of their rules can become refusals of the write verb itself, and which only make sense across the whole record? -7. **Import.** How are the decision records (265 files on `main` on 2026-10-10, numbered up to 0279), the designs and the playbooks imported without +7. **Import.** How are the decision records (274 on `main` on 2026-10-11, numbered up to 0289), the designs and the playbooks imported without losing their history, their numbers or their links? Is the import repeatable, as the issue tracker's is? 8. **What git gives today.** Review on a pull request, line comments, a history of every change with @@ -92,3 +101,9 @@ How this repository works (ADR 0019) and how the development cycle is checked (A `records` module (ADR 0153) and the generated decision index (ADR 0248), the issue tracker and its counters (ADR 0275, to-be 53), approval on a proving channel (ADR 0274), the playbooks in `00-META/process/`, the hq checks in `00-META/checks/`, and the glossary. + +## Documents + +- [01 — Findings](01-findings.md): what exists, measured on 2026-10-11; where the truth lives (the store + for each record's structure and rules, this repository for its reasoning); issues whole in the issue + tracker; the order of work; and what is still open. diff --git a/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md b/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md new file mode 100644 index 00000000..dacb0bfd --- /dev/null +++ b/01-RESEARCH/045-a-knowledge-base-the-mesh-writes/01-findings.md @@ -0,0 +1,295 @@ +# 01 — Findings + +Measured on 2026-10-11 against `main` of this repository, the records module and the issue tracker as they +answer through the mesh MCP server, and the hq checks in `00-META/checks/`. Section 0 is what exists; section 1 is where the truth lives, which +decides the rest; sections 2 to 9 answer the overview's questions and the operator's additions; section 10 is +the order of work, and section 11 what is still open. + +## The purpose, in the operator's words + +The operator gave the purpose on 2026-10-11: the decisions and the rest of the record should become **shared +knowledge that agents reach through tools**, instead of reading markdown files, and a session should no longer +have to start in this repository's folder to know what was decided. A session still starts here for other work, +such as writing a record. Later that day the operator widened the scope to **light records** +(meeting notes, architectural drawings, blog posts) and then put them last; asked for a structured store of +the decisions and research efforts; and decided that an issue's content belongs in the issue tracker, not in +this repository. + +That purpose moves the weight of the overview's questions. Where the truth is stored (option A or B) matters +less than **what an answer looks like**: an agent needs the rule that holds now, with the records it rests on, +in one call and a few kilobytes. + +An acceptance sentence for the whole effort: *a session started in any folder on any node asks "what holds +about the merge window, and why" and gets, in one call of under 4 KiB, the rules in force, the records they +come from, what changed them since, and the commit it was read at.* + +## 0. What exists today, measured + +| Fact | Measured | +|---|---| +| Records on `main` | 274 decision records, 44 research efforts (119 files), 55 to-be and 16 as-is designs, 8 playbooks, 361 issue narratives (479 files): 951 files, and the glossary | +| Size | 6.9 MB of markdown, about 1.7 million tokens at four bytes a token; decision records alone 2.6 MB (about 650,000 tokens) | +| One question, today | "what holds about the merge window": `records_search` gives 26 places in 11 documents; reading the five records and the design they point to is 85 KB, about 21,000 tokens | +| The decision index | `records_decisions` answers 77 KB in one piece; the agent's tool-answer limit refused it inline, and it had to be read from a file in parts | +| What `records_search` does | a literal, case-insensitive phrase match giving document, line and heading; no ranking, no notion of which record is still in force | +| Frontmatter of a decision | `topic`, `status`, `date`, `deciders` on all 274; `extends` on 229 (182 with the folder prefix, 47 without); `supersedes-in-part` on 15 (21 links), `supersedes` on 8, `superseded-by` on 9, `rests-on` on 2, `amended` on 1 | +| Status of a decision | 265 `accepted`, 9 `superseded`; no record is marked as changed in part, though 21 links say so | +| Supersession said only in prose | 6 records say they supersede, amend or replace another and carry no field for it; 7 records carry a "superseded" marker in the body, of which only 1 has the field or the status | +| Progressive insights | 39 records carry the dated marker, 55 markers in all; 2 more use an older form | +| Change after writing | 189 of 271 traceable decision records were changed after the commit that created them; 34 paths were changed five times or more | +| Pace | commits touching decision records: 87 in the last ten days of August, 160 in September, 274 in the first eleven days of October | +| Number collisions | 16 decision numbers were taken twice by sessions working at once (2026-09-25 to 10-10), each repaired by a renumbering commit; 15 numbers on `main` are gaps; no research number collided on `main` (the 043 collision of the overview never reached it) | +| A decision's rules | 263 records have a `Decision` section; only 37 of them write it as numbered rules with a bold lead (198 rules in all); the rest write prose | +| Drawings | no file but markdown in the repository; no diagram source; 22 designs carry diagrams drawn in text characters | +| Knowledge with no home | the repository is public, and its README says what only makes sense to someone who knows this installation "is either a note in the wrong place or a disclosure"; one agent's private notes hold 89 files for one repository on one node, and issue 367 (a session's memory stays on the node where it was learned) is open | +| The hq checks | 33 rules: 25 judge one record, 4 the whole record, 4 the prose (section 7) | + +Three things follow. + +1. **The cost of an answer is the problem, not where the files are.** 85 KB for one question, and an index + too large for one tool answer, mean a session reads in parts or reads less than it should. Moving the + markdown into a store without changing the shape of the answer would keep that cost. +2. **The data that makes an answer cheap is missing or loose.** What a record decided is prose in 226 of 263 + records. Which records changed which is half in fields, half in prose: 6 records say it only in prose, and + 6 more carry a body marker that no field or status agrees with. + Status says `accepted` for records that 21 later links change in part. +3. **The records change after they are written, and that is reviewed.** 189 of 271 decision records were + edited after creation, every edit through a pull request with a human approval. Whatever is built must keep + that review for what governs the mesh. + +## 1. Where the truth lives: what a clean start would choose + +The operator asked on 2026-10-11 which option this effort would choose with no prior setup. The answer +decides the rest, so it comes first. + +**The unit the mesh reasons about is a rule, not a file.** A decision is a handful of rules. A later decision +changes one of them: ADR 0282 replaces one rule of ADR 0276 and leaves the others standing. Git and markdown +know files only, so that change can be written only as prose. The earlier record keeps `accepted`, and every +reader must read both records to learn what holds. Section 0's loose links all have this one cause: 21 partial +supersessions with no reverse, 6 said only in prose, 6 markers no field agrees with. + +**A store can hold what a repository cannot.** In a store, each rule is an entry with its own status and links +("in force", "changed by rule 2 of 0282"). A write that leaves the record inconsistent is refused when it is +made, not found by a check afterwards. Numbers come from one counter. "What holds now" is one query. + +**What git does best is review of prose.** The reasoning behind a decision (its context, the options weighed, +its consequences) is read and commented on by people. Agents rarely ask for it. They ask what holds. + +So a clean start divides **by layer, not by kind**: + +- **the store is the truth for the structure** of every governing record: its number, title, kind, topic, + status, its rules one by one, and its typed links, at the level of the rule where a link is about a rule; +- **this repository keeps the reasoning** as markdown under the store's number, reviewed in pull requests. + +That is the division the issue tracker already made for issues +([ADR 0275](../../02-DECISIONS/0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md)): +the tracker holds number, status and links, and hq held the narrative. Section 9 takes issues one step further, +on the operator's word: their narrative leaves this repository too. + +Option A (git stays the truth, and a module's write verbs open pull requests, with numbers from a counter) was weighed and is not chosen. It keeps review, but it +keeps the rule-level links in prose and the refusals after the fact, which is the cause of section 0's mess. +Option B in full (the reasoning moves into the store too) is not chosen either: it would rebuild review, line +comments, history and diff for prose, which git already gives. + +## 2. Kinds and their lifecycles + +| Kind | Lifecycle | Notes | +|---|---|---| +| decision | proposed → accepted → superseded; a rule can be changed or superseded on its own; a dated progressive insight corrects a fact | 274 records | +| research | active → graduated or abandoned | 44 efforts | +| design | proposed → designed → in-progress → implemented, in one of two layers (to-be, as-is) | 71 documents; a later slice (section 10) | +| playbook | in force, or retired | 8, with no frontmatter today; a later slice | +| glossary entry | in force, or retired with its replacement | one file today; a later slice | + +The first slice holds decisions and research, the two the operator named. + +A **guideline** or a **policy**, which the overview named, has no record of its own today. What the operator +calls a policy is written as a decision; what an agent should do is a shared instruction section (ADR 0274) or +a personal rule ([ADR 0273](../../02-DECISIONS/0273-the-agents-home-is-the-modules-and-the-operators-personal-rules-and-lessons-are-registered-through-it.md)). +No new kind is proposed for either. + +**Light records** (meeting notes, drawings, blog posts) were added to the scope by the operator on 2026-10-11 +and put last by the operator the same day. They govern nothing, need no review to exist, and are private by +default; a drawing's file belongs in an object-store bucket +([ADR 0288](../../02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md)) +with a text description an agent reads. They are a later slice (section 10) and shape nothing in the first. + +## 3. Numbering from one counter + +The 16 concurrent collisions all come from reading the next number off a repository that many branches write +at once. The issue tracker hands out numbers by compare-and-set on a counter key +([to-be 53](../../03-DESIGN/01-to-be/53-the-issue-tracker.md) §4); the knowledge base asks it for every number. + +- **One sequence per kind**: decision, research, and later design. Numbers keep their meaning in conversation + ("ADR 0276"). +- **A reserved number is spent.** A proposal that is abandoned keeps its number with status `withdrawn`, who + reserved it and when. Gaps already exist (15 on `main`) and harm nothing; a reused number would. +- **The counter starts above the highest number on `main` and on every open branch**, as the tracker's did. +- **The counters stay where ADR 0275 put them.** Its rule 8 decided that the issue tracker hands out decision + and research numbers too, in its second slice; to-be 53 §4 so far names only the issue counter. The knowledge + base's `reserve` asks the tracker for the number, so one counter holds every sequence and no accepted record + is superseded for it. + +## 4. Rules and typed links + +**A rule** is one short statement a decision makes, numbered within its record ("0276 rule 1"), with its own +status: `in force`, `changed by` a rule of a later record, or `superseded`. 37 of 263 decision sections already +write numbered rules with a bold lead (198 rules); the other 226 write prose, from which the import drafts the +rules (section 7). + +The links the first slice holds, each stored in both directions: + +| Link | From → to | Today | +|---|---|---| +| supersedes | record → record | `supersedes` / `superseded-by`, 9 of 9 mirrored | +| changes | rule or record → rule | `supersedes-in-part` on 15 records, one direction; 6 more only in prose | +| extends | record → record | `extends` on 229, two spellings | +| became | research → decision or design | `became` | +| decided by | design → decision | `decisions` | +| corrects | progressive insight → record | the dated marker, 55 in 39 records | +| cites | any → any | prose links: 1,785 decision to decision, 469 decision to issue | + +"What holds about X" is then: the rules matching X whose status is `in force`, each with its record and the +changes that brought it to its present wording. + +## 5. Approval + +A decision governs how the mesh is built and run, so it needs a human approval before it holds. + +- **Proposing** reserves a number and writes the structure with status `proposed`. Any agent may propose. +- **Accepting** happens when the pull request carrying the record's reasoning merges with its human approval + and both required checks. The knowledge base sees the merge on the bus and moves the record and its rules to + `accepted` and `in force`, and every rule it changes to `changed by`. The merge is the warrant; a second + approval beside it would split the evidence. +- **A progressive insight** is a correction of fact and goes the same way: a pull request on the reasoning, + and the structure recorded on merge. +- A **structural write without a pull request** (a link added, a rule's wording fixed) is refused for an + accepted record. Changing what an accepted record says is a new decision, as it is today. + +## 6. Public and private + +This repository is public and stays so. The reasoning of a decision or research effort stays here, public. +The structure in the store is derived from public records and may be exported here as generated pages, like +the decision index of [ADR 0248](../../02-DECISIONS/0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md), +generated when read and never stored. + +What changes is the issues' narrative (section 9): it moves into the tracker, which is the mesh's own and not +public. The public repository loses those narratives. ADR 0275 kept them public on purpose, citing *the mesh +remembers*; the tracker remembers them as well, but only for the mesh. Whether a resolved issue's narrative is +ever exported publicly is a question for the decision that supersedes that part of ADR 0275. + +Knowledge about this installation (one agent's 89 private notes, issue 367) stays out of scope. Agents' lessons +and rules belong to the agent module's register (ADR 0273). + +## 7. Checks as refusals at write, and the import + +Of the 33 rules the hq checks enforce, **25 judge one record** and become refusals of the knowledge base's +write verbs: a number taken twice, a supersession not mirrored, a cited record not accepted, a record without a +topic, a research effort graduated without `became`. Two need a look beyond the record: a write that +supersedes must check who cites what it supersedes. **4 judge the whole record** and **4 judge the prose**; +those stay checks on this repository, run on every pull request. As frontmatter leaves this repository +(section 8), the 25 leave `records.py` and `cycle.py` with it, and a check is added that a record's reasoning +names a number the knowledge base holds. + +**The import** reads this repository at a commit through the records module and writes each decision and +research effort at its own number: status, topic, date, deciders, links, and its rules. Like the tracker's, it +creates what is absent, never overwrites, never advances a counter, and a second run changes nothing and says +so. Two things a reading cannot recover alone: + +- **Loose links.** The 6 supersessions said only in prose, the 6 markers no field agrees with, the 47 bare + `extends` targets and the 21 one-way partial supersessions are repaired in this repository first, by one + pull request, so the import reads clean input. +- **The rules of 226 decisions written as prose.** The import drafts them through the agent's judgement + ([ADR 0278](../../02-DECISIONS/0278-the-issue-tracker-checks-every-new-issue-for-a-duplicate-through-a-judgement-the-agent-makes-without-tools.md)) + and stores them as `drafted`, not `in force`. A topic's drafts are confirmed by the operator or a reviewer in + one pass. An answer marks a drafted rule as drafted, and never presents it as the record's word. + +## 8. What leaves this repository, and what stays + +| | Today, in this repository | After | +|---|---|---| +| a decision's number, status, topic, links | frontmatter | the knowledge base; the file keeps the number in its name and heading | +| a decision's rules | prose in the `Decision` section | the knowledge base; the section stays as prose for the reasoning | +| a decision's context, options, consequences | markdown | unchanged | +| a research effort's status and `became` | frontmatter | the knowledge base | +| a research effort's documents | markdown | unchanged | +| the decision index | generated from frontmatter when read | generated from the knowledge base when read | +| an issue's report, diagnosis, resolution | `04-ISSUES/NNN-slug/` for 361 issues | the issue tracker (section 9) | + +Frontmatter leaves a file only after the import has proven it holds the same values, one kind at a time, as +ADR 0275 did for issues' status. + +## 9. Issues live in the issue tracker, whole + +The operator decided on 2026-10-11 that an issue's content belongs in the issue tracker, not in this +repository. Measured on `main`: 361 issue folders, 1.9 MB; a folder's median is 4.4 KB and the largest +25.9 KB, well under a tracker value's 256 KiB. Every folder has `00-report.md`; 107 have a diagnosis, 9 a +resolution, 1 a progress file. Since ADR 0275, new issues live in the tracker already; the highest issue folder +is 364. + +What the tracker needs to hold it: + +- **an issue's documents**: the report it already holds, plus a diagnosis and a resolution as named documents + of the issue, each with its author and date, revised in place with history (notes stay append-only); +- **an import** of the 361 folders, repeatable, keeping each issue's number and dates; +- **links** from records: 250 files in this repository mention a path into `04-ISSUES/` 765 times, counted + as every `04-ISSUES/NNN` path in the text (markdown links alone are fewer, about 660). Those mentions become + plain "issue N" references that the knowledge base and the tracker resolve, rewritten in one pull request + once the import is proven; the link check in `records.py` then refuses a new link into `04-ISSUES/`. + +This supersedes part of ADR 0275 (hq keeps the narrative of a design or governance issue), so it needs its own +decision record. + +## 10. Slices + +1. **Clean the input** (this repository, one pull request): repair the loose links of section 7, so the import + reads clean records. In review as hq pull request 305, which declares every partial supersession in both + records and adds the `partial-supersession` check. Beside it, a shared instruction section for every + session: take a decision or research number above every one on `main` and on open branches until the + counter exists, and name a partial supersession in a field, never only in prose. +2. **The knowledge base, decisions and research**: a Go module serving one seat held once for the mesh, on + the control-node beside the issue tracker. The import, the rules and links, and the answers; numbers come from the issue tracker (section 3): + + | Verb | Answers | + |---|---| + | `ask` | the records and rules that bear on a question, ranked: number, kind, status, title, and whether each rule still holds or what changed it | + | `get` | one record: its rules, its links both ways, what changed it since; its reasoning's text only when asked | + | `holds` | for a topic or a record, the rules in force now, each with the record it comes from | + | `reserve` | the next number of a kind, from the issue tracker's counter (section 3), spent whether or not the record lands | + | `propose` | a new record's structure, status `proposed` | + | `import` | this repository's decisions and research at a commit, repeatable | + + Storage is a database in the store, provided to the module: the structure is small, but `ask` needs ranked + full-text search, which the store's server does and a bucket does not. Every answer is bounded in size. +3. **Frontmatter leaves this repository**, kind by kind, once the import is proven equal. +4. **Issues whole in the tracker** (section 9): documents, import, links rewritten, folders removed. +5. **Designs, playbooks and glossary entries** into the knowledge base the same way. +6. **Light records**: notes, posts and drawings. + +How a session reaches it: the knowledge base's tools reach every session through the mesh MCP server, and one +shared instruction section, approved by ADR 0274's route, tells every session to ask the knowledge base what +holds before reading files. A session needs this repository's folder only to write a record's reasoning. + +Proved by the acceptance sentence at the top: the merge window question, asked from a session outside this +repository, answered in one call under 4 KiB, naming ADR 0276's rules in force and the one ADR 0282 changed. + +## 11. What needs the operator + +Decided by the operator on 2026-10-11: the store is the truth for structure and this repository for reasoning +(section 1); light records come last; issues live whole in the tracker (section 9). + +Still open: + +1. **Drafted rules (section 7)**: the 226 prose decisions get their rules drafted by the agent's judgement and + confirmed a topic at a time. Who confirms: the operator, or an independent reviewing agent with the operator + seeing a summary? +2. **Issues' public narrative (section 6)**: once issues leave this repository, is a resolved design or + governance issue ever exported here publicly, or does it stay the mesh's alone? + +## Words + +The glossary has **decision record**, **design**, **playbook** and **the record** (the records module). This +effort proposes, for a decision to settle through the glossary's own rules: **knowledge base** for the module +and its seat; **rule** for one statement a decision makes, with its own status; **governing record** for a kind +reviewed before it holds; and **note**, **post** and **drawing** for the light kinds. *Policy* stays the word +for a module's upgrade policy, and *guideline* is not proposed, since section 2 finds no record of that kind.