jochen
81aad4189b
The operator asked whether the issue tracker could estimate how hard an issue is and so which model an agent should fix it with. Framed as questions, not answers.
6.4 KiB
status, initiated, touches, became
| status | initiated | touches | became | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| active | 2026-10-10 |
|
043 — An issue's complexity, as a hint to which model should solve it
What
Whether the issue tracker can estimate how hard an issue is to solve, and use that estimate to suggest which model an agent should solve it with: a smaller, cheaper model for a simple fix, the strongest available for a change to the core of the mesh. The idea is the operator's, from 2026-10-10, and he named it a research topic, not a build.
The tracker already asks a model one question about every new issue: the duplicate check (ADR 0278) reads the new issue's report against the backlog, through the agent module's tool that answers a structured question with no tools of its own. An estimate of complexity could be one more answer from the same kind of judgement. Whether it should be, and whether its answer is worth acting on, is what this effort is for.
Why
Agents solve most issues on the mesh, and every session spends a licence's allowance. Today the model a session runs is whatever the session was started with: nothing in an issue says what it needs, and nothing records afterwards which model solved it, how long it took or whether its fix held. A strong model on a one-line fix spends more than it has to; a weak model on a change to the controller or the node-engine can produce a fix that looks done, passes a careless review, and breaks a node that updates unattended. The second cost is far larger than the first, so a hint is only worth having if it is right where it matters most.
The questions
- What "complexity" means here. Lines changed, repositories touched, whether a core component or
the store's schema is involved, whether the cause is known yet, whether a decision is needed first?
Are these one scale or several? Which of them can be read from a report, before anyone has
diagnosed the issue, and which only become known once it is
located? An estimate made atopenmay have to be revised atlocated, and the question is whether the first estimate is worth making at all. - Who estimates, and when. A judgement at
open, beside the duplicate check; one atlocated, when the owner is known; the agent that diagnoses the issue, in its own words; or the operator. Can the owner named inlocated-inalone carry most of the signal (a module's own tool versus the controller)? - Where the hint lives. A field stored on each issue, set and changed through a verb with who set it and why, or a value derived each time from the issue's fields and a setting that maps classes of work to models? A stored hint can be overruled per issue; a derived one changes everywhere when the setting changes, which matters when a newer model arrives. Is the hint a model's name, or a tier that a setting maps to whatever model holds that tier today?
- How the outcome is measured, so the choice rests on evidence. What the mesh would have to record
per solved issue: the model used, tokens and wall time spent, whether the pull request passed its
independent review and both required checks (
mesh/merge-gate,mesh/repo-check) the first time, how many rounds of rework it took, and whether the issue was reopened or a follow-up opened against the same cause. Which of these does the mesh already hold (the tracker's notes and links, the forge's review history, the delivery's checks), and which are recorded nowhere today? Without this, any mapping from complexity to model is an assertion. - The risk of a weak model on a core change. Failure must be loud: how does a session that was given too weak a model say so, rather than hand over a fix that reports success? Should a hint ever lower the model below what the session was started with, or only raise it? Should some owners (the controller, the node-engine, the store, the foundation) be outside the hint entirely, always solved with the strongest model? Is review by a stronger model than the author's a cheaper safeguard than choosing the author well?
- How it relates to the fields being added to the tracker. Severity says how much an issue hurts; what an issue needs says what must happen before it can move. Complexity says how hard the fix is. Are these three separate fields, or is complexity derivable from the other two and the owner? A severe issue can be simple, and a minor one can sit in the core.
- Who acts on the hint. Does an agent session read it and start a sub-agent with that model; does the operator see it when choosing what to hand off; does a future dispatcher assign work by it? A hint nobody reads changes nothing, and the effort should name its reader before anything is built.
- Prior art. Routing a request to the cheapest model that answers it well is studied for single questions (model cascades and routers that learn from preference data); estimating a task's difficulty is studied for software issues in agent benchmarks, and in teams as estimation by size or points. What carries over to a repair done by an agent across a pull request, review and delivery, and what does not?
Words
The glossary has no word for an issue's estimated difficulty, nor for a suggested model. This effort uses complexity and model hint as plain descriptions until a decision settles them, and proposes them then through the glossary's own rules.
What it touches
The issue tracker's fields and verbs (to-be 53), the judgement the duplicate check already makes (ADR 0278), the agent module and the licences it spends (ADR 0206), the issues playbook (03), and, if outcomes are recorded, the forge's reviews and the delivery's required checks as sources of evidence.