Compare commits

Author SHA1 Message Date
jochen 05c0396f6b mesh-reads: drop the binary a local build left in the tree (issue 418)
mesh/merge-gate fail: builds new: modules/mesh-reads, sent nowhere; no bus step; a manifest the change touches fails the module check: modules/mesh-reads/m…
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/delivery stopped: the pull request closed unmerged
2026-10-12 14:04:40 +02:00
jochen c36677b9e5 mesh-reads: read a node's files and accounts as an account that holds no secret (hq ADR 0310, issue 418)
Reading a file, searching one, listing a directory, the owner and mode of each
component of a path and looking up an account went through the login shell's
execute as the operator's account, which sees the operator's keys and the
agent's credentials, and execute is not served on the control-node. This module
holds node-reads on every node, runs as mesh-reads (no login, no group beside
its own), refuses the node's secret places by name before any read (the
controller's list and the known credential files), withholds any line shaped
like a secret through mesh-sdk's outward rule, and bounds every answer.
2026-10-12 14:04:12 +02:00
mesh-admin b82aa1691c Merge pull request 'mesh-knowledge: the review asks changes for a decision nothing would cite, so its acceptance never turns the module unhealthy (hq issue 504)' (#237) from fix/504-propose-refuses-uncited-record into main 2026-10-12 11:44:58 +00:00
mesh-admin 3fe19c9c0e Merge pull request 'messenger: a walk's held messages are told only once its message was sent, handed on only as a first hold would take them, and its start and end compared on the controller's clock (hq issue 538)' (#238) from fix/538-walk-hold-told-handover-clock into main 2026-10-12 11:41:47 +00:00
jochen 424fc5a281 messenger: keep what a walk held owed in the state until its message was sent, so a restart says a held warning too (issue 538, review 39)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A warning held on purpose and lost with the walk's message at a restart was turned
into history and never told. Name the walk in the told line from its own item.
2026-10-12 13:35:48 +02:00
jochen 56a9a1b082 mesh-knowledge README: say a failed whole-set rule is a warning to the operator and leaves the module healthy (hq issue 504)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
2026-10-12 13:27:06 +02:00
jochen df94be0e60 messenger: mark a walk's held messages told only once its message was sent, hand on only what a first hold would take, compare a walk's start and end on the controller's clock (issue 538)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A message lost with a restart before it was sent left what it carried marked told,
so it reached the operator only as its reminder hours later. A later walk took over
silenced, already said or asked messages that ADR 0308 section 3 never holds. A start
capped at the router's now was compared with the controller's moments.
2026-10-12 13:26:41 +02:00
jochen 8f5e8bcb6e mesh-knowledge: tell a failed whole-set rule to the operator as a warning naming the records, and leave the module healthy (hq issue 504)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The operator decided on 2026-10-12 that a rule of the record set that fails
is a fault of the records, which their authors repair, not the module
failing: as issue 508 decided for a proposal never reviewed, health says
only whether the module works, and a finding must not hold the module's
own rollouts. The findings go through the notice the knowledge base
already sends (operator-channel.notify, a warning), once per finding,
kept as told across restarts, and cleared when the rule holds again.
2026-10-12 13:25:09 +02:00
jochen ca3c3e6dfd mesh-knowledge tests: set the local zone once before any goroutine, not under a running bus server (race found by the repository check on #237)
inZone wrote time.Local while the bus test's nats server read it from a
goroutine that outlives its test; the race detector the repository check
now runs failed TestAReviewOwedOutlivesARestartAndItsLastTryIsSaid.
2026-10-12 13:25:09 +02:00
jochen 2051d8b80b mesh-knowledge: ask changes in the review for a decision nothing would cite, so its acceptance never turns the module unhealthy (hq issue 504)
The whole-set rule cited ran only after acceptance: the review approved a
decision with no home and no record linking it, the rule then failed, and
the module's health turned red for a fault its author could only repair
after the fact. hq held the rule at the record's own merge. The review now
judges it ahead, before the judgement: such a decision stays proposed with
a changes finding naming the rule. It is not refused at propose, because
no record can link a number only reserved.
2026-10-12 13:25:09 +02:00
mesh-admin 3e84581927 Merge pull request 'messenger: hold what a delivery walk touches, tell its end once, and say nothing again for a restart (hq ADR 0308, issues 532, 531)' (#230) from adr-0308-hold-during-walks into main 2026-10-12 11:21:16 +00:00
mesh-admin 4d9cb0827f Merge pull request 'gitea forge bundle: the parent the forge used, protection rules as the forge matches them, size and time bounds, one HTTP client (issue 539)' (#236) from fix/539-forge-commit-hardening into main 2026-10-12 10:58:07 +00:00
jochen ac90b35b49 guard.ts: cover every branch for a rule Go's regexp reads otherwise, and hold both matchers to one table (issue 539)
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
Review 30 found guard.ts missing branches the forge protects: Go reads \d, \w,
\pL, a class opening with ] or ^, and [[:digit:]] otherwise than JavaScript.
A rule using any of them now covers every branch, so the guard refuses more
than the forge, never less. test/rule-cases.json holds the forge's answers,
read by the Go and TypeScript tests alike.
2026-10-12 12:51:22 +02:00
jochen f2d765320a gitea forge bundle: name the parent the forge used, match protection rules as the forge does, bound a commit's size and a search's time (issue 539)
The forge's contents and diffpatch endpoints take a branch, never a commit to
build on, so the parent is read from the commit the forge made, and a branch
that moved under the commit is said. Blobs are read at the commit read first,
not at the branch name. Protection rules match as Gitea 1.27 matches them
([...], {a,b}, escapes, a plain name in any case), in the Go bundle and in
guard.ts alike. One commit carries at most 4 MiB per file or patch and 16 MiB
together; code search stops at its deadline and says so. The HTTP client the
two Go bundles copied is one package, internal/forgehttp.
2026-10-12 12:51:22 +02:00
mesh-admin c5abdf35ed Merge pull request 'A module whose tests could not run fails the repository check (issue 385)' (#231) from fix/385-an-untestable-module-fails-the-check into main 2026-10-12 10:48:11 +00:00
mesh-admin 9d6bd2d475 Merge pull request 'gitea: a recheck's pending goes through the one writer of mesh/merge-gate (issue 456)' (#234) from fix/456-a-recheck-writes-through-the-one-writer into main 2026-10-12 10:32:00 +00:00
mesh-admin d06733c490 Merge pull request 'mesh-knowledge: the bus test passes on any day (issue 540)' (#235) from fix/540-knowledge-test-clock into main 2026-10-12 10:31:18 +00:00
jochen 7628b0e5b9 mesh-knowledge: give the bus test's bundle a fixed clock, so it passes on any day (issue 540)
mesh/delivery delivered
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The bundle dated a reservation from the real day while the proposal kept the
fixture's 2026-10-11, so the date rule refused it on every later day.
2026-10-12 12:23:38 +02:00
jschoubben 5053e96d4e Route a recheck's pending through the gitea module's one writer of mesh/merge-gate
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
The recheck's pending was written from the tools' bundle, its own process, past the
writer's per-head chain: the first run's verdict mark stayed set, so a rechecked head
never said where its new run builds, and an old run's verdict in flight or redelivered
could land over the pending. The status tool is now served from the watcher's bundle
and writes the merge check's pending through the writer, and a verdict whose ask
(checked.id) predates the check now pending is dropped, weighed after a restart against
the forge's own history of the gate (novox/hq issue 456).
2026-10-12 12:20:23 +02:00
mesh-admin b707f27911 Merge pull request 'gitea: an agent's commit on a branch of its own, and the reads around it (issue 375)' (#233) from fix/375-a-gitea-commit-tool into main 2026-10-12 10:14:05 +00:00
jochen a92c08fd5b messenger: keep what a walk held until it is said, and open no walk whose end came first (review of #230)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
- No channel ready at a walk's end or bound: what it held is left unsent and tried
  again every minute, as any other message, instead of lost with the walk.
- At a walk's end or bound, a message another open walk touches is held on by
  that walk, even one started after it was held (hq ADR 0308 §2).
- A walk read back past its forget is released at its bound first, then forgotten.
- A walk's end is kept a day: a start heard after it (same started_at, or not
  later than the end) opens nothing, also after the router restarts; a later
  start under the same id still opens.
- A walk's plain end, when its words cannot be said, leaves what it held to be
  said on its own instead of marking it told.
- The restart pass no longer skips a held message: one with an open ask is never
  held (ADR 0308 §3), so its answers are made to work again.
2026-10-12 12:07:38 +02:00
jochen 9eb2ea8d69 gitea: let an agent put a commit on a branch of its own, and read history, compares, reviews, directories and code (issue 375)
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
An agent could not get a finished change onto the forge without someone
pushing by hand, so the branch / pull request / approval rule could not be
followed. A Go bundle serves gitea_commit, held to gitea_api's rule (never the
default branch, never a protected one), and the summarised reads agents
otherwise made through gitea_api as raw JSON.
2026-10-12 12:06:23 +02:00
jochen de8a866189 Fail the check when a module's tests could not run, and take a pinned SDK from its clone beside (issue 385)
mesh/delivery delivered
mesh/delivery-group group fix/385-an-untestable-module-fails-the-check delivered: every member is delivered
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh, merge-check-typescript.sh): main's version judged it; the change's judges the pull r…
A module whose dependencies could not be fetched was said NOT TESTED and the
repository check passed, so a merge on it would have landed untested code.
The build seat clones the SDK beside every check already: a forge requirement
the module cache lacks is now taken from that clone through a go.mod of its
own, and a module that still cannot be built fails the check by name and why.
A TypeScript module whose dependencies the toolchain lacks fails the same way.
2026-10-12 12:03:50 +02:00
jschoubben 991ffcfda0 messenger: run a walk's bound from its started_at, as the controller's walk events say it (hq ADR 0308 §5)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 11:56:30 +02:00
jschoubben 2927cb2ceb messenger: hold what a delivery walk touches and tell its end once, and say nothing again for a restart (hq ADR 0308, issues 532, 531)
A delivery's own restarts reached the operator as notifications while it ran, and every restart of the
tool runner popped the four silenced messages up again on the desk, twice. The router now reads the
controller's walk start and end on plan-moved, holds what an open walk touches until its end or bound,
tells the end as one message, and makes the restart pass once per restart from its own state, leaving
silenced, withheld, answered and held messages as they are.
2026-10-12 11:55:35 +02:00
mesh-admin da296cacc6 Merge pull request 'mesh-delivery: give a waiting walk its word or say what holds it (issues 529, 362)' (#229) from fix/529-a-waiting-walk-is-answered-or-named into main 2026-10-12 09:55:20 +00:00
jschoubben 1de89cd702 Name a walk of nothing-to-move merges instead of letting it go (issue 529)
mesh/delivery delivered
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named delivered: every member is delivered
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The review showed the give-the-word rule skipped a failed or missing check
and group order, and let go moves nothing judged; to-be 47 has no such path.
Such a walk is now said, by stalled with walks, as held by nothing, why.
2026-10-12 11:33:51 +02:00
jschoubben 2bee70a478 Give a waiting walk its word or say what holds it (issues 529, 362)
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named checking: 0 of 2 member(s) ready
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
A merge whose plan sends nothing is delivered at once, yet the controller
can still open a walk for it; no live delivery then held the walk and it
waited for a person (the nzbgeek case). Give such a walk its word.

Name each held delivery in the status, and let stalled say, for every
waiting walk, what holds it, so the controller's plans line points at the
person's decision instead of a hand start.
2026-10-12 11:24:30 +02:00
mesh-admin 6546c05f6e Merge pull request 'mesh-issues: the issues are kept in the log changes, and the bucket is their fold (hq ADR 0300, issues 501, 502)' (#228) from adr-0300-issues-on-a-log into main 2026-10-12 08:46:10 +00:00
jochen d413da203c mesh-issues: resume an interrupted carry by revision and fault one issue instead of the whole start, so no kept value is lost and the tracker always starts (review of PR 228)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
An open's lost answer is judged by the issue's first entry; an entry that does not
fold faults its issue on start; a value written beside the log is never folded over;
one issue is kept in at most 252 KiB so its whole operation fits a log entry.
2026-10-12 10:35:57 +02:00
jochen d33f3e3f71 mesh-issues: pin mesh-sdk at main 9747f3e, where the log and the history are both merged
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 10:22:19 +02:00
jochen dcc7510cd8 mesh-issues: carry each value the issues bucket kept, read through the runtime's history, so the log begins with all the bucket still knew (hq ADR 0300 §6, ADR 0307)
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 10:20:51 +02:00
jochen 39da2cd793 mesh-issues: keep the issues bucket at 8 values until the carry can read them, so the bus drops nothing before it is carried (hq ADR 0300 §6)
mesh/merge-gate fail: builds mesh-issues → novox; no bus step; a manifest the change touches fails the module check: modules/mesh-issues/module.json: this …
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 02:43:58 +02:00
jochen 983d434623 mesh-issues: keep every issue in the log changes and the bucket as its fold, so the record outlives the bucket's cap and a week of events (hq ADR 0300, issues 501, 502)
mesh/merge-gate fail: builds mesh-issues → novox; no bus step; a manifest the change touches fails the module check: modules/mesh-issues/module.json: this …
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery superseded: a newer head of the same pull request
Every write appends one operation at the issue's last sequence before the
bucket is written; a refused append writes nothing. The start carries what
the bucket holds, refolds what is behind and indexes every issue in memory,
so list answers from memory with a ranked search and a before cursor.
2026-10-12 02:41:54 +02:00
mesh-admin 8a92a47bf7 Merge pull request 'Hold a leaf delivery to three minutes in times (hq ADR 0304 decision 4, issue 525)' (#227) from feat/525-leaf-budget-three-minutes into main 2026-10-12 00:02:25 +00:00
jschoubben 2711e812c8 Hold a leaf delivery to three minutes in times (hq ADR 0304 decision 4, issue 525)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
mesh/delivery-group group feat/525-leaf-budget-three-minutes delivered: every member is delivered
ADR 0304 lowered a leaf module's delivery budget from five minutes to
three; times says each class against the controller's same budget.
2026-10-12 01:45:14 +02:00
mesh-admin a0fb555f79 Merge pull request 'Read the tunnel through sudo where the tool runner is not root (hq issue 522)' (#226) from fix/522-tunnel-escalates into main 2026-10-11 22:07:54 +00:00
jschoubben b1280f81a2 Read the tunnel through sudo where the tool runner is not root (hq issue 522)
mesh/merge-gate pass: builds netcheck → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
On novox the tool runner runs as the operator's account, and wg show
answered 'Operation not permitted'. Root is the module's concern: the
command runs as given when the process is root, else through sudo -n,
which the sudo module allows the runtime's account, as nftables does.
2026-10-12 00:05:04 +02:00
mesh-admin d2dff99bcf Merge pull request 'Show the tunnel's peers as each machine sees them, and say on the bus when they move (hq issue 522)' (#225) from feat/522-tunnel-peers into main 2026-10-11 21:55:14 +00:00
jschoubben c31e29f9a6 Keep watching the tunnel after a failed look, and remember peers' names
mesh/merge-gate pass: builds netcheck → ace, g14; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
2026-10-11 23:51:40 +02:00
jschoubben 4fc769ed29 Show the tunnel's peers as each machine sees them, and say on the bus when they move (hq issue 522)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds netcheck → ace, g14; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
ace's tunnel leg lost most packets while its own check called the tunnel
healthy, and nothing on the mesh could say which address and port the
hub saw for each peer, which is what tells a home router's port mix-up
from anything else. netcheck_tunnel answers each peer's endpoint, last
handshake and traffic, never a key, and netcheck now says when a
peer's endpoint changes or its handshake goes stale or fresh again.
2026-10-11 23:50:17 +02:00
mesh-admin 11be8a548a Merge pull request 'mesh-delivery: a stop ends only its own change on a shared walk (hq issue 459, ADR 0299)' (#221) from feat/459-a-stop-re-walks-the-others into main 2026-10-11 19:49:16 +00:00
mesh-admin f7fdb5e87b Merge pull request 'restic: nothing declared has no overdue night; a count's age is judged by awake time (issue 489)' (#224) from fix/489-nothing-declared-is-healthy into main 2026-10-11 19:31:05 +00:00
jschoubben 7f05fad58a restic: judge no night overdue where nothing is declared, and a count's age by the time the machine was awake
mesh/merge-gate pass: builds restic → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A machine that backs up nothing read as missing its night after 26 hours up, and a laptop's count, last
written before it slept, read as stale on waking until the next look (novox/hq issue 489).
2026-10-11 21:10:11 +02:00
mesh-admin fca2f92216 Merge pull request 'mesh-issues: reserve's description says it asks the knowledge base (issue 493)' (#223) from fix/493-reserve-says-what-it-reads into main 2026-10-11 19:07:27 +00:00
jschoubben c0710e2db7 Say that reserve asks the knowledge base for the highest number, not hq (hq issue 493)
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
Since #214 reserve reads knowledge_highest; its description still named hq's folders.
2026-10-11 20:53:21 +02:00
mesh-admin 3fa076829f Merge pull request 'mesh-knowledge, claude-code: a long record is reviewed within a bound sized to it, and a proposal waiting on its author leaves the module healthy (hq issue 508)' (#222) from fix/508-a-long-record-is-reviewed into main 2026-10-11 18:41:53 +00:00
jschoubben fcab25604e Review a long record within a bound sized to it, and let a proposal waiting on its author leave the knowledge base healthy (hq issue 508)
mesh/merge-gate pass: builds claude-code, mesh-knowledge → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (…
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A judgement over one bus call's 30 seconds is collected by its ticket from
claude-code, so an eleven-clause record can be reviewed; a proposal out of
review tries is told to the operator, not raised as the module's fault, so
it no longer holds the module's own rollouts.
2026-10-11 20:32:33 +02:00
jschoubben 3d7f9a3e66 mesh-delivery: say a shared-module leave-out truly, hold back only what was stopped by an act, and say a held walk where it is read (review of #221)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
mesh/delivery-group group feat/459-a-stop-re-walks-the-others delivered: every member is delivered
A merge left out for sharing a module read as containing the stopped merge; the
safety net also held back merges stranded by their walk, which nobody stopped;
a held walk said why only in the log; and a stop of a merge in a batch did not
ask the batch.
2026-10-11 20:27:55 +02:00
jschoubben 8d7f60c9e2 mesh-delivery: never ship a stopped merge walked again, and stop only once the walk is (review of #221, hq issue 459)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery-group group feat/459-a-stop-re-walks-the-others checking: 2 of 3 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
A merge walked again into a batch and then stopped by its group stayed in that
batch, which was later let go. Any walk or batch that names a stopped
delivery's merge is now asked to leave it out and is never let go; a refused
leave-out stops the whole walk. A person's stop marks the delivery stopped only
after its walk is, and the answer says what stays on the machines and what
will still carry the change.
2026-10-11 20:16:37 +02:00
jschoubben aeab5b80b4 mesh-delivery: a stop ends only its own change on a shared walk (hq issue 459, ADR 0299)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery-group group feat/459-a-stop-re-walks-the-others checking: 2 of 3 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
A group stop ended the stopped member's walk, and with it every other delivery
the batch put on that walk. It now asks delivery-stop to leave the stopped
merges out, so the others are walked again; a person's stop on a shared walk
first says what ending the walk would take down and acts only on a way.
2026-10-11 19:59:43 +02:00
mesh-admin 977e7a6fd2 Merge pull request 'mesh-knowledge: a review owed says when it is due, survives a restart, and its last try is told to the operator (hq issue 497)' (#219) from fix/497-a-review-owed-outlives-a-restart into main 2026-10-11 17:49:34 +00:00
jschoubben c394ecd9fd mesh-knowledge: give a never-reviewed notice its resolved line when raised, and test a refused clear and local time in a set zone (review of #219)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
The operator's channel edits a cleared message with the resolved line it was raised with, so one sent only with
the clear was never shown.
2026-10-11 19:41:45 +02:00
jschoubben d45761678a mesh-knowledge: clear a never-reviewed notice once the record moves on, key it by version and run of tries, say times locally (review of #219)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A notice left open kept telling the operator a record was proposed and not in force after it was proposed again
or accepted, and a later version's notice merged into it.
2026-10-11 19:36:14 +02:00
jschoubben 9cfb5f9edb mesh-knowledge: keep when an owed review is due, say it in the history, and tell the operator once after its last try (hq issue 497)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A failed review's try count was already kept with the version, but not when it was asked again, so a 10-minute
pause read as a review lost to a restart; and after the last try only the history and the health said so.
2026-10-11 19:24:07 +02:00
113 changed files with 13245 additions and 719 deletions
+3
View File
@@ -0,0 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/deps
go 1.22
+255
View File
@@ -0,0 +1,255 @@
// Command deps makes a module's dependencies available to the catalogue's check, or says why it cannot
// (novox/hq issue 385).
//
// A module's tests run only when its dependencies can be fetched. The build seat clones what a check
// needs with the forge credential and forgets that credential before the check runs, so a module pinning
// an SDK commit the seat's module cache does not hold cannot fetch it from the forge. The SDK is cloned
// beside the check all the same (MESH_CHECK_BESIDE, mesh-controller's besideRefs): every requirement of the
// mesh's own forge is taken from the clone of its repository there, at the commit or the tag its version
// names, through a go.mod of its own (-modfile) — the module's own files are not changed, and no forge
// credential is added.
//
// Usage: deps -beside <dir> -work <dir> <module dir>. It prints the go.mod to build the module with, or
// nothing when the module fetches what it needs; when it cannot, it prints why and exits 1, and the check
// fails: a module whose tests could not run is a failure, never a note.
package main
import (
"archive/tar"
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
)
// forge is the prefix of the Go modules of the mesh's own forge: <forge><repository>[/<directory>].
const forge = "git.novox.be/novox/"
func main() {
beside := flag.String("beside", "", "where the repositories beside the check are cloned (MESH_CHECK_BESIDE)")
work := flag.String("work", os.TempDir(), "where the go.mod and the sources taken from beside are written")
flag.Parse()
if flag.NArg() != 1 {
fmt.Println("usage: deps -beside <dir> -work <dir> <module dir>")
os.Exit(2)
}
modfile, err := resolve(flag.Arg(0), *beside, *work)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
if modfile != "" {
fmt.Println(modfile)
}
}
// resolve is the go.mod the module at dir builds with: "" when its own fetches everything, else one
// beside it in work that takes every requirement of the mesh's forge from its clone beside the check.
func resolve(dir, beside, work string) (string, error) {
out, err := goIn(dir, "mod", "download")
if err == nil {
return "", nil
}
fetching := lastLine(out)
required, err := forgeRequirements(dir)
if err != nil {
return "", err
}
if len(required) == 0 {
return "", fmt.Errorf("its dependencies cannot be fetched: %s", fetching)
}
if beside == "" {
return "", fmt.Errorf("its dependencies cannot be fetched (%s), and nothing was cloned beside the check to "+
"take %s from", fetching, repositories(required))
}
into, err := os.MkdirTemp(work, filepath.Base(dir)+"-")
if err != nil {
return "", err
}
modfile := filepath.Join(into, "go.mod")
if err := copyFile(filepath.Join(dir, "go.mod"), modfile); err != nil {
return "", err
}
if err := copyFile(filepath.Join(dir, "go.sum"), filepath.Join(into, "go.sum")); err != nil &&
!errors.Is(err, os.ErrNotExist) {
return "", err
}
for _, r := range required {
source, err := export(beside, r, into)
if err != nil {
return "", fmt.Errorf("its dependencies cannot be fetched (%s), and %s@%s cannot be taken from the "+
"clone beside the check: %w", fetching, r.Path, r.Version, err)
}
if out, err := goIn(dir, "mod", "edit", "-replace", r.Path+"@"+r.Version+"="+source, modfile); err != nil {
return "", fmt.Errorf("%s@%s cannot be taken from the clone beside the check: %s", r.Path, r.Version,
lastLine(out))
}
}
if out, err := goIn(dir, "mod", "download", "-modfile="+modfile); err != nil {
return "", fmt.Errorf("its dependencies cannot be fetched even with %s taken from the clone beside the "+
"check: %s", repositories(required), lastLine(out))
}
return modfile, nil
}
// requirement is one module the module requires.
type requirement struct {
Path string
Version string
}
// forgeRequirements is every requirement of the module at dir that comes from the mesh's own forge.
func forgeRequirements(dir string) ([]requirement, error) {
out, err := goIn(dir, "mod", "edit", "-json")
if err != nil {
return nil, fmt.Errorf("its go.mod cannot be read: %s", lastLine(out))
}
var mod struct{ Require []requirement }
if err := json.Unmarshal(out, &mod); err != nil {
return nil, fmt.Errorf("its go.mod cannot be read: %w", err)
}
var on []requirement
for _, r := range mod.Require {
if strings.HasPrefix(r.Path, forge) {
on = append(on, r)
}
}
return on, nil
}
// repositoryOf splits a module path of the forge into its repository and the directory in it.
func repositoryOf(path string) (string, string) {
repo, sub, _ := strings.Cut(strings.TrimPrefix(path, forge), "/")
return repo, sub
}
func repositories(rs []requirement) string {
var names []string
for _, r := range rs {
repo, _ := repositoryOf(r.Path)
names = append(names, repo)
}
return strings.Join(names, ", ")
}
// pseudo is a pseudo-version's commit: v0.1.14-0.20261012082113-9747f3e1b1ac → 9747f3e1b1ac.
var pseudo = regexp.MustCompile(`[-.]\d{14}-([0-9a-f]{12})$`)
// refOf is the ref a version names in its repository: a pseudo-version's commit, else the tag a module in
// a directory is released under (<directory>/<version>).
func refOf(sub, version string) string {
version = strings.TrimSuffix(version, "+incompatible")
if m := pseudo.FindStringSubmatch(version); m != nil {
return m[1]
}
if sub != "" {
return sub + "/" + version
}
return version
}
// export writes the module r names, at its version, from its repository's clone beside the check into
// into, and answers its directory.
func export(beside string, r requirement, into string) (string, error) {
repo, sub := repositoryOf(r.Path)
clone := filepath.Join(beside, repo)
if _, err := os.Stat(filepath.Join(clone, ".git")); err != nil {
return "", fmt.Errorf("no clone of %s is beside the check", repo)
}
ref := refOf(sub, r.Version)
commit, err := exec.Command("git", "-C", clone, "rev-parse", "--verify", "--quiet", ref+"^{commit}").Output()
if err != nil {
return "", fmt.Errorf("the clone of %s holds no %s (a commit or tag not pushed to the forge?)", repo, ref)
}
at := strings.TrimSpace(string(commit))
args := []string{"-C", clone, "archive", "--format=tar", at}
if sub != "" {
args = append(args, "--", sub)
}
var stderr bytes.Buffer
archive := exec.Command("git", args...)
archive.Stderr = &stderr
raw, err := archive.Output()
if err != nil {
return "", fmt.Errorf("git archive of %s at %s: %v %s", repo, ref, err, strings.TrimSpace(stderr.String()))
}
root := filepath.Join(into, repo+"@"+at[:12])
if err := untar(raw, root); err != nil {
return "", err
}
source := filepath.Join(root, filepath.FromSlash(sub))
if _, err := os.Stat(filepath.Join(source, "go.mod")); err != nil {
return "", fmt.Errorf("%s at %s has no go.mod in %q", repo, ref, sub)
}
return source, nil
}
// untar writes a tar's directories and regular files under root.
func untar(raw []byte, root string) error {
r := tar.NewReader(bytes.NewReader(raw))
for {
h, err := r.Next()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
name := filepath.Clean(filepath.FromSlash(h.Name))
if name == "." || strings.HasPrefix(name, "..") || filepath.IsAbs(name) {
continue
}
path := filepath.Join(root, name)
switch h.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(path, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
body, err := io.ReadAll(r)
if err != nil {
return err
}
if err := os.WriteFile(path, body, 0o644); err != nil {
return err
}
}
}
}
func copyFile(from, to string) error {
raw, err := os.ReadFile(from)
if err != nil {
return err
}
return os.WriteFile(to, raw, 0o644)
}
// goIn runs the go command in dir.
func goIn(dir string, args ...string) ([]byte, error) {
cmd := exec.Command("go", args...)
cmd.Dir = dir
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.Output()
if err != nil {
return append(out, stderr.Bytes()...), err
}
return out, nil
}
// lastLine is the last line a command said, which is where go says what failed.
func lastLine(out []byte) string {
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
+177
View File
@@ -0,0 +1,177 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// offline is a Go that fetches nothing: what the build seat is once the forge credential is gone, for a pin
// its module cache does not hold (novox/hq issue 385).
func offline(t *testing.T) {
t.Helper()
t.Setenv("GOPROXY", "off")
t.Setenv("GOFLAGS", "")
t.Setenv("GOWORK", "off")
// Not GOPRIVATE: a private module is fetched directly by git, around GOPROXY, with whatever credential
// the machine running the test holds.
t.Setenv("GOPRIVATE", "")
t.Setenv("GONOPROXY", "")
t.Setenv("GOSUMDB", "off")
t.Setenv("GOMODCACHE", t.TempDir())
}
func git(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t",
"GIT_COMMITTER_EMAIL=t@t", "GIT_AUTHOR_DATE=2026-10-12T08:21:13Z", "GIT_COMMITTER_DATE=2026-10-12T08:21:13Z")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
}
return strings.TrimSpace(string(out))
}
func write(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// sdkBeside is the SDK's repository cloned beside a check, its Go module under go/, one commit tagged
// go/v0.1.7. It answers the beside root and the commit.
func sdkBeside(t *testing.T) (string, string) {
t.Helper()
beside := t.TempDir()
repo := filepath.Join(beside, "mesh-sdk")
write(t, filepath.Join(repo, "go", "go.mod"), "module git.novox.be/novox/mesh-sdk/go\n\ngo 1.22\n")
write(t, filepath.Join(repo, "go", "sdk", "sdk.go"), "package sdk\n\nfunc Hello() string { return \"hello\" }\n")
write(t, filepath.Join(repo, "README.md"), "the SDK\n")
git(t, repo, "init", "--quiet")
git(t, repo, "add", ".")
git(t, repo, "commit", "--quiet", "-m", "sdk")
git(t, repo, "tag", "go/v0.1.7")
return beside, git(t, repo, "rev-parse", "HEAD")
}
// moduleRequiring is a catalogue module whose code and test use the SDK at the version given.
func moduleRequiring(t *testing.T, version string) string {
t.Helper()
dir := filepath.Join(t.TempDir(), "messenger")
write(t, filepath.Join(dir, "go.mod"), "module git.novox.be/novox/mesh-catalog/modules/messenger\n\ngo 1.22\n\n"+
"require git.novox.be/novox/mesh-sdk/go "+version+"\n")
write(t, filepath.Join(dir, "main.go"), "package main\n\nimport \"git.novox.be/novox/mesh-sdk/go/sdk\"\n\n"+
"func main() { println(sdk.Hello()) }\n")
return dir
}
func builds(t *testing.T, dir, modfile string) {
t.Helper()
args := []string{"vet"}
if modfile != "" {
args = append(args, "-modfile="+modfile)
}
cmd := exec.Command("go", append(args, "./...")...)
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("the module does not build with %q: %v\n%s", modfile, err, out)
}
}
// A module that needs nothing it cannot fetch is left as it is: no other go.mod.
func TestAModuleThatFetchesItsDependenciesNeedsNothingBeside(t *testing.T) {
offline(t)
dir := filepath.Join(t.TempDir(), "plain")
write(t, filepath.Join(dir, "go.mod"), "module example.com/plain\n\ngo 1.22\n")
write(t, filepath.Join(dir, "main.go"), "package main\n\nfunc main() {}\n")
modfile, err := resolve(dir, "", t.TempDir())
if err != nil || modfile != "" {
t.Fatalf("resolve = %q, %v; want nothing", modfile, err)
}
}
// The case of issue 385: the module pins an SDK commit the build seat cannot fetch. It is taken from the
// SDK's clone beside the check, and the module builds against it.
func TestAPinnedSDKCommitIsTakenFromTheCloneBeside(t *testing.T) {
offline(t)
beside, commit := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261012082113-"+commit[:12])
modfile, err := resolve(dir, beside, t.TempDir())
if err != nil {
t.Fatalf("resolve: %v", err)
}
if modfile == "" {
t.Fatal("resolve said the module fetches its dependencies, with nothing to fetch them from")
}
builds(t, dir, modfile)
if raw, _ := os.ReadFile(filepath.Join(dir, "go.mod")); strings.Contains(string(raw), "replace") {
t.Fatalf("the module's own go.mod was changed:\n%s", raw)
}
}
// A released SDK is found by the tag it is released under: go/<version>.
func TestATaggedSDKIsTakenFromTheCloneBeside(t *testing.T) {
offline(t)
beside, _ := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.7")
modfile, err := resolve(dir, beside, t.TempDir())
if err != nil {
t.Fatalf("resolve: %v", err)
}
builds(t, dir, modfile)
}
// A commit the clone beside does not hold (a branch never pushed) cannot be tested, and says which.
func TestACommitTheCloneBesideDoesNotHoldIsSaid(t *testing.T) {
offline(t)
beside, _ := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261010132341-4f3efc98e0c0")
_, err := resolve(dir, beside, t.TempDir())
if err == nil {
t.Fatal("resolve found a commit nothing holds")
}
for _, want := range []string{"git.novox.be/novox/mesh-sdk/go@v0.1.14-0.20261010132341-4f3efc98e0c0", "4f3efc98e0c0"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the reason %q does not name %s", err, want)
}
}
}
// With nothing cloned beside (a check by hand without the forge), the reason says that.
func TestNoCloneBesideIsSaid(t *testing.T) {
offline(t)
_, commit := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261012082113-"+commit[:12])
for _, beside := range []string{"", t.TempDir()} {
_, err := resolve(dir, beside, t.TempDir())
if err == nil {
t.Fatalf("resolve with %q beside found what is not there", beside)
}
if !strings.Contains(err.Error(), "mesh-sdk") {
t.Errorf("the reason %q does not name the repository it needed", err)
}
}
}
func TestTheRefOfAVersion(t *testing.T) {
cases := []struct{ sub, version, want string }{
{"go", "v0.1.14-0.20261012082113-9747f3e1b1ac", "9747f3e1b1ac"},
{"go", "v0.0.0-20261012082113-9747f3e1b1ac", "9747f3e1b1ac"},
{"go", "v0.1.7", "go/v0.1.7"},
{"", "v1.2.0", "v1.2.0"},
{"", "v2.0.0+incompatible", "v2.0.0"},
}
for _, c := range cases {
if got := refOf(c.sub, c.version); got != c.want {
t.Errorf("refOf(%q, %q) = %q, want %q", c.sub, c.version, got, c.want)
}
}
}
+7 -3
View File
@@ -9,14 +9,17 @@
#
# 1. its dependencies are the toolchain's own (/app/node_modules: the mesh's SDK, Node's types and what the
# runtime carries), copied beside it, never fetched: the check holds no credential for the package
# registry. A module that depends on a package the toolchain does not hold is said NOT CHECKED, by name;
# registry. A module that depends on a package the toolchain does not hold **fails the check**, by name:
# it could not be type-checked or tested, and what could not run is a failure, never a note (novox/hq
# issue 385);
# 2. every source that is not a test is type-checked with the toolchain's compiler and the flags a build
# compiles with — the same check the build makes, before the merge rather than in the walk;
# 3. its tests (test/*.test.ts) run under Node's own runner with types stripped, as its own `npm test` does
# after its build: the sources are compiled to dist/ and beside themselves first, so a test that imports
# the compiled module, or a source that imports its sibling by its .js name, finds it.
#
# Said, never passed silently: a module with no tests is said NOT TESTED on every run that touches it.
# Said, never passed silently: a module with no tests is said NOT TESTED on every run that touches it —
# nothing of it could fail. A module whose tests exist and could not run fails (novox/hq issue 385).
#
# By hand: MESH_CHECK_MODULES=gitea sh merge-check-typescript.sh, in the toolchain image (it reads
# /app/node_modules), or with TOOLCHAIN_MODULES pointing at a node_modules that holds the same.
@@ -54,7 +57,8 @@ for m in $touched; do
require("fs").existsSync(process.argv[2] + "/" + n));
console.log(names.filter((n) => n !== "typescript" && !held(n)).join(" "));' "$copy/package.json" "$deps")
if [ -n "$missing" ]; then
echo "NOT CHECKED: $d depends on $missing, which the toolchain does not hold and the check cannot fetch"
echo "FAILED: $d could not be checked — it depends on $missing, which the toolchain does not hold and the check cannot fetch (novox/hq issue 385)"
failed="$failed $m"
continue
fi
mkdir -p "$copy/node_modules"
+23 -6
View File
@@ -14,8 +14,11 @@
# it shares a rule with refused (MESH_GATE is the controller the mesh runs) — and the count of
# long-running resources without `health` held to the number in health-undeclared, which only goes down;
# 2. the Go tests of every module the change touches that has them, under the race detector when the
# toolchain has a C compiler — and a module whose dependencies cannot be fetched here is said as not
# tested, never passed silently. A module written in TypeScript is type-checked and tested by
# toolchain has a C compiler. A requirement of the mesh's own forge the module cache does not hold (a
# pinned SDK commit) is taken from its repository's clone beside the check (checks/deps); a module whose
# dependencies still cannot be had **fails the check**, by name and why — its tests could not run, and a
# test that could not run is a failure, never a note (novox/hq issue 385). A module written in
# TypeScript is type-checked and tested by
# merge-check-typescript.sh, declared above, in the TypeScript toolchain (novox/hq issues 302, 311);
# 3. the words (novox/hq ADR 0244): no word the glossary retired for the tools, as copied in
# retired-words, in any text a module's tools can show an agent — every string in its own code that is
@@ -77,16 +80,26 @@ if [ -n "${MESH_CHECK_MODULES:-}" ]; then
else
touched=$(printf '%s\n' "${MESH_CHECK_CHANGED:-}" | tr ',' '\n' | sed -n 's#^modules/\([^/]*\)/.*#\1#p' | sort -u)
fi
# A module's dependencies: fetched, or a forge requirement taken from its clone beside the check
# (MESH_CHECK_BESIDE); its tests run against them (checks/deps, novox/hq issue 385). The helper's own test
# runs first.
(cd checks/deps && go test -count=1 .)
deps=$(mktemp -d)
trap 'rm -rf "$deps"' EXIT
(cd checks/deps && go build -o "$deps/deps" .)
untestable=""
for m in $touched; do
[ -d "modules/$m" ] || continue
if [ ! -f "modules/$m/go.mod" ]; then
[ -f "modules/$m/package.json" ] && echo "modules/$m is TypeScript: merge-check-typescript.sh type-checks and tests it"
continue
fi
if ! (cd "modules/$m" && GOPRIVATE=git.novox.be go mod download >/dev/null 2>&1); then
echo "NOT TESTED: modules/$m — its dependencies cannot be fetched by the build seat"
if ! modfile=$(GOPRIVATE=git.novox.be "$deps/deps" -beside "${MESH_CHECK_BESIDE:-}" -work "$deps" "modules/$m"); then
echo "FAILED: modules/$m could not be tested — $modfile"
untestable="$untestable modules/$m"
continue
fi
[ -n "$modfile" ] && echo "modules/$m takes its forge dependencies from the clones beside the check"
echo "testing modules/$m"
unformatted=$(cd "modules/$m" && gofmt -l .)
if [ -n "$unformatted" ]; then
@@ -95,6 +108,10 @@ for m in $touched; do
fi
cgo=0
[ -n "$race" ] && cgo=1
(cd "modules/$m" && CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go vet ./... &&
CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test $race -count=1 ./...)
(cd "modules/$m" && CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go vet ${modfile:+"-modfile=$modfile"} ./... &&
CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test ${modfile:+"-modfile=$modfile"} $race -count=1 ./...)
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
+4 -1
View File
@@ -199,7 +199,10 @@ only a language model can, for another module — the issue tracker's duplicate
its own), in an empty directory, the question on its input: no built-in tools, no tool servers, plugins, hooks or
instruction files, no session kept, so it can only answer. It answers the structured answer, the model, the node
and the seconds, within 25 seconds — the process is ended past that, under the bus's 30 — and one at a time on a
node. Refused: an empty question or one over 64 KiB, a schema that is not an object, a model not `sonnet`, `haiku`
node. A caller may ask a longer bound with `within` (seconds, at most 120), for a long question (hq issue 508): a
judgement not done in 20 seconds is then answered `pending` with a `ticket`, runs on to its bound, and
`claude_code_judged` (`ticket`) collects it, waiting up to 20 seconds a call. A ticket is answered once; an answer
nobody collects is dropped ten minutes after its bound, and none survives a restart of the module. Refused: an empty question or one over 64 KiB, a schema that is not an object, a model not `sonnet`, `haiku`
or `opus`, and a node holding no licence. The question is never logged.
## Settings
+148 -7
View File
@@ -12,12 +12,15 @@ package main
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -27,6 +30,14 @@ const (
// JudgeBound is how long one judgement may take, waiting for another included: under the bus's 30 seconds
// a tool call is answered in, so the caller hears this module's refusal rather than a timeout.
JudgeBound = 25 * time.Second
// MostJudgeBound is the longest bound a caller may ask with within: a long record's review takes longer than
// one bus call allows (novox/hq issue 508), and is then collected by its ticket.
MostJudgeBound = 120 * time.Second
// JudgeWait is how long one call waits for a judgement asked with a bound past JudgeBound before it answers
// the ticket: well under the bus's 30 seconds, so the call is always answered.
JudgeWait = 20 * time.Second
// JudgedKept is how long an answer nobody collected is kept past its judgement's bound.
JudgedKept = 10 * time.Minute
// MostQuestion is the largest question taken.
MostQuestion = 64 << 10
// MostSchema is the largest answer schema taken.
@@ -43,14 +54,29 @@ type JudgeAsk struct {
Question string
Schema map[string]any
Model string
// Within is the bound the caller asks; JudgeBound when zero. Past JudgeBound the call answers a ticket if the
// judgement is not done within JudgeWait, and claude_code_judged collects it (novox/hq issue 508).
Within time.Duration
}
// Judged is one answer.
type Judged struct {
Answer json.RawMessage `json:"answer"`
Answer json.RawMessage `json:"answer,omitempty"`
Model string `json:"model"`
Node string `json:"node"`
Seconds float64 `json:"seconds"`
// Ticket names a judgement asked with a bound past JudgeBound; Pending says it is not done yet, and
// claude_code_judged with the ticket collects it.
Ticket string `json:"ticket,omitempty"`
Pending bool `json:"pending,omitempty"`
}
// running is one judgement asked with a bound past JudgeBound, until it is collected.
type running struct {
done chan struct{}
out *Judged
err error
until time.Time
}
// Judge runs judgements on this node, one at a time.
@@ -63,7 +89,13 @@ type Judge struct {
Now func() time.Time
// Bound is how long one judgement may take; JudgeBound unless a test says less.
Bound time.Duration
turn chan struct{}
// Wait is how long a call waits for a long judgement before it answers its ticket; JudgeWait unless a test
// says less.
Wait time.Duration
turn chan struct{}
mu sync.Mutex
tickets map[string]*running
}
// NewJudge is the judge of this node.
@@ -118,11 +150,25 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
"bindings say which node holds which", j.P.Node)
}
start := j.Now()
bound := j.Bound
if bound <= 0 {
bound = JudgeBound
}
if a.Within > MostJudgeBound {
return nil, fmt.Errorf("within is %s; a judgement takes at most %s", a.Within, MostJudgeBound)
}
if a.Within > bound {
return j.start(string(schema), model, question, a.Within)
}
if a.Within > 0 {
bound = a.Within
}
return j.run(string(schema), model, question, bound)
}
// run is one judgement within its bound, waiting for its turn included.
func (j *Judge) run(schema, model, question string, bound time.Duration) (*Judged, error) {
start := j.Now()
ctx, cancel := context.WithTimeout(context.Background(), bound)
defer cancel()
select {
@@ -131,7 +177,7 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
case <-ctx.Done():
return nil, fmt.Errorf("another judgement held %s's agent for all of %s; ask again", j.P.Node, bound)
}
out, err := j.Exec(ctx, j.Argv(string(schema), model), []byte(question))
out, err := j.Exec(ctx, j.Argv(schema, model), []byte(question))
if ctx.Err() != nil {
return nil, fmt.Errorf("the agent on %s did not answer within %s; the judgement was ended", j.P.Node, bound)
}
@@ -145,6 +191,68 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
return &Judged{Answer: answer, Model: model, Node: j.P.Node, Seconds: j.Now().Sub(start).Round(100 * time.Millisecond).Seconds()}, nil
}
// start runs a judgement whose bound is past what one bus call allows, and waits for it as long as a call may: its
// answer when it is done by then, its ticket otherwise (novox/hq issue 508).
func (j *Judge) start(schema, model, question string, bound time.Duration) (*Judged, error) {
var b [12]byte
_, _ = rand.Read(b[:])
ticket := hex.EncodeToString(b[:])
r := &running{done: make(chan struct{}), until: j.Now().Add(bound + JudgedKept)}
j.mu.Lock()
j.forget()
if j.tickets == nil {
j.tickets = map[string]*running{}
}
j.tickets[ticket] = r
j.mu.Unlock()
go func() {
r.out, r.err = j.run(schema, model, question, bound)
close(r.done)
}()
return j.Collect(ticket)
}
// Collect waits as long as a call may for the judgement under a ticket: its answer, why it failed, or that it is
// still pending. A ticket is answered once; an answer nobody collects is kept JudgedKept past its bound.
func (j *Judge) Collect(ticket string) (*Judged, error) {
j.mu.Lock()
j.forget()
r := j.tickets[ticket]
j.mu.Unlock()
if r == nil {
return nil, fmt.Errorf("no judgement on %s under ticket %q: it was collected, kept past its time, or this module "+
"restarted since; ask the judgement again", j.P.Node, ticket)
}
wait := j.Wait
if wait <= 0 {
wait = JudgeWait
}
select {
case <-r.done:
case <-time.After(wait):
return &Judged{Node: j.P.Node, Ticket: ticket, Pending: true}, nil
}
j.mu.Lock()
delete(j.tickets, ticket)
j.mu.Unlock()
if r.err != nil {
return nil, r.err
}
out := *r.out
out.Ticket = ticket
return &out, nil
}
// forget drops the answers nobody collected in time. Called with mu held.
func (j *Judge) forget() {
now := j.Now()
for t, r := range j.tickets {
if now.After(r.until) {
delete(j.tickets, t)
}
}
}
// answerOf reads print mode's result: the structured answer, or why there is none.
func answerOf(out []byte) (json.RawMessage, error) {
var r struct {
@@ -213,17 +321,21 @@ func judgeTool(j *Judge) stdio.Tool {
"object); optionally the model (" + strings.Join(JudgeModels, ", ") + "; default " + JudgeModels[0] + "). The agent " +
"answers as the account agents run as here, under the licence this node is bound to, with no tools, " +
"no tool servers, no instructions and no session kept — it can only answer — within 25 seconds or " +
"not at all. One at a time on a node. Refused: an empty or larger than 64 KiB question, a schema that " +
"not at all, unless within asks longer (at most 120 seconds, for a long question): then, when it is not done in " +
"20 seconds, it answers pending with a ticket, and claude_code_judged collects the answer (novox/hq issue 508). " +
"One at a time on a node. Refused: an empty or larger than 64 KiB question, a schema that " +
"is not an object, an unknown model, a node holding no licence. The question is never logged.",
Input: map[string]any{
"question": str("the question, with everything the answer needs"),
"schema": map[string]any{"type": "object", "description": "the JSON schema the answer follows"},
"model": map[string]any{"type": "string", "enum": JudgeModels, "description": "the model (default " + JudgeModels[0] + ")"},
"within": map[string]any{"type": "number", "description": "the bound in seconds, at most 120 (default 25); " +
"past 25 a judgement not done in 20 seconds answers pending with a ticket for claude_code_judged"},
},
Run: func(a map[string]any) (any, error) {
for k := range a {
if k != "question" && k != "schema" && k != "model" {
return nil, fmt.Errorf("claude_code_judge takes question, schema and model; not %s", k)
if k != "question" && k != "schema" && k != "model" && k != "within" {
return nil, fmt.Errorf("claude_code_judge takes question, schema, model and within; not %s", k)
}
}
var ask JudgeAsk
@@ -241,6 +353,35 @@ func judgeTool(j *Judge) stdio.Tool {
return nil, errors.New("model is text")
}
}
if v, given := a["within"]; given && v != nil {
secs, ok := v.(float64)
if !ok || secs <= 0 {
return nil, errors.New("within is a number of seconds")
}
ask.Within = time.Duration(secs * float64(time.Second))
}
return j.Judge(ask)
}}
}
// judgedTool collects a judgement asked with a bound past one bus call's (novox/hq issue 508).
func judgedTool(j *Judge) stdio.Tool {
return stdio.Tool{Name: "claude_code_judged",
Description: "The answer of a judgement claude_code_judge answered pending with a ticket (novox/hq issue 508): " +
"waits up to 20 seconds, then answers the judgement's answer, why it failed, or pending again. A ticket is " +
"answered once; one nobody collects is dropped ten minutes after its bound, and none survives a restart of " +
"this module.",
Input: map[string]any{"ticket": str("the ticket claude_code_judge answered")},
Run: func(a map[string]any) (any, error) {
for k := range a {
if k != "ticket" {
return nil, fmt.Errorf("claude_code_judged takes ticket; not %s", k)
}
}
ticket, ok := a["ticket"].(string)
if !ok || ticket == "" {
return nil, errors.New("claude_code_judged needs ticket, as claude_code_judge answered it")
}
return j.Collect(ticket)
}}
}
@@ -166,3 +166,102 @@ func TestTheToolTakesOnlyItsArguments(t *testing.T) {
t.Fatalf("%s", raw)
}
}
// A long question may ask a longer bound than one bus call allows (novox/hq issue 508): the call answers a ticket
// when the judgement is not done within the call's wait, the judgement runs on to its own bound, and
// claude_code_judged collects its answer.
func TestALongJudgementIsCollectedByItsTicket(t *testing.T) {
var deadline time.Time
j := testJudge(true, true, func(ctx context.Context, _ []string, _ []byte) ([]byte, error) {
deadline, _ = ctx.Deadline()
select {
case <-time.After(400 * time.Millisecond):
return []byte(`{"structured_output":{"verdict":"none"}}`), nil
case <-ctx.Done():
return nil, ctx.Err()
}
})
j.Wait = 50 * time.Millisecond
start := time.Now()
out, err := j.Judge(JudgeAsk{Question: "a long record", Schema: verdictSchema, Within: 90 * time.Second})
if err != nil {
t.Fatal(err)
}
if !out.Pending || out.Ticket == "" || len(out.Answer) != 0 || out.Node != "control" {
t.Fatalf("not a ticket: %+v", out)
}
var got *Judged
for i := 0; i < 100; i++ {
got, err = j.Collect(out.Ticket)
if err != nil {
t.Fatal(err)
}
if !got.Pending {
break
}
}
if got.Pending || string(got.Answer) != `{"verdict":"none"}` || got.Ticket != out.Ticket {
t.Fatalf("%+v", got)
}
// Read once the judgement is collected: its end is what orders the agent's write before this read.
if left := deadline.Sub(start); left < 80*time.Second || left > 91*time.Second {
t.Fatalf("the judgement's bound is %s, not the 90s asked", left)
}
if _, err := j.Collect(out.Ticket); err == nil || !strings.Contains(err.Error(), "no judgement") {
t.Fatalf("a ticket collected twice: %v", err)
}
}
// A long judgement that fails says why when it is collected, and its bound is never past MostJudgeBound.
func TestALongJudgementIsBoundedAndSaysWhyItFailed(t *testing.T) {
j := testJudge(true, true, func(ctx context.Context, _ []string, _ []byte) ([]byte, error) {
<-ctx.Done()
return nil, ctx.Err()
})
if _, err := j.Judge(JudgeAsk{Question: "q", Schema: verdictSchema, Within: MostJudgeBound + time.Second}); err == nil ||
!strings.Contains(err.Error(), "at most") {
t.Fatalf("a bound past the most taken: %v", err)
}
j.Wait = 20 * time.Millisecond
j.Bound = 30 * time.Millisecond // the test's stand-in for JudgeBound: within is past it, so a ticket
out, err := j.Judge(JudgeAsk{Question: "q", Schema: verdictSchema, Within: 200 * time.Millisecond})
if err != nil || !out.Pending {
t.Fatalf("%+v %v", out, err)
}
for i := 0; i < 100; i++ {
got, err := j.Collect(out.Ticket)
if err != nil {
if !strings.Contains(err.Error(), "did not answer within") {
t.Fatalf("%v", err)
}
return
}
if !got.Pending {
t.Fatalf("answered: %+v", got)
}
}
t.Fatal("never ended")
}
// The tools take within and ticket, and nothing else.
func TestTheToolsTakeWithinAndTicket(t *testing.T) {
j := testJudge(true, true, func(context.Context, []string, []byte) ([]byte, error) {
return []byte(`{"structured_output":{"verdict":"none"}}`), nil
})
out, err := judgeTool(j).Run(map[string]any{"question": "q", "schema": verdictSchema, "within": float64(60)})
if err != nil {
t.Fatal(err)
}
if got := out.(*Judged); got.Pending || string(got.Answer) != `{"verdict":"none"}` {
t.Fatalf("a quick long judgement is answered in its call: %+v", got)
}
if _, err := judgeTool(j).Run(map[string]any{"question": "q", "schema": verdictSchema, "within": "long"}); err == nil {
t.Fatal("within as text was taken")
}
if _, err := judgedTool(j).Run(map[string]any{"ticket": "nope"}); err == nil || !strings.Contains(err.Error(), "no judgement") {
t.Fatalf("%v", err)
}
if _, err := judgedTool(j).Run(map[string]any{"ticket": "x", "by": "y"}); err == nil {
t.Fatal("an unknown argument was taken")
}
}
+2 -1
View File
@@ -355,7 +355,8 @@ func tools(p Paths, servers ServerState, view *ServerView, config ConfigState, c
Description: "Read-only: everything registered through this module that a Claude Code session runs or obeys — the tool servers, the hooks, the settings and the skills, subagents, commands, output styles and instructions — by key, with who registered it, when, and whether it is still rendered (novox/hq issue 340), beside what the operator set for this node at the controller's terminal (managed_settings, mcp_servers). `by` is the node whose tool runner served the call, never the caller.",
Run: func(map[string]any) (any, error) { return Registrations(p, servers, config) }})
out = append(out, proposalTools(p)...)
out = append(out, judgeTool(NewJudge(p)))
judge := NewJudge(p)
out = append(out, judgeTool(judge), judgedTool(judge))
return append(out, configTools(p, config, configView)...)
}
+2 -1
View File
@@ -67,7 +67,8 @@
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge"
"claude_code_judge",
"claude_code_judged"
],
"data": {
"own": [
+51 -1
View File
@@ -2,7 +2,7 @@
The forge, and the mesh's npm package registry: it claims the `git` and `npm-package-registry` seats.
Its tools, its events and the provisioner of registry accounts are the TypeScript bundle (`index.ts`,
`tools/`, `provisioner/`). This README covers the Go bundle beside it.
`tools/`, `provisioner/`). This README covers the two Go bundles beside it.
## Announcing pull requests to the merge check (novox/hq to-be 45 §9, issue 347)
@@ -28,6 +28,56 @@ Every new head of an open pull request is announced as `pull.updated`, once, and
- **A pull request or issue number that is not one is refused** before the forge is asked: a tool called with a
misnamed argument sent `pulls/NaN`.
## An agent's commit and the reads around it (novox/hq issue 375)
An agent that finished a change on a branch had no way to put it on the forge: the module had no commit or
push tool, `gitea_api` refused the forge's patch endpoint, and its only file write took one whole file
base64-encoded. The Go bundle `cmd/forge` serves the commit, and the reads an agent otherwise made through
`gitea_api` as raw JSON, summarised.
| tool | | what |
|---|---|---|
| `gitea_commit` | a | one commit on a branch of its own, created from `base` (default the default branch) when absent, carrying either `files` (each `{path, content \| content_base64 \| delete: true}`) or `patch` (a unified diff, as `git diff` writes it); answers the commit and its parent |
| `gitea_commits` | r | a ref's history, newest first, optionally only commits touching a path: sha, author, date, subject |
| `gitea_compare` | r | what head holds that base does not: its commits and the files they touch |
| `gitea_pull_reviews` | r | a pull request's reviews — state, the commit reviewed and whether it is the head now, stale, dismissed — with their line comments |
| `gitea_list_dir` | r | a directory's entries at a ref: name, type, a file's size |
| `gitea_search_code` | r | the lines of a ref's text files matching a literal string or a regular expression, optionally under a path |
**Where a commit may go** is the rule `gitea_api` already holds for a file written (novox/hq ADR 0266):
never the repository's default branch, never a branch a protection rule names. A rule covers a branch as the
forge itself decides it (Gitea 1.27, novox/hq issue 539): a plain name covers that branch in any case; a glob
(`*`, `**`, `?`, `[abc]`, `[!abc]`, `{a,b}`, `\` escaping) covers case-sensitively with `/` as separator; a glob
the forge cannot compile covers its own text. `cmd/forge/rules.go` and `guard.ts` hold this one rule. The rules are read before anything is sent; rules that cannot be read refuse the
commit. Nothing about merging changes: a merge still needs an independent review at the head and both
required checks green, through `gitea_merge_pull_request`.
**How the commit is made.** `files` become one call to the forge's multi-file contents endpoint: a path the
ref holds is updated, one it does not is created, and an update or delete carries the blob read just
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. The blobs are read at
the commit the branch (or, for a new branch, its base) was read at, never at its name. A `patch` goes to the
forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than the
file. Neither endpoint takes a commit to build on, only a branch (Gitea 1.27), so the parent answered is the
one the forge says it made the commit on; when the branch moved between the read and the commit, the answer
says so. One commit carries at most 200 files, 4 MiB per file and 16 MiB together, or a patch of 4 MiB. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
local commit's: the merge check judges the head on the forge. After the write the branch is read back, and a
branch not at the commit the forge named is a failure, never a success.
**Code search** has no endpoint on the forge, so the ref's archive (`/archive/{ref}.tar.gz`, at most 64 MiB)
is read once and searched in memory; a binary file is passed over, a file over 4 MiB is named under
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found. A
search that reaches the call's deadline stops there and answers what it found, with `stopped_at_deadline`.
It reaches the forge through the same HTTP client as the npm-registry bundle (`internal/forgehttp`), and one
tool call takes at most 50 seconds.
`go test ./cmd/forge` runs it against a fake forge: the files of one commit created, updated and deleted on
a new branch from its base; an existing branch committed on; the default branch, a protected branch and
unreadable rules refused before anything is sent; a patch sent to the forge; a branch read back at another
commit failing; a branch or base that moved under the commit said, with the parent the forge used; a file
changed under the commit refused; a commit over its size limits refused; protection globs matched as the
forge matches them; a code search stopped at its deadline; every malformed ask refused; and each read
summarised.
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
Every publish adds a version to the registry, and until this nothing removed one. The Go bundle
+13
View File
@@ -465,6 +465,19 @@ export class GiteaClient {
return out;
}
/** Every status of one context on a commit, newest first, as the forge keeps each one it was given: what tells
* when the check now pending on a head was opened, after a restart (novox/hq issue 456). */
async contextHistory(owner: string, repo: string, sha: string, context: string): Promise<HeadCheck[]> {
const raw = await this.request<any>(`${repoPath(owner, repo)}/statuses/${commitSegment(sha)}?sort=highestindex&limit=50`);
const out: HeadCheck[] = [];
for (const s of Array.isArray(raw) ? raw : []) {
if (s?.context !== context) continue;
out.push({ state: String(s.status ?? s.state ?? ""), description: String(s.description ?? ""),
updated_at: String(s.created_at ?? s.updated_at ?? "") });
}
return out;
}
/** A commit's statuses whole, as the forge combines them for a ref (a sha, or the start of one): the commit
* it resolved to and every status with its description, link, times and setter (novox/hq ADR 0239). */
async combinedStatus(owner: string, repo: string, ref: string): Promise<{ sha: string; state: string; statuses: unknown[] }> {
+406
View File
@@ -0,0 +1,406 @@
package main
import (
"context"
"encoding/base64"
"errors"
"fmt"
"net/http"
"net/url"
"strings"
"unicode/utf8"
)
// gitea_commit (novox/hq issue 375): one commit on a branch of its own, so an agent follows the branch / pull
// request / approval rule without anyone pushing by hand. By the rule gitea_api already enforces for a file
// written (novox/hq ADR 0266): never the repository's default branch, never a branch a protection rule names.
// Nothing about merging changes — that still needs a review at the head and both required checks green
// (gitea_merge_pull_request).
// FileChange is one file of a commit: its new content, as text or base64, or its deletion.
type FileChange struct {
Path string
Content *string
ContentBase64 *string
Delete bool
}
// CommitAsk is one gitea_commit call.
type CommitAsk struct {
Owner, Repo string
Branch string
Base string // the branch or commit a new branch starts from; the default branch when empty
Message string
Files []FileChange
Patch string // a unified diff, as git diff writes it, instead of Files
}
// Committed is what gitea_commit made.
type Committed struct {
Repository string `json:"repository"`
Branch string `json:"branch"`
Created bool `json:"created"`
Base string `json:"base,omitempty"`
Parent string `json:"parent"`
Commit string `json:"commit"`
Files []CommittedFile `json:"files,omitempty"`
Said []string `json:"said"`
}
// CommittedFile is one file the commit changed, and how.
type CommittedFile struct {
Path string `json:"path"`
Operation string `json:"operation"`
}
// What one commit carries at most (novox/hq issue 539): files, one file's content, one patch, and every
// file's content together, in bytes as committed (content_base64 decoded). Larger work is several commits.
const (
maxFiles = 200
maxFileBytes = 4 << 20
maxPatchBytes = 4 << 20
maxCommitBytes = 16 << 20
)
// readCommitAsk reads the tool's arguments.
func readCommitAsk(args map[string]any) (CommitAsk, error) {
a := CommitAsk{Owner: text(args, "owner"), Repo: text(args, "repo"), Branch: text(args, "branch"),
Base: text(args, "base"), Message: strings.TrimSpace(raw(args, "message"))}
a.Patch = raw(args, "patch")
if a.Branch == "" {
return a, refuse("branch is required: the branch the commit goes on, a branch of its own (created from base when absent). Nothing was sent")
}
if a.Message == "" {
return a, refuse("message is required: the commit's message. Nothing was sent")
}
files, given := args["files"]
if given && files != nil {
list, ok := files.([]any)
if !ok {
return a, refuse("files is a list of {path, content | content_base64 | delete: true}. Nothing was sent")
}
for i, item := range list {
m, ok := item.(map[string]any)
if !ok {
return a, refuse("files[%d] is not an object {path, content | content_base64 | delete: true}. Nothing was sent", i)
}
fc := FileChange{Path: text(m, "path")}
if v, ok := m["content"]; ok && v != nil {
s, ok := v.(string)
if !ok {
return a, refuse("files[%d].content is text. Nothing was sent", i)
}
fc.Content = &s
}
if v, ok := m["content_base64"]; ok && v != nil {
s, ok := v.(string)
if !ok {
return a, refuse("files[%d].content_base64 is text. Nothing was sent", i)
}
fc.ContentBase64 = &s
}
if v, ok := m["delete"]; ok && v != nil {
b, ok := v.(bool)
if !ok {
return a, refuse("files[%d].delete is true or false. Nothing was sent", i)
}
fc.Delete = b
}
a.Files = append(a.Files, fc)
}
}
switch {
case len(a.Files) == 0 && strings.TrimSpace(a.Patch) == "":
return a, refuse("give files (each {path, content | content_base64 | delete: true}) or patch (a unified diff, as git diff writes it). Nothing was sent")
case len(a.Files) > 0 && strings.TrimSpace(a.Patch) != "":
return a, refuse("give files or patch, not both: one commit carries one of them. Nothing was sent")
case len(a.Files) > maxFiles:
return a, refuse("%d files is more than the %d one commit carries. Nothing was sent", len(a.Files), maxFiles)
case len(a.Patch) > maxPatchBytes:
return a, refuse("the patch is %s, more than the %s one commit carries: split it into several commits. Nothing was sent", mib(len(a.Patch)), mib(maxPatchBytes))
}
seen := map[string]bool{}
total := 0
for i, fc := range a.Files {
if strings.HasPrefix(fc.Path, "/") {
return a, refuse("files[%d].path %q is relative to the repository's root, without a leading slash. Nothing was sent", i, fc.Path)
}
if _, err := segments("file path", fc.Path); err != nil {
return a, err
}
if seen[fc.Path] {
return a, refuse("%s is named twice in files. Nothing was sent", fc.Path)
}
seen[fc.Path] = true
given := 0
for _, g := range []bool{fc.Content != nil, fc.ContentBase64 != nil, fc.Delete} {
if g {
given++
}
}
if given != 1 {
return a, refuse("files[%d] (%s) needs exactly one of content, content_base64 or delete: true. Nothing was sent", i, fc.Path)
}
if fc.Content != nil && !utf8.ValidString(*fc.Content) {
return a, refuse("files[%d].content (%s) is not UTF-8 text: give it as content_base64. Nothing was sent", i, fc.Path)
}
size := 0
switch {
case fc.Content != nil:
size = len(*fc.Content)
case fc.ContentBase64 != nil:
if base64.StdEncoding.DecodedLen(len(*fc.ContentBase64)) > maxFileBytes+3 {
size = base64.StdEncoding.DecodedLen(len(*fc.ContentBase64))
break
}
raw, err := base64.StdEncoding.DecodeString(*fc.ContentBase64)
if err != nil {
return a, refuse("files[%d].content_base64 (%s) is not base64: %v. Nothing was sent", i, fc.Path, err)
}
size = len(raw)
}
if size > maxFileBytes {
return a, refuse("files[%d] (%s) is %s, more than the %s one file of a commit may be. Nothing was sent", i, fc.Path, mib(size), mib(maxFileBytes))
}
total += size
}
if total > maxCommitBytes {
return a, refuse("the files are %s together, more than the %s one commit carries: split them into several commits. Nothing was sent", mib(total), mib(maxCommitBytes))
}
return a, nil
}
// mib says a size in bytes in MiB, to a tenth.
func mib(n int) string { return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20)) }
// targetRefused says why a commit may not go on a branch: the default branch, or one a protection rule names.
func targetRefused(target, defaultBranch string, rules []Protection) error {
if target == defaultBranch {
return refuse("a commit on %s, the repository's default branch, is refused: a change reaches it by a pull request "+
"that passes its checks (novox/hq ADR 0266). Put it on a branch of its own. Nothing was sent", target)
}
var guarding []string
for _, r := range rules {
if n := r.name(); n != "" && ruleCovers(n, target) {
guarding = append(guarding, n)
}
}
if len(guarding) > 0 {
return refuse("a commit on %s is refused: the protection rule %s guards it (novox/hq ADR 0266). Put it on a "+
"branch of its own. Nothing was sent", target, strings.Join(guarding, ", "))
}
return nil
}
// commit puts one commit on the asked branch, after the judgement, and reads the branch back.
func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
repo, err := repoPath(a.Owner, a.Repo)
if err != nil {
return nil, err
}
if _, err := segments("branch", a.Branch); err != nil {
return nil, err
}
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.DefaultBranch == "" {
return nil, fmt.Errorf("the forge did not say %s/%s's default branch, so whether %s is it cannot be judged. Nothing was sent", a.Owner, a.Repo, a.Branch)
}
// Read the rules, or refuse: a branch the rules could not be read for is never written (fails closed).
var rules []Protection
if err := f.GetJSON(ctx, repo+"/branch_protections", &rules); err != nil {
return nil, fmt.Errorf("the protection rules of %s/%s could not be read, so the commit was not judged and nothing was sent: %w", a.Owner, a.Repo, err)
}
if err := targetRefused(a.Branch, r.DefaultBranch, rules); err != nil {
return nil, err
}
out := &Committed{Repository: a.Owner + "/" + a.Repo, Branch: a.Branch}
existing, err := f.branch(ctx, repo, a.Branch)
if err != nil {
return nil, err
}
// The files are judged at the commit just read, never at a branch name that could move between two
// reads (novox/hq issue 539): a file the branch changes meanwhile is refused by the forge, whose update
// and delete carry the blob read here.
var ref string
if existing != nil {
out.Parent = existing.Commit.ID
ref = existing.Commit.ID
if a.Base != "" {
out.Said = append(out.Said, fmt.Sprintf("%s exists, at %s: base %s was not used", a.Branch, short(existing.Commit.ID), a.Base))
}
} else {
out.Created = true
out.Base = a.Base
if out.Base == "" {
out.Base = r.DefaultBranch
}
b, err := f.branch(ctx, repo, out.Base)
if err != nil {
return nil, err
}
if b == nil {
return nil, refuse("%s does not exist, and its base %s is not a branch of %s/%s. Nothing was sent", a.Branch, out.Base, a.Owner, a.Repo)
}
out.Parent = b.Commit.ID
ref = b.Commit.ID
}
// Where the forge takes the commit: on the branch, or on a new one from the base's commit.
on := map[string]any{"message": a.Message}
if out.Created {
on["branch"] = out.Base
on["new_branch"] = a.Branch
} else {
on["branch"] = a.Branch
}
// The forge takes no commit to build on, only a branch (Gitea 1.27: neither /contents nor /diffpatch
// carries a last commit), so the parent is read from what it made, never assumed (novox/hq issue 539).
var made struct {
Commit madeCommit `json:"commit"`
}
if a.Patch != "" {
on["content"] = a.Patch
if err := f.SendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the patch: %w", err)
}
} else {
var ops []map[string]any
for _, fc := range a.Files {
sha, err := f.blobAt(ctx, repo, fc.Path, ref)
if err != nil {
return nil, err
}
op := map[string]any{"path": fc.Path}
switch {
case fc.Delete && sha == "":
return nil, refuse("%s is to be deleted, but %s has no such file at %s. Nothing was sent", fc.Path, a.Owner+"/"+a.Repo, short(ref))
case fc.Delete:
op["operation"], op["sha"] = "delete", sha
case sha == "":
op["operation"] = "create"
default:
op["operation"], op["sha"] = "update", sha
}
if !fc.Delete {
if fc.Content != nil {
op["content"] = base64.StdEncoding.EncodeToString([]byte(*fc.Content))
} else {
op["content"] = *fc.ContentBase64
}
}
ops = append(ops, op)
out.Files = append(out.Files, CommittedFile{Path: fc.Path, Operation: op["operation"].(string)})
}
on["files"] = ops
if err := f.SendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the commit: %w", err)
}
}
if made.Commit.SHA == "" {
return nil, fmt.Errorf("the forge answered the commit on %s without saying the commit it made: read the branch (gitea_list_branches) before trying again", a.Branch)
}
out.Commit = made.Commit.SHA
read := out.Parent
parent, err := f.parentOf(ctx, repo, made.Commit)
if err != nil {
return nil, fmt.Errorf("the forge made %s on %s, but which commit it went on could not be read: %w", out.Commit, a.Branch, err)
}
out.Parent = parent
if parent != read {
moved := "branch " + a.Branch
if out.Created {
moved = "base " + out.Base
}
how := "each file it changes was checked against " + short(read) + " by its blob, so none of them changed in between"
if a.Patch != "" {
how = "the forge applied the patch there, and it refuses a patch whose context changed"
}
out.Said = append(out.Said, fmt.Sprintf("the %s moved from %s to %s between the read and the commit, so the "+
"commit went on %s: %s. What else changed in between: gitea_compare %s...%s",
moved, short(read), short(parent), short(parent), how, short(read), short(parent)))
}
// Evidence over assertion: the branch is read back, and it must be at the commit the forge said.
after, err := f.branch(ctx, repo, a.Branch)
switch {
case err != nil:
return nil, fmt.Errorf("the forge made %s on %s, but the branch could not be read back: %w", out.Commit, a.Branch, err)
case after == nil:
return nil, fmt.Errorf("the forge said it made %s on %s, but the branch does not exist", out.Commit, a.Branch)
case after.Commit.ID != out.Commit:
return nil, fmt.Errorf("the forge said it made %s on %s, but the branch is at %s: something else moved it", out.Commit, a.Branch, after.Commit.ID)
}
what := "on"
if out.Created {
what = "on the new branch"
}
out.Said = append(out.Said, fmt.Sprintf("%s committed %s %s of %s, after %s; the branch is there now. A merge "+
"still needs a review at this head and both required checks green", short(out.Commit), what, a.Branch, out.Repository, short(out.Parent)))
return out, nil
}
// madeCommit is the commit the forge answers a write with.
type madeCommit struct {
SHA string `json:"sha"`
Parents []struct {
SHA string `json:"sha"`
} `json:"parents"`
}
// parentOf is the commit the forge made its commit on: from its answer, or read from the commit when the
// answer does not say it.
func (f *Forge) parentOf(ctx context.Context, repo string, made madeCommit) (string, error) {
if len(made.Parents) == 0 {
var read madeCommit
if err := f.GetJSON(ctx, repo+"/git/commits/"+url.PathEscape(made.SHA)+"?stat=false&verification=false&files=false", &read); err != nil {
return "", err
}
made = read
}
if len(made.Parents) != 1 || made.Parents[0].SHA == "" {
return "", fmt.Errorf("the forge says %s has %d parents, not one", made.SHA, len(made.Parents))
}
return made.Parents[0].SHA, nil
}
// blobAt is the blob sha of a file at a ref, "" when there is no file there.
func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, error) {
seg, err := segments("file path", path)
if err != nil {
return "", err
}
var entry struct {
Type string `json:"type"`
SHA string `json:"sha"`
}
var list []any
body, err := f.Do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return "", nil
}
return "", err
}
if jsonInto(body, &list) == nil {
return "", refuse("%s is a directory, not a file. Nothing was sent", path)
}
if err := jsonInto(body, &entry); err != nil {
return "", fmt.Errorf("the forge's answer about %s is not readable: %w", path, err)
}
if entry.Type != "file" && entry.Type != "symlink" {
return "", refuse("%s is a %s, not a file. Nothing was sent", path, entry.Type)
}
return entry.SHA, nil
}
func short(sha string) string {
if len(sha) > 12 {
return sha[:12]
}
return sha
}
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"context"
"errors"
"fmt"
"net/url"
"regexp"
"strings"
"gitea/internal/forgehttp"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the npm-registry bundle beside it).
// Forge is the forge's HTTP interface.
type Forge struct {
forgehttp.Client
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
c, missing := forgehttp.FromEnv()
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
}
c.Timeout = deadline
return &Forge{Client: c}, nil
}
// errNotFound is the forge answering 404.
var errNotFound = forgehttp.ErrNotFound
// maxBody is the largest JSON answer read from the forge; maxArchive the largest archive code search reads.
const (
maxBody = forgehttp.MaxBody
maxArchive = 64 << 20
)
// Refused is a refusal of policy or of an argument, in the words the caller sees.
type Refused struct{ msg string }
func (r *Refused) Error() string { return r.msg }
func refuse(format string, a ...any) error { return &Refused{fmt.Sprintf(format, a...)} }
var nameShape = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.-]*$`)
// repoPath is /api/v1/repos/{owner}/{repo} for names that cannot become another path.
func repoPath(owner, repo string) (string, error) {
for _, v := range [][2]string{{"owner", owner}, {"name", repo}} {
if !nameShape.MatchString(v[1]) || v[1] == "." || v[1] == ".." {
return "", refuse("%q is not a repository %s", v[1], v[0])
}
}
return "/api/v1/repos/" + owner + "/" + repo, nil
}
// segments encodes a value that is several path segments — a branch like fix/1-x, a file's path — refusing
// any segment that could make the request another path: empty, `.` or `..`, or holding `?`, `#`, `%`, `\`
// or a control character (the same rule as the TypeScript bundle's pathSegments, guard.ts).
func segments(what, value string) (string, error) {
parts := strings.Split(value, "/")
for i, p := range parts {
if p == "" || p == "." || p == ".." || strings.ContainsAny(p, "?#%\\") || strings.IndexFunc(p, func(r rune) bool { return r < 0x20 || r == 0x7f }) >= 0 {
return "", refuse("%q is not a %s this tool puts in a path. Nothing was sent", value, what)
}
parts[i] = url.PathEscape(p)
}
return strings.Join(parts, "/"), nil
}
// Repo is what the tools read of a repository.
type Repo struct {
FullName string `json:"full_name"`
DefaultBranch string `json:"default_branch"`
Empty bool `json:"empty"`
}
// Protection is one branch protection rule, as far as covering a branch goes.
type Protection struct {
RuleName string `json:"rule_name"`
BranchName string `json:"branch_name"`
}
func (p Protection) name() string {
if p.RuleName != "" {
return p.RuleName
}
return p.BranchName
}
// Branch is one branch with the commit it points at.
type Branch struct {
Name string `json:"name"`
Commit struct {
ID string `json:"id"`
} `json:"commit"`
}
// branch is the named branch, nil when the forge has none of that name.
func (f *Forge) branch(ctx context.Context, repo, name string) (*Branch, error) {
seg, err := segments("branch", name)
if err != nil {
return nil, err
}
var b Branch
if err := f.GetJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
if errors.Is(err, errNotFound) {
return nil, nil
}
return nil, err
}
return &b, nil
}
+744
View File
@@ -0,0 +1,744 @@
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"testing"
"gitea/internal/forgehttp"
)
// fakeForge is the forge's HTTP interface as far as these tools use it.
type fakeForge struct {
t *testing.T
mu sync.Mutex
password string
branches map[string]string // branch -> commit
trees map[string]map[string]string // commit -> path -> content
rules []Protection
rulesBroken bool
staleRead bool // the branch is not moved by a write: a forge that says what it did not do
writes []map[string]any
patches []map[string]any
next int
archive []byte
pullHead string
reviews []map[string]any
comments map[string][]map[string]any
commits []map[string]any
dir []map[string]any
// moves: a branch moved by someone else right after it is first read — to a commit, with its tree.
moves map[string]move
noParents bool // the forge's answer to a write does not say the parents
parentsBroken bool // and GET git/commits fails
parents map[string]string // commit -> parent, as GET git/commits says it
}
type move struct {
to string
tree map[string]string
}
func newFake(t *testing.T) *fakeForge {
return &fakeForge{
t: t,
password: "s3cret-never-shown",
branches: map[string]string{"main": "c0", "fix/1-old": "c1"},
trees: map[string]map[string]string{
"c0": {"README.md": "hello\n", "docs/a.md": "a\n"},
"c1": {"README.md": "hello\n", "docs/a.md": "a on fix\n"},
},
rules: []Protection{{RuleName: "main"}, {RuleName: "release/*"}},
}
}
func blob(content string) string { return fmt.Sprintf("%x", sha1.Sum([]byte(content))) }
func (f *fakeForge) serve() *Forge {
dir := f.t.TempDir()
pw := filepath.Join(dir, "admin.secret")
if err := os.WriteFile(pw, []byte("s3cret-never-shown\n"), 0o600); err != nil {
f.t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(f.handle))
f.t.Cleanup(srv.Close)
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}}
}
func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
if u, p, ok := r.BasicAuth(); !ok || u != "mesh-admin" || p != f.password {
http.Error(w, "unauthorized", 401)
return
}
reply := func(v any) { _ = json.NewEncoder(w).Encode(v) }
const repo = "/api/v1/repos/acme/app"
path := r.URL.EscapedPath()
switch {
case r.Method == "GET" && path == repo:
reply(map[string]any{"full_name": "acme/app", "default_branch": "main"})
case r.Method == "GET" && path == repo+"/branch_protections":
if f.rulesBroken {
http.Error(w, "boom", 500)
return
}
reply(f.rules)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/branches/"):
name := strings.TrimPrefix(r.URL.Path, repo+"/branches/")
c, ok := f.branches[name]
if !ok {
http.NotFound(w, r)
return
}
reply(map[string]any{"name": name, "commit": map[string]any{"id": c}})
if m, ok := f.moves[name]; ok {
delete(f.moves, name)
f.trees[m.to] = m.tree
f.branches[name] = m.to
}
case r.Method == "GET" && (path == repo+"/contents" || path == repo+"/contents/docs"):
reply(f.dir)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/contents/"):
p := strings.TrimPrefix(r.URL.Path, repo+"/contents/")
ref := r.URL.Query().Get("ref")
if c, isBranch := f.branches[ref]; isBranch {
ref = c
}
content, ok := f.trees[ref][p]
if !ok {
http.NotFound(w, r)
return
}
reply(map[string]any{"type": "file", "path": p, "sha": blob(content)})
case r.Method == "POST" && path == repo+"/contents":
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
f.writes = append(f.writes, body)
from := f.branches[body["branch"].(string)]
tree := map[string]string{}
for k, v := range f.trees[from] {
tree[k] = v
}
for _, raw := range body["files"].([]any) {
op := raw.(map[string]any)
p := op["path"].(string)
old, had := tree[p]
switch op["operation"] {
case "create":
if had {
http.Error(w, "exists", 422)
return
}
case "update", "delete":
if !had || op["sha"] != blob(old) {
http.Error(w, "sha does not match", 409)
return
}
}
if op["operation"] == "delete" {
delete(tree, p)
continue
}
c, _ := base64.StdEncoding.DecodeString(op["content"].(string))
tree[p] = string(c)
}
f.next++
made := fmt.Sprintf("n%d", f.next)
f.trees[made] = tree
target := body["branch"].(string)
if nb, ok := body["new_branch"].(string); ok && nb != "" {
target = nb
}
if !f.staleRead {
f.branches[target] = made
}
reply(map[string]any{"commit": f.made(made, from)})
case r.Method == "POST" && path == repo+"/diffpatch":
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
f.patches = append(f.patches, body)
f.next++
made := fmt.Sprintf("p%d", f.next)
from := f.branches[body["branch"].(string)]
target := body["branch"].(string)
if nb, ok := body["new_branch"].(string); ok && nb != "" {
target = nb
}
f.branches[target] = made
reply(map[string]any{"commit": f.made(made, from)})
case r.Method == "GET" && strings.HasPrefix(path, repo+"/git/commits/"):
sha := strings.TrimPrefix(path, repo+"/git/commits/")
p, ok := f.parents[sha]
if !ok || f.parentsBroken {
http.NotFound(w, r)
return
}
reply(map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": p}}})
case r.Method == "GET" && path == repo+"/commits":
f.writes = append(f.writes, map[string]any{"query": r.URL.RawQuery})
reply(f.commits)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/compare/"):
reply(map[string]any{"total_commits": len(f.commits), "commits": f.commits})
case r.Method == "GET" && path == repo+"/pulls/7":
reply(map[string]any{"head": map[string]any{"sha": f.pullHead}})
case r.Method == "GET" && path == repo+"/pulls/7/reviews":
reply(f.reviews)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/pulls/7/reviews/") && strings.HasSuffix(path, "/comments"):
id := strings.TrimSuffix(strings.TrimPrefix(path, repo+"/pulls/7/reviews/"), "/comments")
reply(f.comments[id])
case r.Method == "GET" && strings.HasPrefix(path, repo+"/archive/"):
_, _ = w.Write(f.archive)
default:
http.NotFound(w, r)
}
}
// made is the forge's answer about a commit it made on a parent, with the parents unless noParents.
func (f *fakeForge) made(sha, parent string) map[string]any {
if f.parents == nil {
f.parents = map[string]string{}
}
f.parents[sha] = parent
if f.noParents {
return map[string]any{"sha": sha}
}
return map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": parent}}}
}
func filesArg(files ...map[string]any) []any {
out := make([]any, len(files))
for i, f := range files {
out[i] = f
}
return out
}
func ask(t *testing.T, args map[string]any) CommitAsk {
t.Helper()
base := map[string]any{"owner": "acme", "repo": "app", "message": "Say why"}
for k, v := range args {
base[k] = v
}
a, err := readCommitAsk(base)
if err != nil {
t.Fatalf("ask refused: %v", err)
}
return a
}
func TestACommitOnANewBranchCreatesUpdatesAndDeletesInOneCommitFromTheBase(t *testing.T) {
f := newFake(t)
a := ask(t, map[string]any{"branch": "fix/375-x", "files": filesArg(
map[string]any{"path": "new.md", "content": "new\n"},
map[string]any{"path": "README.md", "content": "hello again\n"},
map[string]any{"path": "docs/a.md", "delete": true},
)})
got, err := commit(context.Background(), f.serve(), a)
if err != nil {
t.Fatal(err)
}
if !got.Created || got.Base != "main" || got.Parent != "c0" || got.Commit != "n1" || f.branches["fix/375-x"] != "n1" {
t.Fatalf("%+v; branches %v", got, f.branches)
}
if len(f.writes) != 1 {
t.Fatalf("%d writes", len(f.writes))
}
w := f.writes[0]
if w["branch"] != "main" || w["new_branch"] != "fix/375-x" || w["message"] != "Say why" {
t.Fatalf("sent %v", w)
}
ops := map[string]string{}
for _, raw := range w["files"].([]any) {
op := raw.(map[string]any)
ops[op["path"].(string)] = op["operation"].(string)
if op["operation"] == "create" && op["sha"] != nil {
t.Error("a created file carried a sha")
}
}
if ops["new.md"] != "create" || ops["README.md"] != "update" || ops["docs/a.md"] != "delete" {
t.Fatalf("ops %v", ops)
}
tree := f.trees["n1"]
if tree["README.md"] != "hello again\n" || tree["new.md"] != "new\n" || tree["docs/a.md"] != "" {
t.Fatalf("tree %v", tree)
}
if f.branches["main"] != "c0" {
t.Fatal("main moved")
}
}
func TestACommitOnAnExistingBranchGoesOnItAndReadsItsFilesThere(t *testing.T) {
f := newFake(t)
a := ask(t, map[string]any{"branch": "fix/1-old", "base": "main",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})})
got, err := commit(context.Background(), f.serve(), a)
if err != nil {
t.Fatal(err)
}
if got.Created || got.Parent != "c1" || f.branches["fix/1-old"] != got.Commit {
t.Fatalf("%+v", got)
}
if _, has := f.writes[0]["new_branch"]; has || f.writes[0]["branch"] != "fix/1-old" {
t.Fatalf("sent %v", f.writes[0])
}
if !strings.Contains(strings.Join(got.Said, " "), "base main was not used") {
t.Errorf("an unused base was not said: %v", got.Said)
}
}
func TestTheDefaultBranchAndProtectedBranchesAreRefusedBeforeAnythingIsSent(t *testing.T) {
for _, branch := range []string{"main", "release/1.0"} {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": branch,
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "refused") || !strings.Contains(err.Error(), "ADR 0266") {
t.Errorf("%s: %v", branch, err)
}
if len(f.writes)+len(f.patches) != 0 {
t.Errorf("%s: something was written", branch)
}
}
}
func TestRulesThatCannotBeReadRefuseTheCommit(t *testing.T) {
f := newFake(t)
f.rulesBroken = true
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/2",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "protection rules") || len(f.writes) != 0 {
t.Fatalf("%v, %d writes", err, len(f.writes))
}
}
func TestAPatchIsAppliedByTheForgeOnANewBranch(t *testing.T) {
f := newFake(t)
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/3", "patch": patch}))
if err != nil {
t.Fatal(err)
}
if len(f.patches) != 1 || f.patches[0]["content"] != patch || f.patches[0]["branch"] != "main" || f.patches[0]["new_branch"] != "fix/3" {
t.Fatalf("%v", f.patches)
}
if got.Commit != "p1" || len(f.writes) != 0 {
t.Fatalf("%+v", got)
}
}
func TestACommitTheBranchDoesNotShowIsAFailure(t *testing.T) {
f := newFake(t)
f.staleRead = true
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "branch is at c1") {
t.Fatalf("%v", err)
}
}
func TestDeletingAFileThatIsNotThereIsRefused(t *testing.T) {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/4",
"files": filesArg(map[string]any{"path": "gone.md", "delete": true})}))
if err == nil || !strings.Contains(err.Error(), "no such file") || len(f.writes) != 0 {
t.Fatalf("%v", err)
}
}
func TestAMissingBaseIsRefused(t *testing.T) {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/5", "base": "nope",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "base nope is not a branch") {
t.Fatalf("%v", err)
}
}
func TestACommitAskThatCannotBeReadIsRefused(t *testing.T) {
one := filesArg(map[string]any{"path": "x", "content": "x"})
for name, args := range map[string]map[string]any{
"no branch": {"files": one},
"no message": {"branch": "b", "message": " ", "files": one},
"neither": {"branch": "b"},
"both": {"branch": "b", "files": one, "patch": "diff"},
"dot path": {"branch": "b", "files": filesArg(map[string]any{"path": "a/../b", "content": "x"})},
"absolute path": {"branch": "b", "files": filesArg(map[string]any{"path": "/etc/x", "content": "x"})},
"twice": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content": "x"}, map[string]any{"path": "x", "delete": true})},
"two contents": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content": "x", "delete": true})},
"no content": {"branch": "b", "files": filesArg(map[string]any{"path": "x"})},
"bad base64": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content_base64": "!!"})},
"files not list": {"branch": "b", "files": "x"},
"path with query": {"branch": "b", "files": filesArg(map[string]any{"path": "a?b", "content": "x"})},
} {
full := map[string]any{"owner": "acme", "repo": "app", "message": "m"}
for k, v := range args {
full[k] = v
}
if _, err := readCommitAsk(full); err == nil {
t.Errorf("%s: taken", name)
}
}
f := newFake(t)
_, err := commit(context.Background(), f.serve(), CommitAsk{Owner: "acme", Repo: "app", Branch: "../x", Message: "m", Patch: "d"})
if err == nil {
t.Error("a branch with a dot segment was taken")
}
}
// The forge's own answers (Gitea 1.27's Match), one table for this bundle and guard.ts alike (novox/hq issue 539).
// `forge` is the forge's answer, which this port must give; `guard` is where guard.ts deliberately covers more.
func TestRuleCoversAnswersAsTheForgeDoes(t *testing.T) {
raw, err := os.ReadFile("../../test/rule-cases.json")
if err != nil {
t.Fatal(err)
}
var cases []struct {
Rule string `json:"rule"`
Branch string `json:"branch"`
Forge bool `json:"forge"`
}
if err := json.Unmarshal(raw, &cases); err != nil {
t.Fatal(err)
}
if len(cases) < 100 {
t.Fatalf("only %d cases", len(cases))
}
for _, c := range cases {
if got := ruleCovers(c.Rule, c.Branch); got != c.Forge {
t.Errorf("%q covers %q: %v, the forge says %v", c.Rule, c.Branch, got, c.Forge)
}
}
}
func TestHistoryAndCompareAreSummarised(t *testing.T) {
f := newFake(t)
mk := func(sha, msg string, files ...string) map[string]any {
var fs []map[string]any
for _, x := range files {
fs = append(fs, map[string]any{"filename": x, "status": "modified"})
}
return map[string]any{"sha": sha, "commit": map[string]any{"message": msg,
"author": map[string]any{"name": "jochen", "date": "2026-10-12T10:00:00Z"}}, "files": fs}
}
f.commits = []map[string]any{mk("a1", "First line\n\nbody", "x.md", "y.md"), mk("a2", "Second", "y.md")}
forge := f.serve()
h, err := history(context.Background(), forge, "acme", "app", "fix/1", "docs", 2, 1)
if err != nil {
t.Fatal(err)
}
if len(h.Commits) != 2 || h.Commits[0].Subject != "First line" || h.Commits[0].Author != "jochen" || !h.More {
t.Fatalf("%+v", h)
}
q := f.writes[0]["query"].(string)
for _, want := range []string{"sha=fix%2F1", "path=docs", "limit=2", "files=false"} {
if !strings.Contains(q, want) {
t.Errorf("query %s lacks %s", q, want)
}
}
c, err := compare(context.Background(), forge, "acme", "app", "main", "fix/1")
if err != nil {
t.Fatal(err)
}
if c.Total != 2 || len(c.Files) != 2 || c.Files[0].Path != "x.md" || c.Commits[1].Subject != "Second" {
t.Fatalf("%+v", c)
}
if _, err := compare(context.Background(), forge, "acme", "app", "a...b", "c"); err == nil {
t.Error("a ref holding ... was compared")
}
}
func TestReviewsSayWhichApproveTheHeadAndCarryTheirLineComments(t *testing.T) {
f := newFake(t)
f.pullHead = "h2"
f.reviews = []map[string]any{
{"id": 1, "user": map[string]any{"login": "reviewer"}, "state": "APPROVED", "commit_id": "h1", "comments_count": 0},
{"id": 2, "user": map[string]any{"login": "reviewer"}, "state": "REQUEST_CHANGES", "commit_id": "h2", "comments_count": 1, "body": "see line"},
{"id": 3, "user": map[string]any{"login": "op"}, "state": "APPROVED", "commit_id": "h2", "comments_count": 0},
}
f.comments = map[string][]map[string]any{"2": {{"path": "a.go", "position": 12, "user": map[string]any{"login": "reviewer"},
"body": "off by one", "resolver": map[string]any{"login": "op"}}}}
r, err := reviews(context.Background(), f.serve(), "acme", "app", 7)
if err != nil {
t.Fatal(err)
}
if len(r.Reviews) != 3 || r.Reviews[0].AtHead || !r.Reviews[1].AtHead || len(r.Reviews[1].Comments) != 1 {
t.Fatalf("%+v", r)
}
lc := r.Reviews[1].Comments[0]
if lc.Path != "a.go" || lc.Line != 12 || lc.Resolved != "op" {
t.Fatalf("%+v", lc)
}
if !strings.Contains(r.Said[0], "1 approve the head") {
t.Fatalf("%v", r.Said)
}
}
func TestADirectoryIsListed(t *testing.T) {
f := newFake(t)
f.dir = []map[string]any{{"name": "a.md", "type": "file", "size": 3}, {"name": "sub", "type": "dir", "size": 0}}
l, err := listDir(context.Background(), f.serve(), "acme", "app", "/docs/", "main")
if err != nil {
t.Fatal(err)
}
if l.Path != "/docs" || len(l.Entries) != 2 || l.Entries[0].Size != 3 || l.Entries[1].Type != "dir" {
t.Fatalf("%+v", l)
}
}
func archive(t *testing.T, files map[string]string) []byte {
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
_ = tw.WriteHeader(&tar.Header{Name: "app/", Typeflag: tar.TypeDir, Mode: 0o755})
for name, content := range files {
if err := tw.WriteHeader(&tar.Header{Name: "app/" + name, Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(content))}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(content))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
func TestCodeSearchReadsTheArchiveOnce(t *testing.T) {
f := newFake(t)
f.archive = archive(t, map[string]string{
"docs/a.md": "one\nthe Forge\nthree forge\n",
"src/b.go": "package b // forge\n",
"bin/x": "forge\x00binary",
"docs/c.txt": "nothing\n",
})
forge := f.serve()
found, err := searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "forge", Max: 100})
if err != nil {
t.Fatal(err)
}
if len(found.Matches) != 2 || found.Ref != "main" || found.Files != 3 {
t.Fatalf("%+v", found)
}
found, _ = searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "forge", IgnoreCase: true, Path: "docs", Max: 100})
if len(found.Matches) != 2 || found.Matches[0].Path != "docs/a.md" || found.Matches[0].Line != 2 {
t.Fatalf("%+v", found)
}
found, _ = searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "^(one|three)", Regex: true, Max: 1})
if len(found.Matches) != 1 || !found.Truncated {
t.Fatalf("%+v", found)
}
if _, err := searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "(", Regex: true, Max: 1}); err == nil {
t.Error("a broken expression was searched")
}
}
func TestAWrongPasswordFailsWithoutSayingIt(t *testing.T) {
f := newFake(t)
forge := f.serve()
f.password = "rotated"
_, err := listDir(context.Background(), forge, "acme", "app", "", "")
if err == nil || !strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "s3cret") {
t.Fatalf("%v", err)
}
}
func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
tools := Tools(func() (*Forge, error) { return nil, fmt.Errorf("unset") })
var served []string
for _, tl := range tools {
served = append(served, tl.Name)
suffix := "(r)"
if tl.Name == "gitea_commit" {
suffix = "(a)"
}
if !strings.HasSuffix(tl.Description, suffix) {
t.Errorf("%s does not end with %s", tl.Name, suffix)
}
if _, err := tl.Run(map[string]any{}); err == nil {
t.Errorf("%s ran without a forge", tl.Name)
}
}
if strings.Join(served, ",") != strings.Join(ToolNames, ",") {
t.Fatalf("served %v, named %v", served, ToolNames)
}
readme, err := os.ReadFile("../../README.md")
if err != nil {
t.Fatal(err)
}
for _, n := range ToolNames {
if !strings.Contains(string(readme), "`"+n+"`") {
t.Errorf("the README does not name %s", n)
}
}
// No name the TypeScript bundle serves: two bundles serving one name is one tool answered twice.
ts, err := os.ReadFile("../../tools/index.ts")
if err != nil {
t.Fatal(err)
}
for _, n := range ToolNames {
if strings.Contains(string(ts), `name: "`+n+`"`) {
t.Errorf("the TypeScript bundle serves %s too", n)
}
}
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Tools []string `json:"tools"`
Build struct {
Artifacts []struct {
From string `json:"from"`
Binary string `json:"binary"`
Loads []string `json:"loads"`
Env map[string]string `json:"env"`
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if len(m.Tools) != 0 {
t.Error("the manifest lists tools: its claims would offer them as both seats' verbs")
}
found := false
for _, a := range m.Build.Artifacts {
if a.From == "cmd/forge" {
found = true
if len(a.Loads) != 1 || a.Loads[0] != a.Binary {
t.Errorf("the bundle loads %v, not its binary %s", a.Loads, a.Binary)
}
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE"} {
if a.Env[k] == "" {
t.Errorf("the bundle is not given %s", k)
}
}
}
}
if !found {
t.Error("the manifest builds no cmd/forge bundle")
}
}
// novox/hq issue 539: the forge takes a branch, never a commit to build on, so the answer's parent is the one
// the forge made the commit on, and a branch that moves between the read and the commit is said.
func TestAnExistingBranchMovedUnderTheCommitIsSaidAndItsParentIsTheOneTheForgeUsed(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "a on fix\n", "other.md": "x\n"}}}
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c2" || f.branches["fix/1-old"] != got.Commit {
t.Fatalf("parent %s, want c2 (the commit the forge built on): %+v", got.Parent, got)
}
said := strings.Join(got.Said, " ")
if !strings.Contains(said, "moved from c1 to c2") || !strings.Contains(said, "after c2") {
t.Fatalf("the move was not said: %v", got.Said)
}
}
func TestAnExistingBranchsFilesAreJudgedAtTheCommitReadNotAtWhereTheBranchMovedTo(t *testing.T) {
f := newFake(t)
// Someone else changes docs/a.md on the branch between the read and the commit: that change is not
// overwritten by a blob read at the branch's new head.
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "theirs\n"}}}
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "mine\n"})}))
if err == nil || !strings.Contains(err.Error(), "sha does not match") {
t.Fatalf("a file changed under the commit was overwritten: %v", err)
}
if f.trees["c2"]["docs/a.md"] != "theirs\n" || f.branches["fix/1-old"] != "c2" {
t.Fatalf("the branch was changed: %v", f.branches)
}
}
func TestAPatchOnABaseThatMovedNamesTheCommitTheForgeAppliedItTo(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"main": {to: "c9", tree: map[string]string{"README.md": "hello\n"}}}
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/6", "patch": patch}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c9" || !strings.Contains(strings.Join(got.Said, " "), "base main moved from c0 to c9") {
t.Fatalf("%+v", got)
}
}
func TestAParentTheAnswerDoesNotSayIsReadFromTheCommit(t *testing.T) {
f := newFake(t)
f.noParents = true
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/7",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err != nil || got.Parent != "c0" {
t.Fatalf("%v %+v", err, got)
}
// A parent that cannot be read is a failure, said with the commit that was made.
f = newFake(t)
f.noParents, f.parentsBroken = true, true
_, err = commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/8",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "the forge made n1 on fix/8, but which commit it went on could not be read") {
t.Fatalf("%v", err)
}
}
func TestACommitLargerThanItsLimitsIsRefusedBeforeAnythingIsSent(t *testing.T) {
big := strings.Repeat("x", maxFileBytes+1)
quarter := strings.Repeat("y", maxFileBytes)
many := filesArg()
for i := 0; i < maxCommitBytes/maxFileBytes+1; i++ {
many = append(many, map[string]any{"path": fmt.Sprintf("f%d", i), "content": quarter})
}
for name, args := range map[string]map[string]any{
"one file": {"files": filesArg(map[string]any{"path": "x", "content": big})},
"one file b64": {"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(big))})},
"the patch": {"patch": "diff --git a/x b/x\n" + big},
"all files": {"files": many},
} {
full := map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b"}
for k, v := range args {
full[k] = v
}
_, err := readCommitAsk(full)
if err == nil || !strings.Contains(err.Error(), "MiB") || !strings.Contains(err.Error(), "Nothing was sent") {
t.Errorf("%s: %v", name, err)
}
}
// At the limit is taken.
if _, err := readCommitAsk(map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b",
"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(quarter))})}); err != nil {
t.Errorf("a file at the limit was refused: %v", err)
}
}
func TestCodeSearchStopsAtItsDeadlineAndSaysSo(t *testing.T) {
gz := archive(t, map[string]string{"a.md": "forge\n", "b.md": "forge\n"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
out := &Found{Matches: []Match{}}
if err := searchArchive(ctx, gz, "", regexp.MustCompile("forge"), 100, out); err != nil {
t.Fatal(err)
}
if !out.Stopped || out.Files != 0 {
t.Fatalf("a search past its deadline went on: %+v", out)
}
// Within its deadline, the same archive is searched whole.
out = &Found{Matches: []Match{}}
if err := searchArchive(context.Background(), gz, "", regexp.MustCompile("forge"), 100, out); err != nil || out.Stopped || len(out.Matches) != 2 {
t.Fatalf("%v %+v", err, out)
}
}
+238
View File
@@ -0,0 +1,238 @@
// forge: the forge module's Go bundle for an agent's commit and the reads around it (novox/hq issue 375).
// A process the node's runtime launches and speaks MCP over stdio to, beside the module's TypeScript bundle
// and its npm-registry bundle. gitea_commit puts one commit on a branch of its own — never the default branch,
// never one a protection rule names (novox/hq ADR 0266) — and the reads answer a commit history, a compare, a
// pull request's reviews with their line comments, a directory's entries and a code search, summarised.
// stdout is the protocol; it says what it says on stderr, and never the forge's credential.
package main
import (
"context"
"fmt"
"math"
"os"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the npm-registry
// bundle for why); the README names these, and a test holds the three together.
var ToolNames = []string{"gitea_commit", "gitea_commits", "gitea_compare", "gitea_pull_reviews", "gitea_list_dir", "gitea_search_code"}
func main() {
if err := stdio.Serve("", Tools(ForgeFromEnv)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// Tools are the bundle's tools over the forge the function gives; asked per call, so a manifest that does not
// set the environment is a refusal naming what is missing, not a bundle that will not start.
func Tools(forge func() (*Forge, error)) []stdio.Tool {
str := func(d string) map[string]any { return map[string]any{"type": "string", "description": d} }
owner := str("the repository's owner, e.g. novox")
repo := str("the repository's name, e.g. hq")
run := func(do func(ctx context.Context, f *Forge, args map[string]any) (any, error)) func(map[string]any) (any, error) {
return func(args map[string]any) (any, error) {
f, err := forge()
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), deadline)
defer cancel()
return do(ctx, f, args)
}
}
return []stdio.Tool{
{
Name: "gitea_commit",
Description: "Put one commit on a branch of its own in a forge repository, creating the branch from base " +
"when it does not exist. Refused on the repository's default branch and on any branch a protection " +
"rule names (novox/hq ADR 0266): a change reaches those only by a pull request whose head an " +
"independent review approved and whose required checks passed (gitea_merge_pull_request). Carries " +
"either files — each {path, content (text) | content_base64 | delete: true}, created or updated as " +
"the branch holds them — or patch, a unified diff as git diff writes it, which the forge applies. " +
"At most 200 files, 4 MiB each and 16 MiB together, or a 4 MiB patch. Answers the commit it made and " +
"the parent the forge made it on, read back from the branch, and says when the branch moved under " +
"it. Replaces git commit and git " +
"push of a feature branch. (a)",
Input: map[string]any{
"type": "object",
"properties": map[string]any{
"owner": owner,
"repo": repo,
"branch": str("the branch the commit goes on, e.g. fix/375-a-gitea-commit-tool; never the default branch or a protected one"),
"base": str("for a branch that does not exist yet: the branch it starts from (default the repository's default branch)"),
"message": str("the commit's message: concise, in the imperative, about why"),
"files": map[string]any{
"type": "array",
"description": "the files the commit changes; give this or patch",
"items": map[string]any{
"type": "object",
"properties": map[string]any{
"path": str("the file's path from the repository's root"),
"content": str("its whole new content, as UTF-8 text"),
"content_base64": str("its whole new content, base64-encoded, for a file that is not text"),
"delete": map[string]any{"type": "boolean", "description": "true to delete the file"},
},
"required": []string{"path"},
},
},
"patch": str("a unified diff of the change, as git diff or git format-patch writes it; give this or files"),
},
"required": []string{"owner", "repo", "branch", "message"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
a, err := readCommitAsk(args)
if err != nil {
return nil, err
}
return commit(ctx, f, a)
}),
},
{
Name: "gitea_commits",
Description: "A repository's commit history at a branch, tag or commit (default the default branch), " +
"newest first, optionally only the commits touching a path: each commit's sha, author, date and " +
"subject line. Replaces git log. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"ref": str("a branch, tag or commit (default the default branch)"),
"path": str("only commits touching this file or directory"),
"limit": map[string]any{"type": "integer", "description": "commits per page, 1 to 100 (default 20)"},
"page": map[string]any{"type": "integer", "description": "the page, from 1 (default 1)"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
limit, err := count(args, "limit", 20, 1, 100)
if err != nil {
return nil, err
}
page, err := count(args, "page", 1, 1, 10000)
if err != nil {
return nil, err
}
return history(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "ref"), strings.Trim(text(args, "path"), "/"), limit, page)
}),
},
{
Name: "gitea_compare",
Description: "What head holds that base does not, in a forge repository: its commits (sha, author, date, " +
"subject) and the files they touch with how. Replaces git log base..head and git diff --stat. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"base": str("the branch, tag or commit compared from, e.g. main"),
"head": str("the branch, tag or commit compared, e.g. fix/375-a-gitea-commit-tool"),
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
return compare(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "base"), text(args, "head"))
}),
},
{
Name: "gitea_pull_reviews",
Description: "A pull request's reviews, oldest first: who, its state (APPROVED, REQUEST_CHANGES, COMMENT, " +
"PENDING), the commit it reviewed and whether that is the pull request's head now, stale or dismissed, " +
"its body, and every line comment it left (file, line, body, who resolved it). (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"number": map[string]any{"type": "integer", "description": "the pull request's number"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
n, err := count(args, "number", 0, 1, math.MaxInt32)
if err != nil {
return nil, err
}
if n == 0 {
return nil, refuse("number is required: the pull request's number")
}
return reviews(ctx, f, text(args, "owner"), text(args, "repo"), n)
}),
},
{
Name: "gitea_list_dir",
Description: "A directory's entries in a forge repository at a branch, tag or commit (default the default " +
"branch): each entry's name, its type (file, dir, symlink, submodule) and a file's size. Replaces ls " +
"and git ls-tree. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"path": str("the directory from the repository's root (default the root)"),
"ref": str("a branch, tag or commit (default the default branch)"),
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
return listDir(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "path"), text(args, "ref"))
}),
},
{
Name: "gitea_search_code",
Description: "Search a forge repository's text files at a branch, tag or commit (default the default " +
"branch) for a literal string, or a regular expression with regex: true, optionally under a path: " +
"each matching line's file, number and text. The forge has no code search of its own, so the " +
"repository's archive is read once and searched; files over 4 MiB are named, not searched. Replaces " +
"git grep. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"query": str("what to find: a literal string, or a Go regular expression with regex: true"),
"regex": map[string]any{"type": "boolean", "description": "true to read query as a regular expression"},
"ignore_case": map[string]any{"type": "boolean", "description": "true to match regardless of case"},
"path": str("only files under this directory, or this one file"),
"ref": str("a branch, tag or commit (default the default branch)"),
"max": map[string]any{"type": "integer", "description": "the most matches answered, 1 to 1000 (default 100)"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
max, err := count(args, "max", 100, 1, 1000)
if err != nil {
return nil, err
}
regex, err := flag(args, "regex")
if err != nil {
return nil, err
}
ignore, err := flag(args, "ignore_case")
if err != nil {
return nil, err
}
return searchCode(ctx, f, SearchAsk{Owner: text(args, "owner"), Repo: text(args, "repo"), Ref: text(args, "ref"),
Query: raw(args, "query"), Path: text(args, "path"), Regex: regex, IgnoreCase: ignore, Max: max})
}),
},
}
}
// text is an argument trimmed; raw is it as given.
func text(args map[string]any, key string) string { return strings.TrimSpace(raw(args, key)) }
func raw(args map[string]any, key string) string {
s, _ := args[key].(string)
return s
}
func count(args map[string]any, key string, fallback, least, most int) (int, error) {
v, given := args[key]
if !given || v == nil {
return fallback, nil
}
x, ok := v.(float64)
if !ok || x != math.Trunc(x) {
return 0, refuse("%s must be a whole number", key)
}
if x < float64(least) || x > float64(most) {
return 0, refuse("%s must be from %d to %d", key, least, most)
}
return int(x), nil
}
func flag(args map[string]any, key string) (bool, error) {
v, given := args[key]
if !given || v == nil {
return false, nil
}
switch b := v.(type) {
case bool:
return b, nil
case string:
if b == "true" || b == "false" {
return b == "true", nil
}
}
return false, refuse("%s is true or false, not %v", key, v)
}
+334
View File
@@ -0,0 +1,334 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"strings"
"time"
)
// The forge's reads an agent otherwise made through gitea_api, answered summarised rather than as the forge's
// JSON (novox/hq issue 375): a commit's history, a compare, a pull request's reviews with their line comments,
// a directory's entries.
// CommitLine is one commit, summarised.
type CommitLine struct {
SHA string `json:"sha"`
Author string `json:"author"`
Date string `json:"date"`
Subject string `json:"subject"`
Files int `json:"files,omitempty"`
}
type forgeCommit struct {
SHA string `json:"sha"`
Commit struct {
Message string `json:"message"`
Author struct {
Name string `json:"name"`
Email string `json:"email"`
Date string `json:"date"`
} `json:"author"`
} `json:"commit"`
Files []struct {
Filename string `json:"filename"`
Status string `json:"status"`
} `json:"files"`
}
func (c forgeCommit) line() CommitLine {
subject, _, _ := strings.Cut(strings.TrimSpace(c.Commit.Message), "\n")
return CommitLine{SHA: c.SHA, Author: c.Commit.Author.Name, Date: c.Commit.Author.Date, Subject: subject, Files: len(c.Files)}
}
// History is a ref's commits, newest first.
type History struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
Path string `json:"path,omitempty"`
Page int `json:"page"`
Commits []CommitLine `json:"commits"`
More bool `json:"more"`
}
func history(ctx context.Context, f *Forge, owner, name, ref, path string, limit, page int) (*History, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
q := url.Values{}
if ref != "" {
q.Set("sha", ref)
}
if path != "" {
q.Set("path", path)
}
q.Set("limit", fmt.Sprint(limit))
q.Set("page", fmt.Sprint(page))
q.Set("stat", "false")
q.Set("verification", "false")
q.Set("files", "false")
var got []forgeCommit
if err := f.GetJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
return nil, err
}
h := &History{Repository: owner + "/" + name, Ref: ref, Path: path, Page: page, Commits: []CommitLine{}, More: len(got) >= limit}
for _, c := range got {
l := c.line()
l.Files = 0
h.Commits = append(h.Commits, l)
}
return h, nil
}
// Comparison is what head holds that base does not.
type Comparison struct {
Repository string `json:"repository"`
Base string `json:"base"`
Head string `json:"head"`
Total int `json:"total_commits"`
Commits []CommitLine `json:"commits"`
Files []FileLine `json:"files"`
Said []string `json:"said"`
}
// FileLine is one file a compare touched, and how the last commit that touched it changed it.
type FileLine struct {
Path string `json:"path"`
Status string `json:"status"`
}
func compare(ctx context.Context, f *Forge, owner, name, base, head string) (*Comparison, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
b, err := segments("base", base)
if err != nil {
return nil, err
}
h, err := segments("head", head)
if err != nil {
return nil, err
}
if strings.Contains(base, "...") || strings.Contains(head, "...") {
return nil, refuse("a base or head holding ... cannot be compared: the forge splits on it. Nothing was sent")
}
var got struct {
Total int `json:"total_commits"`
Commits []forgeCommit `json:"commits"`
}
if err := f.GetJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
return nil, err
}
c := &Comparison{Repository: owner + "/" + name, Base: base, Head: head, Total: got.Total, Commits: []CommitLine{}, Files: []FileLine{}}
at := map[string]int{}
for _, fc := range got.Commits {
c.Commits = append(c.Commits, fc.line())
for _, file := range fc.Files {
if i, seen := at[file.Filename]; seen {
c.Files[i].Status = file.Status
continue
}
at[file.Filename] = len(c.Files)
c.Files = append(c.Files, FileLine{Path: file.Filename, Status: file.Status})
}
}
c.Said = append(c.Said, fmt.Sprintf("%s holds %d commits %s does not, touching %d files", head, got.Total, base, len(c.Files)))
if got.Total > len(got.Commits) {
c.Said = append(c.Said, fmt.Sprintf("the forge listed %d of the %d commits; the files are those of the commits listed", len(got.Commits), got.Total))
}
c.Said = append(c.Said, "a file's status is the last listed commit's change to it; for the lines, gitea_pull_request_diff or gitea_get_file")
return c, nil
}
// Review is one review of a pull request with its line comments.
type Review struct {
ID int64 `json:"id"`
By string `json:"by"`
State string `json:"state"`
Commit string `json:"commit"`
AtHead bool `json:"at_head"`
Submitted string `json:"submitted,omitempty"`
Stale bool `json:"stale,omitempty"`
Dismissed bool `json:"dismissed,omitempty"`
Official bool `json:"official,omitempty"`
Body string `json:"body,omitempty"`
Comments []LineComment `json:"comments"`
}
// LineComment is one comment a review left on a line.
type LineComment struct {
Path string `json:"path"`
Line int `json:"line,omitempty"`
OldLine int `json:"old_line,omitempty"`
By string `json:"by"`
Body string `json:"body"`
Commit string `json:"commit,omitempty"`
Resolved string `json:"resolved_by,omitempty"`
}
// Reviews is a pull request's reviews, oldest first.
type Reviews struct {
Repository string `json:"repository"`
Number int `json:"number"`
Head string `json:"head"`
Reviews []Review `json:"reviews"`
Said []string `json:"said"`
}
type forgeUser struct {
Login string `json:"login"`
}
func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Reviews, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
pr := fmt.Sprintf("%s/pulls/%d", repo, number)
var pull struct {
Head struct {
SHA string `json:"sha"`
} `json:"head"`
}
if err := f.GetJSON(ctx, pr, &pull); err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no pull request %d", owner, name, number)
}
return nil, err
}
out := &Reviews{Repository: owner + "/" + name, Number: number, Head: pull.Head.SHA, Reviews: []Review{}}
type forgeReview struct {
ID int64 `json:"id"`
User forgeUser `json:"user"`
State string `json:"state"`
Body string `json:"body"`
CommitID string `json:"commit_id"`
Submitted string `json:"submitted_at"`
Stale bool `json:"stale"`
Dismissed bool `json:"dismissed"`
Official bool `json:"official"`
Comments int `json:"comments_count"`
}
const page = 50
var all []forgeReview
for p := 1; ; p++ {
var batch []forgeReview
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
return nil, err
}
all = append(all, batch...)
if len(batch) < page || p >= 20 {
break
}
}
approvedAtHead := 0
for _, r := range all {
rv := Review{ID: r.ID, By: r.User.Login, State: r.State, Commit: r.CommitID, AtHead: r.CommitID != "" && r.CommitID == pull.Head.SHA,
Submitted: r.Submitted, Stale: r.Stale, Dismissed: r.Dismissed, Official: r.Official, Body: r.Body, Comments: []LineComment{}}
if r.Comments > 0 {
var cs []struct {
Path string `json:"path"`
Position int `json:"position"`
Original int `json:"original_position"`
User forgeUser `json:"user"`
Body string `json:"body"`
CommitID string `json:"commit_id"`
Resolver *forgeUser `json:"resolver"`
}
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
return nil, fmt.Errorf("review %d's line comments could not be read: %w", r.ID, err)
}
for _, c := range cs {
lc := LineComment{Path: c.Path, Line: c.Position, OldLine: c.Original, By: c.User.Login, Body: c.Body, Commit: c.CommitID}
if c.Resolver != nil {
lc.Resolved = c.Resolver.Login
}
rv.Comments = append(rv.Comments, lc)
}
}
if rv.AtHead && r.State == "APPROVED" && !r.Dismissed && !r.Stale {
approvedAtHead++
}
out.Reviews = append(out.Reviews, rv)
}
out.Said = append(out.Said, fmt.Sprintf("%d reviews; %d approve the head %s", len(out.Reviews), approvedAtHead, short(pull.Head.SHA)))
return out, nil
}
// Entry is one entry of a directory.
type Entry struct {
Name string `json:"name"`
Type string `json:"type"`
Size int64 `json:"size,omitempty"`
}
// Listing is a directory's entries at a ref.
type Listing struct {
Repository string `json:"repository"`
Path string `json:"path"`
Ref string `json:"ref,omitempty"`
Entries []Entry `json:"entries"`
}
func listDir(ctx context.Context, f *Forge, owner, name, path, ref string) (*Listing, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
path = strings.Trim(path, "/")
at := repo + "/contents"
if path != "" {
seg, err := segments("directory path", path)
if err != nil {
return nil, err
}
at += "/" + seg
}
if ref != "" {
at += "?ref=" + queryEscape(ref)
}
body, err := f.Do(ctx, "GET", at, nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no %q at %s", owner, name, path, orDefault(ref))
}
return nil, err
}
var got []struct {
Name string `json:"name"`
Type string `json:"type"`
Size int64 `json:"size"`
}
if err := jsonInto(body, &got); err != nil {
return nil, refuse("%s is a file, not a directory: gitea_get_file reads it", path)
}
l := &Listing{Repository: owner + "/" + name, Path: "/" + path, Ref: ref, Entries: []Entry{}}
for _, e := range got {
en := Entry{Name: e.Name, Type: e.Type}
if e.Type == "file" {
en.Size = e.Size
}
l.Entries = append(l.Entries, en)
}
return l, nil
}
func orDefault(ref string) string {
if ref == "" {
return "the default branch"
}
return ref
}
func jsonInto(body []byte, into any) error { return json.Unmarshal(body, into) }
// queryEscape is a query value as the forge reads it.
func queryEscape(v string) string { return url.QueryEscape(v) }
// deadline is how long one tool call may take: inside a module's 60-second ask.
const deadline = 50 * time.Second
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"errors"
"regexp"
"strings"
)
// ruleCovers is whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
// models/git/protected_branch.go Match and modules/glob, novox/hq issue 539): a name without a glob character
// covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any
// run within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]`
// one character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge
// cannot compile covers its own text exactly. The TypeScript bundle's ruleCovers (guard.ts) is the same rule.
func ruleCovers(rule, branch string) bool {
if !strings.ContainsAny(rule, `*?\[]{}`) {
return strings.EqualFold(rule, branch)
}
re, err := globRegexp(rule)
if err != nil {
return rule == branch
}
return re.MatchString(branch)
}
// globRegexp is the forge's glob as a regular expression, compiled as modules/glob compiles it with `/` as
// the separator.
func globRegexp(glob string) (*regexp.Regexp, error) {
g := []rune(glob)
var b strings.Builder
pos := 0
var compile func(sub bool) error
chars := func() error {
b.WriteByte('[')
if pos < len(g) && g[pos] == '!' {
pos++
b.WriteByte('^')
}
for pos < len(g) {
c := g[pos]
pos++
switch {
case c == ']':
b.WriteByte(']')
return nil
case c == '\\':
if pos >= len(g) {
return errors.New("unterminated character class escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
default:
b.WriteRune(c)
}
}
return errors.New("unterminated character class")
}
compile = func(sub bool) error {
for pos < len(g) {
c := g[pos]
pos++
if sub && c == '}' {
b.WriteByte(')')
return nil
}
switch c {
case '*':
if pos < len(g) && g[pos] == '*' {
pos++
b.WriteString(".*")
} else {
b.WriteString("[^/]*")
}
case '?':
b.WriteString("[^/]")
case '+':
b.WriteString(`\+`)
case '[':
if err := chars(); err != nil {
return err
}
case '{':
b.WriteByte('(')
if err := compile(true); err != nil {
return err
}
case ',':
if sub {
b.WriteByte('|')
} else {
b.WriteByte(',')
}
case '\\':
if pos >= len(g) {
return errors.New("no character to escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
case '.', '^', '$', '(', ')', '|':
b.WriteByte('\\')
b.WriteRune(c)
default:
b.WriteRune(c)
}
}
return nil
}
b.WriteByte('^')
if err := compile(false); err != nil {
return nil, err
}
b.WriteByte('$')
return regexp.Compile(b.String())
}
+206
View File
@@ -0,0 +1,206 @@
package main
import (
"archive/tar"
"bufio"
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"regexp"
"strings"
)
// gitea_search_code: the forge's API has no code search, so the repository's archive at a ref is read once
// and searched here (novox/hq issue 375). One request, nothing kept.
// Match is one line that matched.
type Match struct {
Path string `json:"path"`
Line int `json:"line"`
Text string `json:"text"`
}
// Found is a search's answer.
type Found struct {
Repository string `json:"repository"`
Ref string `json:"ref"`
Query string `json:"query"`
Matches []Match `json:"matches"`
Files int `json:"files_searched"`
Skipped []string `json:"skipped,omitempty"`
Truncated bool `json:"truncated"`
Stopped bool `json:"stopped_at_deadline,omitempty"`
Said []string `json:"said"`
}
// SearchAsk is one gitea_search_code call.
type SearchAsk struct {
Owner, Repo, Ref, Query, Path string
Regex, IgnoreCase bool
Max int
}
const (
maxFileSearched = 4 << 20 // a larger file is skipped and named
maxLineShown = 300
)
func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
repo, err := repoPath(a.Owner, a.Repo)
if err != nil {
return nil, err
}
if a.Query == "" {
return nil, refuse("query is required. Nothing was sent")
}
pattern := a.Query
if !a.Regex {
pattern = regexp.QuoteMeta(pattern)
}
if a.IgnoreCase {
pattern = "(?i)" + pattern
}
re, err := regexp.Compile(pattern)
if err != nil {
return nil, refuse("query is not a regular expression Go reads: %v. Nothing was sent", err)
}
ref := a.Ref
if ref == "" {
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.Empty || r.DefaultBranch == "" {
return &Found{Repository: a.Owner + "/" + a.Repo, Query: a.Query, Matches: []Match{}, Said: []string{"the repository is empty"}}, nil
}
ref = r.DefaultBranch
}
seg, err := segments("ref", ref)
if err != nil {
return nil, err
}
prefix := strings.Trim(a.Path, "/")
if prefix != "" {
if _, err := segments("path", prefix); err != nil {
return nil, err
}
}
body, err := f.Do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no ref %s", a.Owner, a.Repo, ref)
}
return nil, err
}
out := &Found{Repository: a.Owner + "/" + a.Repo, Ref: ref, Query: a.Query, Matches: []Match{}}
if err := searchArchive(ctx, body, prefix, re, a.Max, out); err != nil {
return nil, fmt.Errorf("the archive of %s at %s could not be read: %w", out.Repository, ref, err)
}
said := fmt.Sprintf("%d matches in %d files searched", len(out.Matches), out.Files)
switch {
case out.Stopped:
said = fmt.Sprintf("the search stopped at its deadline of %s after %d files, with %d matches: the rest of the "+
"archive was not searched. Narrow with path", deadline, out.Files, len(out.Matches))
case out.Truncated:
said = fmt.Sprintf("the first %d matches, and more were found: narrow with path or the query", len(out.Matches))
}
out.Said = append(out.Said, said)
if len(out.Skipped) > 0 {
out.Said = append(out.Said, fmt.Sprintf("%d files larger than %d MiB were not searched: they are under skipped", len(out.Skipped), maxFileSearched>>20))
}
return out, nil
}
// searchArchive reads a gzipped tar the forge made — every path under one top directory — and collects the
// lines that match, from text files under the prefix. It stops at the call's deadline (novox/hq issue 539):
// what it found by then is kept, and out.Stopped says the rest was not searched.
func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
zr, err := gzip.NewReader(bytes.NewReader(gz))
if err != nil {
return err
}
tr := tar.NewReader(ctxReader{ctx, zr})
stopped := func(err error) bool {
if ctx.Err() != nil && (err == nil || errors.Is(err, ctx.Err())) {
out.Stopped = true
return true
}
return false
}
for {
if stopped(nil) {
return nil
}
h, err := tr.Next()
if err == io.EOF {
return nil
}
if stopped(err) {
return nil
}
if err != nil {
return err
}
if h.Typeflag != tar.TypeReg {
continue
}
_, path, found := strings.Cut(h.Name, "/")
if !found || path == "" {
continue
}
if prefix != "" && path != prefix && !strings.HasPrefix(path, prefix+"/") {
continue
}
if h.Size > maxFileSearched {
out.Skipped = append(out.Skipped, path)
continue
}
content, err := io.ReadAll(io.LimitReader(tr, maxFileSearched+1))
if stopped(err) {
return nil
}
if err != nil {
return err
}
head := content
if len(head) > 8000 {
head = head[:8000]
}
if bytes.IndexByte(head, 0) >= 0 {
continue // binary
}
out.Files++
sc := bufio.NewScanner(bytes.NewReader(content))
sc.Buffer(make([]byte, 64<<10), maxFileSearched+1)
for n := 1; sc.Scan(); n++ {
line := sc.Text()
if !re.MatchString(line) {
continue
}
if len(out.Matches) >= max {
out.Truncated = true
return nil
}
if len(line) > maxLineShown {
line = strings.ToValidUTF8(line[:maxLineShown], "") + "…"
}
out.Matches = append(out.Matches, Match{Path: path, Line: n, Text: line})
}
}
}
// ctxReader is a reader that ends when its context does: decompressing is the long part of a search.
type ctxReader struct {
ctx context.Context
r io.Reader
}
func (c ctxReader) Read(p []byte) (int, error) {
if err := c.ctx.Err(); err != nil {
return 0, err
}
return c.r.Read(p)
}
+20 -104
View File
@@ -2,45 +2,33 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"time"
"gitea/internal/forgehttp"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin
// account, which the vault delivered into a file this module owns (novox/hq ADR 0086). Basic
// authentication, so this bundle mints nothing and keeps nothing: the TypeScript bundle's token is its
// own. The password is read when a call is made and never logged, answered or put in an error.
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the forge bundle beside it). Basic authentication, so this bundle mints
// nothing and keeps nothing: the TypeScript bundle's token is its own.
// Forge is the forge's HTTP interface.
type Forge struct {
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
Owner string // the package registry's owner, as the seat serves it
HTTP *http.Client
forgehttp.Client
Owner string // the package registry's owner, as the seat serves it
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
f := &Forge{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
Owner: os.Getenv("MESH_NPM_OWNER"),
}
var missing []string
for name, v := range map[string]string{"MESH_GITEA_URL": f.URL, "MESH_GITEA_ADMIN_USER": f.User,
"MESH_GITEA_ADMIN_PASSWORD_FILE": f.PasswordFile, "MESH_NPM_OWNER": f.Owner} {
if v == "" {
missing = append(missing, name)
}
c, missing := forgehttp.FromEnv()
f := &Forge{Client: c, Owner: os.Getenv("MESH_NPM_OWNER")}
if f.Owner == "" {
missing = append(missing, "MESH_NPM_OWNER")
}
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
@@ -49,86 +37,14 @@ func ForgeFromEnv() (*Forge, error) {
}
// errNotFound is the forge answering 404.
var errNotFound = errors.New("not found")
// statusError is an answer the forge gave that is not a success, said without the request's credential.
type statusError struct {
method, path string
status int
said string
}
func (e *statusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
}
func (f *Forge) client() *http.Client {
if f.HTTP != nil {
return f.HTTP
}
return &http.Client{Timeout: 20 * time.Second}
}
func (f *Forge) password() (string, error) {
raw, err := os.ReadFile(f.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
var errNotFound = forgehttp.ErrNotFound
// maxBody is the largest answer read from the forge.
const maxBody = 32 << 20
const maxBody = forgehttp.MaxBody
// do sends one request and answers the body; 404 is errNotFound.
// do sends one request without a body and answers the body; 404 is errNotFound.
func (f *Forge) do(ctx context.Context, method, path string) ([]byte, error) {
password, err := f.password()
if err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, nil)
if err != nil {
return nil, err
}
req.SetBasicAuth(f.User, password)
req.Header.Set("Accept", "application/json")
res, err := f.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
// One byte past the cap is read, so a body the cap cut is told from one that fits it exactly: a
// cut body is refused, never parsed as if it were whole.
body, err := io.ReadAll(io.LimitReader(res.Body, maxBody+1))
if err != nil {
return nil, err
}
if len(body) > maxBody {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, maxBody>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(body))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &statusError{method, path, res.StatusCode, said}
}
return body, nil
}
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
body, err := f.do(ctx, http.MethodGet, path)
if err != nil {
return err
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s is not readable: %w", path, err)
}
return nil
return f.Do(ctx, method, path, nil, maxBody)
}
// PackageVersion is one version of one package, as the forge lists it.
@@ -146,7 +62,7 @@ func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
for page := 1; page <= 1000; page++ {
var batch []PackageVersion
path := fmt.Sprintf("/api/v1/packages/%s?type=npm&page=%d&limit=%d", url.PathEscape(f.Owner), page, pageSize)
if err := f.getJSON(ctx, path, &batch); err != nil {
if err := f.GetJSON(ctx, path, &batch); err != nil {
return nil, err
}
all = append(all, batch...)
@@ -166,7 +82,7 @@ func (f *Forge) Size(ctx context.Context, name, version string) (int64, error) {
var files []struct {
Size int64 `json:"size"`
}
if err := f.getJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
if err := f.GetJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
return 0, err
}
var total int64
@@ -182,7 +98,7 @@ func (f *Forge) DistTags(ctx context.Context, name string) (map[string]string, e
DistTags map[string]string `json:"dist-tags"`
}
path := fmt.Sprintf("/api/packages/%s/npm/%s", url.PathEscape(f.Owner), url.PathEscape(name))
if err := f.getJSON(ctx, path, &meta); err != nil {
if err := f.GetJSON(ctx, path, &meta); err != nil {
return nil, err
}
return meta.DistTags, nil
@@ -208,7 +124,7 @@ func (f *Forge) Repositories(ctx context.Context) ([]Repository, error) {
var found struct {
Data []Repository `json:"data"`
}
if err := f.getJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
if err := f.GetJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
return nil, err
}
all = append(all, found.Data...)
@@ -239,7 +155,7 @@ func (f *Forge) Files(ctx context.Context, repo Repository) ([]string, error) {
}
path := fmt.Sprintf("/api/v1/repos/%s/git/trees/%s?recursive=true&page=%d&per_page=1000",
repoPath(repo.FullName), url.PathEscape(repo.DefaultBranch), page)
if err := f.getJSON(ctx, path, &tree); err != nil {
if err := f.GetJSON(ctx, path, &tree); err != nil {
return nil, err
}
for _, e := range tree.Tree {
@@ -15,6 +15,8 @@ import (
"sync"
"testing"
"time"
"gitea/internal/forgehttp"
)
// fakeForge answers the forge's interface from maps, and records every delete.
@@ -38,7 +40,7 @@ func (f *fakeForge) serve() *Forge {
if err := os.WriteFile(file, []byte(f.password+"\n"), 0o600); err != nil {
f.t.Fatal(err)
}
return &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
}
var (
@@ -501,7 +503,7 @@ func TestAnAnswerTheCapCutIsRefused(t *testing.T) {
if err := os.WriteFile(file, []byte("x\n"), 0o600); err != nil {
t.Fatal(err)
}
f := &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
f := &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
if _, err := f.do(context.Background(), http.MethodGet, "/big"); err == nil || !strings.Contains(err.Error(), "not read whole") {
t.Fatalf("a cut answer: %v", err)
}
+47
View File
@@ -13,6 +13,9 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus } from "./client.js";
import { checkName } from "./guard.js";
/** How a command is run: docker on the forge's machine, or a test's. */
export type Runner = (file: string, args: string[]) => Promise<{ stdout: string; code: number }>;
@@ -106,3 +109,47 @@ export function deliveryStatus(context: string, state: string, description: stri
return { state: state as "pending" | "success" | "error" | "failure" | "warning", context, description: d,
...(target && /^https?:\/\//.test(target) ? { target_url: target } : {}) };
}
/** What the status tool writes with: the forge, and the gate writer the merge check's statuses go through. */
export interface StatusWriters {
setCommitStatus(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void>;
gates: { rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> };
}
/**
* The `gitea_commit_status` tool. **The merge check's pending goes through the module's one writer** (novox/hq issue
* 456): a recheck's pending on mesh/merge-gate or mesh/repo-check takes its turn on the head's chain (gate.ts), so an
* old run's verdict in flight lands before it and a redelivered one is dropped. Served from the watcher's bundle
* (index.ts), the process that holds that writer: each bundle the runtime serves is its own process. The mesh's other
* statuses (mesh/delivery, mesh/delivery-group) are written as they were.
*/
export function commitStatusTool(writers: StatusWriters): ToolDefinition {
return {
name: "gitea_commit_status",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. The merge check's statuses (mesh/merge-gate, mesh/repo-check) take pending only — a recheck putting them back until the fresh verdict sets them — on the head's whole sha, in their turn after the check's earlier writes.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit" },
context: { type: "string", description: "the status's name, mesh/…" },
state: { type: "string", description: "pending | success | error | failure | warning" },
description: { type: "string", description: "one line" },
target_url: { type: "string", description: "the page it links to (the pull request)" },
},
run: async (args) => {
const owner = checkName("owner", args.owner), repo = checkName("name", args.repo), sha = String(args.sha ?? "");
const status = deliveryStatus(String(args.context), String(args.state), String(args.description ?? ""),
args.target_url ? String(args.target_url) : undefined);
if (mergeCheckContexts.has(status.context)) {
// The writer keeps a head by its whole sha, as the forge announced it: a short one would be another head.
if (!/^([0-9a-f]{40}|[0-9a-f]{64})$/i.test(sha)) {
throw new Error(`${JSON.stringify(args.sha)} is not a head's whole sha: the merge check's status names the head whole. Nothing was sent`);
}
await writers.gates.rechecked(owner, repo, sha, status);
} else {
await writers.setCommitStatus(owner, repo, sha, status);
}
return { set: status };
},
};
}
+115 -5
View File
@@ -8,6 +8,14 @@
// between its read and its write. So this module is the one writer of mesh/merge-gate, and every write of one head's
// gate goes through one chain: a verdict is marked the moment it is heard, before it waits its turn, and a "building"
// still waiting behind or taking its turn after that mark is dropped, never written.
//
// **A recheck's pending is written here too** (novox/hq issue 456). mesh-delivery's recheck puts the gate back to
// pending through the gitea_commit_status tool, which this module serves from the watcher's own bundle so that it
// reaches this writer: the pending clears the verdict mark, so the new run's "building" is said, and queues behind an
// old verdict in flight, so that verdict lands first. A verdict is weighed by the ask it answers — the controller's
// checked.id, `build-<unix nanoseconds>` of when the check was asked — against when the check now pending was opened:
// an older run's verdict, redelivered after its write failed, is dropped, never written over the pending. After a
// restart, when the check was opened is read from the forge's own history of the gate.
import { CHECK_CONTEXT, pendingWords } from "./pulls.js";
import type { CommitStatus } from "./client.js";
@@ -27,15 +35,53 @@ export interface GateForge {
setCommitStatus(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void>;
/** The head's mesh/merge-gate, newest: its state and when it was set; undefined when it has none. */
gateOf(owner: string, repo: string, sha: string): Promise<{ state: string; updated_at?: string } | undefined>;
/** The head's mesh/merge-gate statuses, every one the forge keeps, newest first (novox/hq issue 456). */
gateHistory(owner: string, repo: string, sha: string): Promise<GateStatus[]>;
}
/** One status of a head's gate as the forge keeps it: its state, its words, and when it was set. */
export interface GateStatus {
state: string;
description?: string;
updated_at?: string;
}
/**
* When a check was asked, from its id as mesh-controller's link.NewBuildID writes it: `build-<unix nanoseconds>`.
* Undefined for an id of any other shape, whose ask time is not known (novox/hq issue 456).
*/
export function askedAt(id: string | undefined): number | undefined {
const m = /^build-(\d{1,20})$/.exec(String(id ?? ""));
if (!m) return undefined;
const ms = Number(BigInt(m[1]) / 1_000_000n);
return ms > 0 ? ms : undefined;
}
/**
* When the check now pending on a head was opened, from the gate's history, newest first: the newest status, past the
* "building" words this writer says only inside an open check, when it is pending; undefined when the gate holds a
* verdict, holds nothing, or its time cannot be read (novox/hq issue 456).
*/
export function openedAt(history: GateStatus[]): number | undefined {
for (const s of history) {
if (s.state !== "pending") return undefined;
if ((s.description ?? "").startsWith(BUILDING)) continue;
const t = Date.parse(s.updated_at ?? "");
return Number.isNaN(t) ? undefined : t;
}
return undefined;
}
/** How far a build machine's clock may stand behind this one's when a start is weighed against the pending words. */
const CLOCK_SLACK_MS = 60_000;
/** How the words of a check a machine is running begin. */
const BUILDING = "building on ";
/** The words of a check a machine is running, in the time the operator reads. */
export function buildingWords(on: string, at: Date): string {
const two = (n: number) => String(n).padStart(2, "0");
return `building on ${on} since ${two(at.getHours())}:${two(at.getMinutes())}`;
return `${BUILDING}${on} since ${two(at.getHours())}:${two(at.getMinutes())}`;
}
/** Whether a repository as the mesh clones it (`http://forge/novox/hq.git`) is owner/repo. */
@@ -50,6 +96,8 @@ interface Head {
pendingAt?: number;
/** A verdict was heard for the head: nothing pending is written after it. */
verdict: boolean;
/** Which verdict set the mark: one dropped in its turn takes the mark back only while it is still its own. */
mark?: object;
/** The chain the head's writes take their turns on. */
chain: Promise<void>;
}
@@ -94,12 +142,38 @@ export class GateWriter {
pending(owner: string, repo: string, sha: string): Promise<void> {
const h = this.head(owner, repo, sha);
const at = this.now();
h.pendingAt = at.getTime();
h.verdict = false;
this.opened(h, at);
return this.said(h, `${owner}/${repo}@${sha.slice(0, 8)}: could not say a check is pending`, () =>
this.forge.setCommitStatus(owner, repo, sha, { state: "pending", context: CHECK_CONTEXT, description: pendingWords(at) }));
}
/** A new check of the head opened now: no verdict heard for it yet, and nothing older than it is its own. */
private opened(h: Head, at: Date): void {
h.pendingAt = at.getTime();
h.verdict = false;
h.mark = undefined;
}
/**
* A recheck's pending, asked through the gitea_commit_status tool (novox/hq issue 456): on mesh/merge-gate it opens
* a new check of the head — the verdict mark cleared, so the new run's "building" is said — and takes its turn
* after every write already asked for, an old verdict in flight included, so it lands over that verdict. Any
* other status of the merge check (mesh/repo-check, which the verdict also writes) takes its turn on the same
* chain. A write that fails fails the caller, and the head is as it was before it.
*/
rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> {
const h = this.head(owner, repo, sha);
const write = () => this.forge.setCommitStatus(owner, repo, sha, status);
if (status.context !== CHECK_CONTEXT || status.state !== "pending") return this.turn(h, write);
const before = { pendingAt: h.pendingAt, verdict: h.verdict, mark: h.mark };
const at = this.now();
this.opened(h, at);
return this.turn(h, write).catch((err) => {
if (h.pendingAt === at.getTime() && !h.verdict) Object.assign(h, before);
throw err;
});
}
/**
* A build machine took an ask of this head: said while no verdict was heard, and only for the check now pending —
* a start from before the pending words were set (a run before a recheck, the stream's history) is not this check's.
@@ -130,10 +204,46 @@ export class GateWriter {
/**
* The verdict: marked at once, so no pending word is written after it, then written in its turn — last. A write
* that fails fails the caller, so the event is offered again.
*
* `id` is the ask it answers (checked.id): a verdict of a check asked before the check now pending was opened is
* an older run's — redelivered after its write failed, or heard late — and is dropped, never written over the
* pending (novox/hq issue 456). When this process opened no check of the head (it restarted since), the opening
* is read from the forge's history of the gate in the verdict's turn; a history that cannot be read writes the
* verdict, as one with no readable id is: a verdict is never lost to what cannot be weighed.
*/
verdict(owner: string, repo: string, sha: string, write: () => Promise<void>): Promise<void> {
verdict(owner: string, repo: string, sha: string, write: () => Promise<void>, id?: string): Promise<void> {
const h = this.head(owner, repo, sha);
const asked = askedAt(id);
const head = `${owner}/${repo}@${sha.slice(0, 8)}`;
if (asked !== undefined && h.pendingAt !== undefined && older(asked, h.pendingAt)) {
console.log(`[gitea] ${head}: the verdict of ${id} is an older run's than the check now pending; not written`);
return Promise.resolve();
}
const before = h.verdict;
const mark = {};
h.verdict = true;
return this.turn(h, write);
h.mark = mark;
return this.turn(h, async () => {
if (asked !== undefined && h.pendingAt === undefined) {
let opened: number | undefined;
try {
opened = openedAt(await this.forge.gateHistory(owner, repo, sha));
} catch (err) {
console.error(`[gitea] ${head}: the gate's history could not be read, so the verdict of ${id} is written ` +
`unweighed — ${err instanceof Error ? err.message : err}`);
}
if (opened !== undefined && older(asked, opened)) {
if (h.mark === mark) h.verdict = before;
console.log(`[gitea] ${head}: the verdict of ${id} is an older run's than the check the forge holds pending; not written`);
return;
}
}
await write();
});
}
}
/** Whether a check asked at `asked` was asked before a check opened at `opened`, past what the clocks may differ by. */
function older(asked: number, opened: number): boolean {
return asked < opened - CLOCK_SLACK_MS;
}
+98 -13
View File
@@ -164,21 +164,106 @@ export function judgeApiCall(method: string, rawPath: string, body: unknown): Ap
};
}
/** Whether a protection rule's name — a branch, or a glob of them (`*`, `**`) — covers a branch. */
/**
* Whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
* models/git/protected_branch.go Match and modules/glob; novox/hq issue 539): a name without a glob character
* covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any run
* within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]` one
* character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge cannot
* compile covers its own text exactly. The forge bundle's ruleCovers (cmd/forge/rules.go) is the same rule, and
* both are held to one table of answers (test/rule-cases.json).
*
* The forge hands the compiled glob to Go's regexp, which reads some things JavaScript's reads otherwise: a
* backslash before a letter or digit (`\d`, `\w`, `\pL`, `\b`, `\A` are classes or anchors there), a class
* that opens with `]` (`[]a]` holds `]` and `a` there, an empty class here) or with `^` (a negation there),
* and a POSIX class (`[[:digit:]]`).
* A rule using any of them is taken to cover every branch: refusing a write the forge would take is safe,
* taking one it guards is not.
*/
export function ruleCovers(rule: string, branch: string): boolean {
if (rule === branch) return true;
if (!/[*?[]/.test(rule)) return false;
let re = "";
for (let i = 0; i < rule.length; i++) {
const c = rule[i];
if (c === "*" && rule[i + 1] === "*") {
re += ".*";
i++;
} else if (c === "*") re += "[^/]*";
else if (c === "?") re += "[^/]";
else re += c.replace(/[.+^${}()|[\]\\]/g, "\\$&");
if (!/[*?\\[\]{}]/.test(rule)) return rule.toLowerCase() === branch.toLowerCase();
if (goReadsOtherwise(rule)) return true;
const re = globRegExp(rule);
return re ? re.test(branch) : rule === branch;
}
/** Whether a glob holds what Go's regexp reads otherwise than JavaScript's (see ruleCovers): fail closed. */
function goReadsOtherwise(rule: string): boolean {
return /\\[\p{L}\p{N}]/u.test(rule) || /\[!?[\]^]/.test(rule) || /\[[^\]]*\[:/.test(rule);
}
/** The forge's glob as a regular expression, compiled as its modules/glob compiles it with `/` as separator. */
function globRegExp(glob: string): RegExp | null {
const g = Array.from(glob);
let out = "";
let pos = 0;
// A character as itself: escaped when it is syntax to a regular expression, and only then — the `u` flag
// refuses any other escape, and `-` may be escaped only inside a class.
const lit = (c: string) => c.replace(/[.*+?^${}()|[\]\\/]/g, "\\$&");
const litInClass = (c: string) => c.replace(/[\]\\[^-]/g, "\\$&");
const chars = (): boolean => {
out += "[";
if (g[pos] === "!") {
pos++;
out += "^";
}
while (pos < g.length) {
const c = g[pos++];
if (c === "]") {
out += "]";
return true;
}
if (c === "\\") {
if (pos >= g.length) return false;
out += litInClass(g[pos++]);
} else out += c === "[" || c === "^" ? "\\" + c : c;
}
return false;
};
const compile = (sub: boolean): boolean => {
while (pos < g.length) {
const c = g[pos++];
if (sub && c === "}") {
out += ")";
return true;
}
switch (c) {
case "*":
if (g[pos] === "*") {
pos++;
out += ".*";
} else out += "[^/]*";
break;
case "?":
out += "[^/]";
break;
case "[":
if (!chars()) return false;
break;
case "{":
out += "(";
if (!compile(true)) return false;
break;
case ",":
out += sub ? "|" : ",";
break;
case "\\":
if (pos >= g.length) return false;
out += lit(g[pos++]);
break;
default:
out += lit(c);
}
}
// An alternative left open is a glob the forge cannot compile.
return !sub;
};
if (!compile(false)) return null;
try {
return new RegExp(`^${out}$`, "u");
} catch {
return null;
}
return new RegExp(`^${re}$`).test(branch);
}
/** Refuses a file written to the default branch or to a branch a protection rule names. */
+18 -1
View File
@@ -17,6 +17,9 @@
// Consumes node-build-agent.started — a build machine took an ask — and says, on the head a pending check is of,
// which machine builds it since when (novox/hq issue 446). This module is the one writer of mesh/merge-gate.
//
// Serves gitea_commit_status from this bundle, not the tools', so a recheck's pending reaches that one writer
// (novox/hq issue 456).
//
// Every pull request the forge holds is announced, whatever its repository: the controller holds the
// module graph and decides what is checked — a repository is never asked to opt in.
//
@@ -29,6 +32,8 @@
// the forge for an immediacy nobody asked for is not.
import { emit, on } from "@novox/mesh-sdk/events";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { commitStatusTool } from "./delivery.js";
import { GiteaClient, movedSince } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, commentFor, headsToAnnounce, statusesFor, type Announced, type Checked,
type RepoCheckFacts, reposToLook } from "./pulls.js";
@@ -344,8 +349,17 @@ async function repoCheckFacts(client: GiteaClient, c: Checked, base: string | un
const gates = new GateWriter({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gateOf: async (owner, repo, sha) => (await (gitea as GiteaClient).headChecks(owner, repo, sha))[CHECK_CONTEXT],
gateHistory: (owner, repo, sha) => (gitea as GiteaClient).contextHistory(owner, repo, sha, CHECK_CONTEXT),
});
// **The status tool is served here, beside the writer** (novox/hq issue 456): the runtime starts each bundle as its own
// process, so a recheck's pending written from the tools' bundle bypassed this writer's chain. Without a forge to
// reach, nothing is served, as the tools' bundle serves nothing then.
registerModuleTools("gitea", () => (gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
})] : []));
// **A build machine took an ask** (the build seat's `started`, novox/hq ADR 0157): said on the head it checks, when
// the head is an open pull request's this module announced. Any other build — a branch, a merge's — says nothing.
async function buildStarted(event: { body: unknown }): Promise<void> {
@@ -392,7 +406,10 @@ if (gitea) {
await on("mesh-controller.checked", (event) => {
const c = (event.body ?? {}) as Checked;
// A head's own verdict takes its turn after every write of that head's gate already asked for (gate.ts).
if (!c.group && c.owner && c.repo && c.commit) return gates.verdict(c.owner, c.repo, c.commit, () => setVerdict(client, event));
// A verdict of a check asked before the one now pending is an older run's, and dropped (novox/hq issue 456).
if (!c.group && c.owner && c.repo && c.commit) {
return gates.verdict(c.owner, c.repo, c.commit, () => setVerdict(client, event), c.id);
}
return setVerdict(client, event);
});
await on("node-build-agent.started", buildStarted);
@@ -0,0 +1,151 @@
// Package forgehttp is the forge's HTTP interface as this module's Go bundles reach it (cmd/forge,
// cmd/npm-registry): the forge on its own machine, as its admin account, whose password the vault delivered
// into a file this module owns (novox/hq ADR 0086), by basic authentication. The password is read when a call
// is made and never logged, answered or put in an error. One copy for both bundles (novox/hq issue 539).
package forgehttp
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
// MaxBody is the largest JSON answer read from the forge.
const MaxBody = 32 << 20
// Client is the forge's HTTP interface.
type Client struct {
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
HTTP *http.Client // when nil, a client with Timeout
Timeout time.Duration // one request's limit when HTTP is nil; 20 s when zero
}
// FromEnv is the forge as the manifest hands it to a bundle, and the names of the variables it left unset.
func FromEnv() (Client, []string) {
c := Client{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
}
var missing []string
for _, kv := range [][2]string{{"MESH_GITEA_URL", c.URL}, {"MESH_GITEA_ADMIN_USER", c.User},
{"MESH_GITEA_ADMIN_PASSWORD_FILE", c.PasswordFile}} {
if kv[1] == "" {
missing = append(missing, kv[0])
}
}
return c, missing
}
// ErrNotFound is the forge answering 404.
var ErrNotFound = errors.New("not found")
// StatusError is an answer the forge gave that is not a success, said without the request's credential.
type StatusError struct {
Method, Path string
Status int
Said string
}
func (e *StatusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.Status, e.Method, e.Path, e.Said)
}
func (c *Client) client() *http.Client {
if c.HTTP != nil {
return c.HTTP
}
timeout := c.Timeout
if timeout <= 0 {
timeout = 20 * time.Second
}
return &http.Client{Timeout: timeout}
}
func (c *Client) password() (string, error) {
raw, err := os.ReadFile(c.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
// Do sends one request, with payload as its JSON body when not nil, and answers the body, read up to limit;
// 404 is ErrNotFound. One byte past the limit is read, so a body the limit cut is told from one that fits it
// exactly: a cut body is refused, never read as if it were whole.
func (c *Client) Do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
password, err := c.password()
if err != nil {
return nil, err
}
var body io.Reader
if payload != nil {
raw, err := json.Marshal(payload)
if err != nil {
return nil, err
}
body = bytes.NewReader(raw)
}
req, err := http.NewRequestWithContext(ctx, method, c.URL+path, body)
if err != nil {
return nil, err
}
req.SetBasicAuth(c.User, password)
req.Header.Set("Accept", "application/json")
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
res, err := c.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
if err != nil {
return nil, err
}
if int64(len(got)) > limit {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, ErrNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(got))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &StatusError{method, path, res.StatusCode, said}
}
return got, nil
}
// GetJSON reads one answer into into.
func (c *Client) GetJSON(ctx context.Context, path string, into any) error {
return c.SendJSON(ctx, http.MethodGet, path, nil, into)
}
// SendJSON sends one request and reads its answer into into, when into is not nil.
func (c *Client) SendJSON(ctx context.Context, method, path string, payload, into any) error {
body, err := c.Do(ctx, method, path, payload, MaxBody)
if err != nil {
return err
}
if into == nil {
return nil
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
}
return nil
}
+16
View File
@@ -257,6 +257,22 @@
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_NPM_OWNER": "novox"
}
},
{
"name": "forge",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/forge",
"binary": "forge",
"loads": [
"forge"
],
"env": {
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GITEA_ADMIN_USER": "mesh-admin",
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret"
}
}
]
},
+164 -4
View File
@@ -18,12 +18,19 @@ const startedOnShanks = { id: "build-1791680941048622697", repository: "http://1
function aForge() {
const kept: Record<string, { state: string; description: string; updated_at: string }> = {};
const written: string[] = [];
/** Every mesh/merge-gate status the forge was given, newest first, as gitea keeps each one. */
const history: { state: string; description: string; updated_at: string }[] = [];
let failNext = 0;
let hold: Promise<void> | null = null;
let release = () => {};
let readHold: Promise<void> | null = null;
let readRelease = () => {};
return {
kept, written,
kept, written, history,
/** The next writes fail: a request the forge did not take. */
failWrites(n = 1) {
failNext = n;
},
/** The next writes wait until released: a request in flight. */
holdWrites() {
hold = new Promise<void>((r) => (release = r));
@@ -34,7 +41,12 @@ function aForge() {
},
async setCommitStatus(_o: string, _r: string, _sha: string, s: { state: string; context: string; description: string }) {
if (hold) await hold;
if (failNext > 0) {
failNext--;
throw new Error("the forge answered 502");
}
kept[s.context] = { state: s.state, description: s.description, updated_at: new Date().toISOString() };
if (s.context === "mesh/merge-gate") history.unshift(kept[s.context]);
written.push(`${s.state} ${s.description}`);
},
/** Reads answer what the forge held when asked, as late as the reads are held: the answer of a request in flight. */
@@ -50,6 +62,9 @@ function aForge() {
if (readHold) await readHold;
return held;
},
async gateHistory(_o: string, _r: string, _sha: string) {
return history.map((h) => ({ ...h }));
},
};
}
@@ -138,14 +153,159 @@ test("a start from before the check now pending (a run before a recheck, the str
await gates.pending("novox", "hq", SHA);
await gates.verdict("novox", "hq", SHA, () =>
forge.setCommitStatus("novox", "hq", SHA, { state: "failure", context: "mesh/merge-gate", description: "error" }));
now = new Date("2026-10-11T01:30:00Z"); // checked again: pending once more
await gates.pending("novox", "hq", SHA);
now = new Date("2026-10-11T01:30:00Z"); // checked again: the recheck's pending, as the tool writes it (issue 456)
await gates.rechecked("novox", "hq", SHA, { state: "pending", context: "mesh/merge-gate", description: "checking again: flaky" });
assert.equal(await gates.started("novox", "hq", SHA, startedOnShanks), false, "the first run's start");
assert.match(forge.kept["mesh/merge-gate"].description, /^waiting in the build queue since /);
assert.equal(forge.kept["mesh/merge-gate"].description, "checking again: flaky");
assert.equal(await gates.started("novox", "hq", SHA, { ...startedOnShanks, at: "2026-10-11T01:31:00Z" }), true,
"the second run's start");
});
// **A recheck's pending goes through the one writer** (novox/hq issue 456). mesh-delivery's recheck puts the gate back
// to pending through the gitea_commit_status tool; before, that write bypassed the head's chain: the first run's
// verdict mark stayed set, so the new run never said "building", and an old run's verdict could land over the pending.
/** The ask of a check, as the controller's link.NewBuildID names it: when it was asked, in unix nanoseconds. */
const askOf = (at: string) => `build-${BigInt(Date.parse(at)) * 1_000_000n}`;
const firstRun = askOf("2026-10-11T01:09:01Z");
const recheckAt = new Date("2026-10-11T01:30:00Z");
const secondRun = askOf("2026-10-11T01:30:02Z");
const checkingAgain = { state: "pending" as const, context: "mesh/merge-gate", description: "checking again: main moved on" };
const writeOf = (forge: ReturnType<typeof aForge>, state: string, description: string) => () =>
forge.setCommitStatus("novox", "hq", SHA, { state, context: "mesh/merge-gate", description });
const gateOf = (forge: ReturnType<typeof aForge>) =>
`${forge.kept["mesh/merge-gate"].state} ${forge.kept["mesh/merge-gate"].description}`;
test("an old run's verdict redelivered after a recheck's pending is not written over it", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.failWrites(); // the first run's verdict is not taken: the event is offered again
await assert.rejects(gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun));
now = recheckAt;
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun); // redelivered
assert.equal(gateOf(forge), "pending checking again: main moved on", forge.written.join(" | "));
// The recheck's own verdict is written.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail: 0 of 4 compose"), secondRun);
assert.equal(gateOf(forge), "failure fail: 0 of 4 compose");
});
test("an old run's verdict in flight when the recheck comes lands before the recheck's pending", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.holdWrites(); // the first run's verdict is being written when the recheck comes
const verdict = gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
await new Promise((r) => setImmediate(r));
now = recheckAt;
const pending = gates.rechecked("novox", "hq", SHA, checkingAgain);
forge.release();
await Promise.all([verdict, pending]);
assert.equal(gateOf(forge), "pending checking again: main moved on", forge.written.join(" | "));
});
test("a rechecked head says where its new run is: building on which machine", async () => {
const { GateWriter, buildingWords } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
now = recheckAt;
await gates.rechecked("novox", "hq", SHA, checkingAgain);
const at = "2026-10-11T01:30:03Z";
assert.equal(await gates.started("novox", "hq", SHA, { ...startedOnShanks, id: secondRun, at }), true);
assert.equal(forge.kept["mesh/merge-gate"].description, buildingWords("shanks", new Date(at)));
});
test("the recheck's own verdict is written, even asked a little before the pending by another machine's clock", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
const gates = new GateWriter(forge, () => recheckAt);
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), askOf("2026-10-11T01:29:30Z"));
assert.equal(gateOf(forge), "success pass");
// A verdict whose ask cannot be read is written: a verdict is never lost to what cannot be weighed.
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail"), "an id of another shape");
assert.equal(gateOf(forge), "failure fail");
});
test("after a restart, an old run's verdict is weighed against the check the forge's history holds open", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
// What the forge kept before this process started: the announcement, the first run's verdict never written,
// the recheck's pending, and the second run's "building" over it.
forge.history.unshift({ state: "pending", description: "waiting in the build queue since 03:08", updated_at: "2026-10-11T01:08:50Z" });
forge.history.unshift({ state: "pending", description: "checking again: main moved on", updated_at: "2026-10-11T01:30:00Z" });
forge.history.unshift({ state: "pending", description: "building on shanks since 03:30", updated_at: "2026-10-11T01:30:05Z" });
forge.kept["mesh/merge-gate"] = forge.history[0];
const gates = new GateWriter(forge, announcedAt);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(forge.kept["mesh/merge-gate"].description, "building on shanks since 03:30", forge.written.join(" | "));
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail"), secondRun);
assert.equal(gateOf(forge), "failure fail");
});
test("a recheck the forge does not take fails its caller, and the head is as it was", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.failWrites();
await assert.rejects(gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun));
now = recheckAt;
forge.failWrites();
await assert.rejects(gates.rechecked("novox", "hq", SHA, checkingAgain));
// The recheck was refused, so its check is not asked: the first run's verdict, offered again, is the head's.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(gateOf(forge), "success pass");
});
test("the status tool writes the merge check's pending through the writer, and the mesh's other statuses as they were", async () => {
const { GateWriter } = await import("../gate.ts");
const { commitStatusTool } = await import("../delivery.ts");
const forge = aForge();
const direct: string[] = [];
const gates = new GateWriter(forge, () => recheckAt);
const tool = commitStatusTool({
setCommitStatus: async (_o, _r, _sha, s) => void direct.push(`${s.context} ${s.state}`),
gates,
});
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/merge-gate", state: "pending", description: "checking again: x" });
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/repo-check", state: "pending", description: "checking again: x" });
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/delivery", state: "success", description: "delivered" });
assert.equal(gateOf(forge), "pending checking again: x");
assert.equal(forge.kept["mesh/repo-check"].state, "pending");
assert.deepEqual(direct, ["mesh/delivery success"]);
// The redelivered first run's verdict is the writer's to drop.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(gateOf(forge), "pending checking again: x");
// A head named short is another head to the writer: refused, nothing sent.
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: SHA.slice(0, 12), context: "mesh/merge-gate", state: "pending" }),
/whole sha/);
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/merge-gate", state: "success" }),
/set by its verdict only/);
});
test("when a check was asked is read from its id; the opening of the check now pending from the gate's history", async () => {
const { askedAt, openedAt } = await import("../gate.ts");
assert.equal(askedAt("build-1791680941048622697"), Date.parse("2026-10-11T01:09:01.048Z"));
assert.equal(askedAt("build-mesh-catalog"), undefined);
assert.equal(askedAt(undefined), undefined);
assert.equal(openedAt([{ state: "pending", description: "building on ace since 03:31", updated_at: "2026-10-11T01:31:00Z" },
{ state: "pending", description: "checking again: x", updated_at: "2026-10-11T01:30:00Z" }]), Date.parse("2026-10-11T01:30:00Z"));
assert.equal(openedAt([{ state: "success", description: "pass", updated_at: "2026-10-11T01:30:00Z" }]), undefined);
assert.equal(openedAt([]), undefined);
});
test("a head this process did not announce (it restarted) is said only while the forge holds its gate pending", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
+29 -1
View File
@@ -7,10 +7,13 @@
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { readFileSync } from "node:fs";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { commitStatusTool } from "../dist/delivery.js";
import { GateWriter } from "../dist/gate.js";
import { judgeApiCall, judgeProtectionChange, ruleCovers, routedPath } from "../dist/guard.js";
interface Sent { method: string; path: string; body: any }
@@ -97,7 +100,13 @@ const server = createServer(async (req, res) => {
await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r()));
after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const tools = getGiteaTools(new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t")));
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
// gitea_commit_status is served from the watcher's bundle, beside the merge check's one writer (novox/hq issue 456).
const tools = [...getGiteaTools(client), commitStatusTool({
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gates: new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [] }),
})];
const tool = (name: string) => tools.find((t) => t.name === name)!;
const writes = () => state.sent.map((s) => `${s.method} ${s.path}`);
@@ -138,6 +147,25 @@ test("gitea_api refuses a file written to the default branch or one a rule guard
assert.deepEqual(writes(), ["PUT /repos/novox/mesh-catalog/contents/modules/sudo/module.json"]);
});
test("a protection rule covers a branch as the forge decides it: classes, alternatives, a plain name in any case (issue 539)", async () => {
// One table with the forge bundle (cmd/forge/rules.go): `forge` is the forge's answer; `guard` is where this
// guard deliberately covers more — a rule Go's regexp reads otherwise than JavaScript's — and never less.
const cases = JSON.parse(readFileSync(new URL("./rule-cases.json", import.meta.url), "utf8")) as
Array<{ rule: string; branch: string; forge: boolean; guard?: boolean }>;
assert.ok(cases.length >= 100);
for (const c of cases) {
assert.ok(c.guard === undefined || (c.guard && !c.forge), `${c.rule}: the guard may only cover more than the forge`);
assert.equal(ruleCovers(c.rule, c.branch), c.guard ?? c.forge, `${c.rule} covers ${c.branch}`);
}
const put = (branch: string) => tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/contents/README.md",
method: "PUT", body: { content: "e30=", sha: "x", branch } });
state.rules.push({ rule_name: "release-[ab]" }, { rule_name: "{stable,lts}/*" });
await assert.rejects(put("release-a"), /protection rule release-\[ab\]/);
await assert.rejects(put("lts/2"), /protection rule \{stable,lts\}\/\*/);
await assert.rejects(put("MAIN"), /protection rule main/);
assert.deepEqual(writes(), []);
});
test("gitea_api still writes what an author writes", async () => {
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/issues/3/comments", method: "POST", body: { body: "hi" } });
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/pulls/7/reviews", method: "POST", body: { event: "COMMENT" } });
+285
View File
@@ -0,0 +1,285 @@
[
{"rule":"main","branch":"main","forge":true,"why":"a plain name covers its branch"},
{"rule":"main","branch":"mainx","forge":false},
{"rule":"main","branch":"MAIN","forge":true,"why":"a plain name covers in any case (EqualFold)"},
{"rule":"Main","branch":"main","forge":true},
{"rule":"release/*","branch":"release/1.0","forge":true,"why":"* stays within a segment"},
{"rule":"release/*","branch":"release/a/b","forge":false},
{"rule":"release/**","branch":"release/a/b","forge":true,"why":"** crosses segments"},
{"rule":"v?","branch":"v1","forge":true},
{"rule":"v?","branch":"v/","forge":false,"why":"? is not the separator"},
{"rule":"a.b","branch":"axb","forge":false,"why":". is itself"},
{"rule":"Feat/*","branch":"feat/x","forge":false,"why":"a glob is matched with case"},
{"rule":"release-[ab]","branch":"release-a","forge":true,"why":"a class"},
{"rule":"release-[ab]","branch":"release-c","forge":false},
{"rule":"release-[!ab]","branch":"release-c","forge":true,"why":"a negated class"},
{"rule":"release-[!ab]","branch":"release-a","forge":false},
{"rule":"v[0-9].x","branch":"v7.x","forge":true,"why":"a range"},
{"rule":"v[0-9].x","branch":"vA.x","forge":false},
{"rule":"{main,master}","branch":"master","forge":true,"why":"alternatives"},
{"rule":"{main,master}","branch":"mainline","forge":false},
{"rule":"rel/{a,b}/*","branch":"rel/b/1","forge":true},
{"rule":"rel/{a,b}/*","branch":"rel/c/1","forge":false},
{"rule":"a\\*","branch":"a*","forge":true,"why":"an escape"},
{"rule":"a\\*","branch":"ab","forge":false},
{"rule":"a+b","branch":"a+b","forge":true,"why":"+ is itself"},
{"rule":"a+b","branch":"aab","forge":false},
{"rule":"{open","branch":"{open","forge":true,"why":"a glob the forge cannot compile covers its own text"},
{"rule":"[open","branch":"[open","forge":true},
{"rule":"[open","branch":"o","forge":false},
{"rule":"release-\\d","branch":"release-1","forge":true,"why":"Go reads \\d as a digit"},
{"rule":"release-\\d","branch":"release-d","forge":false,"guard":true},
{"rule":"v\\w*","branch":"v2","forge":true},
{"rule":"main\\s","branch":"main","forge":false,"guard":true},
{"rule":"[]a]","branch":"a","forge":true,"why":"a class opening with ] holds ]"},
{"rule":"[]a]","branch":"]","forge":true},
{"rule":"[]a]","branch":"b","forge":false,"guard":true},
{"rule":"rel-[\\d]","branch":"rel-1","forge":true},
{"rule":"v[\\pL]","branch":"va","forge":true,"why":"\\p is a Unicode class"},
{"rule":"v\\pL","branch":"va","forge":true},
{"rule":"[[:alpha:]]","branch":"a","forge":true,"why":"a POSIX class"},
{"rule":"rel-[[:digit:]]","branch":"rel-1","forge":true},
{"rule":"rel-[[:digit:]]","branch":"rel-x","forge":false,"guard":true},
{"rule":"[^+]*","branch":"-[1","forge":true,"why":"a class opening with ^ is negated in Go"},
{"rule":"d[^](","branch":"d[^](","forge":true},
{"rule":"]?","branch":":,1},","forge":false},
{"rule":"Ab,","branch":"Ab,","forge":true},
{"rule":"\\:[}-|","branch":",{","forge":false},
{"rule":"^,*.d::","branch":"/x","forge":false},
{"rule":"","branch":"A*:","forge":false},
{"rule":"!|A{","branch":":.{]","forge":false},
{"rule":"","branch":"]éb","forge":false},
{"rule":"/{)w","branch":"/{)w","forge":true},
{"rule":"*^w]","branch":"}!.","forge":false},
{"rule":"*),^?*d","branch":"*),^?*d","forge":true},
{"rule":"A)^}!","branch":"","forge":false},
{"rule":"}^d?","branch":"1\\.","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"$?[","branch":"$?[","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"+","forge":true},
{"rule":"","branch":"},[","forge":false},
{"rule":"xA[d:a)","branch":"]1.!bx","forge":false},
{"rule":"^/}(","branch":"","forge":false},
{"rule":"-|x.x1?","branch":"-|x.x1?","forge":true},
{"rule":"x+)A]-A","branch":"x+)A]-A","forge":true},
{"rule":"x*a","branch":"x*a","forge":true},
{"rule":"$xb","branch":"}é","forge":false},
{"rule":"wAx?/a,","branch":"","forge":false},
{"rule":"[,","branch":"[,","forge":true},
{"rule":"}!","branch":"}!","forge":true},
{"rule":"*:|\\w","branch":"","forge":false,"guard":true},
{"rule":"","branch":"","forge":true},
{"rule":"é","branch":"é","forge":true},
{"rule":"..(","branch":"{\\{+","forge":false},
{"rule":"*|{","branch":"1.}-","forge":false},
{"rule":"!+,","branch":"}++-*","forge":false},
{"rule":"x\\({","branch":"*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-{+-","branch":"?-{+-","forge":true},
{"rule":"/-{é|$","branch":"b!.[","forge":false},
{"rule":"a-+dxA","branch":"1AA,xA","forge":false},
{"rule":"]|(","branch":"]|(","forge":true},
{"rule":"}","branch":"{!x{d","forge":false},
{"rule":"/{*](","branch":"!]","forge":false},
{"rule":"($+{-}|","branch":"/","forge":false},
{"rule":"(-b.-1}","branch":"*A*+,1","forge":false},
{"rule":"1[d|)wa","branch":"+}[!}}","forge":false},
{"rule":"w","branch":":.","forge":false},
{"rule":"?}\\*","branch":"","forge":false},
{"rule":"?$*","branch":"?$*","forge":true},
{"rule":"","branch":"!/{béa","forge":false},
{"rule":"","branch":"A1","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"!a$1/b{","branch":"!a$1/b{","forge":true},
{"rule":":dww/|d","branch":":dww/|d","forge":true},
{"rule":"!|*}","branch":"xéxé{b","forge":false},
{"rule":"1é||","branch":"a+.","forge":false},
{"rule":"/+],","branch":"!-\\[{é","forge":false},
{"rule":"a}(","branch":"a}(","forge":true},
{"rule":"|A?.","branch":"}.","forge":false},
{"rule":"d$a^d","branch":"*{","forge":false},
{"rule":"-ba$+A","branch":"-ba$+A","forge":true},
{"rule":"-.|,?a!","branch":"-.|,?a!","forge":true},
{"rule":"A","branch":"A","forge":true},
{"rule":"w","branch":"w","forge":true},
{"rule":".!1!+a","branch":".!1!+a","forge":true},
{"rule":"+$w:}","branch":"+$w:}","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":")]","branch":")]","forge":true},
{"rule":"a})^,}1","branch":"a})^,}1","forge":true},
{"rule":":[:!:","branch":":[:!:","forge":true},
{"rule":"w?w:]1!","branch":"\\.]-","forge":false},
{"rule":"*$/","branch":"*$/","forge":true},
{"rule":"é](","branch":"é](","forge":true},
{"rule":"x$a/","branch":"x$a/","forge":true},
{"rule":"*)b+[^","branch":"*)b+[^","forge":true},
{"rule":"/]/?","branch":"/]/?","forge":true},
{"rule":"{x,A","branch":"!Ab","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"-,","branch":"xé}A[.","forge":false},
{"rule":"w)Ad","branch":"w)Ad","forge":true},
{"rule":":??","branch":":??","forge":true},
{"rule":"x/xé|","branch":"x/xé|","forge":true},
{"rule":"","branch":"1!","forge":false},
{"rule":"!![é[","branch":"x","forge":false},
{"rule":"wb$","branch":"wb$","forge":true},
{"rule":")","branch":")","forge":true},
{"rule":"Aw:)(1","branch":"{b\\,","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"1*}?d","branch":"1*}?d","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-b.","branch":"?-b.","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"d","branch":",aé1","forge":false},
{"rule":"+","branch":"+{","forge":false},
{"rule":"]d","branch":"]}:/bA","forge":false},
{"rule":"*?+:|","branch":"!é.","forge":false},
{"rule":"}$d","branch":"}$d","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"a$+b)b","branch":"d}","forge":false},
{"rule":"?$[/A","branch":"","forge":false},
{"rule":"é){","branch":"é){","forge":true},
{"rule":"$(d","branch":"{é}","forge":false},
{"rule":"b(","branch":"b(","forge":true},
{"rule":"^^+]","branch":"^^+]","forge":true},
{"rule":",)","branch":",)","forge":true},
{"rule":"d-.A-","branch":"d-.A-","forge":true},
{"rule":"$","branch":"\\/","forge":false},
{"rule":"}+[\\|\\$","branch":"x[!b","forge":false},
{"rule":"(d1aé!é","branch":"A[-","forge":false},
{"rule":"x","branch":"1","forge":false},
{"rule":"x","branch":"dx","forge":false},
{"rule":"]|)]","branch":"]|)]","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"[.*?","branch":"[.*?","forge":true},
{"rule":"[?","branch":"[?","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"),/","branch":"a1,,1","forge":false},
{"rule":"}+","branch":"}+","forge":true},
{"rule":"$d\\,d","branch":"$d\\,d","forge":false},
{"rule":"d1","branch":"d1","forge":true},
{"rule":"-!^.:","branch":"-!^.:","forge":true},
{"rule":"x","branch":"+].A11","forge":false},
{"rule":"","branch":"-]","forge":false},
{"rule":"}]}.","branch":"","forge":false},
{"rule":"^{wx","branch":"]1a!","forge":false},
{"rule":"^","branch":"*]x,","forge":false},
{"rule":"(:x*x","branch":"","forge":false},
{"rule":"|[/*a/","branch":"}","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"1\\^","branch":"!é!x{","forge":false},
{"rule":"(.b,AA+","branch":"(.b,AA+","forge":true},
{"rule":"xx","branch":"","forge":false},
{"rule":"d)^):","branch":"/}!","forge":false},
{"rule":"|","branch":",é*]*[","forge":false},
{"rule":"d*d:*1","branch":"\\b,","forge":false},
{"rule":":]","branch":"[","forge":false},
{"rule":"([|!\\^é","branch":"([|!\\^é","forge":true},
{"rule":"$[w)x-*","branch":"","forge":false},
{"rule":")","branch":")","forge":true},
{"rule":".d^)?","branch":"x\\é","forge":false},
{"rule":"1|{w","branch":"1|{w","forge":true},
{"rule":"é$,(]$$","branch":"é$,(]$$","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"","forge":false},
{"rule":"]","branch":"","forge":false},
{"rule":"+:a:","branch":"+:a:","forge":true},
{"rule":"/*","branch":"d.}","forge":false},
{"rule":"|/ba1-","branch":"|/ba1-","forge":true},
{"rule":"","branch":"+","forge":false},
{"rule":"A","branch":"A","forge":true},
{"rule":",-!]","branch":",-!]","forge":true},
{"rule":"$*d!)","branch":"","forge":false},
{"rule":")é+x.d","branch":")é+x.d","forge":true},
{"rule":"éa$\\[1","branch":"1/,","forge":false},
{"rule":"}?1|]d","branch":"}?1|]d","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"-1.*+","branch":"-1.*+","forge":true},
{"rule":"Aw|{","branch":"Aw|{","forge":true},
{"rule":"adAAAa","branch":"A","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"d/^{:","branch":"d/^{:","forge":true},
{"rule":"é:(,^","branch":"é:(,^","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"!ébé({+","branch":"!ébé({+","forge":true},
{"rule":"x","branch":"x","forge":true},
{"rule":"|","branch":",:*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"x|","branch":"x|","forge":true},
{"rule":"a","branch":"aaxéb,","forge":false},
{"rule":"/w-),1é","branch":"/w-),1é","forge":true},
{"rule":"|*)^,","branch":"|*)^,","forge":true},
{"rule":"){é.!b.","branch":"a-}:","forge":false},
{"rule":"(\\","branch":"(\\","forge":true},
{"rule":".w(\\","branch":"+","forge":false},
{"rule":"/d[","branch":"/d[","forge":true},
{"rule":"(}$x!}","branch":"a,bb","forge":false},
{"rule":"w{","branch":"w{","forge":true},
{"rule":"[+a\\A(+","branch":"[+a\\A(+","forge":true},
{"rule":"(1^{)bb","branch":"(1^{)bb","forge":true},
{"rule":"^1","branch":"é-}","forge":false},
{"rule":"*,+}1):","branch":"*,+}1):","forge":true},
{"rule":"[A[(.x,","branch":"]db!*","forge":false},
{"rule":"","branch":"Ad}","forge":false},
{"rule":".","branch":".","forge":true},
{"rule":"w!é{x","branch":"é.{","forge":false},
{"rule":"$\\^","branch":"bx]+","forge":false},
{"rule":"1)","branch":"+*d","forge":false},
{"rule":"x)}/])","branch":"]","forge":false},
{"rule":",a}.1}","branch":",a}.1}","forge":true},
{"rule":"-a{(","branch":"-a{(","forge":true},
{"rule":"1","branch":"1","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"}[é$","branch":"]d-!","forge":false},
{"rule":"xé{é+?","branch":"xé{é+?","forge":true},
{"rule":"/A1!","branch":"xAA\\::","forge":false},
{"rule":"1][b-.(","branch":"{/-b![","forge":false},
{"rule":"(d}(","branch":"(d}(","forge":true},
{"rule":"{*}","branch":"!ébd","forge":true},
{"rule":"()^^)","branch":"()^^)","forge":true},
{"rule":"1(1A{$","branch":"*,[","forge":false},
{"rule":"A:!^\\","branch":"A:!^\\","forge":true},
{"rule":"-]:b]","branch":"-]:b]","forge":true},
{"rule":",","branch":",","forge":true},
{"rule":"[-:]","branch":"[-:]","forge":false},
{"rule":"","branch":"é:b1:d","forge":false},
{"rule":"1\\!]?d","branch":"","forge":false},
{"rule":"","branch":"bb!]+","forge":false},
{"rule":"/da","branch":"b]d[","forge":false},
{"rule":",]","branch":",]","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"a]{A-","branch":"-+","forge":false},
{"rule":"A?A^1","branch":"A?A^1","forge":true},
{"rule":"\\-$","branch":"","forge":false},
{"rule":"+","branch":"*[}[","forge":false},
{"rule":",b?{dw","branch":",b?{dw","forge":true},
{"rule":"d[$b1","branch":"/","forge":false},
{"rule":"})|xAA","branch":"é,!","forge":false},
{"rule":"]--","branch":"]--","forge":true},
{"rule":"],/b","branch":"],/b","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"*aé","branch":"*aé","forge":true},
{"rule":"{","branch":"{","forge":true},
{"rule":":aé","branch":"","forge":false},
{"rule":".\\}","branch":".\\}","forge":false},
{"rule":"wA]*","branch":"\\]!d{+","forge":false},
{"rule":"a/1x$/","branch":"[1!:/A","forge":false},
{"rule":"[.wb}d.","branch":"-!*..a","forge":false},
{"rule":".[{()a","branch":"+}","forge":false},
{"rule":"]/!:,","branch":"{Aéa","forge":false},
{"rule":"é{ba[-","branch":"],,1d/","forge":false},
{"rule":"(w*","branch":"1}.!","forge":false},
{"rule":"{?","branch":"1","forge":false},
{"rule":")","branch":",+!","forge":false},
{"rule":":","branch":"{/+A][","forge":false},
{"rule":".1{)","branch":"a\\bA","forge":false},
{"rule":"w-1+x1-","branch":"*]1}bb","forge":false},
{"rule":"xb$$","branch":"éx:]-:","forge":false},
{"rule":"++$!?*$","branch":"bA","forge":false},
{"rule":"]{","branch":"[xd*x.","forge":false}
]
+1 -1
View File
@@ -426,7 +426,7 @@ test("the tools register once there is a way to a token, and the first call mint
"gitea_list_branches",
"gitea_delete_branch",
"gitea_branch_protection_get", "gitea_branch_protection_set",
"gitea_note_append", "gitea_delivery_view", "gitea_commit_statuses", "gitea_open_pulls", "gitea_contains", "gitea_commit_status",
"gitea_note_append", "gitea_delivery_view", "gitea_commit_statuses", "gitea_open_pulls", "gitea_contains",
"gitea_list_labels", "gitea_create_label",
"gitea_api",
],
+3 -20
View File
@@ -11,7 +11,7 @@
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { emit } from "@novox/mesh-sdk/events";
import { GiteaClient, type BranchProtection } from "../client.js";
import { appendNote, deliveryStatus, run, viewBody, viewComment } from "../delivery.js";
import { appendNote, run, viewBody, viewComment } from "../delivery.js";
import { containedFrom, isCommit, mergeableUnder, newestByContext } from "../checks.js";
import {
MERGE_METHODS, Refused, checkName, judgeApiCall, judgeContentsWrite, judgeDelete, judgeProtectionChange, pathSegments, routedPath, sentQuery,
@@ -639,25 +639,8 @@ function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return { commit, in: within, contains, ahead };
},
},
{
name: "gitea_commit_status",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. The merge check's statuses (mesh/merge-gate, mesh/repo-check) take pending only — a recheck putting them back until the fresh verdict sets them.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit" },
context: { type: "string", description: "the status's name, mesh/…" },
state: { type: "string", description: "pending | success | error | failure | warning" },
description: { type: "string", description: "one line" },
target_url: { type: "string", description: "the page it links to (the pull request)" },
},
run: async (args) => {
const status = deliveryStatus(String(args.context), String(args.state), String(args.description ?? ""),
args.target_url ? String(args.target_url) : undefined);
await gitea.setCommitStatus(String(args.owner), String(args.repo), String(args.sha), status);
return { set: status };
},
},
// gitea_commit_status is served from the watcher's bundle (index.ts), beside the merge check's one writer
// (novox/hq issue 456).
// ---- Labels ----
{
@@ -366,6 +366,7 @@ func (h *Holder) words() []func() {
}
sort.Strings(walks)
var asks []func()
h.waitsWhy = map[string]string{}
for _, walk := range walks {
want := h.turning[walk]
var waiting []string
@@ -377,11 +378,20 @@ func (h *Holder) words() []func() {
}
if len(waiting) > 0 {
sort.Strings(waiting)
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s carried by it not at its turn", walk,
strings.Join(waiting, ", "))
h.waitsWhy[walk] = strings.Join(waiting, ", ") + " carried by it not at its turn"
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s", walk, h.waitsWhy[walk])
continue
}
ids, why := append([]string(nil), want.ids...), strings.Join(want.whys, "; ")
// **Never the word for a walk that carries a stopped change** (ADR 0299, review of #221): it is asked to
// leave the merge out first (leaveStopped).
if stopped, _ := h.carriesStopped(walk); len(stopped) > 0 {
h.waitsWhy[walk] = "it carries " + strings.Join(stopped, ", ") + ", whose delivery was stopped: it is " +
"asked to leave that merge out first"
h.Logf("[mesh-delivery] the walk %s is not let go: it carries %s, whose delivery was stopped", walk,
strings.Join(stopped, ", "))
continue
}
if h.asking[walk] {
continue // its word is being asked
}
@@ -412,3 +422,94 @@ func (h *Holder) words() []func() {
h.turning = nil
return asks
}
// endedMovingNothing is whether a delivery ended delivered with nothing for a walk to move: merged, its plan
// moving no module the mesh holds, and no walk followed.
func endedMovingNothing(d *Delivery) bool {
return d.State == Delivered && d.MergedAs != "" && d.Walk == nil && d.MovesNothing()
}
// endedAsNothingToMove are the deliveries a walk answers that ended delivered as nothing for a walk to move (novox/hq
// issues 362, 529: the nzbgeek case); empty when it answers any other. Such a walk is held by nothing: said, never
// let go by this owner. Under the lock.
func (h *Holder) endedAsNothingToMove(w Walk) []string {
var out []string
for _, d := range h.deliveries {
_, answers := walkAnswers(w, d)
if !answers && (d.Walk == nil || d.Walk.ID != w.ID) {
continue
}
if !endedMovingNothing(d) {
return nil
}
out = append(out, d.ID)
}
sort.Strings(out)
return out
}
// WalksWaiting is `stalled` asked with `walks` (novox/hq issue 529): for each walk waiting for this owner's word,
// what holds it, in the shape of a stalled line naming the walk — each held delivery it carries (State held, its
// reason and the verb that releases it), else why its word was not given at the last tick (State published), else
// that no live delivery holds it (State unheld). A walk whose word is being given is not listed. The controller's
// `plans` says these instead of pointing at a hand start; the self-check never raises a condition for them.
func (h *Holder) WalksWaiting() []StalledLine {
h.mu.Lock()
defer h.mu.Unlock()
now := h.Now()
var out []StalledLine
for id, s := range h.walks {
w := s.walk
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.asking[id] {
continue
}
waited := now.Sub(w.Created).Round(time.Second).String()
var held []StalledLine
live := false
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != id {
continue
}
live = true
if d.State == Held {
held = append(held, StalledLine{ID: d.ID, Number: d.Number, State: Held, Walk: id, For: waited,
Bound: "0s", H2: "none: held for a person — the operator's", Says: HeldLine(d)})
}
}
switch {
case len(held) > 0:
out = append(out, held...)
case h.waitsWhy[id] != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: "not let go yet: " + h.waitsWhy[id]})
case !live:
says := "no delivery that has not ended follows it or is answered by it, so nothing gives it its word: " +
"`mesh-delivery.show` for the deliveries of " + w.Repository
if ended := h.endedAsNothingToMove(w); len(ended) > 0 {
// The nzbgeek case (novox/hq issues 362, 529): its merges' deliveries ended as nothing for a walk to
// move, yet the walk moves modules — nothing judged those, so nothing here lets it go.
says = "held by nothing: its merges' deliveries ended as nothing-to-move (" + strings.Join(ended, ", ") +
"), yet the walk moves " + strings.Join(walkModules(w), ", ") + " — nothing gives it its word"
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: id, For: waited,
Bound: "0s", H2: "none: no delivery holds the walk — the operator's", Says: says})
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Walk != out[j].Walk {
return out[i].Walk < out[j].Walk
}
return out[i].ID < out[j].ID
})
return out
}
// walkModules are the modules a walk's record names, sorted.
func walkModules(w Walk) []string {
out := make([]string, 0, len(w.Modules))
for m := range w.Modules {
out = append(out, m)
}
sort.Strings(out)
return out
}
@@ -94,8 +94,18 @@ type fakeController struct {
asked int
delivers []string
stops []string
plans int
down bool
// withouts are the stops that left merges out (novox/hq ADR 0299): "<walk> without <merge>,…".
withouts []string
// refuseWithout refuses every stop with merges to leave out, as a controller before ADR 0299 does; mergedAt are
// the merges' times, by commit, for what a later merge contains.
refuseWithout bool
mergedAt map[string]time.Time
// refuseStop refuses every whole stop too; moves are the modules each merge moves, by commit, for what a left-out
// merge shares (ADR 0299).
refuseStop bool
moves map[string][]string
plans int
down bool
// onCheck runs as a check is asked, before it is answered: what the forge held at that moment.
onCheck func()
// planned is what Plan answers instead of "builds app", when set.
@@ -103,7 +113,8 @@ type fakeController struct {
}
func newFakeController() *fakeController {
return &fakeController{walks: map[string]Walk{}, held: true}
return &fakeController{walks: map[string]Walk{}, held: true, mergedAt: map[string]time.Time{},
moves: map[string][]string{}}
}
func (c *fakeController) Plan(repository, base, head string, paths, dirs, removed []string) (*DeliveryPlan, error) {
@@ -177,7 +188,13 @@ func (c *fakeController) Stop(walk, why, by string) error {
return errors.New("down")
}
c.stops = append(c.stops, walk)
if w, known := c.walks[walk]; known {
if c.refuseStop {
return errors.New("the controller refuses")
}
if w, known := c.walks[walk]; known && !w.Open() {
// As the controller: only a walk that is building or rolling is stopped (`delivery-stop`).
return fmt.Errorf("%s is already %s", walk, w.State)
} else if known {
w.State = walkFailed
if w.Delivery == nil {
w.Delivery = &WalkDelivery{}
@@ -188,6 +205,128 @@ func (c *fakeController) Stop(walk, why, by string) error {
}
return nil
}
// StopWithout ends a walk without the merges named, as the controller does (novox/hq ADR 0299): each merge named,
// and each later merge of its repository by merge time (mergedAt), is left out; the others are put in a batch,
// "<walk>-again", assembling until a test cuts it. Given a batch, the merges left out go to a stopped record,
// "<walk>-left", and the batch goes on with the others. refuseWithout refuses every such stop, as an older
// controller does.
func (c *fakeController) StopWithout(walk, why, by string, without []string) (StopAnswer, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return StopAnswer{}, errors.New("down")
}
c.withouts = append(c.withouts, walk+" without "+strings.Join(without, ","))
if c.refuseWithout {
return StopAnswer{}, errors.New(`delivery-stop takes no argument "without"`)
}
w, known := c.walks[walk]
if !known || !(w.Open() || w.IsBatch()) {
return StopAnswer{}, fmt.Errorf("%s is no open walk or batch", walk)
}
wd := WalkDelivery{}
if w.Delivery != nil {
wd = *w.Delivery
}
named := func(m WalkMerge) bool { return containsString(without, m.Repository+"@"+m.Commit) }
contains := func(m WalkMerge) string {
for _, s := range wd.Merges {
if named(s) && s.Repository == m.Repository && c.mergedAt[m.Commit].After(c.mergedAt[s.Commit]) {
return s.Commit
}
}
return ""
}
shares := func(m WalkMerge) (string, []string) {
for _, s := range wd.Merges {
if named(s) {
var both []string
for _, x := range c.moves[m.Commit] {
if containsString(c.moves[s.Commit], x) {
both = append(both, x)
}
}
if len(both) > 0 {
return s.Commit, both
}
}
}
return "", nil
}
a := StopAnswer{Stopped: walk}
var again, left []WalkMerge
for _, m := range wd.Merges {
s, both := shares(m)
switch {
case named(m):
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit})
left = append(left, m)
case len(both) > 0:
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: s, Shares: both})
left = append(left, m)
case contains(m) != "":
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: contains(m)})
left = append(left, m)
default:
m.Carried = ""
again = append(again, m)
a.WalkedAgain = append(a.WalkedAgain, m.Repository+"@"+m.Commit)
}
}
if len(a.LeftOut) == 0 {
return StopAnswer{}, fmt.Errorf("%s answers no merge %s", walk, strings.Join(without, ", "))
}
if w.IsBatch() && len(again) == 0 {
// Every merge left out: the batch itself is stopped, as the controller does.
wd.Stopped, wd.StoppedWhy, wd.LeftOut, wd.Batch = by, why, a.LeftOut, nil
w.Delivery, w.State = &wd, walkFailed
w.Revision++
c.walks[walk] = w
return a, nil
}
if w.IsBatch() {
rid := walk + "-left"
c.walks[rid] = Walk{ID: rid, Repository: left[0].Repository, Branch: "main", Commit: left[0].Commit,
Created: w.Created, State: walkFailed, Revision: 1, Modules: map[string]*WalkModule{},
Delivery: &WalkDelivery{Stopped: by, StoppedWhy: why, Merges: left, LeftOut: a.LeftOut, WalkedAgainBy: walk}}
wd.Merges = again
w.Delivery = &wd
w.Repository, w.Commit = again[0].Repository, again[0].Commit
w.Revision++
c.walks[walk] = w
a.Stopped, a.WalkedAgainBy = rid, walk
return a, nil
}
if len(again) > 0 {
id := walk + "-again"
closes := w.Created.Add(time.Minute)
c.walks[id] = Walk{ID: id, Repository: again[0].Repository, Branch: "main", Commit: again[0].Commit,
Created: w.Created.Add(time.Minute), State: walkAssembling, Revision: 1,
Modules: map[string]*WalkModule{}, Delivery: &WalkDelivery{Merges: again, Batch: &WalkBatch{ClosesAt: &closes}}}
wd.WalkedAgainBy, a.WalkedAgainBy = id, id
}
wd.Stopped, wd.StoppedWhy, wd.LeftOut = by, why, a.LeftOut
w.Delivery = &wd
w.State = walkFailed
w.Note = "stopped by " + by + ": " + why
w.Revision++
c.walks[walk] = w
return a, nil
}
// cut makes a batch the walk under the same id, waiting for its word, as the controller cuts it.
func (c *fakeController) cut(id string) Walk {
c.mu.Lock()
w := c.walks[id]
c.mu.Unlock()
wd := *w.Delivery
wd.Batch, wd.Awaits = nil, "mesh-delivery"
w.Delivery = &wd
w.State, w.Tiers, w.Modules = walkBuilding, [][]string{{"app"}}, map[string]*WalkModule{"app": {}}
return c.put(w)
}
func (c *fakeController) Walks(walk string) (bool, []Walk, error) {
c.mu.Lock()
defer c.mu.Unlock()
+268 -42
View File
@@ -55,8 +55,18 @@ type Holder struct {
walks map[string]seenWalk
orphans map[string]time.Time
turning map[string]*turnWant
// waitsWhy is, by walk, why the last tick did not give a waiting walk its word: a delivery it carries not at
// its turn, or a stopped change it carries (novox/hq issue 529). Said by WalksWaiting.
waitsWhy map[string]string
// asking are the walks whose word is being asked right now, outside the lock: not asked twice meanwhile.
asking map[string]bool
// leaving are, by walk, the merges a stop gathered to leave out of it, asked once (novox/hq ADR 0299).
leaving map[string][]string
// leaveAsked is when a walk was last asked to leave out the stopped merges it carries (leaveStopped).
leaveAsked map[string]leaveAsk
// leaveRefused is, by walk, what the controller answered when it refused both to leave a stopped merge out and
// to stop the walk whole.
leaveRefused map[string]string
}
// unmatched is a walk no delivery is known for yet: its merge not heard, or merged before this owner held.
@@ -96,6 +106,7 @@ func (h *Holder) init() {
h.written = map[string][]byte{}
h.dirtyGroup = map[string]bool{}
h.unmatched = map[string]unmatched{}
h.waitsWhy = map[string]string{}
h.askTried = map[string]time.Time{}
h.walkAsked = map[string]time.Time{}
h.unchecked = map[string]unchecked{}
@@ -642,6 +653,15 @@ func (h *Holder) linkWalk(d *Delivery, w Walk, carried string) {
d.LetGo, d.LetGoWhy = time.Time{}, "" // a new walk waits for a word of its own
}
d.Walk = w.Seen()
// Stopped without some of its merges (novox/hq ADR 0299): this delivery's merge was left out, maybe for
// containing a stopped one, or is walked again by the batch the record names.
if w.Delivery != nil && (w.Delivery.WalkedAgainBy != "" || len(w.Delivery.LeftOut) > 0) {
if o, out := w.leftOut(d.Repository, d.MergedAs); out {
d.Walk.Contains, d.Walk.Shares = o.Contains, o.Shares
} else {
d.Walk.WalkedAgainBy = w.Delivery.WalkedAgainBy
}
}
if carried != "" {
d.AnsweredBy = carried
}
@@ -821,6 +841,9 @@ func (h *Holder) Tick() {
for _, ask := range h.plan() {
ask()
}
for _, ask := range h.leaveStopped() {
ask()
}
}
// plan works out, under the lock, what to ask; each ask applies its own answer.
@@ -1807,7 +1830,13 @@ func mayCarry(c, d *Delivery, module string) bool {
c.Plan != nil && slices.Contains(c.Plan.Moved, module)
}
// stopInGroup stops a member after one that failed, and its walk if it has one open and carries nothing else live.
// stopInGroup stops a member after one that failed, and leaves its merge out of its walk if it has one open.
//
// **Only its own change** (novox/hq issue 459, ADR 0299): one walk carries every merge of its batch, so ending the
// member's walk ended every other delivery on it, which nobody stopped. Its merge is left out instead: the
// controller ends the walk and walks the other merges again without it, and without each later merge of its
// repository, which contains it. Every member a pass stops on one walk is gathered first, and the walk is asked
// once, by the first member's ask, after the pass: a member is never walked again because its stop came second.
func (h *Holder) stopInGroup(d *Delivery, why string) func() {
if d == nil || d.State.Final() {
return nil
@@ -1818,69 +1847,266 @@ func (h *Holder) stopInGroup(d *Delivery, why string) func() {
return nil
}
h.moved(d, []Transition{t})
if d.Walk == nil || d.Walk.State == walkDone || d.Walk.State == walkFailed || d.Walk.State == walkSuperseded {
// Its open walk; or, its walk stopped for another merge, the batch or walk that walks its merge again (review
// of #221): a merge walked again and then stopped by its group is left out there too.
var walk string
switch {
case d.Walk == nil:
return nil
case d.Walk.State == walkBuilding || d.Walk.State == walkRolling:
walk = d.Walk.ID
case d.walkedAgain():
walk = d.Walk.WalkedAgainBy
default:
return nil
}
walk, id := d.Walk.ID, d.ID
if !h.leave(walk, d) {
return nil // the first member left out of this walk asks for every one
}
return func() {
if _, _, err := h.endWalk(walk, id, why, byOwner); err != nil {
h.Logf("[mesh-delivery] the walk %s could not be stopped: %v", walk, err)
without, shared := h.leaving459(walk)
if !shared {
if err := h.Controller.Stop(walk, why, byOwner); err != nil {
h.Logf("[mesh-delivery] the walk %s could not be stopped: %v", walk, err)
}
return
}
h.Logf("[mesh-delivery] %s", h.stopLeaving(walk, why, byOwner, without))
}
}
// walkRiders are the live deliveries a walk carries, as "id (state)" in id order: those that follow it, and those
// its record answers and that follow it now. Under the lock.
func (h *Holder) walkRiders(walk string) []string {
// stopLeaving asks the controller to stop a walk or batch without merges (ADR 0299) and says what came of it. A
// refusal — an older controller takes no `without`, or the leave-out could not be made — falls back to stopping the
// whole walk: the stopped change is held back, and every delivery on it ends stopped with the reason (review of
// #221). Outside the lock.
func (h *Holder) stopLeaving(walk, why, by string, without []string) string {
h.mu.Lock()
h.askedToLeave(walk)
h.mu.Unlock()
a, err := h.Controller.StopWithout(walk, why, by, without)
if err == nil {
return fmt.Sprintf("the walk %s is stopped without %s; %s", walk, strings.Join(without, ", "), againWords(a))
}
refused := err
if err := h.Controller.Stop(walk, fmt.Sprintf("%s, and it could not be left out alone (%v)", why, refused), by); err != nil {
h.mu.Lock()
if h.leaveRefused == nil {
h.leaveRefused = map[string]string{}
}
h.leaveRefused[walk] = fmt.Sprintf("leaving it out (%v), and stopping the walk whole (%v)", refused, err)
h.mu.Unlock()
return fmt.Sprintf("the walk %s could not be stopped without %s (%v), nor whole (%v): `plans stop %s --why …` "+
"ends it by hand", walk, strings.Join(without, ", "), refused, err, walk)
}
return fmt.Sprintf("the walk %s could not be stopped without %s (%v), so it was stopped whole: every delivery on it "+
"ends stopped", walk, strings.Join(without, ", "), refused)
}
// carriesStopped are the merges a walk or batch carries whose deliveries were stopped, as `<repository>@<commit>`,
// and why the first was: a record that names a stopped delivery's merge and has not left it out would deliver a
// stopped change (review of #221). Under the lock.
func (h *Holder) carriesStopped(walk string) ([]string, string) {
s, known := h.walks[walk]
if !known || !(s.walk.Open() || s.walk.IsBatch()) || s.walk.Delivery == nil || len(s.walk.Delivery.Merges) == 0 {
return nil, ""
}
var out []string
for _, d := range h.deliveries {
if d.State.Final() {
why := ""
ids := make([]string, 0, len(h.deliveries))
for id := range h.deliveries {
ids = append(ids, id)
}
slices.Sort(ids)
for _, id := range ids {
d := h.deliveries[id]
// Stopped by a person or by its group, the acts that hold a change back: never one ended by what its walk
// did (the observed rows, which name nobody), such as a merge stranded after its walk (issue 474).
if d.State != Stopped || d.MergedAs == "" || len(d.Transitions) == 0 {
continue
}
on := d.Walk != nil && d.Walk.ID == walk
if !on && known && follows(d, s.walk) {
_, on = walkAnswers(s.walk, d)
if last := d.Transitions[len(d.Transitions)-1]; last.Event != EvStop || last.By == "" {
continue
}
if on {
out = append(out, fmt.Sprintf("%s (%s)", d.ID, d.State))
// By the merges it names alone: its own commit may be another's merge it no longer answers.
if !slices.ContainsFunc(s.walk.Delivery.Merges, func(m WalkMerge) bool {
return strings.EqualFold(m.Repository, d.Repository) && sameCommit(m.Commit, d.MergedAs)
}) {
continue
}
if _, out := s.walk.leftOut(d.Repository, d.MergedAs); out {
continue
}
out = append(out, d.Repository+"@"+d.MergedAs)
if why == "" {
why = fmt.Sprintf("%s was stopped: %s", d.ID, lastWhyOf(d))
}
}
sort.Strings(out)
return out
return out, why
}
// endWalk asks the controller to end a walk on a stopped delivery's word, unless the walk still carries a live
// delivery. Since the merge window (ADR 0276) one walk carries every merge of its batch, across repositories and
// groups, and the controller ends a walk whole: ending it for one stopped delivery ended the rollout of every other
// delivery it carries, for a reason not theirs (novox/hq issue 459). While one rides it, the walk goes on — what
// the stopped delivery's walk had asked still builds and is sent with it — and the stopped delivery says so on its
// commit, naming what it goes on for. The last delivery stopped on it ends it. Not under the lock.
func (h *Holder) endWalk(walk, stopped, why, by string) (ended bool, goesOn []string, err error) {
// lastWhyOf is the reason of a delivery's last transition.
func lastWhyOf(d *Delivery) string {
if len(d.Transitions) == 0 {
return ""
}
return d.Transitions[len(d.Transitions)-1].Why
}
// leaveAsk is when a walk was last asked to leave merges out, and at which revision of its record.
type leaveAsk struct {
at time.Time
revision int64
}
// askedToLeave notes an ask to leave merges out of a walk, at its record's revision: under the lock.
func (h *Holder) askedToLeave(walk string) {
if h.leaveAsked == nil {
h.leaveAsked = map[string]leaveAsk{}
}
h.leaveAsked[walk] = leaveAsk{at: h.Now(), revision: h.walks[walk].walk.Revision}
}
// leaveStopped asks, for every open walk or batch that carries a stopped delivery's merge, that the merge be left
// out (ADR 0299, review of #221): a merge walked again and then stopped, or stopped while its batch assembled, never
// ships. Asked once per record of it while it is carried; such a walk is never given its word (words).
func (h *Holder) leaveStopped() []func() {
h.mu.Lock()
goesOn = h.walkRiders(walk)
if len(goesOn) > 0 {
line := fmt.Sprintf("%s delivery %s is stopped, and its walk %s goes on: it also carries %s, which were not "+
"stopped; it is ended once the last delivery it carries is (novox/hq issue 459)",
h.Now().UTC().Format(time.RFC3339), stopped, walk, strings.Join(goesOn, ", "))
if d := h.deliveries[stopped]; d != nil {
now := h.Now()
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: oneLine(line), Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) {
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.MergedAs, Line: oneLine(line), Since: now})
defer h.mu.Unlock()
now := h.Now()
var asks []func()
held := map[string]string{}
for id := range h.walks {
if without, _ := h.carriesStopped(id); len(without) > 0 {
said := fmt.Sprintf("its walk %s is not let go: it carries %s, whose delivery was stopped, and is asked to "+
"leave it out (novox/hq ADR 0299)", id, strings.Join(without, ", "))
if r := h.leaveRefused[id]; r != "" {
said += "; the controller refused: " + r
}
held[id] = said
}
}
// What the waiting deliveries say: `waits`, and `stalled` past their bound.
for _, d := range h.deliveries {
said := ""
if d.Walk != nil && !d.State.Final() {
said = held[d.Walk.ID]
}
if d.HeldBack != said {
d.HeldBack = said
h.keep(d)
}
h.Logf("[mesh-delivery] the walk %s is not ended for %s: it goes on for %s (novox/hq issue 459)", walk, stopped,
strings.Join(goesOn, ", "))
h.mu.Unlock()
return false, goesOn, nil
}
h.mu.Unlock()
if err := h.Controller.Stop(walk, why, by); err != nil {
return false, nil, err
for id, seen := range h.walks {
without, why := h.carriesStopped(id)
if len(without) == 0 {
delete(h.leaveRefused, id)
continue
}
// Once per record of it: the record that answers an ask is newer. Asked again after five minutes, should
// the ask have gone nowhere.
if a, asked := h.leaveAsked[id]; asked && seen.walk.Revision <= a.revision && now.Sub(a.at) < 5*time.Minute {
continue
}
walk := id
asks = append(asks, func() { h.Logf("[mesh-delivery] %s", h.stopLeaving(walk, why, byOwner, without)) })
}
return true, nil, nil
return asks
}
// leave gathers a delivery's merge to leave out of a walk (ADR 0299), for a caller holding the lock; true for the
// first of that walk, whose ask asks for all.
func (h *Holder) leave(walk string, d *Delivery) bool {
if h.leaving == nil {
h.leaving = map[string][]string{}
}
first := len(h.leaving[walk]) == 0
if m := d.Repository + "@" + d.MergedAs; d.MergedAs != "" && !containsString(h.leaving[walk], m) {
h.leaving[walk] = append(h.leaving[walk], m)
}
return first
}
// leaving459 takes the merges gathered to leave out of a walk, and whether the walk carries anything else: another
// merge it names, or a delivery not ended that follows it. Taken under the lock, outside any.
func (h *Holder) leaving459(walk string) ([]string, bool) {
h.mu.Lock()
defer h.mu.Unlock()
without := h.leaving[walk]
delete(h.leaving, walk)
return without, len(h.othersOn(walk, without)) > 0
}
// othersOn are what a walk carries besides the merges named: each delivery not ended that follows it, by id, and
// each merge it names that no such delivery holds, as `<repository>@<commit>`.
func (h *Holder) othersOn(walk string, without []string) []string {
out := map[string]bool{}
held := map[string]bool{}
for _, d := range h.deliveries {
if d.Walk == nil || d.Walk.ID != walk || d.MergedAs == "" {
continue
}
m := d.Repository + "@" + d.MergedAs
held[strings.ToLower(m)] = true
if !d.State.Final() && !containsString(without, m) {
out[d.ID] = true
}
}
if s, known := h.walks[walk]; known && s.walk.Delivery != nil {
for _, m := range s.walk.Delivery.Merges {
key := m.Repository + "@" + m.Commit
if !held[strings.ToLower(key)] && !slices.ContainsFunc(without, func(w string) bool { return strings.EqualFold(w, key) }) {
out[key] = true
}
}
}
keys := make([]string, 0, len(out))
for k := range out {
keys = append(keys, k)
}
slices.Sort(keys)
return keys
}
// againWords is what a stop that left merges out did, in a person's words: the walk that walks the others again,
// what it left out (the controller's word on what contains a stopped merge), what stays on the machines, and what
// will still deliver a stopped change (ADR 0299).
func againWords(a StopAnswer) string {
var left []string
for _, o := range a.LeftOut {
s := o.Repository + "@" + shortOf(o.Commit, 12)
switch {
case len(o.Shares) > 0:
s += " (moves " + strings.Join(o.Shares, ", ") + ", as " + shortOf(o.Contains, 12) + " does)"
case o.Contains != "":
s += " (contains " + shortOf(o.Contains, 12) + ")"
}
left = append(left, s)
}
var out string
switch {
case a.WalkedAgainBy != "":
out = fmt.Sprintf("%s walks its other merges again: %s; left out: %s", a.WalkedAgainBy,
strings.Join(a.WalkedAgain, ", "), orNone(strings.Join(left, ", ")))
case len(left) > 0:
out = "left out: " + strings.Join(left, ", ") + "; nothing else to walk"
default:
out = "nothing else to walk"
}
if len(a.Withheld) > 0 {
out += "; its walk withholds " + strings.Join(a.Withheld, ", ")
}
if len(a.Kept) > 0 {
var kept []string
for _, k := range a.Kept {
kept = append(kept, k.Module+" on "+strings.Join(k.Machines, ", "))
}
out += "; what the stopped walk already sent stays on the machines: " + strings.Join(kept, "; ")
}
if len(a.StillCarriedBy) > 0 {
out += "; still to deliver the stopped change when walked, a revert's to hold back: " +
strings.Join(a.StillCarriedBy, ", ")
}
return out
}
// GroupState is a group's state, derived from its members and its composed check — never set apart (novox/hq
@@ -1,80 +1,475 @@
package main
import (
"fmt"
"strings"
"testing"
"time"
)
// novox/hq issue 459: since the merge window (ADR 0276) one walk carries every merge of its batch, across
// repositories. Stopping a group member asked the controller to end that member's walk, and with it the rollout of
// every other delivery the walk carries, though none of them is in the group or was stopped. The shape is
// foldShape (issue 445's review of mesh-catalog #203): group member B and C, a delivery of no group, on one walk,
// plan-2. A stop now ends only the stopped delivery; the walk is ended only once no live delivery rides it, and
// while it goes on the stopped delivery says so, naming what it goes on for.
// novox/hq issue 459, ADR 0299: since the merge window (ADR 0276) one walk carries every merge of its batch. A group
// stop asked the controller to end the stopped member's walk, and with it every other delivery on it, which no one
// stopped. Now a group stop asks `delivery-stop` with the stopped members' merges to leave out: the walk ends, and
// the controller walks its other merges again in a fresh batch; a later merge of a stopped merge's repository
// contains it and ends with it, named. A person's stop of a delivery on a shared walk answers first with what
// ending the walk would take down, and acts only on a way given: `again` or `walk`.
// B is stopped by its group (the forge says C does not contain A's merge): plan-2 is not ended, C takes its turn
// on it, and B says its walk goes on for C.
func TestAGroupStopLeavesAWalkThatCarriesAnotherDelivery(t *testing.T) {
s := foldShape()
s.forge = map[string]bool{mergedA + " in " + mergedC: false}
w := s.open(t)
stopped445(t, w, "does not contain its merge")
if containsString(w.ctl.stops, "plan-2") {
t.Fatalf("plan-2 was ended %v, and with it C, which is in no group and was not stopped", w.ctl.stops)
// shape459 is a world of a delivery group feat/g (its members in the repositories group names, in that order) and
// lone deliveries (the repositories alone names), each checked and merged. The group's first member is on a walk of
// its own, plan-a; every other merge is carried by one walk, plan-2, waiting for its word.
type shape459 struct {
*world
ids, heads, merges map[string]string // by repository
}
func open459(t *testing.T, group, alone []string) *shape459 {
t.Helper()
w := newWorld(t)
s := &shape459{world: w, ids: map[string]string{}, heads: map[string]string{}, merges: map[string]string{}}
n := 0
add := func(repo, branch string) {
n++
head := fmt.Sprintf("%02d%s", n, strings.Repeat("h", 14))
s.ids[repo], s.heads[repo] = IDOf(repo, head), head
s.merges[repo] = fmt.Sprintf("%02d%s", n, strings.Repeat("m", 10))
w.h.PullUpdated(pr(repo, n, head, branch, "x"))
w.h.Checked(verdict(repo, n, head, "pass"))
}
if d := w.delivery(cID); d.State != Delivering || !containsString(w.ctl.delivers, "plan-2") {
t.Fatalf("C is %s, let go %v; want it delivering on plan-2", d.State, w.ctl.delivers)
for _, r := range group {
add(r, "feat/g")
}
goesOn459(t, w, headB, cID)
// The walk delivers what it carries: C is delivered, and B stays stopped.
end445(w, "plan-2", walkDone)
if b, c := w.state(bID), w.state(cID); b != Stopped || c != Delivered {
t.Fatalf("once plan-2 is done, B is %s and C is %s; want B stopped and C delivered", b, c)
for _, r := range alone {
add(r, "fix/"+r[strings.LastIndex(r, "/")+1:])
}
w.settleAll()
w.h.Checked(CheckedEvent{Group: "feat/g", ID: "check-1", Verdict: "pass", Summary: "compose"})
w.settleAll()
for i, r := range append(append([]string{}, group...), alone...) {
w.h.PullMerged(merged(r, i+1, s.heads[r], s.merges[r]))
}
first := aWalk("plan-a", group[0], s.merges[group[0]], true)
first.Delivery.Merges = []WalkMerge{{Repository: group[0], Commit: s.merges[group[0]]}}
w.h.WalkMoved(w.ctl.put(first))
two := aWalk("plan-2", alone[0], s.merges[alone[0]], true)
two.Created = two.Created.Add(time.Minute)
for _, r := range append(append([]string{}, group[1:]...), alone...) {
two.Delivery.Merges = append(two.Delivery.Merges, WalkMerge{Repository: r, Commit: s.merges[r]})
}
w.h.WalkMoved(w.ctl.put(two))
w.settleAll()
return s
}
// failFirst fails the group's first member's walk: the members after it are stopped by their group.
func (s *shape459) failFirst() {
a := s.ctl.walk("plan-a")
a.State, a.Note = walkFailed, "the first member failed its gate on anchor"
s.h.WalkMoved(s.ctl.put(a))
s.settleAll()
}
// heard is the controller's records of the stopped walk and the walk that walks its merges again, as plan-moved says
// them, and then a walk done.
func (s *shape459) heard(walks ...string) {
for _, id := range walks {
s.h.WalkMoved(s.ctl.walk(id))
}
s.settleAll()
}
// cutAgain is the controller cutting the batch that walks a stopped walk's merges again, as plan-moved says it.
func (s *shape459) cutAgain(id string) {
s.h.WalkMoved(s.ctl.cut(id))
s.settleAll()
}
func (s *shape459) done(walk string) {
r := s.ctl.walk(walk)
sent := s.now
for m := range r.Modules {
r.Modules[m] = &WalkModule{State: "built", SentAt: &sent}
}
r.State = walkDone
s.h.WalkMoved(s.ctl.put(r))
s.settleAll()
}
func lastWhy(d *Delivery) string {
if len(d.Transitions) == 0 {
return ""
}
return d.Transitions[len(d.Transitions)-1].Why
}
// A group member stopped on a walk it shares with a lone delivery: the walk is stopped without the member's merge,
// never whole, and the lone delivery follows the walk that walks its merge again and is delivered by it.
func TestAGroupStopWalksTheOtherDeliveriesAgainWithoutIt(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
if containsString(s.ctl.delivers, "plan-2") {
t.Fatalf("plan-2 was let go before b's turn: %v", s.ctl.delivers)
}
s.failFirst()
if b := s.delivery(s.ids["novox/b"]); b.State != Stopped || !strings.Contains(lastWhy(b), s.ids["novox/a"]) {
t.Fatalf("b is %s (%q); want it stopped by its group, naming a", b.State, lastWhy(b))
}
if containsString(s.ctl.stops, "plan-2") {
t.Fatalf("plan-2 was ended whole %v, and with it z, which is in no group and was not stopped", s.ctl.stops)
}
if want := "plan-2 without novox/b@" + s.merges["novox/b"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
s.heard("plan-2", "plan-2-again")
if containsString(s.ctl.delivers, "plan-2-again") {
t.Fatal("a batch not yet cut was given its word")
}
s.cutAgain("plan-2-again")
z := s.delivery(s.ids["novox/z"])
if z.State.Final() || z.Walk == nil || z.Walk.ID != "plan-2-again" || !containsString(s.ctl.delivers, "plan-2-again") {
t.Fatalf("z is %s on %+v, let go %v; want it following plan-2-again, let go", z.State, z.Walk, s.ctl.delivers)
}
s.done("plan-2-again")
if z, b := s.state(s.ids["novox/z"]), s.state(s.ids["novox/b"]); z != Delivered || b != Stopped {
t.Fatalf("once plan-2-again is done, z is %s and b %s; want z delivered and b stopped", z, b)
}
}
// A person stops B while B and C deliver on plan-2: the walk goes on for C, and the answer says so. Then the person
// stops C too: nothing live rides plan-2 any more, and it is ended.
func TestAPersonsStopEndsTheWalkOnlyWithItsLastDelivery(t *testing.T) {
s := foldShape()
s.forge = map[string]bool{mergedA + " in " + mergedC: true}
w := s.open(t)
letGo445(t, w, "plan-2")
said, err := w.h.Stop(bID, "not wanted after all", "jochen")
// Two members after the failed one, both on the shared walk: one ask leaves both out, so neither is walked again
// because its own stop came second.
func TestAGroupStopLeavesOutEveryMemberItStopsOnOneWalk(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b", "novox/c"}, []string{"novox/z"})
s.failFirst()
if len(s.ctl.withouts) != 1 || !strings.Contains(s.ctl.withouts[0], "novox/b@"+s.merges["novox/b"]) ||
!strings.Contains(s.ctl.withouts[0], "novox/c@"+s.merges["novox/c"]) {
t.Fatalf("the controller was asked %v; want one stop of plan-2 leaving out b and c", s.ctl.withouts)
}
s.heard("plan-2", "plan-2-again")
again := s.ctl.walk("plan-2-again")
if len(again.Delivery.Merges) != 1 || again.Delivery.Merges[0].Repository != "novox/z" {
t.Fatalf("plan-2-again walks %+v; want z's merge alone", again.Delivery.Merges)
}
for _, r := range []string{"novox/b", "novox/c"} {
if st := s.state(s.ids[r]); st != Stopped {
t.Fatalf("%s is %s", r, st)
}
}
}
// A lone delivery whose merge is a later merge of the stopped member's repository contains the member's merge: it
// is left out and ends stopped, naming the merge it contains; the other lone delivery is walked again.
func TestAGroupStopEndsAMergeThatContainsTheStoppedOne(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z", "novox/b2"})
// b2 is a later merge of b's repository: the walk carries it for b.
s.retarget("novox/b2", "novox/b")
s.failFirst()
s.heard("plan-2", "plan-2-again")
s.cutAgain("plan-2-again")
b2 := s.delivery(s.ids["novox/b2"])
if b2.State != Stopped || !strings.Contains(lastWhy(b2), shortOf(s.merges["novox/b"], 8)) ||
!strings.Contains(lastWhy(b2), "contains") {
t.Fatalf("b2 is %s (%q); want it stopped, naming b's merge it contains", b2.State, lastWhy(b2))
}
if z := s.delivery(s.ids["novox/z"]); z.Walk == nil || z.Walk.ID != "plan-2-again" || z.State.Final() {
t.Fatalf("z is %s on %+v; want it following plan-2-again", z.State, z.Walk)
}
}
// retarget makes the delivery of repo `later` a later merge of repository `of`: same repository, carried by plan-2
// for the merge of `of`.
func (s *shape459) retarget(later, of string) {
s.t.Helper()
s.h.mu.Lock()
d := s.h.deliveries[s.ids[later]]
d.Repository, d.MergedAt = of, d.MergedAt.Add(time.Minute)
s.h.mu.Unlock()
s.ctl.mu.Lock()
s.ctl.mergedAt[s.merges[of]], s.ctl.mergedAt[s.merges[later]] = d.MergedAt.Add(-time.Minute), d.MergedAt
s.ctl.mu.Unlock()
two := s.ctl.walk("plan-2")
merges := []WalkMerge{}
for _, m := range two.Delivery.Merges {
switch m.Repository {
case of:
m.Carried = s.merges[later]
case later:
m.Repository = of
}
merges = append(merges, m)
}
wd := *two.Delivery
wd.Merges = merges
two.Delivery = &wd
s.h.WalkMoved(s.ctl.put(two))
s.settleAll()
}
// shared is two lone deliveries, y and z, let go together on one walk, plan-2.
func shared459(t *testing.T) *shape459 {
t.Helper()
s := open459(t, []string{"novox/a"}, []string{"novox/y", "novox/z"})
for _, r := range []string{"novox/y", "novox/z"} {
if st := s.state(s.ids[r]); st != Delivering {
t.Fatalf("%s is %s, let go %v; want both delivering on plan-2", r, st, s.ctl.delivers)
}
}
return s
}
// A person's stop of y, which shares its walk with z, does nothing at first: it names z and the two ways.
func TestAPersonsStopOfASharedWalkAsksFirst(t *testing.T) {
s := shared459(t)
said, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "")
if err == nil {
t.Fatalf("the stop acted at once: %q", said)
}
for _, want := range []string{s.ids["novox/z"], "plan-2", "again", "walk", "not stopped"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the stop said %q; want it to name %q", err, want)
}
}
if st := s.state(s.ids["novox/y"]); st != Delivering || len(s.ctl.stops)+len(s.ctl.withouts) != 0 {
t.Fatalf("y is %s, the controller was asked %v %v; want nothing done", st, s.ctl.stops, s.ctl.withouts)
}
if _, err := s.h.StopWay(s.ids["novox/y"], "x", "jochen", "sideways"); err == nil {
t.Fatal("a way that is neither again nor walk was taken")
}
}
// With `again`, y is stopped, the walk is stopped without y's merge, and z goes back to published and follows the
// walk that walks it again.
func TestAPersonsStopAgainWalksTheOthersAgain(t *testing.T) {
s := shared459(t)
said, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again")
if err != nil {
t.Fatal(err)
}
w.settleAll()
if w.state(bID) != Stopped {
t.Fatalf("B is %s", w.state(bID))
if !strings.Contains(said, "plan-2-again") || !strings.Contains(said, "novox/z@") {
t.Fatalf("the stop said %q; want it to name the walk that walks z again", said)
}
if containsString(w.ctl.stops, "plan-2") {
t.Fatalf("plan-2 was ended %v while it carries C", w.ctl.stops)
if want := "plan-2 without novox/y@" + s.merges["novox/y"]; !containsString(s.ctl.withouts, want) ||
containsString(s.ctl.stops, "plan-2") {
t.Fatalf("the controller was asked %v, ended whole %v; want %q", s.ctl.withouts, s.ctl.stops, want)
}
if s := w.state(cID); s != Delivering {
t.Fatalf("C is %s, want delivering", s)
s.heard("plan-2")
if z := s.delivery(s.ids["novox/z"]); z.State != Published {
t.Fatalf("z is %s once plan-2 was stopped and walked again; want published, waiting for that walk", z.State)
}
if !strings.Contains(said, "goes on") || !strings.Contains(said, cID) {
t.Fatalf("the stop said %q; want it to say plan-2 goes on for C", said)
s.heard("plan-2-again")
s.cutAgain("plan-2-again")
if z := s.delivery(s.ids["novox/z"]); z.Walk == nil || z.Walk.ID != "plan-2-again" || z.State != Delivering {
t.Fatalf("z is %s on %+v; want it delivering on plan-2-again", z.State, z.Walk)
}
goesOn459(t, w, headB, cID)
if _, err := w.h.Stop(cID, "this one too", "jochen"); err != nil {
t.Fatal(err)
}
w.settleAll()
if !containsString(w.ctl.stops, "plan-2") {
t.Fatalf("plan-2 was not ended once every delivery it carries was stopped: %v", w.ctl.stops)
if st := s.state(s.ids["novox/y"]); st != Stopped {
t.Fatalf("y is %s", st)
}
}
// goesOn459 is whether the stopped delivery at head said on its commit that its walk goes on, naming other.
func goesOn459(t *testing.T, w *world, head, other string) {
t.Helper()
for _, l := range w.forge.notesOn(head) {
if strings.Contains(l, "plan-2 goes on") && strings.Contains(l, other) {
// With `walk`, the whole walk is stopped, and z ends stopped, naming y's stop.
func TestAPersonsStopOfTheWalkStopsEveryDelivery(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "walk"); err != nil {
t.Fatal(err)
}
if !containsString(s.ctl.stops, "plan-2") || len(s.ctl.withouts) != 0 {
t.Fatalf("the controller was asked %v %v; want plan-2 ended whole", s.ctl.stops, s.ctl.withouts)
}
s.heard("plan-2")
z := s.delivery(s.ids["novox/z"])
if z.State != Stopped || !strings.Contains(lastWhy(z), s.ids["novox/y"]) {
t.Fatalf("z is %s (%q); want it stopped, naming y's stop", z.State, lastWhy(z))
}
}
// The reviewer's case (review of #221): group [a, b]; y and z ride plan-2 with b. y is stopped `way: again`, and the
// batch that walks plan-2's others again carries b and z. Then a fails: b is stopped by its group, and its merge is
// left out of that batch — never shipped by it — and z alone is walked and delivered.
func TestAMergeWalkedAgainAndThenStoppedByItsGroupNeverShips(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/y", "novox/z"})
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.heard("plan-2", "plan-2-again")
s.failFirst()
if b := s.state(s.ids["novox/b"]); b != Stopped {
t.Fatalf("b is %s", b)
}
if want := "plan-2-again without novox/b@" + s.merges["novox/b"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
s.heard("plan-2-again", "plan-2-again-left")
s.cutAgain("plan-2-again")
again := s.ctl.walk("plan-2-again")
for _, m := range again.Delivery.Merges {
if m.Repository == "novox/b" {
t.Fatalf("the walk %s still carries b's stopped merge: %+v", again.ID, again.Delivery.Merges)
}
}
if !containsString(s.ctl.delivers, "plan-2-again") {
t.Fatalf("z's walk was not let go: %v", s.ctl.delivers)
}
s.done("plan-2-again")
if z := s.state(s.ids["novox/z"]); z != Delivered {
t.Fatalf("z is %s", z)
}
}
// Whatever brought it there, a walk that carries a stopped delivery's merge is asked to leave it out, and is never
// given its word while it carries it.
func TestAWalkCarryingAStoppedMergeIsAskedToLeaveItOutAndNeverLetGo(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
// A walk the controller made that carries y's merge anyway, waiting for its word, with z on it.
x := aWalk("plan-x", "novox/z", s.merges["novox/z"], true)
x.Created = x.Created.Add(10 * time.Minute)
x.Delivery.Merges = []WalkMerge{{Repository: "novox/y", Commit: s.merges["novox/y"]},
{Repository: "novox/z", Commit: s.merges["novox/z"]}}
s.ctl.refuseWithout = true // it cannot be left out: the walk is stopped whole
s.h.WalkMoved(s.ctl.put(x))
s.settleAll()
if want := "plan-x without novox/y@" + s.merges["novox/y"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
if containsString(s.ctl.delivers, "plan-x") {
t.Fatalf("a walk carrying a stopped merge was let go: %v", s.ctl.delivers)
}
if !containsString(s.ctl.stops, "plan-x") {
t.Fatalf("refused the leave-out, the walk was not stopped whole: %v", s.ctl.stops)
}
}
// A group stop the controller refuses to make without the merge (an older controller) stops the whole walk: the
// stopped change is held back, and the delivery riding with it ends stopped, saying why.
func TestARefusedLeaveOutStopsTheWholeWalk(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
s.ctl.refuseWithout = true
s.failFirst()
if len(s.ctl.withouts) != 1 || !containsString(s.ctl.stops, "plan-2") {
t.Fatalf("asked %v, ended whole %v; want the leave-out tried, then plan-2 ended whole", s.ctl.withouts, s.ctl.stops)
}
s.heard("plan-2")
if z := s.delivery(s.ids["novox/z"]); z.State != Stopped || !strings.Contains(lastWhy(z), "could not be left out") {
t.Fatalf("z is %s (%q); want it stopped, saying the leave-out was refused", z.State, lastWhy(z))
}
}
// A person's stop the controller cannot make leaves the delivery as it was, and says how to go on.
func TestAPersonsStopThatCannotBeMadeStopsNothing(t *testing.T) {
s := shared459(t)
s.ctl.refuseWithout = true
_, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again")
if err == nil || !strings.Contains(err.Error(), "is not stopped") || !strings.Contains(err.Error(), "plans stop") {
t.Fatalf("the refused stop said %v", err)
}
if st := s.state(s.ids["novox/y"]); st != Delivering {
t.Fatalf("y is %s; want it delivering, its stop not made", st)
}
}
// What a stop that left merges out says: the walk that walks the others, what it left out and what each contains,
// as the controller says it, what stays on which machines, and what will still deliver the stopped change.
func TestAStopSaysWhatStaysAndWhatStillCarriesIt(t *testing.T) {
said := againWords(StopAnswer{Stopped: "plan-2", WalkedAgainBy: "plan-3", WalkedAgain: []string{"novox/z@zz"},
LeftOut: []WalkLeftOut{{Repository: "novox/b", Commit: "b1b1b1b1b1b1"},
{Repository: "novox/b", Commit: "b2b2b2b2b2b2", Contains: "b1b1b1b1b1b1"}},
Kept: []WalkKept{{Module: "lab", Machines: []string{"anchor"}}},
StillCarriedBy: []string{"novox/b@b3b3 (in plan-4)"}})
for _, want := range []string{"plan-3", "novox/z@zz", "b2b2b2b2b2b2 (contains b1b1b1b1b1b1)", "lab on anchor",
"stays on the machines", "novox/b@b3b3 (in plan-4)", "revert"} {
if !strings.Contains(said, want) {
t.Fatalf("the stop said %q; want %q in it", said, want)
}
}
}
// A merge left out for moving a module the stopped merge moves, from another repository, ends stopped saying so —
// never that it contains the stopped merge (review of #221).
func TestALeftOutMergeSharingAModuleSaysSoTruly(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
s.ctl.mu.Lock()
s.ctl.moves[s.merges["novox/b"]], s.ctl.moves[s.merges["novox/z"]] = []string{"lab"}, []string{"lab"}
s.ctl.mu.Unlock()
s.failFirst()
s.heard("plan-2")
z := s.delivery(s.ids["novox/z"])
why := lastWhy(z)
if z.State != Stopped || !strings.Contains(why, "moves lab, as the stopped merge "+shortOf(s.merges["novox/b"], 12)) ||
strings.Contains(why, "contains the stopped merge") {
t.Fatalf("z is %s (%q); want it stopped, saying it moves lab as b's merge does", z.State, why)
}
}
// A person's first answer names the deliveries that would be left out for moving a module this merge moves, and says
// the controller may leave out more.
func TestAPersonsFirstAnswerNamesWhatSharesItsModules(t *testing.T) {
s := shared459(t)
_, err := s.h.StopWay(s.ids["novox/y"], "x", "jochen", "")
if err == nil || !strings.Contains(err.Error(), "for moving a module this merge moves: "+s.ids["novox/z"]+" (moves app)") ||
!strings.Contains(err.Error(), "may leave out more") {
t.Fatalf("the first answer said %v", err)
}
}
// The safety net holds back only what a person or a group stopped: a delivery ended by what its walk did (the
// observed rows, which name nobody) is not asked out of a walk that carries its merge.
func TestTheSafetyNetLeavesAStrandedStopAlone(t *testing.T) {
s := shared459(t)
s.h.mu.Lock()
y := s.h.deliveries[s.ids["novox/y"]]
y.State = Stopped
y.Transitions = append(y.Transitions, Transition{At: s.now, From: Published, To: Stopped, Event: EvStop,
Why: "its walk ended done before its word"})
s.h.mu.Unlock()
s.settleAll()
if len(s.ctl.withouts) != 0 {
t.Fatalf("a stop nobody asked for was held back: %v", s.ctl.withouts)
}
}
// A walk the safety net holds — its leave-out and its whole stop both refused — says why on the deliveries waiting on
// it, in `waits`, and in `stalled` past their bound.
func TestAHeldWalkIsSaidOnItsDeliveriesAndStalls(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.ctl.refuseWithout, s.ctl.refuseStop = true, true
x := aWalk("plan-x", "novox/z", s.merges["novox/z"], true)
x.Created = x.Created.Add(10 * time.Minute)
x.Delivery.Merges = []WalkMerge{{Repository: "novox/y", Commit: s.merges["novox/y"]},
{Repository: "novox/z", Commit: s.merges["novox/z"]}}
s.h.WalkMoved(s.ctl.put(x))
s.settleAll()
z := s.delivery(s.ids["novox/z"])
if !strings.Contains(z.HeldBack, "plan-x") || !strings.Contains(z.HeldBack, "refused") {
t.Fatalf("z says it is held back by %q; want plan-x and the refusal", z.HeldBack)
}
if w := s.h.waitsFor(z); w != z.HeldBack {
t.Fatalf("z waits for %q", w)
}
s.later(Bounds[Published].For + time.Minute)
s.settleAll()
for _, l := range s.h.Stalled() {
if l.ID == z.ID && strings.Contains(l.Says, "plan-x") {
return
}
}
t.Fatalf("the stopped delivery's notes %q do not say plan-2 goes on for %s", w.forge.notesOn(head), other)
t.Fatalf("z is not listed stalled naming plan-x: %+v", s.h.Stalled())
}
// A person's stop of a delivery whose merge waits in a batch asks that batch now to leave it out, and says so.
func TestAPersonsStopOfAMergeInABatchAsksTheBatch(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.heard("plan-2", "plan-2-again")
said, err := s.h.StopWay(s.ids["novox/z"], "this one too", "jochen", "")
if err != nil {
t.Fatal(err)
}
if want := "plan-2-again without novox/z@" + s.merges["novox/z"]; !containsString(s.ctl.withouts, want) ||
!strings.Contains(said, "batch plan-2-again") {
t.Fatalf("said %q, asked %v; want the batch asked now, and said", said, s.ctl.withouts)
}
}
@@ -0,0 +1,142 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 362's note of 2026-10-11 (the nzbgeek case) and issue 529: mesh-media-catalog #18 added a module
// whose delivery plan said "sent nowhere". This owner ended its delivery at once, "nothing for a walk to move", but
// the controller opened a walk for the merge that moved modules to a machine and waited for this owner's word.
// With its delivery ended, nothing gave the walk its word: it held 39 minutes and two walks queued behind it.
const (
media = "novox/mesh-media-catalog"
headNzb = "abababab00001818"
mergeNzb = "cdcdcdcd00001818"
nzbWalkID = "plan-1791748193836841649"
)
var atNzb = time.Date(2026, 10, 11, 19, 43, 0, 0, time.UTC)
// nzbgeekMerged is #18 checked on a plan that builds a new module and sends it nowhere, and merged: its delivery
// ends at once, nothing for a walk to move.
func nzbgeekMerged(t *testing.T) *world {
t.Helper()
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb.Add(-time.Minute))
p := pr(media, 18, headNzb, "feat/nzbgeek", "modules/nzbgeek/module.json")
w.h.PullUpdated(p)
v := verdict(media, 18, headNzb, "pass")
v.Plan = &DeliveryPlan{Repository: media, Base: "main", New: []string{"modules/nzbgeek"},
Summary: "builds new: modules/nzbgeek, sent nowhere; no bus step"}
w.h.Checked(v)
w.later(time.Minute)
w.h.PullMerged(mergedAt(p, mergeNzb, atNzb))
w.settleAll()
if d := w.delivery(IDOf(media, headNzb)); d.State != Delivered {
t.Fatalf("#18 is %s, not delivered as nothing for a walk to move", d.State)
}
return w
}
// nzbWalk is the walk the controller cut for the merge's batch: it names the merge and moves nzbgeek and nzbget.
func nzbWalk(created time.Time, merges ...WalkMerge) Walk {
w := Walk{ID: nzbWalkID, Repository: media, Branch: "main", Commit: mergeNzb, Created: created, State: walkBuilding,
Tiers: [][]string{{"nzbgeek", "nzbget"}}, Modules: map[string]*WalkModule{"nzbgeek": {}, "nzbget": {}},
Delivery: &WalkDelivery{Awaits: "mesh-delivery", Merges: merges}}
return w
}
// **A walk answering only merges that ended as nothing for a walk to move is named as held by nothing** (novox/hq
// issues 362 and 529): never let go by this owner — nothing judged what it moves — and said, by `stalled` with
// `walks`, with why, so the controller's `plans` says so instead of pointing nowhere.
func TestAWalkOfNothingToMoveMergesIsNamedAsHeldByNothing(t *testing.T) {
w := nzbgeekMerged(t)
w.later(90 * time.Second)
w.h.WalkMoved(w.ctl.put(nzbWalk(w.now, WalkMerge{Repository: media, Commit: mergeNzb})))
w.settleAll()
if len(w.ctl.delivers) != 0 {
t.Fatalf("a walk no live delivery holds was let go: %v", w.ctl.delivers)
}
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != nzbWalkID || lines[0].State != Unheld ||
!strings.Contains(lines[0].Says, "ended as nothing-to-move") || !strings.Contains(lines[0].Says, IDOf(media, headNzb)) ||
!strings.Contains(lines[0].Says, "nzbget") {
t.Fatalf("the walk is said as %+v", lines)
}
}
// **The status names each held delivery, why, and the verb that releases it** (novox/hq issue 529): "held: 1"
// alone pointed a reader nowhere.
func TestTheStatusNamesAHeldDeliveryAndWhy(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb)
p := pr(media, 22, headNzb, "adr-0307-state-history", "modules/nzbgeek/module.json")
w.h.PullUpdated(p)
w.h.PullMerged(mergedAt(p, mergeNzb, atNzb)) // merged before any verdict: held for a person
w.settleAll()
id := IDOf(media, headNzb)
d := w.delivery(id)
if d.State != Held {
t.Fatalf("#22 is %s", d.State)
}
held, _ := w.h.Status("listening")["held"].([]string)
if len(held) != 1 || !strings.Contains(held[0], id) || !strings.Contains(held[0], d.HeldWhy) ||
!strings.Contains(held[0], "mesh-delivery.release") {
t.Fatalf("the status says held %q", held)
}
}
// **A walk waiting for this owner's word is told why** (novox/hq issue 529): asked with `walks`, `stalled` says of
// each walk waiting for its word what holds it — a held delivery, naming it, its reason and `mesh-delivery.release`,
// or the deliveries not at their turn — so the controller's `plans` points at the person's decision, not at a hand
// start.
func TestAWaitingWalkIsToldWhatHoldsIt(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atDu.Add(5*time.Second))
cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json")
du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json")
w.h.PullUpdated(cc)
w.h.PullUpdated(du)
w.h.Checked(verdictMoving(cat, 101, headCC, "claude-code"))
w.h.PullMerged(mergedAt(cc, mergeCC, atCC))
w.h.PullMerged(mergedAt(du, mergeDu, atDu)) // unchecked: held for a person
walk := catWalk("plan-9", mergeDu, atDu.Add(5*time.Second), "claude-code", "dunst")
walk.Delivery.Merges = []WalkMerge{{Repository: cat, Commit: mergeCC, Carried: mergeDu}, {Repository: cat, Commit: mergeDu}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
idDu := IDOf(cat, headDu)
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != "plan-9" || lines[0].State != Held || lines[0].ID != idDu ||
!strings.Contains(lines[0].Says, w.delivery(idDu).HeldWhy) || !strings.Contains(lines[0].Says, "mesh-delivery.release") {
t.Fatalf("the waiting walk is said as %+v", lines)
}
// Not by default: `stalled` stays what the self-check raises conditions for.
for _, l := range w.h.Stalled() {
if l.Walk == "plan-9" {
t.Fatalf("stalled lists the waiting walk without being asked: %+v", l)
}
}
// Released, its word is asked: nothing holds it any more.
if _, err := w.h.Release(idDu, "checked by hand", "jochen"); err != nil {
t.Fatal(err)
}
w.h.WalkMoved(w.ctl.walk("plan-9"))
w.settleAll()
if lines := w.h.WalksWaiting(); len(lines) != 0 {
t.Fatalf("a walk let go is still said waiting: %+v", lines)
}
}
// A walk waiting while a delivery it carries is not at its turn says so, naming the delivery — what was said only
// in this owner's journal (novox/hq issue 529, note 2).
func TestAWaitingWalkSaysWhichDeliveryIsNotAtItsTurn(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb)
w.h.mu.Lock()
w.h.waitsWhy = map[string]string{"plan-7": "novox/x@1 (published) carried by it not at its turn"}
w.h.walks["plan-7"] = seenWalk{walk: aWalk("plan-7", "novox/x", "1111", true), at: atNzb}
w.h.mu.Unlock()
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != "plan-7" || !strings.Contains(lines[0].Says, "not at its turn") {
t.Fatalf("the waiting walk is said as %+v", lines)
}
}
@@ -205,7 +205,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: seat + "times",
Description: "Delivery time (novox/hq ADR 0282): from a merge to every machine of its walk running the build. " +
"Each class's budget (a leaf module five minutes, a core module ten), the last days' median and worst, " +
"Each class's budget (a leaf module three minutes, a core module ten), the last days' median and worst, " +
"how many within and over, the newest delivery of each class, every delivery over its budget with its " +
"longest phase, and the delivered ones whose time is not known and why. Given a delivery's id, its time " +
"phase by phase: window, queued, word, build, send-first, judgement, send-rest, between-tiers, apply — " +
@@ -268,8 +268,17 @@ func tools(h *Holder, l *listening) []stdio.Tool {
"and what healer H2 may do once it has passed; and the machine steps' table.",
Run: func(map[string]any) (any, error) { return TableText(), nil }},
{Name: seat + "stalled",
Description: "Every delivery held past its state's bound, with the transition the table lets healer H2 take.",
Run: func(map[string]any) (any, error) { return h.Stalled(), nil }},
Description: "Every delivery held past its state's bound, with the transition the table lets healer H2 take. " +
"With walks: true, instead each walk waiting for this owner's word and what holds it — a held delivery, " +
"its reason and the verb that releases it, or the deliveries not at their turn (novox/hq issue 529).",
Input: map[string]any{"type": "object", "properties": map[string]any{
"walks": map[string]any{"type": "boolean", "description": "the waiting walks and what holds each, instead"}}},
Run: func(a map[string]any) (any, error) {
if walks, _ := a["walks"].(bool); walks {
return h.WalksWaiting(), nil
}
return h.Stalled(), nil
}},
{Name: seat + "recheck",
Description: "Check a rejected or ready delivery again: its merge check statuses put back to pending on the forge, " +
"so nothing merges on the old verdict; proposed again, and its check asked. With why.",
@@ -293,14 +302,18 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: seat + "stop",
Description: "Stop a delivery that is not final, with why: its walk is ended through the controller, and " +
"in a delivering group every member after it is stopped too, naming it.",
"in a delivering group every member after it is stopped too, naming it. When its walk carries other " +
"deliveries, it first answers what ending the walk would take down, and acts only on a way: `again` " +
"ends the walk and walks the others again without this merge, `walk` stops the whole walk (novox/hq ADR 0299).",
Input: map[string]any{"type": "object", "properties": map[string]any{"id": str("the delivery's id"),
"why": str("why, kept with the transition")}, "required": []string{"id", "why"}},
"why": str("why, kept with the transition"),
"way": str("on a walk shared with other deliveries: again (the others walked again without it) or walk " +
"(the whole walk stopped)")}, "required": []string{"id", "why"}},
Run: func(a map[string]any) (any, error) {
if err := need(a, "id", "why"); err != nil {
return nil, err
}
return h.Stop(strArg(a, "id"), strArg(a, "why"), aPerson)
return h.StopWay(strArg(a, "id"), strArg(a, "why"), aPerson, strArg(a, "way"))
}},
{Name: seat + "close",
Description: "Healer H2's verb: for a delivery held past its bound, read its walk again and take the " +
@@ -71,7 +71,10 @@ type Delivery struct {
// TurnAwaits says, for a group member whose turn waits for a later merge to carry a superseded member before it,
// which merges and until when; TurnAwaitsUntil is that moment, which bounds the wait (ADR 0276 decision 9,
// novox/hq issue 445). `show` shows both, and `stalled` says them past that moment.
TurnAwaits string `json:"turn_awaits,omitempty"`
TurnAwaits string `json:"turn_awaits,omitempty"`
// HeldBack is why the walk it waits on is not let go: it carries a merge whose delivery was stopped, and is
// asked to leave it out (novox/hq ADR 0299). Said by `waits`, and by `stalled` past its bound.
HeldBack string `json:"held_back,omitempty"`
TurnAwaitsUntil *time.Time `json:"turn_awaits_until,omitempty"`
// ContainedIn is the forge's answer, by a later merge commit, to whether that commit contains this one's
// merge (gitea_contains): asked once per pair, and kept (ADR 0270).
@@ -383,6 +386,11 @@ type WalkDelivery struct {
// TakenOverBy is, on a walk folded before it started, the walk that took it over and answers every merge
// it answered.
TakenOverBy string `json:"taken_over_by,omitempty"`
// WalkedAgainBy is, on a walk stopped without some of its merges, the batch that walks its other merges again,
// and LeftOut the merges it left out (novox/hq ADR 0299): a delivery of a merge left out ends stopped with it,
// one of a merge walked again follows that batch's walk.
WalkedAgainBy string `json:"walked_again_by,omitempty"`
LeftOut []WalkLeftOut `json:"left_out,omitempty"`
// Batch is set while the record is a batch, not yet a walk: its state `assembling` or `queued`. A batch, once
// cut, is the walk under the same id: its next record is `building`. A merge this owner reads in a batch is
// not held as unwalked while the batch is heard of.
@@ -398,6 +406,30 @@ type WalkMerge struct {
Carried string `json:"carried,omitempty"`
}
// WalkLeftOut is a merge a stop left out (novox/hq ADR 0299): a merge stopped, or a later merge of its
// repository's branch that contains it, Contains naming the stopped merge.
type WalkLeftOut struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Contains string `json:"contains,omitempty"`
// Shares are the modules it moves that a stopped merge moves too, when that is why it was left out: Contains
// then names that stopped merge, which it does not contain.
Shares []string `json:"shares,omitempty"`
}
// leftOut is the merge of a repository a stop left out, when it did.
func (w Walk) leftOut(repository, merge string) (WalkLeftOut, bool) {
if w.Delivery == nil || merge == "" {
return WalkLeftOut{}, false
}
for _, o := range w.Delivery.LeftOut {
if strings.EqualFold(o.Repository, repository) && sameCommit(o.Commit, merge) {
return o, true
}
}
return WalkLeftOut{}, false
}
// WalkBatch is a batch's window: when it closes unless another merge arrives, when it closes at the latest,
// and, once closed, the walk it waits behind.
type WalkBatch struct {
@@ -454,18 +486,27 @@ func (w Walk) carriedFor(repository, merge string) (string, bool) {
// WalkSeen is what a delivery keeps of its walk.
type WalkSeen struct {
ID string `json:"id"`
Commit string `json:"commit"`
State string `json:"state"`
Note string `json:"note,omitempty"`
Revision int64 `json:"revision"`
Waits bool `json:"waits,omitempty"`
WaitedFor string `json:"waited_for,omitempty"`
GoBy string `json:"go_by,omitempty"`
StoppedBy string `json:"stopped_by,omitempty"`
Tier int `json:"tier"`
Tiers int `json:"tiers"`
Updated time.Time `json:"updated"`
ID string `json:"id"`
Commit string `json:"commit"`
State string `json:"state"`
Note string `json:"note,omitempty"`
Revision int64 `json:"revision"`
Waits bool `json:"waits,omitempty"`
WaitedFor string `json:"waited_for,omitempty"`
GoBy string `json:"go_by,omitempty"`
StoppedBy string `json:"stopped_by,omitempty"`
// StoppedWhy is why it was stopped; WalkedAgainBy, on a walk stopped without some merges, the batch that walks
// this delivery's merge again; Contains, when this delivery's merge was left out for containing a stopped
// merge, that merge (novox/hq ADR 0299).
StoppedWhy string `json:"stopped_why,omitempty"`
WalkedAgainBy string `json:"walked_again_by,omitempty"`
Contains string `json:"contains,omitempty"`
// Shares are, when this delivery's merge was left out for moving modules a stopped merge moves, those modules:
// built at the branch, which holds the stopped merge, they would deliver it (ADR 0299).
Shares []string `json:"shares,omitempty"`
Tier int `json:"tier"`
Tiers int `json:"tiers"`
Updated time.Time `json:"updated"`
// Branch is the branch the walk's commit was merged into; empty in a record kept before it was read.
Branch string `json:"branch,omitempty"`
// Created is when the controller made the walk: a delivery follows the newest walk that answers it.
@@ -487,7 +528,7 @@ func (w Walk) Seen() *WalkSeen {
s.WaitedFor = d.Awaits
s.Waits = d.Awaits != "" && d.Go == nil
s.GoBy = d.By
s.StoppedBy = d.Stopped
s.StoppedBy, s.StoppedWhy = d.Stopped, d.StoppedWhy
}
s.Branch = w.Branch
s.Created = w.Created
@@ -511,6 +552,33 @@ func (d *Delivery) endedUnstarted() bool {
return d.State == Published && d.Walk != nil && d.Walk.State == walkDone && !d.Walk.Started()
}
// walkStopped is whether the walk it follows was stopped through this owner, and walkedAgain whether its merge is
// walked again by another walk since (novox/hq ADR 0299).
func (d *Delivery) walkStopped() bool {
return d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy != ""
}
func (d *Delivery) walkedAgain() bool { return d.walkStopped() && d.Walk.WalkedAgainBy != "" }
// stoppedWords is why a delivery ends with the walk it follows stopped: the stop, and the stopped merge its own
// contains when it was left out for that.
func (d *Delivery) stoppedWords() string {
s := fmt.Sprintf("its walk %s was stopped by %s", d.Walk.ID, d.Walk.StoppedBy)
if d.Walk.StoppedWhy != "" {
s += ": " + d.Walk.StoppedWhy
}
switch {
case len(d.Walk.Shares) > 0:
s += fmt.Sprintf("; its merge moves %s, as the stopped merge %s does: built at the branch, which holds that "+
"merge, it would deliver it, so it was left out with it (novox/hq ADR 0299)", strings.Join(d.Walk.Shares, ", "),
shortOf(d.Walk.Contains, 12))
case d.Walk.Contains != "":
s += fmt.Sprintf("; its merge contains the stopped merge %s, so it cannot be walked without delivering it, "+
"and was left out with it (novox/hq ADR 0299)", shortOf(d.Walk.Contains, 12))
}
return s
}
// walkMovedItsModules is whether the walk it follows built every module its plan moves, and whether that is known:
// from the modules the walk's record names, or, for a record kept before they were, from a walk of no tier, which
// moved nothing. Unknown when its plan is not known.
@@ -61,6 +61,10 @@ type Controller interface {
Deliver(walk, why string) error
// Stop ends a walk, said as stopped by whom.
Stop(walk, why, by string) error
// StopWithout ends a walk and walks its other merges again without the merges named, each
// `<repository>@<commit>`, and without each later merge of their repository, which contains one (novox/hq
// ADR 0299).
StopWithout(walk, why, by string, without []string) (StopAnswer, error)
// Walks are the walks the controller keeps — one, given its id — and whether this seat has a holder on record.
Walks(walk string) (bool, []Walk, error)
}
@@ -269,6 +273,41 @@ func (c seatController) Stop(walk, why, by string) error {
return err
}
// StopAnswer is what the controller's `delivery-stop` with merges to leave out answers (novox/hq ADR 0299): the walk
// stopped, the batch that walks its other merges again with those merges, and every merge left out.
type StopAnswer struct {
Stopped string `json:"stopped"`
WalkedAgainBy string `json:"walked_again_by,omitempty"`
WalkedAgain []string `json:"walked_again,omitempty"`
LeftOut []WalkLeftOut `json:"left_out,omitempty"`
// Kept are the modules the stopped walk sent that nothing walks again, on the machines named; Withheld the
// modules the walk that walks the others withholds; StillCarriedBy the merges elsewhere that will deliver a
// stopped change when walked — a later merge of its repository, or one moving its modules (ADR 0299).
Kept []WalkKept `json:"kept,omitempty"`
Withheld []string `json:"withheld,omitempty"`
StillCarriedBy []string `json:"still_carried_by,omitempty"`
}
// WalkKept is a module a stopped walk left running its build, and where.
type WalkKept struct {
Module string `json:"module"`
Machines []string `json:"machines"`
}
func (c seatController) StopWithout(walk, why, by string, without []string) (StopAnswer, error) {
raw, err := c.verb("delivery-stop", map[string]any{"plan": walk, "why": why, "by": by,
"without": strings.Join(without, ",")})
if err != nil {
return StopAnswer{}, err
}
var a StopAnswer
if json.Unmarshal(raw, &a) != nil || a.Stopped == "" {
return StopAnswer{}, fmt.Errorf("the controller's delivery-stop did not say what it left out: %s",
lastLine(string(raw)))
}
return a, nil
}
func (c seatController) Walks(walk string) (bool, []Walk, error) {
args := map[string]any{}
if walk != "" {
@@ -84,7 +84,7 @@ const (
EvRetire Event = "retire" // a person's word that a held delivery is history, never to be walked
EvCovered Event = "covered" // a later merge into the same trunk delivered every module it moves
EvLifted Event = "lifted" // what held it for a person is no longer so: its group's composed check passed, or a walk now answers its merge
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed: it waits for a walk of its own commit
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed, or was stopped for another merge (ADR 0299): it waits for the walk that walks it again
)
// Facts are what a guard reads beyond the delivery itself: the moment, and who asks, for an act.
@@ -259,6 +259,9 @@ var Table = []Row{
if d.Walk != nil && d.Walk.State == walkFailed && d.carriedLater() {
return errors.New("its walk, which carried a later commit for it, failed: it waits for a walk of its own commit")
}
if d.walkStopped() {
return errors.New("its walk was stopped: it waits for the walk that walks its merge again, or ends stopped")
}
return unless(d.Walk != nil && d.Walk.Started(), "its walk waits")
}},
// **A walk that ended before its word settles its delivery** (novox/hq issue 474): a batch's walk the
@@ -284,11 +287,21 @@ var Table = []Row{
}},
{From: []State{Published}, Event: EvStop, To: Stopped,
Guard: "its walk ended done before its word without building every module it moves, and no later merge that " +
"contains it delivered them: no start can come, and nothing delivered it (novox/hq issue 474)",
"contains it delivered them: no start can come, and nothing delivered it (novox/hq issue 474); or the walk " +
"it waits on was stopped through this owner and walks its merge again nowhere — left out, or the walk " +
"ended whole (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
if d.walkStopped() && !d.walkedAgain() {
return nil
}
return unless(d.endedUnstarted() && d.Stranded != "", "its walk has not ended unstarted, or something may cover it")
},
Says: func(d *Delivery) string { return d.Stranded }},
Says: func(d *Delivery) string {
if d.walkStopped() {
return d.stoppedWords()
}
return d.Stranded
}},
{From: []State{Held}, Event: EvGo, To: Delivering,
Guard: "its walk started on a word that was not this owner's: the controller's own path, which waits for " +
"nobody, or a person's `plans go`",
@@ -332,10 +345,15 @@ var Table = []Row{
return "covered: a later merge that contains it was delivered: " + d.Carrier
}},
{From: []State{Delivering}, Event: EvStop, To: Stopped,
Guard: "its walk was stopped through this owner",
Guard: "its walk was stopped through this owner, and walks its merge again nowhere: left out, or the walk ended " +
"whole (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
return unless(d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy != "", "its walk was not stopped")
}},
if d.walkedAgain() {
return fmt.Errorf("its walk was stopped without it, and %s walks its merge again", d.Walk.WalkedAgainBy)
}
return unless(d.walkStopped(), "its walk was not stopped")
},
Says: func(d *Delivery) string { return d.stoppedWords() }},
{From: []State{Delivering}, Event: EvFailed, To: Failed,
Guard: "its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine",
Holds: func(d *Delivery, f Facts) error {
@@ -344,12 +362,20 @@ var Table = []Row{
}},
{From: []State{Delivering}, Event: EvRewalk, To: Published,
Guard: "its walk failed carrying a later merge of its branch for it: a failed walk marks no merge delivered, and this " +
"merge waits, published, for a walk of its own commit (novox/hq ADR 0276 decision 3)",
"merge waits, published, for a walk of its own commit (novox/hq ADR 0276 decision 3); or its walk was stopped " +
"for another merge, and its merge is walked again without that one (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
if d.walkedAgain() {
return nil
}
return unless(d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy == "" && d.carriedLater(),
"its walk did not fail, or carried its own commit")
},
Says: func(d *Delivery) string {
if d.walkedAgain() {
return fmt.Sprintf("its walk %s was stopped for another merge (%s): %s walks its merge again without that one",
d.Walk.ID, d.Walk.StoppedWhy, d.Walk.WalkedAgainBy)
}
return fmt.Sprintf("its walk %s, which carried %s for it, failed: it waits for a walk of its own commit",
d.Walk.ID, shortOf(d.AnsweredBy, 12))
}},
@@ -542,6 +568,12 @@ func boundOf(d *Delivery) (Bound, bool) {
if d.waitsForItsOwnWalk() {
return rewalkBound, true
}
if (d.State == Published || d.State == Delivering) && d.HeldBack != "" {
// Its walk carries a stopped merge and is not let go (ADR 0299): said with why, past the bound of its state.
b := Bounds[d.State]
b.For, b.Says = Bounds[Published].For, d.HeldBack // a walk not let go waits for a word, whatever its state
return b, true
}
if d.State == Published && d.TurnAwaits != "" && d.TurnAwaitsUntil != nil {
// Its turn waits for a later merge to carry a member before it: bounded by that carrier's own bound, and
// said with it, never the generic "waits for its turn or its word" (novox/hq issue 445).
@@ -13,8 +13,8 @@ import (
// delivery time from its own merge, its class and its phases, and `times` answers the week's median and worst
// per class against its budget. Measurement only: nothing here holds, releases or changes a delivery.
// The budgets by class (ADR 0282 decision 1).
var budgets = map[string]time.Duration{"leaf": 5 * time.Minute, "core": 10 * time.Minute}
// The budgets by class (ADR 0282 decision 1, the leaf's as ADR 0304 decision 4 changed it).
var budgets = map[string]time.Duration{"leaf": 3 * time.Minute, "core": 10 * time.Minute}
// classes in the order `times` says them.
var classes = []string{"leaf", "core"}
@@ -68,8 +68,8 @@ func TestADeliveryKeepsItsTimeFromItsOwnMerge(t *testing.T) {
if d.Time == nil {
t.Fatalf("no delivery time kept: %+v", d)
}
if d.Time.TookMS != (280*time.Second).Milliseconds() || d.Time.Class != "leaf" || d.Time.Over {
t.Fatalf("time %dms (want 280000, from its own merge), class %s, over %v", d.Time.TookMS, d.Time.Class, d.Time.Over)
if d.Time.TookMS != (280*time.Second).Milliseconds() || d.Time.Class != "leaf" || !d.Time.Over {
t.Fatalf("time %dms (want 280000, from its own merge, over a leaf's three minutes), class %s, over %v", d.Time.TookMS, d.Time.Class, d.Time.Over)
}
var sum int64
for _, p := range d.Time.Phases {
@@ -133,9 +133,13 @@ func TestADeliveryKeepsItsTimeFromItsOwnMerge(t *testing.T) {
t.Fatalf("merged after its walk's end: %+v", dt)
}
// Exactly its budget is within it.
p = phasesOf("leaf", from, ph{"build", 5*time.Minute + 10*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); dt.Over || dt.TookMS != 300000 {
t.Fatalf("five minutes is within a leaf's budget: %+v", dt)
p = phasesOf("leaf", from, ph{"build", 3*time.Minute + 10*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); dt.Over || dt.TookMS != 180000 {
t.Fatalf("three minutes is within a leaf's budget: %+v", dt)
}
p = phasesOf("leaf", from, ph{"build", 3*time.Minute + 11*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); !dt.Over {
t.Fatalf("a second past three minutes is over a leaf's budget: %+v", dt)
}
// Seen again, unchanged: not written again (issue 326).
before := w.store.puts
@@ -174,8 +178,8 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
return walkedOnce(w, n, "head"+string(rune('a'+n)), "merge"+string(rune('a'+n))+"zzzz",
phasesOf("leaf", t0.Add(at), ph{"window", 10 * time.Second}, ph{"judgement", d}))
}
leaf(1, 2*time.Minute, 0)
leaf(2, 4*time.Minute, time.Hour)
leaf(1, 1*time.Minute, 0)
leaf(2, 2*time.Minute, time.Hour)
over := leaf(3, 6*time.Minute, 2*time.Hour)
walkedOnce(w, 4, "headcore", "mergekore1", phasesOf("core", t0.Add(3*time.Hour), ph{"judgement", 9 * time.Minute}))
w.later(4 * time.Hour)
@@ -184,8 +188,8 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
t.Fatalf("classes: %+v", a.Classes)
}
l := a.Classes[0]
// Each delivery's time is from its own merge, ten seconds after the walk's: 2m, 4m, 6m.
if l.Count != 3 || l.Within != 2 || l.Over != 1 || l.MedianMS != (4*time.Minute).Milliseconds() ||
// Each delivery's time is from its own merge, ten seconds after the walk's: 1m, 2m, 6m.
if l.Count != 3 || l.Within != 2 || l.Over != 1 || l.MedianMS != (2*time.Minute).Milliseconds() ||
l.WorstMS != (6*time.Minute).Milliseconds() || l.WorstID != over {
t.Fatalf("leaf: %+v", l)
}
@@ -198,7 +202,7 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
if len(a.Over) != 1 || a.Over[0].ID != over {
t.Fatalf("over: %+v", a.Over)
}
if !strings.Contains(a.Line, "leaf: median 4m0s, worst 6m0s, 2 of 3 within 5m0s") {
if !strings.Contains(a.Line, "leaf: median 2m0s, worst 6m0s, 2 of 3 within 3m0s") {
t.Fatalf("the line: %s", a.Line)
}
// Past the window, nothing.
+182 -19
View File
@@ -29,6 +29,9 @@ type Line struct {
// waitsFor is what a delivery waits for, in words.
func (h *Holder) waitsFor(d *Delivery) string {
if d.HeldBack != "" && !d.State.Final() {
return d.HeldBack
}
switch d.State {
case Proposed:
return "its check"
@@ -326,10 +329,19 @@ func (h *Holder) Release(id, why, by string) (string, error) {
if d.Walk != nil && d.Walk.Waits {
walk = d.Walk.ID
}
var stopped []string
if walk != "" {
stopped, _ = h.carriesStopped(walk)
}
h.mu.Unlock()
if walk == "" {
return fmt.Sprintf("%s is delivering, on %s's word", id, by), nil
}
if len(stopped) > 0 {
// Never the word for a walk that carries a stopped change (ADR 0299): asked to leave it out first.
return "", fmt.Errorf("%s is delivering on %s's word, and its walk %s is not let go yet: it carries %s, whose "+
"delivery was stopped, and is asked to leave it out first", id, by, walk, strings.Join(stopped, ", "))
}
if err := h.Controller.Deliver(walk, "released by "+by+": "+why); err != nil {
return "", fmt.Errorf("%s is delivering on %s's word, and its walk %s could not be let go yet (%v): the next "+
"tick asks again", id, by, walk, err)
@@ -346,25 +358,98 @@ func (h *Holder) Release(id, why, by string) (string, error) {
return fmt.Sprintf("%s is delivering on %s's word; its walk %s is let go", id, by, walk), nil
}
// Stop is the `stop` verb: a delivery stopped, its walk ended through the controller once it carries no other live
// delivery (novox/hq issue 459), and in a group every member after it stopped too.
func (h *Holder) Stop(id, why, by string) (string, error) {
// Stop is the `stop` verb with no way given: see StopWay.
func (h *Holder) Stop(id, why, by string) (string, error) { return h.StopWay(id, why, by, "") }
// The ways a person's stop of a delivery on a shared walk takes (novox/hq ADR 0299).
const (
wayAgain = "again" // the walk ended, its other merges walked again without this one
wayWalk = "walk" // the whole walk stopped, every delivery on it with it
)
// StopWay is the `stop` verb: a delivery stopped, its walk ended through the controller, and in a group every
// member after it stopped too.
//
// **A stop never takes other deliveries down quietly** (novox/hq issue 459, ADR 0299): one walk carries every merge
// of its batch. When the delivery's open walk carries anything else, the stop does nothing at first: it says what
// ending the walk would take down — each other delivery, and which of them contain this merge and end either way —
// and the two ways, given with `way`: `again` ends the walk and walks the others again without this merge and the
// merges that contain it; `walk` stops the whole walk, and every delivery on it ends stopped, naming this stop.
func (h *Holder) StopWay(id, why, by, way string) (string, error) {
if way != "" && way != wayAgain && way != wayWalk {
return "", fmt.Errorf("way %q: %q or %q", way, wayAgain, wayWalk)
}
h.mu.Lock()
d := h.deliveries[id]
if d == nil {
h.mu.Unlock()
return "", fmt.Errorf("no delivery %q", id)
}
// Its guard is read on a copy: the delivery is stopped only once its walk is (review of #221).
trial := *d
if _, err := Apply(&trial, EvStop, true, Facts{Now: h.Now(), Why: why, By: by}); err != nil {
h.mu.Unlock()
return "", err
}
var walk string
if d.Walk != nil && (d.Walk.State == walkBuilding || d.Walk.State == walkRolling) {
walk = d.Walk.ID
}
merge := d.Repository + "@" + d.MergedAs
// A merge that waits in a batch not yet cut: the batch is asked to leave it out once it is stopped (ADR 0299).
var batch string
if b, batched := h.batchOf(d); batched && walk == "" {
batch = b.ID
}
var others []string
if walk != "" {
others = h.othersOn(walk, []string{merge})
}
if len(others) > 0 && way == "" {
said := h.takenDown(d, walk, others)
h.mu.Unlock()
return "", errors.New(said)
}
h.mu.Unlock()
var did string
switch {
case walk == "":
case len(others) == 0 || way == wayWalk:
whole := why
if len(others) > 0 {
whole = fmt.Sprintf("%s was stopped, and its whole walk with it: %s", id, why)
}
if err := h.Controller.Stop(walk, whole, by); err != nil {
return "", fmt.Errorf("%s is not stopped: its walk %s could not be ended yet: %v — `plans stop %s --why …` "+
"ends it by hand", id, walk, err, walk)
}
did = fmt.Sprintf("; its walk %s is ended", walk)
if len(others) > 0 {
did = fmt.Sprintf("; its walk %s is ended whole, and every delivery on it with it: %s", walk,
strings.Join(others, ", "))
}
default:
a, err := h.Controller.StopWithout(walk, why, by, []string{merge})
if err != nil {
return "", fmt.Errorf("%s is not stopped: its walk %s could not be ended without it: %v — `way: %s` stops "+
"the whole walk, and `plans stop %s --why …` ends it by hand", id, walk, err, wayWalk, walk)
}
did = fmt.Sprintf("; its walk %s is ended without it, and %s", walk, againWords(a))
}
h.mu.Lock()
d = h.deliveries[id]
if d == nil || d.State.Final() {
h.mu.Unlock()
return fmt.Sprintf("%s ended meanwhile%s", id, did), nil
}
t, err := Apply(d, EvStop, true, Facts{Now: h.Now(), Why: why, By: by})
if err != nil {
h.mu.Unlock()
return "", err
}
h.moved(d, []Transition{t})
var walk string
if d.Walk != nil && (d.Walk.State == walkBuilding || d.Walk.State == walkRolling) {
walk = d.Walk.ID
}
var asks []func()
if g := h.groups[d.Group]; g != nil && g.Closed {
asks = h.groupTurns(g, h.membersOf(g))
@@ -375,21 +460,73 @@ func (h *Holder) Stop(id, why, by string) (string, error) {
for _, a := range asks {
a()
}
if walk == "" {
return fmt.Sprintf("%s is stopped, on %s's word", id, by), nil
if batch != "" {
// Asked now, not at the next tick; and the batch's walk is never let go while it carries the merge.
did = fmt.Sprintf("; its merge waits in the batch %s, which was asked now to leave it out and is never let go "+
"while it carries it: %s", batch, h.stopLeaving(batch, why, by, []string{merge}))
}
// The walk is ended only when it carries no other live delivery: since the merge window it may carry merges of
// other repositories and groups, which a stop of this one does not stop (novox/hq issue 459).
ended, goesOn, err := h.endWalk(walk, id, why, by)
if err != nil {
return "", fmt.Errorf("%s is stopped, and its walk %s could not be ended yet: %v — `plans stop %s --why …` "+
"ends it by hand", id, walk, err, walk)
return fmt.Sprintf("%s is stopped, on %s's word%s", id, by, did), nil
}
// takenDown is what ending a delivery's shared walk would take down, and the two ways: for a caller holding the
// lock (ADR 0299).
func (h *Holder) takenDown(d *Delivery, walk string, others []string) string {
var ride, contain []string
for _, o := range others {
od := h.deliveries[o]
if od != nil && containsMerge(od, d) {
contain = append(contain, o)
continue
}
ride = append(ride, o)
}
if !ended {
return fmt.Sprintf("%s is stopped, on %s's word; its walk %s goes on for %s, which it also carries and which "+
"were not stopped", id, by, walk, strings.Join(goesOn, ", ")), nil
s := fmt.Sprintf("%s is not stopped: its walk %s carries other changes, and ending the walk would take them down: %s.",
d.ID, walk, orNone(strings.Join(ride, ", ")))
if len(contain) > 0 {
s += fmt.Sprintf(" These contain its merge and end with it either way: %s.", strings.Join(contain, ", "))
}
return fmt.Sprintf("%s is stopped, on %s's word; its walk %s is ended", id, by, walk), nil
// A module is built at its branch, which holds this merge: a delivery moving a module this one moves would
// deliver it, and the controller leaves it out too (ADR 0299). Its plan is what is known here; the controller
// decides, and its answer names each.
var share []string
for _, o := range ride {
if od := h.deliveries[o]; od != nil && od.Plan != nil && d.Plan != nil {
if both := sharedModules(od.Plan.Moved, d.Plan.Moved); len(both) > 0 {
share = append(share, fmt.Sprintf("%s (moves %s)", o, strings.Join(both, ", ")))
}
}
}
if len(share) > 0 {
s += fmt.Sprintf(" With `way: %s` these are left out too, for moving a module this merge moves: %s.", wayAgain,
strings.Join(share, ", "))
}
s += fmt.Sprintf(" With `way: %s` the controller may leave out more for sharing a module, and its answer names each.",
wayAgain)
return s + fmt.Sprintf(" What the walk already sent stays where it was sent; the stop's answer names it."+
" Choose a way: `way: %s` ends the walk and walks the others again without this merge; "+
"`way: %s` stops the whole walk and every delivery on it.", wayAgain, wayWalk)
}
// sharedModules are the modules in both lists, sorted.
func sharedModules(a, b []string) []string {
var out []string
for _, m := range a {
if containsString(b, m) && !containsString(out, m) {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// containsMerge is whether o is a later merge of d's repository and branch, which contains d's merge on a linear
// trunk: merged after it, or the merge the walk carries for it.
func containsMerge(o, d *Delivery) bool {
if o.ID == d.ID || !strings.EqualFold(o.Repository, d.Repository) || !strings.EqualFold(o.Base, d.Base) || o.MergedAs == "" {
return false
}
return o.MergedAt.After(d.MergedAt) || d.AnsweredBy != "" && sameCommit(d.AnsweredBy, o.MergedAs) &&
!sameCommit(d.AnsweredBy, d.MergedAs)
}
// Close is healer H2's verb: for a delivery past its bound, the walk's record is read again and the
@@ -498,6 +635,18 @@ func (h *Holder) Status(listening string) map[string]any {
out := map[string]any{"deliveries": byState, "groups": len(h.groups), "owed": owed, "unkept": len(h.dirty),
"walks-unmatched": len(h.unmatched), "events": listening, "started": h.started.Format(time.RFC3339),
"seat-on-record": h.held}
// Each held delivery by name, with why and the verb that releases it (novox/hq issue 529): a count alone
// pointed a reader at nothing.
var held []string
for _, d := range h.deliveries {
if d.State == Held {
held = append(held, HeldLine(d))
}
}
if len(held) > 0 {
sort.Strings(held)
out["held"] = held
}
if !h.reconciled.IsZero() {
out["walks-read"] = h.reconciled.Format(time.RFC3339)
}
@@ -606,3 +755,17 @@ func withoutKinds(owed []Effect, kinds ...string) []Effect {
}
return out
}
// HeldLine says a held delivery in one line: which, since when, why, its walk when one waits, and the verb that
// releases it (novox/hq issue 529).
func HeldLine(d *Delivery) string {
why := d.HeldWhy
if why == "" {
why = lastWhyOf(d)
}
s := fmt.Sprintf("%s held since %s: %s", d.ID, d.Since.UTC().Format(time.RFC3339), why)
if d.Walk != nil && d.Walk.Waits {
s += "; its walk " + d.Walk.ID + " waits for it"
}
return s + " — a person decides: `mesh-delivery.release` (id " + d.ID + ", why) lets it go, `mesh-delivery.stop` ends it"
}
+77 -18
View File
@@ -19,7 +19,7 @@ what it refuses.
|---|---|---|
| `open` | `title`, `report`, optionally `located-in`, `record` | checks the issue, then takes the next number and keeps it under that number; answers the number |
| `get` | `number` | the issue whole: fields, notes, links |
| `list` | optionally `status`, `owner`, `phrase`, `limit`, `severity`, `needs`, `ready`, `flag` (`delivered-not-resolved` or `abandoned`) | short rows, newest first: number, status, owner, title, and each flag that holds |
| `list` | optionally `status`, `owner`, `phrase`, `limit`, `severity`, `needs`, `ready`, `flag` (`delivered-not-resolved` or `abandoned`), `search` (words, ranked), `before` (a number, the cursor) | short rows, newest first, or best first for a `search`: number, status, owner, title, and each flag that holds; `next`, the `before` that reads on, when more match |
| `move` | `number`, `status`, and what that status needs (`by` for `diagnosing` and `fixing`, optionally `take-over`) | changes the status under the development cycle's rules |
| `note` | `number`, `text` | appends a note; a note is never edited |
| `link` | `number`, `kind`, `target`, optionally `remove` | adds or removes a typed link |
@@ -167,7 +167,7 @@ It is a verb on the seat, callable by any agent, because it can do only this:
`imported` event. What it writes comes from hq's `main` as the records module holds it, which is reviewed
in pull requests.
- **What it keeps is bounded** as every verb's text is: the title 200 bytes, the report 64 KiB, one issue
256 KiB; a report that trips a credential's shape is refused. An entry of `located-in` or `fixed-by` hq
252 KiB; a report that trips a credential's shape is refused. An entry of `located-in` or `fixed-by` hq
wrote is prose at times, so it is taken within a reason's bound (2000 bytes), not a line's.
- **An owner hq named that the registered list does not know is kept as written and marked**, in
`located-in-unverified` on the issue and `unverified-owners` in the answer, never refused: the list is
@@ -200,14 +200,15 @@ in one instance or two — never take one number. A write another opener won is
bounded number of times with a short random wait; past the bound the open is refused and takes no number.
- **A refused open takes no number**: an issue is checked before its number is taken.
- **A number is never handed out twice, and may be skipped.** A number taken and then not kept — the bus
failing between the counter and the issue — stays unused, and the answer says which. A write whose answer
was lost is recognised as its own when tried again, so it leaves no gap; when it cannot be confirmed either
way, the answer says the issue may have been written and to `get` that number before opening it again,
never that nothing was opened.
- **An issue is never overwritten.** It is kept by a write that is refused when the key exists; if the
counter ever hands out a number already kept, the open is refused loudly, the number stays unused, and
nothing is overwritten.
- **A number is never handed out twice, and may be skipped.** A number taken and then not taken by the log —
the bus failing between the counter and the issue's first operation, or a full log — is spent, and the open
answers the failure (hq ADR 0300 §2). An append whose answer was lost is recognised as its own by the open's
id when tried again or read back, so it leaves no gap; when it cannot be confirmed either way, the answer says
the issue may have been written and to `get` that number before opening it again, never that nothing was
opened.
- **An issue is never overwritten.** Its first operation is appended at `expect` 0, refused when the log
already holds one for that number; if the counter ever hands out a number already kept, the open is refused
loudly, the number stays unused, and nothing is overwritten.
- **The counter starts once**, the first time a number is taken: at the setting `first-number`, or after
the highest issue already kept, whichever is higher, made only if it does not exist. The setting is never
read again, so a number never moves.
@@ -242,16 +243,63 @@ sessions writing a record at once never take one number.
`decision-floor` or `research-floor` (the controller's settings verb) to the highest number on any open
branch of hq; once the counter has started, the floor is not read again.
## State and events
## The log, the bucket and the events (hq ADR 0300)
The states, declared in the manifest and made by the controller; the module never makes a bucket
(ADR 0201): `issues`, one key per issue by number, keeping 8 values per key; `counters`, one value per key
(`issue`, `decision`, `research`); `grooming`, the backlog's pass; `reservations`, one key per reserved number.
Neither expires. One issue is kept in at most 256 KiB; a note that would pass it is refused.
**The record is the log `changes`** (declared in the manifest's `logs`, made by the controller, hq ADR 0297),
one key per issue by number. Every write — open, move, note, link, set, the triage, a delivery's word on a
fix, a warning said or cleared, the import — is one operation appended under the issue's number with what
changed, who acted and when, at the sequence of the last operation the issue was folded at
(`AppendAt`). Another operation in between refuses it: nothing is written, the log is folded again and the
change applied to that, a bounded number of times. A write the log refused writes nothing; only an open's
number, once taken, stays spent. An operation that begins an issue (`opened`, `imported`, `carried`) holds it
whole; every other one names the fields given a new value (`set`), the lists added to at their end (`add`:
a note is kept as itself, not as every note again) and the fields taken away (`unset`). The fold applies them
as written, so a new kind of operation needs nothing new in the fold.
Every change is an event: `mesh-issues.opened`, `.moved`, `.noted`, `.linked`, `.imported`, `.reserved`, each with the
number, what changed, who changed it and when. The events are the audit trail. A change whose event the bus did not take
is kept, and its answer says the event was not said (`announced: false`).
**The bucket `issues` is the fold of the log**, each issue carrying `folded`, the sequence
of its last operation. It is written after the append, at the revision read; one whose write fails after the
log took the change is behind, and caught up at its next change (the stale append is refused and the log
folded) or at the next start. Agents and views watch it as before. It still keeps 8 values per key, so the bus
cannot drop the older values before the carry has read them; it goes to one value per key (ADR 0300 §3) in a
follow-up, once the carry has run with a read of those values.
**On start**, the tracker watches its own bucket, carries what the bucket holds and the log does not (below),
reads the whole log page by page, folds every issue whose last operation is newer than its `folded` — an
emptied bucket is rebuilt whole — and raises the issue counter to the highest number in the log where it is
below. Until then a `list` reads the bucket and a write carries the issue it changes first.
**The switch** (ADR 0300 §6) is the carry, run on every start and before any write of an issue not carried:
for every issue the bucket holds with no `folded`, each value the bucket still keeps of it, read through the
runtime's `mesh/state.history` (hq ADR 0307), is appended oldest first as a `carried` operation holding the
whole issue and the bucket's revision of that value, and the fold written. It resumes rather than restarts: a
kept value at or below the highest revision a carried operation holds, or one the tracker wrote itself (it has
`folded`), is known to the log, so an interrupted carry, or two instances carrying one issue at once, carry each
value once. Matched by revision, not content, because the same content may be kept twice. A value written beside
the log after the carry, by an instance from before the switch, is carried as one more operation and is never
written over by a fold. A deletion among the kept values is passed over and counted, since the tracker never
deletes an issue and every value after one holds the issue whole.
An issue that cannot be carried — the log refusing it, or a value this module cannot read — is a fault, named
with its reason in the start line; the others are carried and the start finishes. That issue stays in the
bucket as it was, read and listed as before, and every write to it is refused with the same reason until it
can be carried, so no operation reaches the log before its carry. One issue is kept in at most 252 KiB, so that
the operation holding it whole fits one entry of the log (256 KiB). An entry of the log that cannot be folded
faults its issue on start: nothing folded past it is written, and its bucket value is left as it is.
**In memory** the tracker holds every folded issue, kept current by the watch and its own writes, and a
full-text index of title, report and notes, ranked by BM25 (title words weigh three), rebuilt on every start.
`list` answers from memory, with `search` ranked best first, and `before` paging the newest-first list past
the 500 rows one answer holds. The index is the module's own: a few thousand issues need a hundred lines, not a
library of dozens of modules.
The other states: `counters`, one value per key (`issue`, `decision`, `research`); `grooming`, the backlog's
pass; `reservations`, one key per reserved number. None expires. One issue is kept in at most 252 KiB; a note
that would pass it is refused.
**The events announce; the log is the audit trail.** Every change is still an event: `mesh-issues.opened`,
`.moved`, `.noted`, `.linked`, `.imported`, `.reserved`, each with the number, what changed, who changed it and
when. A change whose event the bus did not take is kept, and its answer says the event was not said
(`announced: false`).
## Who acted
@@ -319,3 +367,14 @@ date, a replay, an empty list, a wontfix): the frontmatter read as written, each
number with the counter untouched, a second run changing nothing, an issue the tracker holds differently
left alone, a folder-backed issue filled once, a folder past the counter or past the first number refused,
a dry run writing nothing, and a report that cannot be read refused alone with why.
The log (hq ADR 0300) is tested in `log_test.go` over a log kept in the test with the runtime's semantics
(`expect`, `ErrChanged`): a write whose append is refused changing nothing, an operation in between not lost, an
open refused at validation taking no number, an append failing after the number spending it and answering the
failure, an emptied bucket restored exactly by the fold on start, a bucket behind its log catching up, the
counter raised to the log, each write saying its event beside its one operation, `before` paging 530 issues
without a gap or a repeat, a ranked search with the index rebuilt on start and kept current by the watch, and
the carry carrying each value kept, oldest first, once, passing over a deletion among them. Over the bus, a
write to an issue the bucket kept before the switch carries its three kept values first, read through the
runtime's history. Over the bus, the stream the controller makes holds one
operation per change of `TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus`.
+169 -1
View File
@@ -12,6 +12,7 @@ import (
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
@@ -48,6 +49,8 @@ type bus struct {
nc *nats.Conn
js jetstream.JetStream
kvs map[string]jetstream.KeyValue
// logs is each declared log's stream, made as the controller makes it (novox/hq ADR 0297).
logs map[string]jetstream.Stream
}
func newBus(t *testing.T) *bus {
@@ -73,7 +76,7 @@ func newBus(t *testing.T) *bus {
if err != nil {
t.Fatal(err)
}
b := &bus{nc: nc, js: js, kvs: map[string]jetstream.KeyValue{}}
b := &bus{nc: nc, js: js, kvs: map[string]jetstream.KeyValue{}, logs: map[string]jetstream.Stream{}}
// The controller's job: a bucket per declared state, with the options the manifest gives it.
m, _ := readManifest(t)
for _, d := range m.stateDeclarations(t) {
@@ -88,9 +91,98 @@ func newBus(t *testing.T) *bus {
}
b.kvs[d.Name] = kv
}
for _, name := range m.Logs {
st, err := js.CreateStream(context.Background(), jetstream.StreamConfig{
Name: "LOG_" + m.Module + "_" + name, Subjects: []string{"mesh.log." + m.Module + "." + name + ".>"},
Storage: jetstream.FileStorage, MaxMsgsPerSubject: -1, Discard: jetstream.DiscardNew, AllowDirect: true,
DenyDelete: true, DenyPurge: true})
if err != nil {
t.Fatal(err)
}
b.logs[name] = st
}
return b
}
// logVerb is the runtime's mesh/log verbs over a stream, as the tool runner serves them (ADR 0297).
func (b *bus) logVerb(ctx context.Context, verb string, params json.RawMessage) (any, error) {
var p struct {
Log string `json:"log"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Expect *uint64 `json:"expect"`
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
}
if err := json.Unmarshal(params, &p); err != nil {
return nil, err
}
st := b.logs[p.Log]
if st == nil {
return nil, fmt.Errorf("the log %q is not declared", p.Log)
}
subject := func(key string) string { return "mesh.log.mesh-issues." + p.Log + "." + key }
entry := func(m *jetstream.RawStreamMsg) map[string]any {
return map[string]any{"sequence": m.Sequence, "key": m.Subject[len(subject("")):], "value": json.RawMessage(m.Data),
"at": m.Time}
}
switch verb {
case "append":
msg := nats.NewMsg(subject(p.Key))
msg.Data = p.Value
if p.Expect != nil {
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(*p.Expect, 10))
}
ack, err := b.js.PublishMsg(ctx, msg)
if err != nil {
return nil, err
}
return map[string]any{"sequence": ack.Sequence}, nil
case "last":
m, err := st.GetLastMsgForSubject(ctx, subject(p.Key))
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
return entry(m), nil
case "read":
filter := subject(">")
if p.Keys != "" {
filter = subject(p.Keys)
}
limit := p.Limit
if limit <= 0 {
limit = 100
}
info, err := st.Info(ctx)
if err != nil {
return nil, err
}
entries := []map[string]any{}
var next uint64
for seq := max(p.From, 1); seq <= info.State.LastSeq; {
m, err := st.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return nil, err
}
if len(entries) == limit {
next = m.Sequence
break
}
entries = append(entries, entry(m))
seq = m.Sequence + 1
}
return map[string]any{"entries": entries, "next": next, "last": info.State.LastSeq}, nil
}
return nil, fmt.Errorf("the runtime answers no mesh/log.%s", verb)
}
// instance is one running bundle, and the runtime's half of its conversation.
type instance struct {
t *testing.T
@@ -213,6 +305,9 @@ func (i *instance) answer(m rpc) {
}
return map[string]any{"decision": 291, "research": 45, "records": 336}, nil
}
if verb, ok := strings.CutPrefix(m.Method, "mesh/log."); ok {
return i.b.logVerb(ctx, verb, m.Params)
}
kv := i.b.kvs[p.State]
if kv == nil {
return nil, fmt.Errorf("the state %q is not declared", p.State)
@@ -239,6 +334,26 @@ func (i *instance) answer(m rpc) {
case "mesh/state.put":
rev, err := kv.Put(ctx, p.Key, p.Value)
return map[string]any{"revision": rev}, err
case "mesh/state.history":
// As the runtime serves it (novox/hq ADR 0307): every value kept, oldest first, puts and deletes.
kept, err := kv.History(ctx, p.Key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return []any{}, nil
}
if err != nil {
return nil, err
}
out := []map[string]any{}
for _, e := range kept {
r := map[string]any{"key": e.Key(), "revision": e.Revision(), "op": "put", "at": e.Created()}
if e.Operation() == jetstream.KeyValuePut {
r["value"] = json.RawMessage(e.Value())
} else {
r["op"] = "delete"
}
out = append(out, r)
}
return out, nil
case "mesh/state.keys":
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
@@ -471,6 +586,42 @@ func TestFortyReservationsAtOnceFromTwoInstancesGetFortyDistinctConsecutiveNumbe
}
}
// The carry over the bus (novox/hq ADR 0300 §6, ADR 0307): values the bucket kept before the switch are read
// through the runtime's history, oldest first, and carried before the write that touches the issue.
func TestAWriteOverTheBusCarriesEveryValueTheBucketKeptFirst(t *testing.T) {
b := newBus(t)
var events []event
var evMu sync.Mutex
kv := b.kvs[IssuesState]
for _, title := range []string{"first", "second", "third"} {
raw, _ := json.Marshal(Issue{Number: 300, Title: title, Status: StatusOpen, Report: "r", OpenedBy: "x", Opening: "o300"})
if _, err := kv.Put(context.Background(), "300", raw); err != nil {
t.Fatal(err)
}
}
i := b.start(t, writeSettings(t, 365), &events, &evMu)
if _, err := i.call("note", map[string]any{"number": 300, "text": "after the switch"}); err != nil {
t.Fatal(err)
}
var titles []string
var ops []string
for seq := uint64(1); ; seq++ {
m, err := b.logs[ChangesLog].GetMsg(context.Background(), seq)
if err != nil {
break
}
var op Operation
_ = json.Unmarshal(m.Data, &op)
ops = append(ops, op.Op)
if op.Issue != nil {
titles = append(titles, op.Issue.Title)
}
}
if fmt.Sprint(ops) != "[carried carried carried noted]" || fmt.Sprint(titles) != "[first second third]" {
t.Fatalf("the log holds %v of %v", ops, titles)
}
}
// One instance, every verb over the bus: what an agent does with an issue from opening to resolving.
func TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus(t *testing.T) {
b := newBus(t)
@@ -531,4 +682,21 @@ func TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus(t *testing.T) {
if fmt.Sprint(kinds) != "[opened noted linked moved moved moved]" {
t.Fatalf("events %v", kinds)
}
// The record is the log (novox/hq ADR 0300 §1): one operation per change, in the stream the controller makes,
// and the issue in the bucket folded at the last of them.
var ops []string
for seq := uint64(1); ; seq++ {
m, err := b.logs[ChangesLog].GetMsg(context.Background(), seq)
if err != nil {
break
}
var op Operation
if err := json.Unmarshal(m.Data, &op); err != nil || m.Subject != "mesh.log.mesh-issues.changes.365" {
t.Fatalf("entry %d on %s: %s", seq, m.Subject, m.Data)
}
ops = append(ops, op.Op)
}
if fmt.Sprint(ops) != fmt.Sprint(kinds) || is.Folded != uint64(len(ops)) {
t.Fatalf("the log holds %v, the issue is folded at %d", ops, is.Folded)
}
}
@@ -0,0 +1,831 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// The tracker's record is its log `changes`, not its bucket (novox/hq ADR 0300 §1). Every write is one
// operation appended under the issue's number, refused when another operation on that issue came in between;
// only then is the folded issue written to the bucket `issues`, which is what the operations add up to (§3).
const ChangesLog = "changes"
// Log is the module's log as the runtime offers it (novox/hq ADR 0297): an append at the key's last sequence,
// refused with stdio.ErrChanged when another entry came in between, and a read in pages.
type Log interface {
AppendAt(key string, value any, expect uint64) (uint64, error)
Read(from uint64, keys string, limit int) (*stdio.LogPage, error)
}
// runtimeLog is Log through the runtime that launched this bundle.
type runtimeLog struct{}
func (runtimeLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
return stdio.Log(ChangesLog).AppendAt(key, value, expect)
}
func (runtimeLog) Read(from uint64, keys string, limit int) (*stdio.LogPage, error) {
return stdio.Log(ChangesLog).Read(from, keys, limit)
}
// errNoLog is a tracker started without its log: the terminal's dry run, which writes nothing.
var errNoLog = errors.New("this tracker runs without its log, so it writes nothing")
// logPage is how many entries one read of the log asks for: the most the runtime gives.
const logPage = 1000
// The kinds of operation. An operation that begins an issue holds it whole: opened, imported, carried.
const (
OpOpened = "opened"
OpImported = "imported"
// OpCarried is a value the bucket kept when the tracker switched to its log (ADR 0300 §6), whole.
OpCarried = "carried"
)
// Operation is one change to one issue, as the log keeps it: what changed, who acted and when (ADR 0300 §1).
// One that begins an issue, or carries a value the bucket kept, holds the issue whole; every other one names
// the fields it gave a new value, the lists it added to at their end, and the fields it took away. The fold
// applies them as they are written, so it needs no knowledge of the kind: a new kind is folded as it comes.
type Operation struct {
Op string `json:"op"`
By string `json:"by"`
At time.Time `json:"at"`
Issue *Issue `json:"issue,omitempty"`
Set map[string]json.RawMessage `json:"set,omitempty"`
Add map[string]json.RawMessage `json:"add,omitempty"`
Unset []string `json:"unset,omitempty"`
// Revision is the bucket's revision of the value a carried operation holds: how a carry knows which kept
// values the log holds already.
Revision uint64 `json:"revision,omitempty"`
}
// Act is who changes an issue and as what kind of operation.
type Act struct {
Op string
By string
}
// fields is an issue as its fields in JSON, without the sequence it was folded at: that is the log's, not
// the issue's.
func fields(is *Issue) (map[string]json.RawMessage, error) {
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
var m map[string]json.RawMessage
if err := json.Unmarshal(raw, &m); err != nil {
return nil, err
}
delete(m, "folded")
return m, nil
}
// operation is what changed from before to after as one operation of act, and false when nothing did.
func operation(act Act, at time.Time, before, after *Issue) (*Operation, bool, error) {
b, err := fields(before)
if err != nil {
return nil, false, err
}
a, err := fields(after)
if err != nil {
return nil, false, err
}
op := &Operation{Op: act.Op, By: act.By, At: at}
for k, v := range a {
old, had := b[k]
if had && bytes.Equal(old, v) {
continue
}
if !had {
// A list left out while empty: its first element is added like any other.
old = json.RawMessage("[]")
}
if tail, ok := appended(old, v); ok {
if op.Add == nil {
op.Add = map[string]json.RawMessage{}
}
op.Add[k] = tail
continue
}
if op.Set == nil {
op.Set = map[string]json.RawMessage{}
}
op.Set[k] = v
}
for k := range b {
if _, still := a[k]; !still {
op.Unset = append(op.Unset, k)
}
}
sort.Strings(op.Unset)
return op, len(op.Set)+len(op.Add)+len(op.Unset) > 0, nil
}
// appended is what a list gained at its end, when after is before with elements added and nothing else
// changed: a note or a link is kept in the log as itself, not as every note again.
func appended(before, after json.RawMessage) (json.RawMessage, bool) {
var b, a []json.RawMessage
if len(after) == 0 || after[0] != '[' || json.Unmarshal(before, &b) != nil || json.Unmarshal(after, &a) != nil || len(a) <= len(b) {
return nil, false
}
for i := range b {
if !bytes.Equal(compact(b[i]), compact(a[i])) {
return nil, false
}
}
tail, err := json.Marshal(a[len(b):])
if err != nil {
return nil, false
}
return tail, true
}
func compact(raw json.RawMessage) []byte {
var buf bytes.Buffer
if json.Compact(&buf, raw) != nil {
return raw
}
return buf.Bytes()
}
// ErrNoBeginning is an operation on an issue the log holds no beginning for.
var ErrNoBeginning = errors.New("the log holds no operation that begins it")
// fold applies one operation to an issue (nil before its first) and answers the issue it adds up to, at seq.
func fold(base *Issue, op *Operation, seq uint64) (*Issue, error) {
var m map[string]json.RawMessage
switch {
case op.Issue != nil:
f, err := fields(op.Issue)
if err != nil {
return nil, err
}
m = f
case base == nil:
return nil, fmt.Errorf("a %s operation at %d: %w", op.Op, seq, ErrNoBeginning)
default:
f, err := fields(base)
if err != nil {
return nil, err
}
m = f
}
for k, v := range op.Set {
m[k] = v
}
for k, tail := range op.Add {
var have, more []json.RawMessage
if old, ok := m[k]; ok {
if err := json.Unmarshal(old, &have); err != nil {
return nil, fmt.Errorf("a %s operation at %d adds to %s, which is not a list", op.Op, seq, k)
}
}
if err := json.Unmarshal(tail, &more); err != nil {
return nil, fmt.Errorf("a %s operation at %d adds to %s what is not a list", op.Op, seq, k)
}
joined, err := json.Marshal(append(have, more...))
if err != nil {
return nil, err
}
m[k] = joined
}
for _, k := range op.Unset {
delete(m, k)
}
raw, err := json.Marshal(m)
if err != nil {
return nil, err
}
var is Issue
if err := json.Unmarshal(raw, &is); err != nil {
return nil, fmt.Errorf("a %s operation at %d folds to no issue: %v", op.Op, seq, err)
}
is.Folded = seq
return &is, nil
}
// foldEntry is fold of one entry of the log.
func foldEntry(base *Issue, e stdio.LogEntry) (*Issue, error) {
var op Operation
if err := json.Unmarshal(e.Value, &op); err != nil {
return nil, fmt.Errorf("the entry at %d is no operation this module reads: %v", e.Sequence, err)
}
return fold(base, &op, e.Sequence)
}
// readKey is every entry of one issue's key from the sequence from on, oldest first, read page by page.
func (t *Tracker) readKey(n uint64, from uint64) ([]stdio.LogEntry, error) {
if t.Log == nil {
return nil, errNoLog
}
var out []stdio.LogEntry
for from != 0 || out == nil {
p, err := t.Log.Read(max(from, 1), keyOf(n), logPage)
if err != nil {
return nil, fmt.Errorf("the log of issue %d could not be read: %v", n, err)
}
if out == nil {
out = []stdio.LogEntry{}
}
out = append(out, p.Entries...)
from = p.Next
}
return out, nil
}
// append adds one operation to an issue at the sequence its fold was read at. A refusal is stdio.ErrChanged:
// another operation on the issue came in between, and nothing was written.
func (t *Tracker) appendOp(n uint64, op *Operation, expect uint64) (uint64, error) {
if t.Log == nil {
return 0, errNoLog
}
raw, err := json.Marshal(op)
if err != nil {
return 0, err
}
return t.Log.AppendAt(keyOf(n), json.RawMessage(raw), expect)
}
// errBegun is a first operation refused because the log already begins that issue, with another one.
var errBegun = errors.New("the log already holds another issue under that number")
// begin appends the first operation of a new issue — expect 0 — and answers its sequence. A first operation
// whose answer was lost is known as its own by the issue's opening id. Any other refusal is errBegun; a
// failure is answered as it is (ADR 0300 §2: the number it was given stays spent).
func (t *Tracker) begin(is *Issue, act Act) (uint64, error) {
op := &Operation{Op: act.Op, By: act.By, At: t.now(), Issue: is}
var err error
for i := 0; i < createAttempts; i++ {
if i > 0 {
t.Sleep(jitter(time.Duration(i) * 50 * time.Millisecond))
}
var seq uint64
seq, err = t.appendOp(is.Number, op, 0)
if err == nil {
return seq, nil
}
if errors.Is(err, stdio.ErrChanged) {
first, lerr := t.firstOf(is.Number)
if lerr != nil {
return 0, fmt.Errorf("the log refused issue %d's first operation and could not be read back (%v): %w",
is.Number, lerr, errBegun)
}
if mine(first, is.Opening) {
return first.Sequence, nil
}
return 0, errBegun
}
}
// An append that timed out may still have landed: read back before saying it did not.
first, lerr := t.firstOf(is.Number)
switch {
case lerr != nil:
return 0, fmt.Errorf("%v, and the log could not be read back (%v): %w", err, lerr, errUnsure)
case mine(first, is.Opening):
return first.Sequence, nil
case first != nil:
return 0, errBegun
}
return 0, err
}
// firstOf is an issue's first entry in the log, the one that begins it, or nil when it has none. The first, not
// the last: an open whose answer was lost may have been followed by another operation on it already, such as the
// triage's.
func (t *Tracker) firstOf(n uint64) (*stdio.LogEntry, error) {
p, err := t.Log.Read(1, keyOf(n), 1)
if err != nil {
return nil, err
}
if len(p.Entries) == 0 {
return nil, nil
}
return &p.Entries[0], nil
}
// errUnsure is a first operation the log may have taken: its answer was lost, and the log could not be read back.
var errUnsure = errors.New("it may have been taken")
// mine says whether a log entry is the first operation of the issue opened by opening.
func mine(e *stdio.LogEntry, opening string) bool {
if e == nil || opening == "" {
return false
}
var op Operation
return json.Unmarshal(e.Value, &op) == nil && op.Issue != nil && op.Issue.Opening == opening
}
// change reads an issue, applies f, appends what changed to the log at the sequence the issue was folded at,
// and then writes the fold to the bucket (ADR 0300 §1). An append another operation won is read and applied
// again, so no note, link or move is lost and none is applied to a stale issue; a refused append writes
// nothing. A change that changes nothing appends nothing.
func (t *Tracker) change(n uint64, act Act, f func(*Issue) error) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := t.current(n)
if err != nil {
return nil, err
}
if is.Folded == 0 {
// Kept before the switch, and not carried yet: carried first, so the log begins it (ADR 0300 §6).
if _, _, err := t.carryOne(n); err != nil {
return nil, fmt.Errorf("issue %d is kept from before the tracker's log and is not carried into it yet, "+
"so nothing is changed: %v", n, err)
}
continue
}
before, err := clone(is)
if err != nil {
return nil, err
}
if err := f(is); err != nil {
return nil, err
}
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
if len(raw) > MostIssueBytes {
return nil, fmt.Errorf("issue %d would be %d bytes; one issue is kept in at most %d (252 KiB). Nothing "+
"was changed: say it shorter, or in a document the issue links", n, len(raw), MostIssueBytes)
}
op, changed, err := operation(act, t.now(), before, is)
if err != nil {
return nil, err
}
if !changed {
return is, nil
}
seq, err := t.appendOp(n, op, is.Folded)
if errors.Is(err, stdio.ErrChanged) {
// Another operation came in between, or the bucket is behind the log: fold what the log holds and
// apply this change again to that.
if _, err := t.catchUp(n); err != nil {
return nil, err
}
t.Sleep(jitter(time.Duration(i+1) * 5 * time.Millisecond))
continue
}
if err != nil {
return nil, fmt.Errorf("issue %d: the log did not take the change (%v), so the bucket was not written. If "+
"the log took it before the failure it is kept and the bucket folds it at the next change or start: "+
"get %d before changing it again", n, err, n)
}
is.Folded = seq
t.store(n, is, rev)
return is, nil
}
return nil, fmt.Errorf("issue %d was changed by another caller %d times while this change was made; nothing was "+
"changed. Try again", n, issueAttempts)
}
// current is an issue as the bucket holds it, and the revision it was read at; an issue the log holds and the
// bucket does not is folded into the bucket first.
func (t *Tracker) current(n uint64) (*Issue, uint64, error) {
is, rev, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) && t.Log != nil {
if _, ferr := t.catchUp(n); ferr != nil {
return nil, 0, ferr
}
is, rev, err = read(t.Store, n)
}
return is, rev, err
}
// store writes the fold of an issue at the revision read, or creates it at 0. The log already holds the
// change, so a write another writer won is folded again from the log, and one that fails is said and left for
// the next change or start, which fold the log again; it is never answered as a change not made.
func (t *Tracker) store(n uint64, is *Issue, rev uint64) {
var err error
if rev == 0 {
_, err = t.Store.Create(IssuesState, keyOf(n), is)
} else {
_, err = t.Store.Update(IssuesState, keyOf(n), is, rev)
}
if err == nil {
t.remember(is)
return
}
if errors.Is(err, stdio.ErrChanged) {
if folded, cerr := t.catchUp(n); cerr == nil {
if folded != nil {
t.remember(folded)
}
return
} else {
err = cerr
}
}
t.remember(is)
t.Logf("[mesh-issues] issue %d is kept in the log at %d and its bucket could not be written (%v): the bucket is "+
"behind until the next change of it or the next start folds it", n, is.Folded, err)
}
// catchUp folds into the bucket every operation of an issue the bucket has not folded, and answers the issue
// as folded (nil when neither holds it). The bucket is written only at the revision read; a write another
// writer won is read again.
func (t *Tracker) catchUp(n uint64) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil && !errors.Is(err, ErrNoIssue) {
return nil, err
}
var from uint64 = 1
if is != nil {
if is.Folded == 0 {
// Not carried yet: what the log holds is no fold of it.
return is, nil
}
from = is.Folded + 1
}
entries, err := t.readKey(n, from)
if err != nil {
return nil, err
}
if len(entries) == 0 {
return is, nil
}
folded := is
for _, e := range entries {
if folded, err = foldEntry(folded, e); err != nil {
return nil, fmt.Errorf("issue %d cannot be folded from its log: %v", n, err)
}
}
if is == nil {
_, err = t.Store.Create(IssuesState, keyOf(n), folded)
} else {
_, err = t.Store.Update(IssuesState, keyOf(n), folded, rev)
}
if err == nil {
t.remember(folded)
return folded, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return nil, fmt.Errorf("issue %d is behind its log in the bucket and could not be written (%v)", n, err)
}
}
return nil, fmt.Errorf("issue %d was written by another writer %d times while it was folded from its log", n, issueAttempts)
}
func clone(is *Issue) (*Issue, error) {
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
var c Issue
return &c, json.Unmarshal(raw, &c)
}
// Carried is what the carry did (ADR 0300 §6), as it happened: the values it appended, not the ones it meant to.
type Carried struct {
// Issues is how many issues it appended values of, Values how many values; Already how many issues the log
// already began whole, which it left as they are.
Issues int `json:"issues"`
Values int `json:"values"`
Already int `json:"already"`
// Deletes is how many deletions among the kept values were passed over: an issue is never deleted by the
// tracker, and each value after one holds the issue whole, so the fold needs none.
Deletes int `json:"deletes,omitempty"`
// Faults is each issue that could not be carried, and why. It stays in the bucket as it was, readable and
// listed; every write to it is refused with the same reason until it can be carried, so nothing reaches the
// log before its carry.
Faults []string `json:"faults,omitempty"`
Said string `json:"said"`
}
// carryOne carries one issue the bucket has not folded from the log: each value the bucket still keeps of it
// (novox/hq ADR 0307), oldest first, as `carried` operations holding the whole issue and the bucket's revision
// of that value, then the fold written to the bucket. Answers how many values it appended and how many
// deletions it passed over.
//
// It resumes rather than restarts. A value is known to the log when its revision is at or below the highest
// revision a carried operation of the issue holds, or when it is a value the tracker wrote itself (it carries
// `folded`): matched by revision, not by content, because the same content may be kept twice (a write that
// changed nothing) and a revision names one value exactly. So an interrupted carry, or another instance carrying
// the same issue at the same moment, carries each value once; a value written beside the log after the carry,
// by an instance from before the switch, is carried as one more operation and so is never overwritten.
func (t *Tracker) carryOne(n uint64) (int, int, error) {
appended := 0
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil {
return appended, 0, err
}
if is.Folded != 0 {
return appended, 0, nil
}
entries, err := t.readKey(n, 1)
if err != nil {
return appended, 0, err
}
var folded *Issue
var expect, known uint64
for _, e := range entries {
if folded, err = foldEntry(folded, e); err != nil {
return appended, 0, fmt.Errorf("issue %d cannot be folded from its log: %v", n, err)
}
expect = e.Sequence
var op Operation
if json.Unmarshal(e.Value, &op) == nil && op.Op == OpCarried {
known = max(known, op.Revision)
}
}
kept, err := t.Store.History(IssuesState, keyOf(n))
if err != nil {
return appended, 0, fmt.Errorf("the values the bucket keeps of issue %d could not be read: %v", n, err)
}
type value struct {
issue *Issue
revision uint64
}
var values []value
deletes := 0
for _, e := range kept {
if e.Op == "delete" {
deletes++
continue
}
var v Issue
if err := json.Unmarshal(e.Value, &v); err != nil {
return appended, deletes, fmt.Errorf("the value of issue %d at revision %d is in a form this module "+
"cannot read (%v): not carried, so that nothing kept is left out", n, e.Revision, err)
}
switch {
case v.Folded != 0:
// The tracker's own fold: the log holds it already, and whatever came before it.
values = nil
known = max(known, e.Revision)
case e.Revision > known:
values = append(values, value{&v, e.Revision})
}
}
if len(kept) == 0 || kept[len(kept)-1].Revision < rev {
// The value read is newer than the history read before it: carried too.
if rev > known {
values = append(values, value{is, rev})
}
}
raced := false
for _, v := range values {
v.issue.Folded = 0
seq, err := t.appendOp(n, &Operation{Op: OpCarried, By: Seat, At: t.now(), Issue: v.issue, Revision: v.revision}, expect)
if errors.Is(err, stdio.ErrChanged) {
raced = true
break
}
if err != nil {
return appended, deletes, fmt.Errorf("issue %d could not be carried into the log (%v); %d of its %d "+
"values not carried yet were carried. Nothing is lost: the bucket still holds it, and the carry "+
"resumes after the last value carried, at the next start or before its next change", n, err, appended, len(values))
}
appended++
expect = seq
if folded, err = fold(nil, &Operation{Op: OpCarried, Issue: v.issue}, seq); err != nil {
return appended, deletes, err
}
}
if raced {
// Another carrier was first: read again; what it appended is known by its revisions.
continue
}
if folded == nil {
return appended, deletes, fmt.Errorf("issue %d keeps no value to carry", n)
}
if _, err := t.Store.Update(IssuesState, keyOf(n), folded, rev); err != nil {
if errors.Is(err, stdio.ErrChanged) {
// Written meanwhile, by a carrier or an instance from before the switch: read again.
continue
}
return appended, deletes, fmt.Errorf("issue %d was carried into the log and its bucket could not be written (%v): "+
"it is folded at the next change or start", n, err)
}
t.remember(folded)
return appended, deletes, nil
}
return appended, 0, fmt.Errorf("issue %d was written by another writer %d times while it was carried", n, issueAttempts)
}
// same says whether two issues hold the same, whatever sequence each was folded at.
func same(a, b *Issue) bool {
fa, err1 := fields(a)
fb, err2 := fields(b)
if err1 != nil || err2 != nil || len(fa) != len(fb) {
return false
}
for k, v := range fa {
if !bytes.Equal(compact(v), compact(fb[k])) {
return false
}
}
return true
}
// Carry carries every issue the bucket holds and has not folded from the log (ADR 0300 §6). An issue that
// cannot be carried is a fault, named with its reason, and the others are carried. Run again, it carries
// nothing more, and resumes an issue whose carry was interrupted after the last value carried.
func (t *Tracker) Carry() (*Carried, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues kept cannot be listed: %v", err)
}
out := &Carried{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
out.Faults = append(out.Faults, err.Error())
continue
}
if is.Folded != 0 {
out.Already++
continue
}
v, d, err := t.carryOne(n)
out.Values += v
out.Deletes += d
if v > 0 {
out.Issues++
} else if err == nil {
out.Already++
}
if err != nil {
t.Logf("[mesh-issues] %v", err)
out.Faults = append(out.Faults, err.Error())
}
}
if out.Values == 0 {
out.Said = fmt.Sprintf("Carried nothing: the log begins every issue of the bucket it could (%d).", out.Already)
} else {
out.Said = fmt.Sprintf("Carried %d values of %d issues into the log, each value the bucket kept, oldest first; "+
"%d issues were there already.", out.Values, out.Issues, out.Already)
}
if out.Deletes > 0 {
out.Said += fmt.Sprintf(" %d deletions among the kept values were passed over: each value after one holds "+
"the issue whole.", out.Deletes)
}
if len(out.Faults) > 0 {
out.Said += fmt.Sprintf(" %d issues could not be carried, and stay as the bucket holds them, every write to "+
"them refused until they can be: %s.", len(out.Faults), strings.Join(out.Faults, "; "))
}
return out, nil
}
// Started is what the tracker did on start (ADR 0300 §3, §6).
type Started struct {
// Entries is how many operations the log holds; Issues how many issues they add up to.
Entries int `json:"entries"`
Issues int `json:"issues"`
// Refolded is how many issues the bucket held behind the log, Rebuilt how many it did not hold at all.
Refolded int `json:"refolded"`
Rebuilt int `json:"rebuilt"`
Carried *Carried `json:"carried"`
// Counter is the issue counter after start, raised to the highest number in the log where it was below.
Counter uint64 `json:"counter"`
Faults []string `json:"faults,omitempty"`
Said string `json:"said"`
}
// Start carries what the bucket holds and the log does not, reads the whole log, folds every issue whose
// last operation is newer than the bucket's fold of it — an emptied bucket is rebuilt whole — keeps the
// counter at or above the highest number in the log, and holds every issue in memory with its index
// (ADR 0300 §3, §5, §6).
func (t *Tracker) Start() (*Started, error) {
out := &Started{}
if t.Log == nil {
return out, errNoLog
}
carried, err := t.Carry()
if err != nil {
return out, fmt.Errorf("the carry could not start (%v); writes carry each issue before they change it", err)
}
out.Carried = carried
for _, f := range carried.Faults {
out.Faults = append(out.Faults, "not carried: "+f)
}
folds := map[uint64]*Issue{}
// broken is each issue an entry of which could not be folded: none of its later entries is folded on top,
// and its bucket value is left as it is, so no fold that skipped an operation is ever written.
broken := map[uint64]bool{}
var highest uint64
for from := uint64(1); from != 0; {
p, err := t.Log.Read(from, "", logPage)
if err != nil {
return out, fmt.Errorf("the log could not be read from %d: %v", from, err)
}
for _, e := range p.Entries {
out.Entries++
n, ok := numberOfKey(e.Key)
if !ok {
out.Faults = append(out.Faults, fmt.Sprintf("the entry at %d has the key %q, which is no issue's number", e.Sequence, e.Key))
continue
}
highest = max(highest, n)
if broken[n] {
continue
}
f, err := foldEntry(folds[n], e)
if err != nil {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d cannot be folded from its log, and its bucket "+
"value is left as it is: %v", n, err))
broken[n] = true
delete(folds, n)
continue
}
folds[n] = f
}
from = p.Next
}
out.Issues = len(folds) + len(broken)
numbers := make([]uint64, 0, len(folds))
for n := range folds {
numbers = append(numbers, n)
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] < numbers[j] })
for _, n := range numbers {
f := folds[n]
is, rev, err := read(t.Store, n)
switch {
case errors.Is(err, ErrNoIssue):
if _, err := t.Store.Create(IssuesState, keyOf(n), f); err != nil && !errors.Is(err, stdio.ErrChanged) {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d could not be rebuilt in the bucket: %v", n, err))
continue
} else if err == nil {
out.Rebuilt++
}
case err != nil:
out.Faults = append(out.Faults, err.Error())
continue
case is.Folded == 0:
// Written beside the log, by an instance from before the switch, after the carry, and not carried again
// (its carry is in the faults): the fold would overwrite that write. Left for its carry, which appends it
// as one more operation before the issue's next change.
out.Faults = append(out.Faults, fmt.Sprintf("issue %d holds a value written beside the log and not carried "+
"yet; it is not refolded over", n))
continue
case is.Folded < f.Folded:
if _, err := t.Store.Update(IssuesState, keyOf(n), f, rev); err != nil && !errors.Is(err, stdio.ErrChanged) {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d could not be refolded in the bucket: %v", n, err))
continue
} else if err == nil {
out.Refolded++
}
}
}
if out.Counter, err = t.raiseCounter(highest); err != nil {
out.Faults = append(out.Faults, err.Error())
}
all, err := t.readAll()
if err != nil {
return out, err
}
t.view().load(all)
out.Said = fmt.Sprintf("The log holds %d operations of %d issues; the bucket was behind on %d and missed %d, now "+
"folded. %s The issue counter is at %d. %d issues are held in memory and indexed.", out.Entries, out.Issues,
out.Refolded, out.Rebuilt, carried.Said, out.Counter, len(all))
if len(out.Faults) > 0 {
out.Said += fmt.Sprintf(" %d faults: %v", len(out.Faults), out.Faults)
}
return out, nil
}
// raiseCounter keeps the issue counter at or above the highest number in the log, so no number the log holds
// is handed out again; a counter not started yet is left to the first open, which starts it above every issue
// the bucket holds.
func (t *Tracker) raiseCounter(highest uint64) (uint64, error) {
for i := 0; i < issueAttempts; i++ {
e, err := t.Store.Get(CountersState, IssueCounter)
if err != nil {
return 0, fmt.Errorf("the issue counter cannot be read: %v", err)
}
if e == nil {
return 0, nil
}
var n uint64
if err := json.Unmarshal(e.Value, &n); err != nil {
return 0, fmt.Errorf("the issue counter holds %s, no number", e.Value)
}
if n >= highest {
return n, nil
}
_, err = t.Store.Update(CountersState, IssueCounter, highest, e.Revision)
if err == nil {
t.Logf("[mesh-issues] the issue counter was at %d, below issue %d in the log: raised to %d", n, highest, highest)
return highest, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return n, fmt.Errorf("the issue counter is at %d, below issue %d in the log, and could not be raised: %v", n, highest, err)
}
}
return 0, errors.New("the issue counter was advanced by others while it was raised to the log's highest number")
}
@@ -91,7 +91,7 @@ func (t *Tracker) sayNotResolved(is *Issue, now time.Time) (bool, error) {
prs, ago, at.Format(time.RFC3339), worker), "", false); err != nil {
return false, err
}
_, err := t.change(is.Number, func(i *Issue) error { i.NotResolvedOpen = true; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.NotResolvedOpen = true; return nil })
return err == nil, err
}
@@ -105,7 +105,7 @@ func (t *Tracker) clearNotResolved(is *Issue) error {
"fix not delivered yet", is.Number, is.Status), true); err != nil {
return err
}
_, err := t.change(is.Number, func(i *Issue) error { i.NotResolvedOpen = false; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.NotResolvedOpen = false; return nil })
return err
}
@@ -167,7 +167,7 @@ func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, on
// the same pull request at most, and an older one is history.
continue
}
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: "delivery", By: Seat}, func(i *Issue) error {
f := fixOf(i, pr)
if f == nil {
i.Fixes = append(i.Fixes, Fix{PullRequest: pr})
@@ -242,7 +242,7 @@ func (t *Tracker) afterFix(n uint64) error {
}
}
claimed := false
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: EventReadyToVerify, By: Seat}, func(i *Issue) error {
claimed = false
if strings.Join(i.VerifyFor, ",") == strings.Join(prs, ",") {
return nil
@@ -290,7 +290,7 @@ func (t *Tracker) clearStuck(is *Issue) error {
if err := t.Notify(stuckKey(is.Number), "", "", "", "", "the fix of issue "+keyOf(is.Number)+" is delivered or no longer waited for", true); err != nil {
return err
}
_, err := t.change(is.Number, func(i *Issue) error { i.StuckOpen = false; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.StuckOpen = false; return nil })
return err
}
@@ -514,7 +514,7 @@ func (t *Tracker) Sweep() (int, error) {
}
}
pr := f.PullRequest
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if g := fixOf(i, pr); g != nil {
g.StuckSaid = true
}
+1 -25
View File
@@ -188,30 +188,6 @@ func hasLink(is *Issue, kind string, target uint64) bool {
return false
}
// allIssues is every issue kept, by number.
func (t *Tracker) allIssues() (map[uint64]*Issue, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues cannot be listed: %v", err)
}
out := map[uint64]*Issue{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
return nil, err
}
out[n] = is
}
return out, nil
}
// unresolvedIssues is every issue at open, diagnosing or located, oldest first.
func (t *Tracker) unresolvedIssues() ([]*Issue, error) {
all, err := t.allIssues()
@@ -538,7 +514,7 @@ func (t *Tracker) apply(newer uint64, v *Verdict) ([]string, error) {
}
}
kept := *v
if _, err := t.change(newer, func(i *Issue) error {
if _, err := t.change(newer, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error {
if i.Triage == nil {
i.Triage = &kept
}
+25 -12
View File
@@ -600,29 +600,42 @@ func differences(kept, folder *Issue) []string {
return d
}
// importCreate writes a new issue under its number, never over one.
// importCreate writes a new issue under its number, never over one: its first operation appended to the log at
// expect 0 (novox/hq ADR 0300 §1), then its fold to the bucket.
func (t *Tracker) importCreate(is *Issue, out *Imported) error {
_, err := t.Store.Create(IssuesState, keyOf(is.Number), is)
if errors.Is(err, stdio.ErrChanged) {
if there, _, err := read(t.Store, is.Number); err == nil && there.Opening != is.Opening {
return fmt.Errorf("an issue %d appeared while the import ran; nothing was overwritten. Run the import again", is.Number)
}
if err != nil {
// A write that timed out may still have landed: read back before saying it did not.
if there, _, rerr := read(t.Store, is.Number); rerr == nil && there.Opening == is.Opening {
t.announceImport(is, out)
return nil
}
return fmt.Errorf("issue %d could not be written: %v", is.Number, err)
seq, err := t.begin(is, Act{Op: OpImported, By: importer(is)})
if errors.Is(err, errBegun) {
return fmt.Errorf("an issue %d appeared while the import ran; nothing was overwritten. Run the import again", is.Number)
}
if errors.Is(err, errUnsure) {
return fmt.Errorf("issue %d may have been written: the log did not confirm it (%v); run the import again, "+
"which finds it the same or creates it", is.Number, err)
}
if err != nil {
return fmt.Errorf("issue %d could not be written: the log did not take it (%v)", is.Number, err)
}
is.Folded = seq
t.store(is.Number, is, 0)
t.announceImport(is, out)
return nil
}
// importer is who ran the import that took an issue from hq.
func importer(is *Issue) string {
if is.Imported != nil && is.Imported.By != "" {
return is.Imported.By
}
return ImportedBy
}
// importFill takes the status and the fields of an issue the tracker holds without a status: opened with a
// folder before slice 3, whose status lived in that folder. Its title, report, opening, notes and links
// stay; links the report names are added.
func (t *Tracker) importFill(from *Issue, out *Imported) error {
is, err := t.change(from.Number, func(is *Issue) error {
is, err := t.change(from.Number, Act{Op: OpImported, By: importer(from)}, func(is *Issue) error {
if is.Status != "" {
return fmt.Errorf("issue %d took a status while the import ran; nothing was changed", is.Number)
}
@@ -663,7 +676,7 @@ func (t *Tracker) announceImport(is *Issue, out *Imported) {
// fillOpen gives an issue the tracker holds without a status — opened with a folder before slice 3, whose
// folder hq does not hold at the import's commit — the status open, so no issue is left without one.
func (t *Tracker) fillOpen(n uint64, commit string, at time.Time, by string) error {
is, err := t.change(n, func(is *Issue) error {
is, err := t.change(n, Act{Op: OpImported, By: by}, func(is *Issue) error {
if is.Status != "" {
return fmt.Errorf("issue %d took a status while the import ran; nothing was changed", is.Number)
}
@@ -1,7 +1,6 @@
package main
import (
"errors"
"os"
"path/filepath"
"reflect"
@@ -304,7 +303,7 @@ func TestTheImportRefusesWhatItMustNotWrite(t *testing.T) {
// An issue that appears under the number while the import runs is never overwritten.
h = newHarness(t)
h.mem.fail[IssuesState] = errors.New("the bus is away")
h.log.failNext = createAttempts
out, err = h.Import(ImportAsk{}, newSample(t))
if err != nil {
t.Fatal(err)
+15 -3
View File
@@ -118,6 +118,9 @@ type Issue struct {
// slice 3: when, by whom, at which commit of hq, from which document. An issue the import wrote has
// OpenedBy ImportedBy, and its Opened is the date its report gave.
Imported *Import `json:"imported,omitempty"`
// Folded is the sequence in the log `changes` of the last operation this issue folds (novox/hq ADR 0300
// §3); 0 for an issue kept before the switch and not carried into the log yet.
Folded uint64 `json:"folded,omitempty"`
}
// ImportedBy is who opened an issue the import wrote: not a caller of `open`, and not a name anybody typed.
@@ -148,9 +151,10 @@ type Link struct {
At time.Time `json:"at"`
}
// MostIssueBytes is the most one issue may be as kept: a value the bus holds (to-be 53 §4). The largest
// issue folder in hq was 25 KiB on 2026-10-10.
const MostIssueBytes = 256 << 10
// MostIssueBytes is the most one issue may be as kept: 252 KiB, so the operation holding it whole (an open, an
// import, a carry) fits in one entry of the log, which takes at most 256 KiB (novox/hq ADR 0297, ADR 0300). The
// largest issue folder in hq was 25 KiB on 2026-10-10.
const MostIssueBytes = 252 << 10
// Entry is one key's value and the revision it was read at.
type Entry struct {
@@ -166,6 +170,9 @@ type Store interface {
Update(state, key string, value any, revision uint64) (uint64, error)
Keys(state string) ([]string, error)
Increment(state, key string, attempts int) (uint64, error)
// History is every value a key still keeps, oldest first, puts and deletes (novox/hq ADR 0307): what the
// carry into the log reads (ADR 0300 §6).
History(state, key string) ([]stdio.StateRevision, error)
}
// runtimeStore is Store through the runtime that launched this bundle.
@@ -185,6 +192,11 @@ func (runtimeStore) Update(state, key string, value any, revision uint64) (uint6
return stdio.State(state).Update(key, value, revision)
}
func (runtimeStore) Keys(state string) ([]string, error) { return stdio.State(state).Keys() }
// History is every value the bucket still keeps of a key, oldest first (novox/hq ADR 0307).
func (runtimeStore) History(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
func (runtimeStore) Increment(state, key string, attempts int) (uint64, error) {
return stdio.State(state).Increment(key, attempts)
}
@@ -0,0 +1,819 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// fakeLog is the module's log kept in this process with the runtime's semantics (novox/hq ADR 0297, mesh-sdk
// go/log.go): an append at expect is refused with stdio.ErrChanged when the key's last sequence is another,
// 0 standing for a key with no entry; a read pages from a sequence, narrowed to one key or none.
type fakeLog struct {
mu sync.Mutex
entries []stdio.LogEntry
// fail, when set, is what every append answers, writing nothing: a full log, or a bus that is down.
fail error
// failNext is how many of the next appends fail, writing nothing, as a bus that does not answer.
failNext int
// lost, when set, is what the next append answers after it was written: an answer lost on the way.
lost error
// between, when set, runs once before the next append is judged: another writer coming in between.
between func()
clock time.Time
}
func newFakeLog() *fakeLog {
return &fakeLog{clock: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)}
}
func (l *fakeLog) lastOf(key string) uint64 {
for i := len(l.entries) - 1; i >= 0; i-- {
if l.entries[i].Key == key {
return l.entries[i].Sequence
}
}
return 0
}
func (l *fakeLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
l.mu.Lock()
if b := l.between; b != nil {
l.between = nil
l.mu.Unlock()
b()
l.mu.Lock()
}
defer l.mu.Unlock()
if l.fail != nil {
return 0, l.fail
}
if l.failNext > 0 {
l.failNext--
return 0, errors.New("the bus did not answer")
}
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
if len(raw) > 256<<10 {
return 0, fmt.Errorf("the value for %s is %d bytes; a log entry holds at most 256 KiB", key, len(raw))
}
if last := l.lastOf(key); last != expect {
return 0, fmt.Errorf("the last entry of %s is %d, not %d: %w", key, last, expect, stdio.ErrChanged)
}
seq := uint64(len(l.entries) + 1)
l.entries = append(l.entries, stdio.LogEntry{Sequence: seq, Key: key, Value: raw, At: l.clock})
if lost := l.lost; lost != nil {
l.lost = nil
return 0, lost
}
return seq, nil
}
func (l *fakeLog) Read(from uint64, keys string, limit int) (*stdio.LogPage, error) {
l.mu.Lock()
defer l.mu.Unlock()
if limit <= 0 {
limit = 100
}
limit = min(limit, 1000)
p := &stdio.LogPage{Entries: []stdio.LogEntry{}, Last: uint64(len(l.entries))}
for _, e := range l.entries {
if e.Sequence < from || (keys != "" && e.Key != keys) {
continue
}
if len(p.Entries) == limit {
p.Next = e.Sequence
break
}
p.Entries = append(p.Entries, e)
}
return p, nil
}
// ops is every operation the log holds of one issue, oldest first.
func (l *fakeLog) ops(t *testing.T, n uint64) []Operation {
t.Helper()
l.mu.Lock()
defer l.mu.Unlock()
var out []Operation
for _, e := range l.entries {
if e.Key != keyOf(n) {
continue
}
var op Operation
if err := json.Unmarshal(e.Value, &op); err != nil {
t.Fatal(err)
}
out = append(out, op)
}
return out
}
// foldCheck holds every issue the bucket folded to what its whole log adds up to, byte for byte: run after every
// test of a harness on this log (the review of PR 228).
func (l *fakeLog) foldCheck(t *testing.T, m *memory) {
l.mu.Lock()
entries := append([]stdio.LogEntry(nil), l.entries...)
l.mu.Unlock()
for k, v := range m.snapshot(IssuesState) {
var is Issue
if json.Unmarshal([]byte(v), &is) != nil || is.Folded == 0 {
continue
}
var f *Issue
var err error
for _, e := range entries {
if e.Key != k || e.Sequence > is.Folded {
continue
}
if f, err = foldEntry(f, e); err != nil {
t.Errorf("issue %s does not fold from its log: %v", k, err)
break
}
}
if raw, _ := json.Marshal(f); err == nil && string(raw) != v {
t.Errorf("issue %s in the bucket is not the fold of its log:\nbucket %s\nfold %s", k, v, raw)
}
}
}
func (l *fakeLog) size() int {
l.mu.Lock()
defer l.mu.Unlock()
return len(l.entries)
}
// another is a second tracker on the same bucket and log: another instance, or the same after a restart.
func (h *harness) another(t *testing.T) *harness {
t.Helper()
o := newHarness(t)
o.mem, o.log = h.mem, h.log
o.Tracker.Store, o.Tracker.Log = h.mem, h.log
o.settings = h.settings
return o
}
// §1: a write whose append is refused changes nothing: not the bucket, not the log, and says no event.
func TestAWriteWhoseAppendIsRefusedChangesNothing(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the bucket fills", "mesh-catalog modules/mesh-issues")
bucket, entries, events := h.mem.snapshot(IssuesState), h.log.size(), len(h.said())
h.log.fail = fmt.Errorf("the log is full: %w", errors.New("maximum bytes exceeded"))
if _, err := h.Note(n, "a note the log refuses"); err == nil || !strings.Contains(err.Error(), "log did not take") {
t.Fatalf("a note whose append is refused answers %v", err)
}
if _, err := h.Move(MoveAsk{Number: n, Status: StatusDiagnosing, By: "fix@g14"}); err == nil {
t.Fatal("a move whose append is refused was answered as made")
}
if got := h.mem.snapshot(IssuesState); fmt.Sprint(got) != fmt.Sprint(bucket) {
t.Fatalf("the bucket changed:\n%v\n%v", bucket, got)
}
if h.log.size() != entries || len(h.said()) != events {
t.Fatalf("the log holds %d entries (was %d), %d events said (was %d)", h.log.size(), entries, len(h.said()), events)
}
}
// §1: an operation that came in between refuses the stale append; the change is read again and applied to
// what the log holds, so neither is lost.
func TestAnOperationInBetweenIsNotLost(t *testing.T) {
h := newHarness(t)
n := h.open(t, "two notes at once", "hq")
other := h.another(t)
h.log.between = func() {
if _, err := other.Note(n, "the other's note"); err != nil {
t.Errorf("the other note: %v", err)
}
}
if _, err := h.Note(n, "this note"); err != nil {
t.Fatal(err)
}
is, err := h.Get(n)
if err != nil {
t.Fatal(err)
}
if len(is.Notes) != 2 || is.Notes[0].Text != "the other's note" || is.Notes[1].Text != "this note" {
t.Fatalf("notes %+v", is.Notes)
}
ops := h.log.ops(t, n)
if len(ops) != 3 || ops[0].Op != OpOpened || ops[1].Op != EventNoted || ops[2].Op != EventNoted {
t.Fatalf("the log of %d: %+v", n, ops)
}
if is.Folded != uint64(h.log.size()) {
t.Fatalf("folded at %d, the log's last is %d", is.Folded, h.log.size())
}
}
// §2: an open refused at validation takes no number.
func TestAnOpenRefusedAtValidationTakesNoNumber(t *testing.T) {
h := newHarness(t)
if _, err := h.Open(OpenAsk{Title: "", Report: "no title"}); err == nil {
t.Fatal("an open without a title was taken")
}
if _, err := h.Open(OpenAsk{Title: "t", Report: "r", LocatedIn: []string{"nowhere"}}); err == nil {
t.Fatal("an open naming an unregistered owner was taken")
}
if c := h.mem.counter(t); c != 0 || h.log.size() != 0 {
t.Fatalf("counter %d, log %d entries", c, h.log.size())
}
if n := h.open(t, "the first"); n != 365 {
t.Fatalf("the first open got %d", n)
}
}
// §2: an append that fails after the number was taken answers the failure and spends that number.
func TestAnAppendFailingAfterTheNumberSpendsIt(t *testing.T) {
h := newHarness(t)
h.log.fail = errors.New("the log is full")
_, err := h.Open(OpenAsk{Title: "refused by a full log", Report: "r"})
if err == nil || !strings.Contains(err.Error(), "365 is spent") || !strings.Contains(err.Error(), "the log is full") {
t.Fatalf("the open answered %v", err)
}
if c := h.mem.counter(t); c != 365 {
t.Fatalf("the counter is at %d: the number was not taken", c)
}
if len(h.mem.snapshot(IssuesState)) != 0 || h.log.size() != 0 || len(h.said()) != 0 {
t.Fatal("an open the log refused left an issue, an entry or an event")
}
h.log.fail = nil
if n := h.open(t, "the next"); n != 366 {
t.Fatalf("the next open got %d, not 366", n)
}
}
// §2: an append whose answer was lost is known as the open's own by its opening id: one issue, one number.
func TestAnAppendWhoseAnswerWasLostIsTheOpensOwn(t *testing.T) {
h := newHarness(t)
h.log.lost = errors.New("timeout")
o, err := h.Open(OpenAsk{Title: "lost answer", Report: "r"})
if err != nil {
t.Fatal(err)
}
if o.Number != 365 || h.log.size() != 1 {
t.Fatalf("number %d, %d entries", o.Number, h.log.size())
}
if is, err := h.Get(365); err != nil || is.Folded != 1 {
t.Fatalf("issue %+v, %v", is, err)
}
}
// writeEverything makes issues with every kind of write: open, move, note, link, set, delivery, triage.
func writeEverything(t *testing.T, h *harness) []uint64 {
t.Helper()
a := h.open(t, "the issues bucket fills toward its cap", "mesh-catalog modules/mesh-issues")
b := h.open(t, "status changes are kept for seven days", "mesh-controller")
c := h.open(t, "an unrelated one")
if _, err := h.Note(a, "measured 10.9 MB of 64 MiB"); err != nil {
t.Fatal(err)
}
if _, err := h.Link(LinkAsk{Number: b, Kind: LinkRelatesTo, Target: keyOf(a)}); err != nil {
t.Fatal(err)
}
if _, err := h.Set(SetAsk{Number: a, Severity: "major", Needs: "decision"}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: a, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"novox/mesh-catalog#1"}}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: c, Status: StatusWontfix, Reason: "not ours"}); err != nil {
t.Fatal(err)
}
if _, err := h.Link(LinkAsk{Number: b, Kind: LinkRelatesTo, Target: keyOf(a), Remove: true}); err != nil {
t.Fatal(err)
}
return []uint64{a, b, c}
}
// §3: the bucket emptied, the fold on start restores every issue exactly.
func TestTheFoldOnStartRestoresAnEmptiedBucketExactly(t *testing.T) {
h := newHarness(t)
writeEverything(t, h)
before := h.mem.snapshot(IssuesState)
if len(before) != 3 {
t.Fatalf("%d issues", len(before))
}
h.mem.empty(IssuesState)
r := h.another(t)
s, err := r.Start()
if err != nil {
t.Fatal(err)
}
if s.Rebuilt != 3 || s.Issues != 3 || len(s.Faults) != 0 {
t.Fatalf("started %+v", s)
}
after := h.mem.snapshot(IssuesState)
for k, v := range before {
if after[k] != v {
t.Errorf("issue %s was\n%s\nand is folded as\n%s", k, v, after[k])
}
}
if len(after) != len(before) {
t.Fatalf("%d issues after, %d before", len(after), len(before))
}
}
// §3: a bucket behind its log (the write after the append failed) catches up on start, and at the next change.
func TestABucketBehindItsLogCatchesUp(t *testing.T) {
h := newHarness(t)
n := h.open(t, "crash between the two", "hq")
h.mem.fail[IssuesState] = errors.New("the bus went away")
if _, err := h.Note(n, "kept in the log only"); err != nil {
t.Fatalf("a note the log took is a note made: %v", err)
}
is, _ := h.Get(n)
if len(is.Notes) != 0 {
t.Fatal("the bucket write was meant to fail")
}
r := h.another(t)
s, err := r.Start()
if err != nil {
t.Fatal(err)
}
if s.Refolded != 1 {
t.Fatalf("started %+v", s)
}
if is, _ = h.Get(n); len(is.Notes) != 1 || is.Folded != 2 {
t.Fatalf("after start: %+v", is)
}
// And at the next change without a start: the stale append is refused, the log folded, the change applied.
h.mem.fail[IssuesState] = errors.New("the bus went away")
if _, err := h.Note(n, "second"); err != nil {
t.Fatal(err)
}
if _, err := h.Note(n, "third"); err != nil {
t.Fatal(err)
}
if is, _ = h.Get(n); len(is.Notes) != 3 || is.Folded != 4 {
t.Fatalf("after the next change: notes %d, folded %d", len(is.Notes), is.Folded)
}
}
// §3: the counter is kept at or above the highest number in the log.
func TestStartRaisesTheCounterToTheLog(t *testing.T) {
h := newHarness(t)
h.open(t, "one")
h.open(t, "two")
h.mem.put(CountersState, IssueCounter, uint64(365))
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if s.Counter != 366 || h.mem.counter(t) != 366 {
t.Fatalf("counter %d, kept %d", s.Counter, h.mem.counter(t))
}
if n := h.open(t, "three"); n != 367 {
t.Fatalf("the next open got %d", n)
}
}
// §4: each write still says its event, alongside its one operation in the log.
func TestEachWriteSaysItsEventAlongsideItsOperation(t *testing.T) {
h := newHarness(t)
type step struct {
event string
do func() error
}
var n uint64
steps := []step{
{EventOpened, func() error {
o, err := h.Open(OpenAsk{Title: "t", Report: "r", LocatedIn: []string{"hq"}})
n = o.Number
return err
}},
{EventNoted, func() error { _, err := h.Note(n, "a note"); return err }},
{EventLinked, func() error { _, err := h.Link(LinkAsk{Number: n, Kind: LinkDecision, Target: "0300"}); return err }},
{EventSet, func() error { _, err := h.Set(SetAsk{Number: n, Severity: "minor"}); return err }},
{EventMoved, func() error { _, err := h.Move(MoveAsk{Number: n, Status: StatusLocated}); return err }},
}
for i, s := range steps {
events, entries := len(h.said()), h.log.size()
if err := s.do(); err != nil {
t.Fatalf("%s: %v", s.event, err)
}
said := h.said()
if len(said) != events+1 || said[len(said)-1].Event != s.event {
t.Fatalf("step %d said %v", i, said[events:])
}
ops := h.log.ops(t, n)
if h.log.size() != entries+1 || ops[len(ops)-1].Op != s.event || ops[len(ops)-1].By != Unproven {
t.Fatalf("step %d: %d entries (was %d), last %+v", i, h.log.size(), entries, ops[len(ops)-1])
}
}
// A note is kept as itself, not as every note again.
if op := h.log.ops(t, n)[1]; op.Set != nil || len(op.Add) != 1 || op.Add["notes"] == nil {
t.Fatalf("the note's operation is %+v", op)
}
}
// §5: before pages through more than 500 issues, without a gap or a repeat.
func TestBeforePagesThroughMoreThanFiveHundredIssues(t *testing.T) {
h := newHarness(t)
const issues = 530
for i := 0; i < issues; i++ {
h.open(t, fmt.Sprintf("issue %d", i))
}
if _, err := h.Start(); err != nil {
t.Fatal(err)
}
seen := map[uint64]bool{}
var last uint64 = 0
var before uint64
pages := 0
for {
l, err := h.List(ListAsk{Limit: 100, Before: before})
if err != nil {
t.Fatal(err)
}
pages++
for _, r := range l.Rows {
if seen[r.Number] {
t.Fatalf("%d repeated", r.Number)
}
if last != 0 && r.Number != last-1 {
t.Fatalf("a gap from %d to %d", last, r.Number)
}
seen[r.Number], last = true, r.Number
}
if l.Next == 0 {
break
}
before = l.Next
}
if len(seen) != issues || pages != 6 || last != 365 {
t.Fatalf("%d issues seen in %d pages, ending at %d", len(seen), pages, last)
}
// Over the tool too, as JSON gives the cursor.
out, err := h.call(t, "list", map[string]any{"before": 400, "limit": 2})
if err != nil {
t.Fatal(err)
}
if l := out.(*Listed); len(l.Rows) != 2 || l.Rows[0].Number != 399 || l.Next != 398 {
t.Fatalf("listed %+v", l)
}
}
// §5: a search is ranked by relevance; the index is rebuilt on start and kept current by the bucket's watch.
func TestASearchIsRankedAndTheIndexRebuiltOnStart(t *testing.T) {
h := newHarness(t)
full := h.open(t, "the issues bucket fills toward its cap")
h.open(t, "a bucket of the controller")
week := h.open(t, "status changes are kept for seven days")
h.open(t, "nothing alike")
if _, err := h.Note(week, "the events stream drops them after a week; the bucket is not involved"); err != nil {
t.Fatal(err)
}
r := h.another(t)
if _, err := r.Start(); err != nil {
t.Fatal(err)
}
if !r.view().ready() {
t.Fatal("the index was not built on start")
}
l, err := r.List(ListAsk{Search: "issues bucket cap"})
if err != nil {
t.Fatal(err)
}
if len(l.Rows) != 3 || l.Rows[0].Number != full {
t.Fatalf("ranked %+v", l.Rows)
}
// A note's words are found: notes are indexed.
if l, _ = r.List(ListAsk{Search: "week"}); len(l.Rows) != 1 || l.Rows[0].Number != week {
t.Fatalf("a note's word: %+v", l.Rows)
}
// Another instance's write reaches this one's index through the watch of the bucket.
if _, err := h.Note(full, "seen again by the operator on shanks"); err != nil {
t.Fatal(err)
}
e, _ := h.mem.Get(IssuesState, keyOf(full))
if err := r.Watched(stdio.StateChange{State: IssuesState, Key: keyOf(full), Op: "put", Value: e.Value}); err != nil {
t.Fatal(err)
}
if l, _ = r.List(ListAsk{Search: "operator"}); len(l.Rows) != 1 || l.Rows[0].Number != full {
t.Fatalf("after the watch: %+v", l.Rows)
}
if _, err := r.List(ListAsk{Search: "bucket", Before: 400}); err == nil {
t.Fatal("a search with before was taken")
}
}
// keptBefore puts an issue in the bucket as the tracker kept it before the switch: values, no log.
func keptBefore(t *testing.T, h *harness, n uint64, titles ...string) {
t.Helper()
for _, title := range titles {
h.mem.put(IssuesState, keyOf(n), Issue{Number: n, Title: title, Status: StatusOpen, Report: "r",
Opened: time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC), OpenedBy: "x", Opening: fmt.Sprint("o", n)})
}
}
// §6: the carry appends each value the bucket keeps, oldest first, as carried operations holding the whole
// issue; run again it carries nothing; a write after it is one entry in the log.
func TestTheCarryIsIdempotentAndCarriesEveryValueKept(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "first title", "second title", "third title")
keptBefore(t, h, 301, "only")
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Issues != 2 || c.Values != 4 || len(c.Faults) != 0 {
t.Fatalf("carried %+v", c)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 {
t.Fatalf("%d operations of 300", len(ops))
}
for i, title := range []string{"first title", "second title", "third title"} {
if ops[i].Op != OpCarried || ops[i].Issue == nil || ops[i].Issue.Title != title {
t.Fatalf("operation %d: %+v", i, ops[i])
}
}
is, _ := h.Get(300)
if is.Title != "third title" || is.Folded != 3 {
t.Fatalf("300 is %+v", is)
}
again, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if again.Issues != 0 || again.Values != 0 || again.Already != 2 || h.log.size() != 4 {
t.Fatalf("carried again %+v, %d entries", again, h.log.size())
}
if _, err := h.Move(MoveAsk{Number: 300, Status: StatusDiagnosing, By: "fix@g14"}); err != nil {
t.Fatal(err)
}
if h.log.size() != 5 || h.log.ops(t, 300)[3].Op != EventMoved {
t.Fatalf("a move after the carry: %d entries", h.log.size())
}
// The start carries too, and finds nothing more.
s, err := h.another(t).Start()
if err != nil || s.Carried.Issues != 0 || h.log.size() != 5 {
t.Fatalf("start %+v, %v", s, err)
}
}
// §6: a deletion among the values the bucket kept is passed over, said, and the values after it carried.
func TestTheCarryPassesOverADeletionAmongTheKeptValues(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "before the deletion")
h.mem.deleteKey(IssuesState, keyOf(300))
keptBefore(t, h, 300, "after the deletion", "now")
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Issues != 1 || c.Values != 3 || c.Deletes != 1 || !strings.Contains(c.Said, "1 deletions") {
t.Fatalf("carried %+v", c)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 || ops[0].Issue.Title != "before the deletion" || ops[2].Issue.Title != "now" {
t.Fatalf("the log of 300: %+v", ops)
}
if is, _ := h.Get(300); is.Title != "now" || is.Folded != 3 {
t.Fatalf("300 is %+v", is)
}
}
// §6: a write to an issue not carried yet carries it first, so its first operation in the log is a carried one.
func TestAWriteBeforeTheCarryCarriesItsIssueFirst(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "before", "now")
if _, err := h.Note(300, "after the switch"); err != nil {
t.Fatal(err)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 || ops[0].Op != OpCarried || ops[1].Op != OpCarried || ops[2].Op != EventNoted {
t.Fatalf("the log of 300: %+v", ops)
}
h.mem.empty(IssuesState)
if _, err := h.another(t).Start(); err != nil {
t.Fatal(err)
}
if is, _ := h.Get(300); is.Title != "now" || len(is.Notes) != 1 {
t.Fatalf("300 folds to %+v", is)
}
}
// The counter behind an issue the bucket kept before the switch: the open refuses, overwrites nothing.
func TestAnOpenOverAnIssueKeptBeforeTheSwitchOverwritesNothing(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 365, "kept")
h.mem.put(CountersState, IssueCounter, uint64(364))
if _, err := h.Open(OpenAsk{Title: "new", Report: "r"}); err == nil || !strings.Contains(err.Error(), "fault") {
t.Fatalf("the open answered %v", err)
}
if is, _ := h.Get(365); is.Title != "kept" {
t.Fatalf("365 is %+v", is)
}
}
// titlesOf is each operation of an issue's log as its kind and, for one holding the issue whole, its title.
func titlesOf(t *testing.T, h *harness, n uint64) string {
t.Helper()
var out []string
for _, op := range h.log.ops(t, n) {
if op.Issue != nil {
out = append(out, op.Op+":"+op.Issue.Title)
} else {
out = append(out, op.Op)
}
}
return strings.Join(out, " ")
}
// §6, the review of PR 228: a carry that fails partway, then a second carry, leaves every kept value carried
// once, oldest first, and each carry says what it really carried.
func TestAnInterruptedCarryResumesAfterTheLastValueCarried(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "v1", "v2", "v3", "v4")
// The bus goes away after the first value is appended.
h.log.between = func() { h.log.between = func() { h.log.fail = errors.New("the bus went away") } }
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Values != 1 || len(c.Faults) != 1 || !strings.Contains(c.Said, "could not be carried") ||
!strings.Contains(c.Faults[0], "the bus went away") {
t.Fatalf("the first carry said %+v", c)
}
if is, _ := h.Get(300); is.Folded != 0 || is.Title != "v4" {
t.Fatalf("after the failure the bucket holds %+v", is)
}
// A write meanwhile is refused, and changes nothing.
if _, err := h.Note(300, "while the bus is away"); err == nil || !strings.Contains(err.Error(), "not carried") {
t.Fatalf("a note on an issue half carried: %v", err)
}
h.log.fail = nil
c, err = h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Values != 3 || c.Issues != 1 || len(c.Faults) != 0 {
t.Fatalf("the second carry said %+v", c)
}
if got := titlesOf(t, h, 300); got != "carried:v1 carried:v2 carried:v3 carried:v4" {
t.Fatalf("the log of 300: %s", got)
}
if is, _ := h.Get(300); is.Title != "v4" || is.Folded != 4 {
t.Fatalf("300 is %+v", is)
}
}
// §6, the review of PR 228: one issue that cannot be carried is a fault, named, and the start carries the others,
// folds, raises the counter and loads memory. Writes to that issue are refused with the reason.
func TestOneIssueThatCannotBeCarriedDoesNotStopTheStart(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "small")
// Kept before the switch at the bucket's limit: whole, in a carried operation, past what one log entry holds.
h.mem.put(IssuesState, "301", Issue{Number: 301, Title: "huge", Status: StatusOpen, Report: strings.Repeat("x", 256<<10-200),
OpenedBy: "x", Opening: "o301"})
keptBefore(t, h, 302, "also small")
s, err := h.Start()
if err != nil {
t.Fatal(err)
}
if s.Carried.Issues != 2 || len(s.Faults) != 1 || !strings.Contains(s.Faults[0], "issue 301") ||
!strings.Contains(s.Faults[0], "256 KiB") || !strings.Contains(s.Said, "issue 301") {
t.Fatalf("started %+v", s)
}
if !h.view().ready() {
t.Fatal("memory was not loaded")
}
l, err := h.List(ListAsk{})
if err != nil || l.Matched != 3 {
t.Fatalf("listed %+v, %v", l, err)
}
if _, err := h.Note(301, "a note"); err == nil || !strings.Contains(err.Error(), "256 KiB") {
t.Fatalf("a note on the issue not carried: %v", err)
}
if is, _ := h.Get(301); is.Title != "huge" || is.Folded != 0 || len(is.Notes) != 0 {
t.Fatal("the issue not carried was changed")
}
if is, _ := h.Get(302); is.Folded == 0 {
t.Fatal("302 was not carried")
}
}
// §6, the review of PR 228: two instances carrying the same issue at the same moment carry each value once.
func TestTwoInstancesCarryingOneIssueAtOnceCarryEachValueOnce(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "v1", "v2", "v3")
other := h.another(t)
var otherSaid *Carried
h.log.between = func() {
c, err := other.Carry()
if err != nil {
t.Errorf("the other carry: %v", err)
}
otherSaid = c
}
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if got := titlesOf(t, h, 300); got != "carried:v1 carried:v2 carried:v3" {
t.Fatalf("the log of 300: %s", got)
}
if otherSaid.Values+c.Values != 3 || len(c.Faults)+len(otherSaid.Faults) != 0 {
t.Fatalf("this carry %+v, the other %+v", c, otherSaid)
}
}
// §6, the review of PR 228: a value written beside the log after the carry, by an instance from before the switch,
// is never overwritten by the fold on start; it is carried as one more operation.
func TestAValueWrittenBesideTheLogIsCarriedNotOverwritten(t *testing.T) {
h := newHarness(t)
n := h.open(t, "carried", "hq")
if _, err := h.Note(n, "after the switch"); err != nil {
t.Fatal(err)
}
// An instance from before the switch writes the bucket: no folded.
is, _ := h.Get(n)
is.Folded, is.Title = 0, "written beside the log"
h.mem.put(IssuesState, keyOf(n), is)
// Its carry fails at this start: the fold is not written over it.
h.log.failNext = 1
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if got, _ := h.Get(n); got.Title != "written beside the log" || got.Folded != 0 || len(s.Faults) != 2 {
t.Fatalf("issue %d is %+v after a start whose carry failed (faults %v)", n, got, s.Faults)
}
// The next start carries it.
if _, err := h.another(t).Start(); err != nil {
t.Fatal(err)
}
got, _ := h.Get(n)
if got.Title != "written beside the log" || got.Folded != 3 || len(got.Notes) != 1 {
t.Fatalf("issue %d is %+v", n, got)
}
if ops := titlesOf(t, h, n); ops != "opened:carried noted carried:written beside the log" {
t.Fatalf("the log: %s", ops)
}
}
// §3, the review of PR 228: an entry that cannot be folded faults its issue on start, and nothing folded past it
// is written.
func TestAnEntryThatCannotBeFoldedLeavesItsIssueAsItIs(t *testing.T) {
h := newHarness(t)
n := h.open(t, "fine")
m := h.open(t, "broken")
if _, err := h.Note(m, "kept"); err != nil {
t.Fatal(err)
}
h.log.mu.Lock()
for i, e := range h.log.entries {
if e.Key == keyOf(m) && e.Sequence == 2 {
h.log.entries[i].Value = json.RawMessage(`{"op":"noted","add":{"notes":"not a list"}}`)
}
}
h.log.mu.Unlock()
before := h.mem.snapshot(IssuesState)[keyOf(m)]
h.mem.empty(IssuesState)
h.mem.put(IssuesState, keyOf(m), json.RawMessage(before))
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if s.Rebuilt != 1 || len(s.Faults) != 1 || !strings.Contains(s.Faults[0], fmt.Sprintf("issue %d cannot be folded", m)) {
t.Fatalf("started %+v", s)
}
if h.mem.snapshot(IssuesState)[keyOf(m)] != before {
t.Fatal("the broken issue's bucket value was written over")
}
if is, _ := h.Get(n); is.Title != "fine" {
t.Fatalf("%d is %+v", n, is)
}
// The broken entry is the test's doing: the fold check would find it, so take the issue out of the bucket.
h.mem.empty(IssuesState)
}
// §2, the review of PR 228: an open whose answer was lost, its issue already changed by another operation, is
// still known as its own by its first entry.
func TestALostOpenIsKnownByItsFirstEntryNotItsLast(t *testing.T) {
h := newHarness(t)
h.log.lost = errors.New("no answer in 30 s")
// After the first operation lands and its answer is lost, another operation follows before the retry.
h.log.between = func() {
h.log.between = func() {
h.log.mu.Lock()
h.log.entries = append(h.log.entries, stdio.LogEntry{Sequence: uint64(len(h.log.entries) + 1), Key: "365",
Value: json.RawMessage(`{"op":"triaged","by":"triage","set":{"severity":"\"minor\""}}`)})
h.log.mu.Unlock()
}
}
o, err := h.Open(OpenAsk{Title: "lost and followed", Report: "r"})
if err != nil || o.Number != 365 {
t.Fatalf("%+v %v", o, err)
}
// The entry added by hand is not one the tracker folded: take the issue out before the fold check.
h.mem.empty(IssuesState)
}
+43 -4
View File
@@ -108,6 +108,8 @@ func main() {
if _, err := t.Settings(); err != nil {
logf("[mesh-issues] the settings cannot be read (%v): every open and move will say so", err)
}
t.Log = runtimeLog{}
go start(t)
go hearOpened(t)
go sweep(t)
if err := stdio.Serve("", Tools(t)); err != nil {
@@ -116,6 +118,30 @@ func main() {
}
}
// start watches the module's own bucket into memory, then carries, folds and indexes (novox/hq ADR 0300 §3, §5,
// §6), each asked again until it is done: the runtime answers only once Serve runs. Until it is done, a list
// reads the bucket, and a write carries the issue it changes first, so no write reaches the log before its
// issue's carry.
func start(t *Tracker) {
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
time.Sleep(wait)
if err := stdio.State(IssuesState).Watch("", t.Watched); err != nil {
logf("[mesh-issues] the bucket is not watched yet (%v); asking again in %s", err, wait)
continue
}
break
}
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
s, err := t.Start()
if err == nil {
logf("[mesh-issues] started: %s", s.Said)
return
}
logf("[mesh-issues] the start did not finish (%v); trying again in %s", err, wait)
time.Sleep(wait)
}
}
// hearOpened subscribes the triage to the tracker's own `opened` and `moved` events and to mesh-delivery's
// `transition` (ADR 0278 §1, ADR 0279 §3), each asked again until the runtime takes it: the runtime answers only
// once Serve runs.
@@ -181,6 +207,9 @@ func (emptyStore) Update(string, string, any, uint64) (uint64, error) {
return 0, errors.New("a dry run at the terminal writes nothing")
}
func (emptyStore) Keys(string) ([]string, error) { return nil, nil }
func (emptyStore) History(string, string) ([]stdio.StateRevision, error) {
return []stdio.StateRevision{}, nil
}
func (emptyStore) Increment(string, string, int) (uint64, error) {
return 0, errors.New("a dry run at the terminal writes nothing")
}
@@ -376,7 +405,8 @@ func Tools(t *Tracker) []stdio.Tool {
"taken over. flag delivered-not-resolved keeps the issues in fixing whose every fix (fixed-by and the " +
"pull requests linked) is delivered; flag abandoned the issues whose worker did nothing on them for a " +
"day with no pull request open, which a move takes over without a reason. Each row says both. Answers " +
"what is open in one call.",
"what is open in one call. search ranks the issues whose title, report or notes hold its words, best first; " +
"before pages the newest-first list: give the answer's next to read on below it.",
Input: schema(nil, map[string]any{
"status": map[string]any{"type": "string", "enum": Statuses, "description": "only issues at this status"},
"owner": text("only issues located in this owner, such as mesh-controller"),
@@ -386,9 +416,11 @@ func Tools(t *Tracker) []stdio.Tool {
"needs": map[string]any{"type": "string", "enum": NeedsKinds, "description": "only issues that need this"},
"ready": map[string]any{"type": "boolean", "description": "only issues ready (true) or waiting (false)"},
"flag": map[string]any{"type": "string", "enum": Flags, "description": "only issues delivered and not resolved, or the abandoned ones"},
"search": text("rank the issues whose title, report or notes hold these words, best first"),
"before": number("only issues numbered below this: the next of the page before, to read on"),
}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("list", m, "status", "owner", "phrase", "limit", "severity", "needs", "ready", "flag")
a, err := readArgs("list", m, "status", "owner", "phrase", "limit", "severity", "needs", "ready", "flag", "search", "before")
if err != nil {
return nil, err
}
@@ -411,6 +443,12 @@ func Tools(t *Tracker) []stdio.Tool {
if ask.Flag, err = a.str("flag"); err != nil {
return nil, err
}
if ask.Search, err = a.str("search"); err != nil {
return nil, err
}
if ask.Before, err = a.number("before", false); err != nil {
return nil, err
}
if v, given := m["ready"]; given && v != nil {
r, err := a.flag("ready")
if err != nil {
@@ -646,13 +684,14 @@ func Tools(t *Tracker) []stdio.Tool {
}},
{Name: Seat + ".reserve",
Description: "Reserve the next decision or research number, before the record is written, and get it: never " +
"one another caller got, and always above every number hq already holds. The answer is the kind and the " +
"one another caller got, and always above the highest number the knowledge base holds (knowledge_highest; " +
"hq issue 493). The answer is the kind and the " +
"number, and the number as a record's name starts with it (0292 for a decision, 046 for research). " +
"Optionally for: what the record is about, in one line. A reserved number is spent forever, whether or " +
"not its record lands, and is never handed out again; who reserved it, when and for what are kept. " +
"Refused, taking no number: a kind that is not decision or research (an issue's number comes from " +
"open), a for that is not one line of plain text or carries a secret, an unknown argument, and the " +
"first number of a kind while what hq holds cannot be read.",
"first number of a kind while the knowledge base does not answer its highest number.",
Input: schema([]string{"kind"}, map[string]any{
"kind": map[string]any{"type": "string", "enum": ReserveKinds, "description": "which number: decision or research"},
"for": text("what the record is about, in one line, such as: the knowledge base holds decisions"),
@@ -22,6 +22,7 @@ type manifest struct {
Consumes []string `json:"consumes"`
Invokes []string `json:"invokes"`
State []json.RawMessage `json:"state"`
Logs []string `json:"logs"`
Tools []string `json:"tools"`
Resources []struct {
ID string `json:"id"`
@@ -87,8 +88,13 @@ func TestItDeclaresAndHoldsTheIssueTrackerSeatOnceForTheMesh(t *testing.T) {
EventTriaged, EventReady, EventReadyToVerify, EventReserved}) {
t.Fatalf("emits %v", m.Emits)
}
// Four states, none expiring: issues keep a short history per key, the counters one value each, the
// backlog's pass of the duplicate check one value (ADR 0278 §6), each reserved number one value (issue 424).
// The record is the log `changes` (ADR 0300 §1). Four states, none expiring: issues the fold of the log (§3),
// still keeping its older values until the carry has read them (they go to one value in a follow-up), the
// counters one value each, the backlog's pass of the duplicate check one value (ADR 0278 §6), each reserved
// number one value (issue 424).
if !reflect.DeepEqual(m.Logs, []string{ChangesLog}) {
t.Fatalf("logs %v", m.Logs)
}
d := m.stateDeclarations(t)
if len(d) != 4 || d[0].Name != IssuesState || d[0].History < 2 || d[0].TTLSeconds != 0 ||
d[1].Name != CountersState || d[1].History > 1 || d[1].TTLSeconds != 0 ||
@@ -22,10 +22,14 @@ type memory struct {
data map[string]map[string]Entry
// fail, when set, is the error the next write of that state answers.
fail map[string]error
// hist is every value each key was given, the last eight kept, as the bucket kept them before the tracker
// switched to its log (novox/hq ADR 0300 §6).
hist map[string]map[string][]Entry
}
func newMemory() *memory {
return &memory{data: map[string]map[string]Entry{IssuesState: {}, CountersState: {}, GroomingState: {}, ReservationsState: {}}, fail: map[string]error{}}
return &memory{data: map[string]map[string]Entry{IssuesState: {}, CountersState: {}, GroomingState: {}, ReservationsState: {}}, fail: map[string]error{},
hist: map[string]map[string][]Entry{}}
}
func (m *memory) bucket(state string) (map[string]Entry, error) {
@@ -71,9 +75,50 @@ func (m *memory) write(state, key string, value any, check func(Entry, bool) boo
}
m.rev++
b[key] = Entry{Value: raw, Revision: m.rev}
if m.hist[state] == nil {
m.hist[state] = map[string][]Entry{}
}
kept := append(m.hist[state][key], b[key])
if len(kept) > 8 {
kept = kept[len(kept)-8:]
}
m.hist[state][key] = kept
return m.rev, nil
}
// History is every value the key still keeps, oldest first, as the SDK's State(...).History answers it: puts and
// deletes, empty for a key that keeps nothing.
func (m *memory) History(state, key string) ([]stdio.StateRevision, error) {
m.mu.Lock()
defer m.mu.Unlock()
out := []stdio.StateRevision{}
for _, e := range m.hist[state][key] {
op, value := "put", e.Value
if e.Value == nil {
op = "delete"
}
out = append(out, stdio.StateRevision{Key: key, Value: value, Revision: e.Revision, Op: op})
}
return out, nil
}
// deleteKey deletes a key, keeping the deletion among its values as the bucket does.
func (m *memory) deleteKey(state, key string) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.data[state], key)
m.rev++
m.hist[state][key] = append(m.hist[state][key], Entry{Revision: m.rev})
}
// empty takes every key out of a state, as an emptied bucket.
func (m *memory) empty(state string) {
m.mu.Lock()
defer m.mu.Unlock()
m.data[state] = map[string]Entry{}
delete(m.hist, state)
}
func (m *memory) Create(state, key string, value any) (uint64, error) {
return m.write(state, key, value, func(_ Entry, there bool) bool { return !there })
}
@@ -154,6 +199,7 @@ type said struct {
type harness struct {
*Tracker
mem *memory
log *fakeLog
mu sync.Mutex
events []said
settings Settings
@@ -162,12 +208,13 @@ type harness struct {
func newHarness(t *testing.T) *harness {
t.Helper()
h := &harness{mem: newMemory(), settings: Settings{FirstNumber: 365, Owners: []string{
h := &harness{mem: newMemory(), log: newFakeLog(), settings: Settings{FirstNumber: 365, Owners: []string{
"hq", "mesh-catalog", "mesh-controller", "mesh-host", "mesh-sdk", "mesh-tools",
"mesh-catalog modules/nats", "mesh-catalog modules/records", "mesh-catalog modules/mesh-issues"}}}
clock := time.Date(2026, 10, 10, 12, 0, 0, 0, time.UTC)
h.Tracker = &Tracker{
Store: h.mem,
Log: h.log,
Emit: func(event string, body any) error {
h.mu.Lock()
defer h.mu.Unlock()
@@ -185,6 +232,11 @@ func newHarness(t *testing.T) *harness {
Logf: t.Logf,
Sleep: func(time.Duration) {},
}
t.Cleanup(func() {
if h.Tracker.Log == Log(h.log) {
h.log.foldCheck(t, h.mem)
}
})
return h
}
+10 -2
View File
@@ -45,6 +45,14 @@ type SetDone struct {
// Set changes an issue's severity, needs or suggested owner, as a person decides.
func (t *Tracker) Set(a SetAsk) (*SetDone, error) { return t.set(t.by(), a, false) }
// setOp is the kind of operation a set is in the log: a person's set, or the triage filling what is empty.
func setOp(onlyEmpty bool) string {
if onlyEmpty {
return EventTriaged
}
return EventSet
}
// set is Set as the one who acts. onlyEmpty fills only a value that is empty, as the triage does: a value a
// person gave meanwhile stands.
func (t *Tracker) set(by string, a SetAsk, onlyEmpty bool) (*SetDone, error) {
@@ -72,7 +80,7 @@ func (t *Tracker) set(by string, a SetAsk, onlyEmpty bool) (*SetDone, error) {
return nil, errors.New("set changes severity, needs or suggested-owner: give at least one")
}
var changed []string
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: setOp(onlyEmpty), By: by}, func(is *Issue) error {
changed = nil
for _, f := range []struct {
name string
@@ -344,7 +352,7 @@ func (t *Tracker) letGo(dep uint64) error {
deps := dependsOn(is)
sort.Slice(deps, func(i, j int) bool { return deps[i] < deps[j] })
claimed := false
if _, err := t.change(w, func(i *Issue) error {
if _, err := t.change(w, Act{Op: EventReady, By: Seat}, func(i *Issue) error {
claimed = false
if equalNumbers(i.ReadyFor, deps) {
return nil
+99 -90
View File
@@ -9,6 +9,7 @@ import (
"math/big"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -19,7 +20,7 @@ import (
// change as an event.
type Tracker struct {
Store Store
// Emit says an event as this module (ADR 0041); the events are the audit trail.
// Emit says an event as this module (ADR 0041); the events announce, the log is the audit trail (ADR 0300 §4).
Emit func(event string, body any) error
Settings func() (Settings, error)
Now func() time.Time
@@ -39,7 +40,11 @@ type Tracker struct {
Notify Notifier
// CallerOf is who called, for the arguments one call was handed (stdio.CallerOf); nil proves nobody.
CallerOf func(map[string]any) string
// Log is the tracker's record, the log `changes` (novox/hq ADR 0300); the bucket `issues` is its fold.
Log Log
groom grooming
held *memoryView
heldOnce sync.Once
}
// The events this module says, one per kind of change.
@@ -122,9 +127,10 @@ type Opened struct {
Warning string `json:"warning,omitempty"`
}
// Open checks an issue, then takes the next number and writes the issue under it (to-be 53 §4). A refused
// issue takes no number. A number taken and then not written — the bus failing between the two — stays
// unused, and the answer says which: numbers may skip, never repeat.
// Open checks an issue, then takes the next number, appends the issue's first operation to the log at expect
// 0, and writes its fold under its number (to-be 53 §4, novox/hq ADR 0300 §2). An issue refused at validation
// takes no number. A number taken and then not taken by the log — the bus failing between the two, or a full
// log — is spent, and the open answers the failure: numbers may skip, never repeat.
func (t *Tracker) Open(a OpenAsk) (*Opened, error) { return t.open(t.by(), a) }
// open is Open as the one who acts.
@@ -174,9 +180,33 @@ func (t *Tracker) open(by string, a OpenAsk) (*Opened, error) {
if record != "" {
is.Record = RecordOf(n, record)
}
if err := t.create(&is); err != nil {
return nil, err
// The first operation, at expect 0 (ADR 0300 §2): one that fails after the number was taken spends the
// number, and the open answers the failure.
var seq uint64
if there, _, rerr := read(t.Store, n); rerr == nil && there.Opening != is.Opening {
// Kept from before the switch and not carried yet: the log does not know it, the bucket does.
err = errBegun
} else {
seq, err = t.begin(&is, Act{Op: OpOpened, By: by})
}
if errors.Is(err, errBegun) {
t.Logf("[mesh-issues] the counter handed out %d and an issue %d is already kept: not overwritten", n, n)
return nil, fmt.Errorf("a fault, not your mistake: the counter handed out %d, and an issue %d is already kept. "+
"Nothing was overwritten and nothing was opened; %d stays unused. Open it again for the next number, "+
"and tell the operator the counter is behind the issues kept", n, n, n)
}
if errors.Is(err, errUnsure) {
t.Logf("[mesh-issues] number %d was taken and the log's answer to the issue was lost: %v", n, err)
return nil, fmt.Errorf("number %d was taken and the log did not confirm the issue (%v): it may have been written. "+
"Call get %d before opening it again; if there is no issue %d, it is spent", n, err, n, n)
}
if err != nil {
t.Logf("[mesh-issues] number %d was taken and the log did not take the issue: %v", n, err)
return nil, fmt.Errorf("number %d was taken and the log did not take the issue (%v): nothing was opened and %d "+
"is spent. Open it again for the next number", n, err, n)
}
is.Folded = seq
t.store(n, &is, 0)
out := &Opened{Number: n, Title: title, Status: is.Status, LocatedIn: located, Record: is.Record, By: is.OpenedBy}
out.Said = fmt.Sprintf("Opened issue %d: %s.", n, title)
if is.Record != "" {
@@ -188,45 +218,6 @@ func (t *Tracker) open(by string, a OpenAsk) (*Opened, error) {
return out, nil
}
// create writes a new issue under its number, never over one: a key that is there already is a fault said
// loudly, and the number stays unused.
func (t *Tracker) create(is *Issue) error {
key := keyOf(is.Number)
var err error
for i := 0; i < createAttempts; i++ {
if i > 0 {
t.Sleep(jitter(time.Duration(i) * 50 * time.Millisecond))
}
_, err = t.Store.Create(IssuesState, key, is)
if err == nil {
return nil
}
if errors.Is(err, stdio.ErrChanged) {
// Ours, written by an attempt whose answer was lost?
there, _, rerr := read(t.Store, is.Number)
if rerr != nil {
t.Logf("[mesh-issues] issue %d is kept and could not be read back: %v", is.Number, rerr)
return fmt.Errorf("issue %d may have been written by this open: the key is taken and could not be read "+
"back (%v). Call get %d before opening it again, so it is not opened twice", is.Number, rerr, is.Number)
}
if there.Opening == is.Opening {
return nil
}
t.Logf("[mesh-issues] the counter handed out %d and an issue %d is already kept: not overwritten", is.Number, is.Number)
return fmt.Errorf("a fault, not your mistake: the counter handed out %d, and an issue %d is already kept. "+
"Nothing was overwritten and nothing was opened; %d stays unused. Open it again for the next number, "+
"and tell the operator the counter is behind the issues kept", is.Number, is.Number, is.Number)
}
}
// A write that timed out may still have landed: say so, never "nothing was opened".
t.Logf("[mesh-issues] number %d was taken and the issue's write was not confirmed: %v", is.Number, err)
if there, _, rerr := read(t.Store, is.Number); rerr == nil && there.Opening == is.Opening {
return nil
}
return fmt.Errorf("number %d was taken and the issue's write was not confirmed (%v): it may have been written. "+
"Call get %d before opening it again; if there is no issue %d, it stays unused", is.Number, err, is.Number, is.Number)
}
// next takes the next issue number: the counter advanced only at the revision read, so two openers — in
// one instance or two — never take one number (ADR 0275 §3).
func (t *Tracker) next() (uint64, error) {
@@ -285,38 +276,6 @@ func (t *Tracker) seed() error {
return nil
}
// change reads an issue, applies f, and writes it only at the revision read; a write another writer won
// is read and applied again, so no note, link or move is lost and none is applied to a stale issue.
func (t *Tracker) change(n uint64, f func(*Issue) error) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil {
return nil, err
}
if err := f(is); err != nil {
return nil, err
}
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
if len(raw) > MostIssueBytes {
return nil, fmt.Errorf("issue %d would be %d bytes; one issue is kept in at most %d (256 KiB). Nothing "+
"was changed: say it shorter, or in a document the issue links", n, len(raw), MostIssueBytes)
}
_, err = t.Store.Update(IssuesState, keyOf(n), json.RawMessage(raw), rev)
if err == nil {
return is, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return nil, fmt.Errorf("issue %d could not be written (%v); nothing was changed", n, err)
}
t.Sleep(jitter(time.Duration(i+1) * 5 * time.Millisecond))
}
return nil, fmt.Errorf("issue %d was changed by another caller %d times while this change was made; nothing was "+
"changed. Try again", n, issueAttempts)
}
// Get is one issue whole.
func (t *Tracker) Get(n uint64) (*Issue, error) {
is, _, err := read(t.Store, n)
@@ -451,7 +410,7 @@ func (t *Tracker) move(by string, a MoveAsk) (*Moved, error) {
var from string
var changed []string
var work workOutcome
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: EventMoved, By: by}, func(is *Issue) error {
changed = nil
from = is.Status
at := t.now()
@@ -610,7 +569,7 @@ func (t *Tracker) note(by string, n uint64, text string) (*Noted, error) {
return nil, err
}
note := Note{By: by, At: t.now(), Text: text}
is, err := t.change(n, func(is *Issue) error {
is, err := t.change(n, Act{Op: EventNoted, By: by}, func(is *Issue) error {
is.Notes = append(is.Notes, note)
touch(is, by, note.At)
return nil
@@ -680,7 +639,7 @@ func (t *Tracker) link(by string, a LinkAsk) (*Linked, error) {
}
}
at := t.now()
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: EventLinked, By: by}, func(is *Issue) error {
for i, l := range is.Links {
if l.Kind == kind && l.Target == target {
if !a.Remove {
@@ -727,6 +686,10 @@ type ListAsk struct {
Ready *bool
// Flag keeps only the issues delivered and not resolved, or the abandoned ones (novox/hq issue 483).
Flag string
// Search ranks the issues whose title, report or notes hold its words, best first (novox/hq ADR 0300 §5).
Search string
// Before keeps only the issues numbered below it: the cursor that pages the newest-first list.
Before uint64
}
// Row is one issue in a list: short enough to read in a chat.
@@ -755,12 +718,15 @@ type Row struct {
// Listed is the answer to a list.
type Listed struct {
Rows []Row `json:"rows"`
Matched int `json:"matched"`
Said string `json:"said"`
Rows []Row `json:"rows"`
Matched int `json:"matched"`
// Next is the before that continues the list where this page ends; 0 when nothing more matches.
Next uint64 `json:"next,omitempty"`
Said string `json:"said"`
}
// List is the issues that match, newest first.
// List is the issues that match, newest first, or ranked by a search: from memory once it is loaded on start
// (novox/hq ADR 0300 §5).
func (t *Tracker) List(a ListAsk) (*Listed, error) {
status := strings.TrimSpace(a.Status)
if status != "" && !known(status) {
@@ -795,6 +761,19 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
return nil, err
}
}
search := ""
if strings.TrimSpace(a.Search) != "" {
if search, err = singleLine("search", a.Search, MostPhrase); err != nil {
return nil, err
}
if len(tokens(search)) == 0 {
return nil, fmt.Errorf("search %q holds no word of two letters or digits", search)
}
if a.Before != 0 {
return nil, errors.New("before pages the newest-first list; a search is ranked best first, so give limit, " +
"or more words, instead")
}
}
limit := a.Limit
if limit <= 0 {
limit = 50
@@ -806,10 +785,30 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
}
now, staleAfter := t.now(), t.workerStaleAfter()
var numbers []uint64
for n := range all {
numbers = append(numbers, n)
if search != "" {
var hits []hit
if v := t.view(); v.ready() {
hits = v.search(search)
} else {
x := newIndex()
for _, is := range all {
x.add(is)
}
hits = x.search(search)
}
for _, h := range hits {
if all[h.Number] != nil {
numbers = append(numbers, h.Number)
}
}
} else {
for n := range all {
if a.Before == 0 || n < a.Before {
numbers = append(numbers, n)
}
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] > numbers[j] })
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] > numbers[j] })
out := &Listed{Rows: []Row{}}
lower := strings.ToLower(phrase)
for _, n := range numbers {
@@ -849,13 +848,23 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
out.Rows = append(out.Rows, row)
}
}
below := ""
if a.Before != 0 {
below = fmt.Sprintf(" below %d", a.Before)
}
switch {
case out.Matched == 0:
out.Said = "No issue matches."
out.Said = "No issue matches" + below + "."
case search != "" && out.Matched > len(out.Rows):
out.Said = fmt.Sprintf("%d issues match the search; the best %d are shown, best first.", out.Matched, len(out.Rows))
case search != "":
out.Said = fmt.Sprintf("%d issues match the search, best first.", out.Matched)
case out.Matched > len(out.Rows):
out.Said = fmt.Sprintf("%d issues match; the newest %d are shown.", out.Matched, len(out.Rows))
out.Next = out.Rows[len(out.Rows)-1].Number
out.Said = fmt.Sprintf("%d issues match%s; the newest %d are shown. The next page is before %d.", out.Matched,
below, len(out.Rows), out.Next)
default:
out.Said = fmt.Sprintf("%d issues match, newest first.", out.Matched)
out.Said = fmt.Sprintf("%d issues match%s, newest first.", out.Matched, below)
}
return out, nil
}
@@ -106,19 +106,19 @@ func TestAnIssuesKeyIsNeverOverwrittenAndTheFaultIsSaid(t *testing.T) {
func TestANumberTakenAndNotWrittenIsSaidAndNeverHandedOutAgain(t *testing.T) {
h := newHarness(t)
h.open(t, "first")
h.mem.fail[IssuesState] = errors.New("the bus did not answer")
h.log.failNext = 1
// One failure is tried again and succeeds: no gap.
if got := h.open(t, "second"); got != 366 {
t.Fatalf("got %d", got)
}
// Failing every time leaves a gap, said.
failing := &failingStore{memory: h.mem, failCreates: createAttempts}
h.Store = failing
// Failing every time spends the number, said (novox/hq ADR 0300 §2).
h.log.failNext = createAttempts
_, err := h.Open(OpenAsk{Title: "third", Report: "r"})
if err == nil || !strings.Contains(err.Error(), "may have been written") || !strings.Contains(err.Error(), "get 367") {
t.Fatalf("a write that never landed: %v", err)
if err == nil || !strings.Contains(err.Error(), "367 is spent") || !strings.Contains(err.Error(), "did not answer") {
t.Fatalf("an append that never landed: %v", err)
}
h.Store = h.mem
// A bucket that takes no write after the log took the issue: the issue is opened, the bucket folds it later.
h.mem.fail[IssuesState] = errors.New("the bus did not answer")
if got := h.open(t, "fourth"); got != 368 {
t.Fatalf("got %d, want 368: a number is never handed out twice", got)
}
@@ -189,17 +189,30 @@ func (u unreadable) Get(state, key string) (*Entry, error) {
return u.memory.Get(state, key)
}
// An open whose write may have landed never says nothing was opened (the review of 2026-10-10).
// unreadableLog loses every append's answer and cannot be read back.
type unreadableLog struct{ *fakeLog }
func (u unreadableLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
_, _ = u.fakeLog.AppendAt(key, value, expect)
return 0, errors.New("no answer in 30 s")
}
func (u unreadableLog) Read(uint64, string, int) (*stdio.LogPage, error) {
return nil, errors.New("no answer in 30 s")
}
// An open whose write may have landed never says nothing was opened (the review of 2026-10-10), in the bucket
// or in the log.
func TestAnOpenWhoseWriteMayHaveLandedNeverSaysNothingWasOpened(t *testing.T) {
h := newHarness(t)
h.Store = lostEvery{h.mem}
if o, err := h.Open(OpenAsk{Title: "t", Report: "r"}); err != nil || o.Number != 365 {
t.Fatalf("a write that landed, every answer lost: %v %v", o, err)
}
h.Store = unreadable{h.mem}
h.Store = h.mem
h.Log = unreadableLog{h.log}
_, err := h.Open(OpenAsk{Title: "t", Report: "r"})
if err == nil || strings.Contains(err.Error(), "nothing was opened") || !strings.Contains(err.Error(), "get 366") {
t.Fatalf("a taken key that cannot be read back: %v", err)
t.Fatalf("an append whose answer was lost and cannot be read back: %v", err)
}
}
@@ -434,7 +447,7 @@ func TestANoteThatWouldMakeTheIssueTooLargeIsRefusedAndNothingChanges(t *testing
for i := 0; i < 40 && err == nil; i++ {
_, err = h.Note(n, note)
}
if err == nil || !strings.Contains(err.Error(), "256 KiB") {
if err == nil || !strings.Contains(err.Error(), "252 KiB") {
t.Fatalf("no refusal: %v", err)
}
e, _ := h.mem.Get(IssuesState, keyOf(n))
@@ -305,7 +305,7 @@ func TestTheSameEventTwiceJudgesOnceAndFinishesWhatIsMissing(t *testing.T) {
o2 := h2.open(t, "the counter skips a number")
n2 := h2.open(t, "a number is skipped")
kept := &Verdict{Verdict: VerdictDuplicate, Issue: o2, Certain: true, Reason: "the same", Severity: "minor", Needs: "none", By: "the agent"}
if _, err := h2.change(n2, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
if _, err := h2.change(n2, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
t.Fatal(err)
}
s2 := withJudge(h2, &stubJudge{err: errors.New("must not be asked")})
@@ -690,7 +690,7 @@ func TestATriagedIssueAPersonResolvedIsNotTouchedAgain(t *testing.T) {
older := h.open(t, "the counter skips a number")
newer := h.open(t, "a number is skipped")
kept := &Verdict{Verdict: VerdictDuplicate, Issue: older, Certain: true, Reason: "same"}
if _, err := h.change(newer, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
if _, err := h.change(newer, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
t.Fatal(err)
}
resolve(t, h, newer)
+291
View File
@@ -0,0 +1,291 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"math"
"sort"
"strings"
"sync"
"unicode"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// memoryView is every folded issue in memory, kept current by watching the module's own bucket, with a full-text
// index of title, report and notes (novox/hq ADR 0300 §5). Until it is loaded on start, the tracker reads
// the bucket as before.
type memoryView struct {
mu sync.RWMutex
loaded bool
issues map[uint64]*Issue
index *index
}
func newMemoryView() *memoryView { return &memoryView{issues: map[uint64]*Issue{}, index: newIndex()} }
// view is the tracker's issues in memory.
func (t *Tracker) view() *memoryView {
t.heldOnce.Do(func() { t.held = newMemoryView() })
return t.held
}
// load holds these issues and rebuilds the index from them. An issue the watch handed over meanwhile is kept
// where it was folded later than the one read.
func (h *memoryView) load(all map[uint64]*Issue) {
h.mu.Lock()
defer h.mu.Unlock()
merged := map[uint64]*Issue{}
for n, is := range all {
merged[n] = is
}
for n, is := range h.issues {
if old := merged[n]; old == nil || old.Folded < is.Folded {
merged[n] = is
}
}
h.issues = merged
h.index = newIndex()
for _, is := range merged {
h.index.add(is)
}
h.loaded = true
}
func (h *memoryView) ready() bool {
h.mu.RLock()
defer h.mu.RUnlock()
return h.loaded
}
// put holds an issue unless what is held was folded later. Before the load it is held for the load to merge.
func (h *memoryView) put(is *Issue) {
h.mu.Lock()
defer h.mu.Unlock()
old := h.issues[is.Number]
if old != nil && old.Folded > is.Folded {
return
}
h.issues[is.Number] = is
if h.loaded {
if old != nil {
h.index.remove(old.Number)
}
h.index.add(is)
}
}
func (h *memoryView) drop(n uint64) {
h.mu.Lock()
defer h.mu.Unlock()
delete(h.issues, n)
if h.loaded {
h.index.remove(n)
}
}
// all is every issue held, each a shallow copy: a caller may set an issue's fields without changing what is held,
// but its lists (notes, links, fixes and the rest) are shared with what is held, so a caller reads them and never
// changes them in place. Every caller in this module only reads; a write goes through change, which reads the
// bucket afresh. A deep copy per list would cost a JSON round trip of every issue on every list.
func (h *memoryView) all() map[uint64]*Issue {
h.mu.RLock()
defer h.mu.RUnlock()
out := make(map[uint64]*Issue, len(h.issues))
for n, is := range h.issues {
c := *is
out[n] = &c
}
return out
}
func (h *memoryView) search(q string) []hit {
h.mu.RLock()
defer h.mu.RUnlock()
return h.index.search(q)
}
// remember holds an issue the tracker just wrote, so a list right after a write shows it before the watch
// hands it back.
func (t *Tracker) remember(is *Issue) {
if is == nil {
return
}
c, err := clone(is)
if err != nil {
return
}
t.view().put(c)
}
// Watched takes one change of the module's own bucket into memory: a put held unless memory holds a later
// fold, a delete dropped (the next start folds it back from the log).
func (t *Tracker) Watched(c stdio.StateChange) error {
n, ok := numberOfKey(c.Key)
if !ok {
return nil
}
if c.Op != "put" {
t.view().drop(n)
return nil
}
var is Issue
if err := json.Unmarshal(c.Value, &is); err != nil {
return fmt.Errorf("issue %d is kept in a form this module cannot read: %v", n, err)
}
t.view().put(&is)
return nil
}
// readAll is every issue the bucket holds, by number, each read by its key.
func (t *Tracker) readAll() (map[uint64]*Issue, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues cannot be listed: %v", err)
}
out := map[uint64]*Issue{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
return nil, err
}
out[n] = is
}
return out, nil
}
// allIssues is every issue kept, by number: from memory once it is loaded, from the bucket before.
func (t *Tracker) allIssues() (map[uint64]*Issue, error) {
if v := t.view(); v.ready() {
return v.all(), nil
}
return t.readAll()
}
// index is a full-text index of the issues' title, report and notes, ranked by BM25. Written here rather than
// taken from a library (bleve): the tracker holds a few thousand issues, the index is rebuilt on every start in
// well under a second, and its whole need — words in, issues ranked out — is a hundred lines, against a
// dependency of dozens of modules for a bundle that has two.
type index struct {
// postings is, for each word, how often it stands in each issue, the title's words counted titleWeight
// times.
postings map[string]map[uint64]int
// lengths is each issue's words, weighted as postings counts them; total their sum.
lengths map[uint64]int
words map[uint64][]string
total int
}
const titleWeight = 3
func newIndex() *index {
return &index{postings: map[string]map[uint64]int{}, lengths: map[uint64]int{}, words: map[uint64][]string{}}
}
// tokens is a text's words: lower case, letters and digits, two characters at least.
func tokens(s string) []string {
var out []string
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) }) {
if len([]rune(w)) >= 2 {
out = append(out, w)
}
}
return out
}
func (x *index) add(is *Issue) {
counts := map[string]int{}
for _, w := range tokens(is.Title) {
counts[w] += titleWeight
}
body := []string{is.Report}
for _, n := range is.Notes {
body = append(body, n.Text)
}
for _, w := range tokens(strings.Join(body, "\n")) {
counts[w]++
}
length := 0
words := make([]string, 0, len(counts))
for w, c := range counts {
p := x.postings[w]
if p == nil {
p = map[uint64]int{}
x.postings[w] = p
}
p[is.Number] = c
length += c
words = append(words, w)
}
x.lengths[is.Number] = length
x.words[is.Number] = words
x.total += length
}
func (x *index) remove(n uint64) {
for _, w := range x.words[n] {
if p := x.postings[w]; p != nil {
delete(p, n)
if len(p) == 0 {
delete(x.postings, w)
}
}
}
x.total -= x.lengths[n]
delete(x.lengths, n)
delete(x.words, n)
}
// hit is one issue a search found, and how well it matched.
type hit struct {
Number uint64
Score float64
}
// search ranks every issue holding at least one of the query's words by BM25: a word few issues hold weighs
// more, a word standing often weighs more with diminishing return, a long issue weighs each word less. Equal
// scores rank the newer first.
func (x *index) search(q string) []hit {
docs := len(x.lengths)
if docs == 0 {
return nil
}
const k1, b = 1.2, 0.75
avg := float64(x.total) / float64(docs)
scores := map[uint64]float64{}
seen := map[string]bool{}
for _, w := range tokens(q) {
if seen[w] {
continue
}
seen[w] = true
p := x.postings[w]
if len(p) == 0 {
continue
}
idf := math.Log(1 + (float64(docs)-float64(len(p))+0.5)/(float64(len(p))+0.5))
for n, tf := range p {
f := float64(tf)
scores[n] += idf * f * (k1 + 1) / (f + k1*(1-b+b*float64(x.lengths[n])/avg))
}
}
out := make([]hit, 0, len(scores))
for n, s := range scores {
out = append(out, hit{n, s})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Score != out[j].Score {
return out[i].Score > out[j].Score
}
return out[i].Number > out[j].Number
})
return out
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-issues
go 1.25.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.51.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf h1:ED/+RszKwm86U7H/NZRbS4b5LzQ8n7rubqKmeFqFtvM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac h1:jnZ/n+fFMo7VQPNNXK1gjStHIJPw3SPbzPxsf/PpZqw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+3
View File
@@ -66,6 +66,9 @@
"records.records_read",
"mesh-knowledge.knowledge_highest"
],
"logs": [
"changes"
],
"state": [
{
"name": "issues",
+23 -7
View File
@@ -99,13 +99,21 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
the record, the records it links and the rules, and answers `approve` or `changes` with findings. The rules it sees
name every kind of link `propose` accepts, from the one list `propose` holds a link to, and the record it sees
shows its state in the knowledge base apart from a research effort's own status (hq issue 492). The background
work asks it at once, before the next step of any drafting, one judgement at a time.
work asks it at once, before the next step of any drafting, one judgement at a time. Every judgement asks a bound
sized to its question: 25 seconds and 2 more per KiB, at most 120 (hq issue 508: an eleven-clause record was never
reviewed in 25). A judgement not done within one bus call is answered pending with a ticket, and collected from
the same node with `claude-code.claude_code_judged`.
- On `changes` the record stays proposed, the findings kept with the version they judged (`get` shows them).
Its author revises it by proposing it again, and that version is reviewed again.
- A judgement that cannot be had (the judge unreachable, or an answer that is not `approve`, or `changes` naming
no finding) leaves the record proposed and says so on it (`get`'s `review`). It is never taken as an
approval. It is asked again after 10, then 20 minutes; after 3 tries the record says its author proposes it
again to have it reviewed again, and the module is unhealthy, naming it, so the mesh raises a condition.
approval. It is owed: its try count and when it is asked again (10, then 20 minutes later) are kept with the
version in the store and said in its history, so the background work takes it up at that time, also after a
restart of the tool runner (hq issue 497). After 3 tries the record says it was the last and its author
proposes it again to have it reviewed again; the operator is told once on the operator's channel (keyed by
record and version, cleared once the record is proposed again, accepted, withdrawn or superseded, and sent
again at the next step when it could not be sent). The module's health names the record and stays healthy:
the record waits on its author, not on the module, so it never holds the module's own rollouts (hq issue 508).
An ask of the operator that cannot be made counts as a try the same way.
- A review whose outcome the database refused to keep is held back for that record alone (10, then 20
minutes, 3 tries, logged as given up), so it never blocks the records behind it.
@@ -124,8 +132,16 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- `partial`: both ends of a change agree;
- `rests-on-live`: no accepted record extends a superseded one.
A failure is kept as a finding and makes the module unhealthy, so the mesh raises a condition naming the rule
and the records. The acceptance stands.
A failure is kept as a finding and told to the operator once, as a warning naming the rule and the records
(`operator-channel.notify`, keyed `knowledge.finding.<rule>.<records>`, sent again at the next step when it
could not be sent, and cleared once the rule holds for them again). It is a fault of the records, which their
authors repair, not of the module: the module's health names it and stays healthy, so it never holds the
module's own rollouts (the operator's decision of 2026-10-12 on hq issue 504). The acceptance stands.
`cited` is also judged ahead, in the review: a decision that names no home and that no accepted (or
superseded) record links is not judged and not accepted. Its review, by the knowledge base itself, asks changes
naming the rule (hq issue 504). It is not refused at propose, because no record can link a number only
reserved: a record citing a new one is proposed after it and accepted, and the new one is proposed again.
## The import
@@ -173,14 +189,14 @@ the consolidated schema for a fresh database is `schema.sql`. A test holds the t
- **the bundle**: the verbs, `knowledge_health`, the operator's answers heard, and once a minute (and at once when
a record is proposed) the router's record read, reservations never proposed withdrawn, the reviews of proposals,
and the drafting worker. It never changes the schema, and
and the drafting worker; and it tells the whole-set findings. It never changes the schema, and
refuses to serve while the schema is behind.
- **`prepare`** (run once): brings the schema to this version.
- **`keep`** (judged by its unit): judges the whole record set every five minutes and keeps the findings.
- **`watch`** (judged by `knowledge_health`, run as the machine's account, the one the runtime runs the bundle
as, so the tool can read what it writes): every minute, reads whether the database serves and what the
findings say, and leaves its verdict in `${dir:state}/health.json`. That is the file the health tool answers
from. A verdict older than three minutes is unhealthy.
from. A verdict older than three minutes is unhealthy; a whole-set finding is named in it but leaves it healthy.
## Settings
@@ -770,11 +770,6 @@ func reviewSays(r *Record) string {
case JudgingUnproven:
return "not reviewed: " + j.Reason
}
s := fmt.Sprintf("not reviewed: %s (try %d of %d); it stays proposed, and no judgement that could not be had counts as an approval", j.Reason, j.Tries, MostReviewTries)
if j.Tries >= MostReviewTries {
s += "; its author proposes it again to have it reviewed again"
} else {
s += "; asked again after a pause"
}
return s
// When it is asked again is said, so a pause is not taken for a review lost (novox/hq issue 497).
return fmt.Sprintf("not reviewed: %s (%s; no judgement that could not be had counts as an approval", j.Reason, j.owedSays())
}
@@ -356,7 +356,7 @@ func (k *KB) tellTopic(ctx context.Context, topic string) {
summary := fmt.Sprintf("The clauses of %d prose decisions on %s are drafted and checked: %d confirmed, %d still drafted.", records, topic, confirmed, drafted)
explanation := "A second judgement confirmed each clause against its record; a clause still drafted is never served as in force. " +
"You may send any record back with the knowledge base's return, and its clauses are drafted again. Nothing needs you otherwise."
if err := k.Notify("knowledge.clauses."+strings.ReplaceAll(topic, " ", "-"), summary, "Clauses drafted for "+topic, explanation); err != nil {
if err := k.Notify("knowledge.clauses."+strings.ReplaceAll(topic, " ", "-"), summary, "Clauses drafted for "+topic, explanation, ""); err != nil {
k.logf("[mesh-knowledge] the operator could not be told about %s's clauses: %v", topic, err)
}
}
@@ -0,0 +1,113 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
)
// novox/hq issue 508: ADR 0297, eleven clauses and eleven links, could not be reviewed within the judgement's 25
// seconds five times running, while short records were reviewed in 14 to 25. A review's bound scales with what is
// judged, up to a cap, and a judgement past one bus call's time is collected by its ticket.
// aRecord is a decision of n clauses, each about size bytes.
func aRecord(n, size int) *Record {
r := &Record{Kind: KindDecision, Number: 297, Title: "Issues are kept as a log of operations", Status: StatusProposed}
for i := 1; i <= n; i++ {
r.Clauses = append(r.Clauses, Clause{N: i, Text: fmt.Sprintf("**Clause %d.** %s", i, strings.Repeat("A word of reasoning. ", size/20))})
}
r.Sections = []Section{{Name: "Context", Text: strings.Repeat("Why it was decided. ", n*size/40)}}
return r
}
// asked is the agent module, played over the calls the knowledge base makes to it.
type asked struct {
keys []string
bodies []map[string]any
pending int // how many times a ticketed judgement answers pending before its answer
refuse bool
}
func (a *asked) ask(key string, body any) (json.RawMessage, error) {
b, _ := body.(map[string]any)
a.keys, a.bodies = append(a.keys, key), append(a.bodies, b)
if strings.HasPrefix(key, "claude-code.claude_code_judge@") {
if _, long := b["within"]; long && a.refuse {
return nil, errors.New("claude_code_judge takes question, schema and model; not within")
}
if _, long := b["within"]; long && a.pending > 0 {
return json.RawMessage(`{"node":"novox","ticket":"t1","pending":true}`), nil
}
return json.RawMessage(`{"answer":{"verdict":"approve","findings":[]},"model":"sonnet","node":"novox"}`), nil
}
if key == "claude-code.claude_code_judged@novox" && b["ticket"] == "t1" {
if a.pending--; a.pending > 0 {
return json.RawMessage(`{"node":"novox","ticket":"t1","pending":true}`), nil
}
return json.RawMessage(`{"answer":{"verdict":"approve","findings":[]},"model":"sonnet","node":"novox","ticket":"t1"}`), nil
}
return nil, fmt.Errorf("nothing answers %s", key)
}
func withinOf(t *testing.T, b map[string]any) time.Duration {
t.Helper()
v, ok := b["within"]
if !ok {
return 0
}
s, ok := v.(int)
if !ok {
t.Fatalf("within is not whole seconds: %#v", v)
}
return time.Duration(s) * time.Second
}
func TestALongRecordsReviewGetsTheLargerBound(t *testing.T) {
short, long := ReviewQuestion(aRecord(2, 200), nil), ReviewQuestion(aRecord(12, 2000), nil)
a := &asked{}
j := judgementOver(a.ask, "novox")
for _, q := range []string{short, long} {
if _, _, err := j(q, reviewSchema, "sonnet"); err != nil {
t.Fatal(err)
}
}
s, l := withinOf(t, a.bodies[0]), withinOf(t, a.bodies[1])
if l <= s || l <= 25*time.Second || l > MostJudgeWithin {
t.Fatalf("a %d-byte review is bounded %s and a %d-byte one %s: the longer must get the larger bound, past 25s and at most %s",
len(short), s, len(long), l, MostJudgeWithin)
}
if got := JudgeWithin(strings.Repeat("x", MostReviewQuestion)); got != MostJudgeWithin {
t.Fatalf("the largest question is bounded %s, not the cap %s", got, MostJudgeWithin)
}
}
// A judgement answered pending is collected by its ticket from the node that holds it, and its answer is the
// judgement's.
func TestAPendingJudgementIsCollectedByItsTicket(t *testing.T) {
a := &asked{pending: 3}
answer, by, err := judgementOver(a.ask, "novox")(ReviewQuestion(aRecord(12, 2000), nil), reviewSchema, "sonnet")
if err != nil {
t.Fatal(err)
}
if string(answer) != `{"verdict":"approve","findings":[]}` || by != "the agent on novox (sonnet)" {
t.Fatalf("%s by %s", answer, by)
}
if strings.Join(a.keys, " ") != "claude-code.claude_code_judge@novox claude-code.claude_code_judged@novox claude-code.claude_code_judged@novox claude-code.claude_code_judged@novox" {
t.Fatalf("asked %q", a.keys)
}
}
// An agent module from before issue 508, which takes no within, is asked again without it: the review is had when
// it fits the old bound, and fails as before when it does not.
func TestAnAgentModuleThatTakesNoWithinIsAskedWithoutIt(t *testing.T) {
a := &asked{refuse: true}
if _, _, err := judgementOver(a.ask, "novox")(ReviewQuestion(aRecord(12, 2000), nil), reviewSchema, "sonnet"); err != nil {
t.Fatal(err)
}
if len(a.bodies) != 2 || withinOf(t, a.bodies[1]) != 0 {
t.Fatalf("asked %v", a.bodies)
}
}
@@ -64,8 +64,12 @@ type Reserver func(kind, title string) (int, error)
// the structured answer and who judged.
type Judgement func(question string, schema map[string]any, model string) (json.RawMessage, string, error)
// Notifier tells the operator's channel something (operator-channel.notify).
type Notifier func(key, summary, headline, explanation string) error
// Notifier tells the operator's channel something (operator-channel.notify). Resolved is the line its message says
// once it is cleared: the channel keeps the one given when it is raised (novox/hq issue 497); "" for none.
type Notifier func(key, summary, headline, explanation, resolved string) error
// Clearer tells the operator's channel that what it was told under key is over, in one resolved line.
type Clearer func(key, resolved string) error
// KB is the knowledge base.
type KB struct {
@@ -75,6 +79,7 @@ type KB struct {
Judge Judgement
Asks AskBus
Notify Notifier
Clear Clearer
Emit func(event string, body any) error
Settings func() (Settings, error)
Now func() time.Time
@@ -509,6 +514,73 @@ func (k *KB) judgeSet(ctx context.Context) []Finding {
return findings
}
// findingPrefix keys the notice of one whole-set finding: the prefix, the rule, and the records it names.
const findingPrefix = "knowledge.finding."
func findingKey(f Finding) string {
names := make([]string, len(f.Records))
for i, r := range f.Records {
names[i] = strings.ReplaceAll(strings.ToLower(r), " ", "-")
}
return findingPrefix + f.Rule + "." + strings.Join(names, ".")
}
// TellFindings, called by the background work and never under the write lock, tells the operator, once, each finding of the rules that judge the whole record set, as a warning
// naming the rule and the records, and clears the notice once the rule holds for them again. A finding is a fault
// of the records, which their authors repair; it is not the module failing, so it leaves the module's health as it
// is and never holds the module's own rollouts (the operator's decision of 2026-10-12 on novox/hq issue 504, as
// issue 508 decided for a proposal never reviewed). A notice that cannot be sent is not kept as told, and is sent
// again at the next step.
func (k *KB) TellFindings() {
if k.Notify == nil || k.Clear == nil {
return
}
ctx := context.Background()
findings, err := k.Store.Findings(ctx)
if err != nil {
k.logf("[mesh-knowledge] the whole-set findings cannot be read to tell them: %v", err)
return
}
now := map[string]bool{}
for _, f := range findings {
key := findingKey(f)
now[key] = true
first, err := k.Store.Told(ctx, key)
if err != nil || !first {
continue
}
records := strings.Join(f.Records, ", ")
summary := fmt.Sprintf("The rule %s, which judges the whole record set, fails: %s.", f.Rule, f.Says)
explanation := "The knowledge base works; its records break a rule. The acceptance stands. The author of " + records +
" repairs it: a decision cited by nothing names its home in a progressive insight (propose with homes and insight), " +
"or a record that cites it is accepted. get and links show the record."
resolved := fmt.Sprintf("The rule %s holds again for %s", f.Rule, records)
if err := k.Notify(key, summary, "A rule of the record set fails: "+records, explanation, resolved); err != nil {
k.logf("[mesh-knowledge] the operator could not be told that %s fails for %s: %v; told again at the next step", f.Rule, records, err)
if uerr := k.Store.Untold(ctx, key); uerr != nil {
k.logf("[mesh-knowledge] and that it was not told could not be kept: %v", uerr)
}
}
}
told, err := k.Store.ToldOf(ctx, findingPrefix)
if err != nil {
k.logf("[mesh-knowledge] the notices of whole-set findings cannot be read: %v", err)
return
}
for _, key := range told {
if now[key] {
continue
}
if err := k.Clear(key, ""); err != nil {
k.logf("[mesh-knowledge] the notice %s could not be cleared: %v; it is cleared again at the next step", key, err)
continue
}
if err := k.Store.Untold(ctx, key); err != nil {
k.logf("[mesh-knowledge] the notice %s was cleared, and that could not be kept: %v", key, err)
}
}
}
// ---- reservations that were never proposed -------------------------------------------------------------------
// Withdraw withdraws every number reserved longer ago than the settings allow and never proposed: it stays spent,
@@ -28,13 +28,21 @@ import (
const bundleVar = "MESH_KNOWLEDGE_TEST_BUNDLE"
// bundleDay is the bundle's clock: the day the fixture's proposals are dated (valid's Date), so a reservation and the
// proposal after it are of one day whatever day the test runs on (issue 540).
var bundleDay = time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC)
func TestMain(m *testing.M) {
if os.Getenv(bundleVar) == "1" {
// The bundle the bus test starts: the knowledge base as the runtime launches it, its store in memory.
k := newKB(newMemStore())
k.Now = func() time.Time { return bundleDay }
serve(k, os.Getenv("MESH_KNOWLEDGE_STATUS"))
os.Exit(0)
}
// Every test runs in a zone that is not UTC, so a time said locally is proven to be, wherever the tests run. It
// is set once, before any goroutine starts: a test that set it while a bus test's server read it raced.
time.Local = testZone
code := m.Run()
stopPostgres()
os.Exit(code)
@@ -397,7 +405,11 @@ type world struct {
router *router
said *said
told []string
clock time.Time
// toldKeys are the keys told under, and cleared what was cleared: key, then its resolved line.
toldKeys []string
cleared [][2]string
resolved []string // the resolved line given with each notice raised
clock time.Time
}
func newWorld(t *testing.T, s Store, records Records) *world {
@@ -406,8 +418,12 @@ func newWorld(t *testing.T, s Store, records Records) *world {
clock: time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC)}
w.k = &KB{Store: s, Records: records, Reserve: w.tracker.reserve, Judge: w.judge.judge, Asks: w.router,
Emit: w.said.emit, Settings: func() (Settings, error) { return Settings{}, nil }, Now: func() time.Time { return w.clock },
Logf: func(f string, a ...any) { t.Logf(f, a...) },
Notify: func(key, summary, headline, explanation string) error { w.told = append(w.told, summary); return nil }}
Logf: func(f string, a ...any) { t.Logf(f, a...) },
Notify: func(key, summary, headline, explanation, resolved string) error {
w.told, w.toldKeys, w.resolved = append(w.told, summary), append(w.toldKeys, key), append(w.resolved, resolved)
return nil
},
Clear: func(key, resolved string) error { w.cleared = append(w.cleared, [2]string{key, resolved}); return nil }}
return w
}
@@ -423,9 +439,13 @@ func valid(n int) ProposeAsk {
{Name: "Consequences", Text: "Merges land sooner."},
{Name: "How it is checked", Text: "The controller's merge-window tests."},
},
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}}}
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}},
Homes: []string{testHome}}
}
// testHome is the hq design valid names as its home: a decision is accepted only cited (novox/hq issue 504).
const testHome = "03-DESIGN/01-to-be/55-the-knowledge-base.md"
// imported is a world whose knowledge base holds the fixture's records.
func imported(t *testing.T, s Store) *world {
t.Helper()
+125 -39
View File
@@ -87,6 +87,7 @@ func newKB(store Store) *KB {
Judge: busJudgement(os.Getenv("MESH_NODE")),
Asks: sdkAskBus{},
Notify: busNotifier,
Clear: busClearer,
Emit: func(event string, body any) error { return stdio.Emit(event, body) },
Settings: func() (Settings, error) { return readSettings(file) },
Logf: logf,
@@ -163,6 +164,8 @@ func background(k *KB) {
logf("[mesh-knowledge] withdrew %d reservation(s) never proposed", n)
}
review()
// The keeper judges the record set in a process of its own, which tells nobody: its findings are told here.
k.TellFindings()
for {
said, err := k.DraftStep()
if err != nil {
@@ -201,32 +204,95 @@ func busReserver(kind, title string) (int, error) {
}
// busJudgement asks the agent module on this node for one judgement (ADR 0278).
func busJudgement(node string) Judgement {
func busJudgement(node string) Judgement { return judgementOver(stdio.Ask, node) }
// The bound a judgement asks scales with its question (novox/hq issue 508): ADR 0297, eleven clauses and eleven
// links, was never reviewed within the agent module's 25 seconds, while short records took 14 to 25. Each KiB of
// question adds JudgePerKiB to JudgeFloor, up to MostJudgeWithin, which the agent module takes at most. A sized
// bound was chosen over asking clause by clause: every judgement pays the agent's start, which alone is most of
// those 14 to 25 seconds, so a dozen clause-sized judgements would each risk the same 25 seconds, take minutes
// together, and still need a judgement of the whole record for what binds its clauses together.
const (
JudgeFloor = 25 * time.Second
JudgePerKiB = 2 * time.Second
MostJudgeWithin = 120 * time.Second
)
// JudgeWithin is the bound a judgement of this question asks.
func JudgeWithin(question string) time.Duration {
return min(JudgeFloor+time.Duration(len(question)>>10)*JudgePerKiB, MostJudgeWithin)
}
// judgementOver asks the agent module through ask. A bound past one bus call's 30 seconds is answered pending with
// a ticket when the judgement is not done in the call, and collected from the node that holds it with
// claude_code_judged (novox/hq issue 508).
func judgementOver(ask func(key string, body any) (json.RawMessage, error), node string) Judgement {
return func(question string, schema map[string]any, model string) (json.RawMessage, string, error) {
key := "claude-code.claude_code_judge"
if node != "" {
key += "@" + node
}
raw, err := stdio.Ask(key, map[string]any{"question": question, "schema": schema, "model": model})
if err != nil {
return nil, "", err
within := JudgeWithin(question)
body := map[string]any{"question": question, "schema": schema, "model": model}
if within > JudgeFloor {
body["within"] = int(within / time.Second)
}
var out struct {
Answer json.RawMessage `json:"answer"`
Model string `json:"model"`
Node string `json:"node"`
raw, err := ask(key, body)
if err != nil && body["within"] != nil && strings.Contains(err.Error(), "not within") {
// An agent module from before issue 508 takes no within: asked again without it, under its own bound.
delete(body, "within")
raw, err = ask(key, body)
}
if err := json.Unmarshal(raw, &out); err != nil || len(out.Answer) == 0 {
return nil, "", fmt.Errorf("the agent module's answer is not a judgement: %.200s", raw)
// Past the judgement's bound the agent module ends it and says so; the deadline only guards against a
// module that answers pending for ever.
deadline := time.Now().Add(within + time.Minute)
for {
if err != nil {
return nil, "", err
}
var out struct {
Answer json.RawMessage `json:"answer"`
Model string `json:"model"`
Node string `json:"node"`
Ticket string `json:"ticket"`
Pending bool `json:"pending"`
}
if err := json.Unmarshal(raw, &out); err != nil {
return nil, "", fmt.Errorf("the agent module's answer is not a judgement: %.200s", raw)
}
if !out.Pending {
if len(out.Answer) == 0 {
return nil, "", fmt.Errorf("the agent module's answer is not a judgement: %.200s", raw)
}
return out.Answer, fmt.Sprintf("the agent on %s (%s)", out.Node, out.Model), nil
}
if out.Ticket == "" || out.Node == "" {
return nil, "", fmt.Errorf("the agent module answered pending with no ticket or node: %.200s", raw)
}
if time.Now().After(deadline) {
return nil, "", fmt.Errorf("the agent on %s still had not answered %s after its bound of %s", out.Node, out.Ticket, within)
}
raw, err = ask("claude-code.claude_code_judged@"+out.Node, map[string]any{"ticket": out.Ticket})
}
return out.Answer, fmt.Sprintf("the agent on %s (%s)", out.Node, out.Model), nil
}
}
// busNotifier tells the operator's channel, as a warning: it is the one severity below urgent.
func busNotifier(key, summary, headline, explanation string) error {
func busNotifier(key, summary, headline, explanation, resolved string) error {
body := map[string]any{"key": key, "subject": "the knowledge base",
"severity": "warning", "summary": summary, "headline": headline, "explanation": explanation}
if resolved != "" {
body["resolved"] = resolved
}
_, err := stdio.Ask("seat:operator-channel.notify", body)
return err
}
// busClearer tells the operator's channel that what it was told under key is over: the message is edited to say
// resolved (novox/hq issue 497).
func busClearer(key, resolved string) error {
_, err := stdio.Ask("seat:operator-channel.notify", map[string]any{"key": key, "subject": "the knowledge base",
"severity": "warning", "summary": summary, "headline": headline, "explanation": explanation})
"clear": true, "resolved": resolved})
return err
}
@@ -329,6 +395,20 @@ func (l *lazyStore) Told(ctx context.Context, topic string) (bool, error) {
}
return s.Told(ctx, topic)
}
func (l *lazyStore) ToldOf(ctx context.Context, prefix string) ([]string, error) {
s, err := l.store(ctx)
if err != nil {
return nil, err
}
return s.ToldOf(ctx, prefix)
}
func (l *lazyStore) Untold(ctx context.Context, key string) error {
s, err := l.store(ctx)
if err != nil {
return err
}
return s.Untold(ctx, key)
}
// ---- prepare, check, dry-run --------------------------------------------------------------------------------
@@ -474,9 +554,11 @@ type HealthAnswer struct {
At time.Time `json:"at,omitempty"`
}
// Health is how the knowledge base is: unhealthy when its database cannot serve, when a rule that judges the
// whole record set fails, or when a proposed record ran out of review tries — the mesh raises a condition naming
// the rule, the records, or the records not reviewed.
// Health is whether the knowledge base works: unhealthy when its database cannot serve. A rule that judges the whole
// record set and fails is a fault of the records, which their authors repair, not of the module: it is told to the
// operator as a warning naming the rule and the records (TellFindings), and named here, but leaves the module
// healthy (the operator's decision of 2026-10-12 on novox/hq issue 504). So does a proposal that ran out of review
// tries (novox/hq issue 508). Neither holds the module's own rollouts.
func (k *KB) Health(ready Healther) HealthAnswer {
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
@@ -498,33 +580,36 @@ func (k *KB) Health(ready Healther) HealthAnswer {
if err != nil {
return HealthAnswer{State: "unhealthy", Why: "the records cannot be read: " + err.Error()}
}
unreviewed := Unreviewed(all)
if len(findings)+len(unreviewed) > 0 {
var parts []string
if len(unreviewed) > 0 {
if len(unreviewed) > 8 {
unreviewed = append(unreviewed[:8], fmt.Sprintf("and %d more", len(unreviewed)-8))
}
parts = append(parts, fmt.Sprintf("%d proposed record(s) could not be reviewed in %d tries and wait for their authors to propose them again: %s",
len(Unreviewed(all)), MostReviewTries, strings.Join(unreviewed, "; ")))
waiting := ""
if unreviewed := Unreviewed(all); len(unreviewed) > 0 {
n := len(unreviewed)
if n > 8 {
unreviewed = append(unreviewed[:8], fmt.Sprintf("and %d more", n-8))
}
if len(says) > 0 {
if len(says) > 8 {
says = append(says[:8], fmt.Sprintf("and %d more", len(says)-8))
}
parts = append(parts, fmt.Sprintf("%d rule(s) that judge the whole record set fail: %s", len(findings), strings.Join(says, "; ")))
}
return HealthAnswer{State: "unhealthy", Why: strings.Join(parts, ". ")}
waiting = fmt.Sprintf("%d proposed record(s) could not be reviewed in %d tries and wait for their authors to propose them again: %s",
n, MostReviewTries, strings.Join(unreviewed, "; "))
}
return HealthAnswer{State: "healthy", Healthy: true, Why: "the database serves, every rule that judges the whole record set holds, and no proposal ran out of review tries"}
why := "the database serves and every rule that judges the whole record set holds"
if len(says) > 0 {
if len(says) > 8 {
says = append(says[:8], fmt.Sprintf("and %d more", len(says)-8))
}
why = fmt.Sprintf("the database serves; %d rule(s) that judge the whole record set fail, told to the operator as a warning "+
"for the records' authors to repair: %s", len(findings), strings.Join(says, "; "))
}
if waiting != "" {
why += "; " + waiting
}
return HealthAnswer{State: "healthy", Healthy: true, Why: why}
}
func healthTool(status string, now func() time.Time) stdio.Tool {
return stdio.Tool{Name: HealthTool,
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version and every " +
"rule that judges the whole record set holds; unhealthy otherwise, naming the rule and the records — a decision accepted " +
"that nothing cites, a supersession or a change said on one end only, a record extending a superseded one — or when a " +
"proposed record ran out of review tries, naming it.",
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version, unhealthy " +
"otherwise. A rule that judges the whole record set and fails — a decision accepted that nothing cites, a supersession " +
"or a change said on one end only, a record extending a superseded one — is named but leaves it healthy: it is told " +
"to the operator as a warning for the records' authors (novox/hq issue 504). So is a proposed record that ran out of " +
"review tries: it waits on its author (novox/hq issue 508).",
Run: func(map[string]any) (any, error) { return ReadStatus(status, now()), nil }}
}
@@ -834,7 +919,8 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
"link to nothing, a clause without a bold lead, a superseded record cited alone, an unknown topic, a retired word. A " +
"proposal is revised by proposing it again; an accepted record changes only by a progressive insight (insight: what " +
"stood before). The knowledge base then reviews each version itself, in a judgement that sees only the record, the " +
"records it links and the rules: on approve it is accepted (a record that needs the operator waits for the operator's " +
"records it links and the rules. A decision that names no home and that no accepted record links is not judged: " +
"its review asks changes for the whole-set rule cited, which its acceptance would break. On approve it is accepted (a record that needs the operator waits for the operator's " +
"Approve), on changes it stays proposed with the findings for its author to revise, and when no judgement can be had " +
"it stays proposed and says so. Nothing proposed is in force.",
Input: schema([]string{"number", "title"}, map[string]any{
@@ -849,7 +935,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
"links": map[string]any{"type": "array", "description": "its links: kind, to (\"0276\", \"0276 §4\", \"research 045\", \"issue 365\", a design's path), from (the clause it goes out from)",
"items": map[string]any{"type": "object", "properties": map[string]any{"kind": map[string]any{"type": "string", "enum": LinkKinds},
"to": text("the target"), "from": integer("the clause of this record it goes out from")}, "required": []string{"kind", "to"}}},
"homes": texts("hq documents that cite it, such as a design's path"),
"homes": texts("hq documents that cite it, such as a design's path; a decision needs one unless an accepted record links it"),
"needs-operator": flag("it holds only once the operator approves it"),
"status": map[string]any{"type": "string", "enum": ResearchStatuses, "description": "a research effort's status"},
"insight": text("for an accepted record: a progressive insight, saying what stood before"),
@@ -76,7 +76,7 @@ func TestItDeclaresAndHoldsTheKnowledgeBaseSeatOnceForTheMesh(t *testing.T) {
if !reflect.DeepEqual(m.Uses, []string{"operator-channel"}) || len(m.Consumes) != 0 {
t.Fatalf("uses %v, consumes %v", m.Uses, m.Consumes)
}
if !reflect.DeepEqual(m.Invokes, []string{"seat:issue-tracker.reserve", "claude-code.claude_code_judge", "records.records_status",
if !reflect.DeepEqual(m.Invokes, []string{"seat:issue-tracker.reserve", "claude-code.claude_code_judge", "claude-code.claude_code_judged", "records.records_status",
"records.records_list", "records.records_read", "seat:operator-channel.notify"}) {
t.Fatalf("invokes %v", m.Invokes)
}
@@ -141,13 +141,8 @@ func (k *KB) askOperator(ctx context.Context, held *Entry, w *Record, by string,
}
unasked := func(why string) (string, error) {
w.Ask = nil
w.Judging = &Judging{State: JudgingFailed, Tries: tries + 1, Reason: why, At: k.now()}
said := fmt.Sprintf("approved by the knowledge base's review, and %s (try %d of %d); it stays proposed", why, tries+1, MostReviewTries)
if tries+1 >= MostReviewTries {
said += ", and its author proposes it again to have it reviewed again"
} else {
said += " and is reviewed again after a pause"
}
w.Judging = failedJudging(tries+1, why, k.now())
said := fmt.Sprintf("approved by the knowledge base's review, and %s (%s", why, w.Judging.owedSays())
e, err := k.Store.Get(ctx, w.Key())
if err != nil || e == nil {
return "", fmt.Errorf("%s: %s, and that could not be kept (%v)", w.Key(), said, err)
@@ -549,6 +549,29 @@ func (s *PgStore) Findings(ctx context.Context) ([]Finding, error) {
return out, rows.Err()
}
func (s *PgStore) ToldOf(ctx context.Context, prefix string) ([]string, error) {
// left() and not like: a prefix may hold an underscore, which like would take for any character.
rows, err := s.pool.Query(ctx, `select topic from told where left(topic, length($1)) = $1 order by topic`, prefix)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var k string
if err := rows.Scan(&k); err != nil {
return nil, err
}
out = append(out, k)
}
return out, rows.Err()
}
func (s *PgStore) Untold(ctx context.Context, key string) error {
_, err := s.pool.Exec(ctx, `delete from told where topic = $1`, key)
return err
}
func (s *PgStore) Told(ctx context.Context, topic string) (bool, error) {
tag, err := s.pool.Exec(ctx, `insert into told (topic) values ($1) on conflict do nothing`, topic)
if err != nil {
@@ -9,8 +9,12 @@ package main
// - On changes the record stays proposed, the findings kept with the version they judged, and its author revises
// it with a new version, which is reviewed again.
// - A judgement that cannot be had — the judge unreachable, or its answer unreadable — leaves the record proposed
// and says so on it. It is never taken as an approval. It is asked again after a pause, up to MostReviewTries;
// after that the record says its author proposes it again to have it reviewed again.
// and says so on it. It is never taken as an approval. It is owed: the try count and when the next try is due
// are kept with the version in the store, so the background work takes it up at that time, also after a restart
// of the tool runner (novox/hq issue 497), up to MostReviewTries. After the last try the record says so, its
// author proposes it again to have it reviewed again, and the operator is told once on the operator's channel.
// The module's health names it and stays healthy: the record waits on its author, and the module works
// (novox/hq issue 508).
// - A record with a version whose writer the runtime did not prove is not reviewed: who wrote it cannot be told,
// and no one but its proven author revises it, so its author proposes it again under a new number.
// - On approve, a record that does not need the operator is accepted at once; one its author marked
@@ -35,6 +39,41 @@ type Judging struct {
Tries int `json:"tries,omitempty"`
Reason string `json:"reason,omitempty"`
At time.Time `json:"at,omitempty"`
// Due is when a failed review is asked again; zero after its last try. It is kept in the store with the
// version, so a restart of the tool runner neither loses the try nor asks it early (novox/hq issue 497).
Due time.Time `json:"due,omitempty"`
}
// ReviewPause is the pause after a failed try of a review, times the tries so far.
const ReviewPause = 10 * time.Minute
// failedJudging is a review that could not be had, the tries'th time: owed again at a time it keeps, unless that
// was its last try.
func failedJudging(tries int, reason string, now time.Time) *Judging {
j := &Judging{State: JudgingFailed, Tries: tries, Reason: reason, At: now}
if tries < MostReviewTries {
j.Due = now.Add(time.Duration(tries) * ReviewPause)
}
return j
}
// due is when a failed review is asked again. A version kept before the time was kept (novox/hq issue 497) has
// none, and is due after the same pause from its last try.
func (j *Judging) due() time.Time {
if !j.Due.IsZero() {
return j.Due
}
return j.At.Add(time.Duration(j.Tries) * ReviewPause)
}
// owedSays is what a failed try says of what follows it: when it is asked again, or that it was the last.
func (j *Judging) owedSays() string {
if j.Tries >= MostReviewTries {
return fmt.Sprintf("try %d of %d, the last); it stays proposed and nothing of it is in force, the operator is told, "+
"and its author proposes it again to have it reviewed again", j.Tries, MostReviewTries)
}
return fmt.Sprintf("try %d of %d); it stays proposed, and it is asked again at %s", j.Tries, MostReviewTries,
j.due().Local().Format("2006-01-02 15:04 MST"))
}
// The states of Judging.
@@ -162,7 +201,7 @@ func (k *KB) reviewDue(e *Entry) bool {
case JudgingDue:
return true
case JudgingFailed:
return w.Judging.Tries < MostReviewTries && k.now().Sub(w.Judging.At) >= time.Duration(w.Judging.Tries)*10*time.Minute
return w.Judging.Tries < MostReviewTries && !k.now().Before(w.Judging.due())
}
return false
}
@@ -217,7 +256,12 @@ func (k *KB) ReviewStep() (string, error) {
if err != nil {
return "", err
}
k.clearUnreviewed(all)
for _, e := range all {
if ranOutOfTries(e.Working()) {
k.tellUnreviewed(e)
continue
}
if !k.reviewDue(e) || k.heldBack(e) {
continue
}
@@ -235,18 +279,128 @@ func (k *KB) ReviewStep() (string, error) {
}
// Unreviewed are the proposed records that ran out of review tries: no judgement could be had, or the operator
// could not be asked. They make the module unhealthy, so the mesh raises a condition naming them.
// could not be asked. The module's health names them and stays healthy (novox/hq issue 508); the operator is told.
func Unreviewed(all []*Entry) []string {
var out []string
for _, e := range all {
w := e.Working()
if w.Status == StatusProposed && w.Judging != nil && w.Judging.State == JudgingFailed && w.Judging.Tries >= MostReviewTries {
if ranOutOfTries(w) {
out = append(out, fmt.Sprintf("%s (%s)", w.Key(), w.Judging.Reason))
}
}
return out
}
func ranOutOfTries(w *Record) bool {
return w.Status == StatusProposed && w.Judging != nil && w.Judging.State == JudgingFailed && w.Judging.Tries >= MostReviewTries
}
// unreviewedPrefix starts the key of every notice of a proposal that ran out of review tries. The notice's key is
// unreviewedPrefix, the record and the version that recorded its last try (knowledge.unreviewed.decision-294.v9), so
// a later version's notice is its own; what the store keeps as told is that key, a space and the last failed try's
// time, which names the run of tries whatever versions are written after it (a link change writes one).
const unreviewedPrefix = "knowledge.unreviewed."
func unreviewedOf(k Key) string { return fmt.Sprintf("%s%s-%d.", unreviewedPrefix, k.Kind, k.Number) }
// tellUnreviewed tells the operator, once for each run of tries, that a proposal ran out of review tries: a history
// line alone is read by nobody, and a proposal never reviewed must not wait in silence (novox/hq issue 497).
func (k *KB) tellUnreviewed(e *Entry) {
if k.Notify == nil {
return
}
ctx := context.Background()
w := e.Working()
at := w.Judging.At.UTC().Format(time.RFC3339Nano)
told, err := k.Store.ToldOf(ctx, unreviewedOf(w.Key()))
if err != nil {
k.logf("[mesh-knowledge] whether the operator was told that %s was never reviewed cannot be read: %v", w.Key(), err)
return
}
for _, t := range told {
if strings.HasSuffix(t, " "+at) {
return
}
}
version := e.CurrentVersion
if e.Pending != nil {
version = e.PendingVersion
}
key := fmt.Sprintf("%sv%d", unreviewedOf(w.Key()), version)
// Kept as told before the notice is sent, so two steps never send it twice; a notice that fails is no longer
// kept as told, and is sent again at the next step: the module's health stays healthy for it (novox/hq issue
// 508), so this notice is the only loud word of it.
if first, err := k.Store.Told(ctx, key+" "+at); err != nil || !first {
return
}
summary := fmt.Sprintf("%s was never reviewed: its review could not be had %d times (last: %s). It stays proposed, and nothing of it is in force.",
w.Key(), w.Judging.Tries, w.Judging.Reason)
explanation := "No review that could not be had counts as an approval. Its author proposes it again to have it reviewed again; " +
"the knowledge base's get and history show it."
// The channel edits a cleared message with the resolved line given here, not the one sent with the clear.
resolved := w.Key().String() + " no longer waits for its review"
if err := k.Notify(key, summary, "A proposal was never reviewed: "+w.Key().String(), explanation, resolved); err != nil {
k.logf("[mesh-knowledge] the operator could not be told that %s was never reviewed: %v; told again at the next step", w.Key(), err)
if uerr := k.Store.Untold(ctx, key+" "+at); uerr != nil {
k.logf("[mesh-knowledge] and that it was not told could not be kept: %v", uerr)
}
}
}
// clearUnreviewed clears every notice of a proposal that ran out of review tries once the record has left that
// state — proposed again, accepted, withdrawn or superseded — so the operator's message never keeps saying it waits
// (novox/hq issue 497). A clear that fails is kept as told and sent again at the next step.
func (k *KB) clearUnreviewed(all []*Entry) {
if k.Clear == nil {
return
}
ctx := context.Background()
told, err := k.Store.ToldOf(ctx, unreviewedPrefix)
if err != nil {
k.logf("[mesh-knowledge] the notices of proposals never reviewed cannot be read: %v", err)
return
}
byKey := map[string]*Entry{}
for _, e := range all {
byKey[unreviewedOf(e.Current.Key())] = e
}
for _, t := range told {
key, at, ok := strings.Cut(t, " ")
if !ok {
continue
}
var e *Entry
for prefix, x := range byKey {
if strings.HasPrefix(key, prefix) {
e = x
break
}
}
resolved := strings.TrimPrefix(key, unreviewedPrefix) + " is no longer held by the knowledge base"
if e != nil {
w := e.Working()
if ranOutOfTries(w) && w.Judging.At.UTC().Format(time.RFC3339Nano) == at {
continue
}
switch {
case w.Status == StatusProposed:
resolved = w.Key().String() + " was proposed again and is reviewed again"
case w.Accepted():
resolved = w.Key().String() + " was reviewed and is " + w.Status
default:
resolved = w.Key().String() + " no longer waits for its review: it is " + w.Status
}
}
if err := k.Clear(key, resolved); err != nil {
k.logf("[mesh-knowledge] the notice that %s was never reviewed could not be cleared: %v; it is cleared again at the next step", key, err)
continue
}
if err := k.Store.Untold(ctx, t); err != nil {
k.logf("[mesh-knowledge] the notice %s was cleared, and that could not be kept: %v", key, err)
}
}
}
// reviewOne asks the judgement on one record's working version, and acts on its verdict. The write lock is held
// only to read and to write, never across the judgement; a record revised while it was judged is judged again.
func (k *KB) reviewOne(key Key) (string, error) {
@@ -254,7 +408,10 @@ func (k *KB) reviewOne(key Key) (string, error) {
k.mu.Lock()
e, err := k.Store.Get(ctx, key)
var linked []*Record
var all []*Entry
var allErr error
if err == nil && e != nil {
all, allErr = k.Store.All(ctx)
seen := map[Key]bool{key: true}
for _, l := range e.Working().Links {
to, ok := l.To.Record()
@@ -276,9 +433,20 @@ func (k *KB) reviewOne(key Key) (string, error) {
var verdict string
var findings []string
by := ReviewBy
if len(question) > MostReviewQuestion {
uncited := ""
if allErr == nil {
uncited = UncitedOnAcceptance(w, all)
}
switch {
case allErr != nil:
err = fmt.Errorf("the record set cannot be read to judge whether a record cites it: %v", allErr)
case uncited != "":
// The whole-set rule cited, judged ahead: changes, asked by the knowledge base itself without a judgement,
// so its acceptance never turns the module's health red for it (novox/hq issue 504).
verdict, findings = VerdictChanges, []string{"the rule cited, which judges the whole record set: " + uncited}
case len(question) > MostReviewQuestion:
err = fmt.Errorf("the record is %d bytes as a question, and a judgement takes at most %d: shorten it", len(question), MostReviewQuestion)
} else {
default:
var answer json.RawMessage
var judge string
answer, judge, err = k.Judge(question, reviewSchema, k.judgeModel())
@@ -312,12 +480,8 @@ func (k *KB) reviewOne(key Key) (string, error) {
}
if err != nil {
// No verdict: the record stays proposed and says so. Never an approval.
w.Judging.State, w.Judging.Reason, w.Judging.At = JudgingFailed, err.Error(), k.now()
w.Judging.Tries++
why := fmt.Sprintf("not reviewed: %v (try %d of %d); it stays proposed", err, w.Judging.Tries, MostReviewTries)
if w.Judging.Tries >= MostReviewTries {
why += ", and its author proposes it again to have it reviewed again"
}
w.Judging = failedJudging(w.Judging.Tries+1, err.Error(), k.now())
why := fmt.Sprintf("not reviewed: %v (%s", err, w.Judging.owedSays())
by = ReviewBy
if serr := save(w, why); serr != nil {
return "", fmt.Errorf("%s: %s, and that could not be kept (%v)", key, why, serr)
@@ -73,21 +73,36 @@ func TestARecordWhoseReviewCannotBeKeptBlocksNoOtherAndIsTriedBoundedly(t *testi
}
}
// A record that ran out of review tries makes the module unhealthy, naming it: the mesh raises a condition.
func TestARecordThatCouldNotBeReviewedMakesTheModuleUnhealthy(t *testing.T) {
// A record that ran out of review tries waits on its author, not on the module: it is told to the operator and leaves
// the module healthy, so it never holds the module's own rollouts (novox/hq issue 508). A notice that could not be
// sent is sent again at the next step, since health no longer stands for it.
func TestARecordThatCouldNotBeReviewedLeavesTheModuleHealthyAndIsTold(t *testing.T) {
w := imported(t, newMemStore())
n := proposed(t, w, "author@g14", false)
w.judge.reviewFail = true
for i := 0; i < MostReviewTries; i++ {
notify := w.k.Notify
failing := true
w.k.Notify = func(key, summary, headline, explanation, resolved string) error {
if failing {
failing = false
return errors.New("the operator's channel did not answer")
}
return notify(key, summary, headline, explanation, resolved)
}
for i := 0; i < MostReviewTries+1; i++ {
w.reviewAll(t)
if h := w.k.Health(nil); i < MostReviewTries-1 && !h.Healthy {
t.Fatalf("unhealthy while tries remain: %s", h.Why)
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy after try %d: %s", i+1, h.Why)
}
w.clock = w.clock.Add(time.Hour)
}
w.reviewAll(t)
h := w.k.Health(nil)
if h.Healthy || !strings.Contains(h.Why, itoaKey(n)) || !strings.Contains(h.Why, "could not be reviewed") {
t.Fatalf("%+v", h)
if !h.Healthy || h.State != "healthy" || !strings.Contains(h.Why, itoaKey(n)) || !strings.Contains(h.Why, "author") {
t.Fatalf("not healthy, or the record waiting on its author not named: %+v", h)
}
if len(w.told) != 1 || !strings.Contains(w.told[0], itoaKey(n)) || !strings.Contains(w.told[0], "never reviewed") {
t.Fatalf("told %q", w.told)
}
}
@@ -0,0 +1,202 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
)
// novox/hq issue 497: a review that could not be had is owed. What is owed is kept in the knowledge base's store,
// with its try count and when its next try is due, so the background work takes it up after a restart of the tool
// runner as it would have without one; the history says when it is asked again, so nobody takes a pause for a
// loss; and after its last try the history says so and the operator is told once. It is never an approval.
func TestAReviewOwedOutlivesARestartAndItsLastTryIsSaid(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
inZone(t)
w := imported(t, s)
n := proposed(t, w, "author@g14", false)
key := Key{KindDecision, n}
lastWhy := func() string {
vs, err := s.Versions(context.Background(), key)
if err != nil || len(vs) == 0 {
t.Fatalf("versions of %s: %v", key, err)
}
return vs[len(vs)-1].Why
}
var due time.Time
for try := 1; try <= MostReviewTries; try++ {
if try > 1 {
// The tool runner restarts: a new knowledge base over the same store, nothing of the old one's memory.
w = newWorld(t, s, fixture())
w.clock = due.Add(-time.Second)
if said := w.reviewAll(t); len(said) != 0 {
t.Fatalf("try %d asked before it was due: %v", try, said)
}
w.clock = due
}
w.judge.reviewFail = true
if said := w.reviewAll(t); len(said) != 1 || !strings.Contains(said[0], "not reviewed") {
t.Fatalf("try %d after a restart was not taken up: %v", try, said)
}
e := mustGet(t, s, key)
if e.Current.Status != StatusProposed || e.Current.Judging == nil || e.Current.Judging.State != JudgingFailed ||
e.Current.Judging.Tries != try || len(e.Current.Reviews) != 0 {
t.Fatalf("try %d is not counted in the store: %s, %+v", try, e.Current.Status, e.Current.Judging)
}
why := lastWhy()
if !strings.Contains(why, fmt.Sprintf("try %d of %d", try, MostReviewTries)) {
t.Fatalf("try %d: the history does not count it: %q", try, why)
}
if try < MostReviewTries {
due = w.clock.Add(time.Duration(try) * 10 * time.Minute)
when := "asked again at " + due.Local().Format("2006-01-02 15:04 MST")
if utc := due.UTC().Format("15:04"); strings.Contains(when, utc) || !strings.Contains(when, "CEST") {
t.Fatalf("the time is not said locally: %q (UTC %s)", when, utc)
}
if !strings.Contains(why, when) {
t.Fatalf("try %d: the history does not say when it is asked again (%q): %q", try, when, why)
}
if g, _ := w.k.Get(GetAsk{Number: n}); !strings.Contains(g.Review, when) {
t.Fatalf("try %d: get does not say when it is asked again (%q): %q", try, when, g.Review)
}
if len(w.told) != 0 {
t.Fatalf("try %d: the operator was told before the last try: %v", try, w.told)
}
}
}
// After its last try: the history says it was the last, and the operator is told, once.
if why := lastWhy(); !strings.Contains(why, "the last") || !strings.Contains(why, "the operator is told") {
t.Fatalf("the last try is not said in the history: %q", why)
}
if len(w.told) != 1 || !strings.Contains(w.told[0], key.String()) || !strings.Contains(w.told[0], "never reviewed") {
t.Fatalf("the operator is not told once of the last try: %v", w.told)
}
w.clock = w.clock.Add(24 * time.Hour)
if said := w.reviewAll(t); len(said) != 0 || len(w.told) != 1 {
t.Fatalf("after the last try: asked %v, told %v", said, w.told)
}
// Another restart neither asks it again nor tells the operator again.
clock := w.clock
w = newWorld(t, s, fixture())
w.clock = clock
if said := w.reviewAll(t); len(said) != 0 || len(w.told) != 0 {
t.Fatalf("after a restart past the last try: asked %v, told %v", said, w.told)
}
// Never an approval.
if e := mustGet(t, s, key); e.Current.Status != StatusProposed || len(w.said.of(EventAccepted)) != 0 {
t.Fatalf("a review never had approved it: %s", e.Current.Status)
}
})
}
// novox/hq issue 497: the notice of a proposal never reviewed says it stays proposed, so it is cleared once that is
// no longer so; and a later version's run of tries is a notice of its own, told once whatever else is written.
func TestProposedAgainAfterTheLastTryClearsTheNotice(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
n := proposed(t, w, "author@g14", false)
key := Key{KindDecision, n}
runOut := func(from int) {
t.Helper()
w.judge.reviewFail = true
for try := from; try <= MostReviewTries; try++ {
if said := w.reviewAll(t); len(said) != 1 {
t.Fatalf("try %d: %v", try, said)
}
w.clock = w.clock.Add(time.Duration(try) * ReviewPause)
}
}
runOut(1)
e := mustGet(t, s, key)
first := fmt.Sprintf("knowledge.unreviewed.decision-%d.v%d", n, e.CurrentVersion)
if len(w.toldKeys) != 1 || w.toldKeys[0] != first || len(w.cleared) != 0 ||
w.resolved[0] != key.String()+" no longer waits for its review" {
t.Fatalf("after the last try: told %v, cleared %v", w.toldKeys, w.cleared)
}
// Another version written while it waits (as a link change on it writes one) tells nothing again.
r := e.Current.Clone()
if err := s.Save(context.Background(), Version{By: "test", At: w.clock, Why: "a link changed", Record: r}, true, false, e.Revision); err != nil {
t.Fatal(err)
}
if w.reviewAll(t); len(w.toldKeys) != 1 || len(w.cleared) != 0 {
t.Fatalf("a version written while it waits: told %v, cleared %v", w.toldKeys, w.cleared)
}
// Proposed again: the notice is cleared, saying so.
if _, err := w.k.Propose("author@g14", valid(n)); err != nil {
t.Fatal(err)
}
w.clock = w.clock.Add(time.Second)
w.reviewAll(t) // its try 1, which fails, and the clearing
w.clock = w.clock.Add(ReviewPause)
if len(w.cleared) != 1 || w.cleared[0][0] != first || !strings.Contains(w.cleared[0][1], "proposed again") {
t.Fatalf("proposed again: cleared %v", w.cleared)
}
// Its own run of tries out is a notice of its own, under its own version.
runOut(2)
if len(w.toldKeys) != 2 || w.toldKeys[1] == first || len(w.cleared) != 1 {
t.Fatalf("a later version ran out: told %v, cleared %v", w.toldKeys, w.cleared)
}
// Proposed again and accepted at last: that notice is cleared too, and nothing is cleared twice, also after
// a restart.
w.judge.reviewFail = false
w.accepted(t, "author@g14", valid(n))
if len(w.cleared) != 2 || w.cleared[1][0] != w.toldKeys[1] || !strings.Contains(w.cleared[1][1], "proposed again") {
t.Fatalf("accepted: cleared %v", w.cleared)
}
clock := w.clock
w = newWorld(t, s, fixture())
w.clock = clock
if w.reviewAll(t); len(w.cleared) != 0 || len(w.told) != 0 {
t.Fatalf("after a restart: told %v, cleared %v", w.told, w.cleared)
}
})
}
// testZone is the local zone every test runs in, set once by TestMain: not UTC.
var testZone = time.FixedZone("CEST", 2*60*60)
// inZone says the test needs the local zone not to be UTC; TestMain set it before any goroutine started. Setting it
// here, under a running bus server's goroutines, was a data race the race detector failed.
func inZone(t *testing.T) {
t.Helper()
if time.Local != testZone {
t.Fatal("the local zone is not the tests' own: TestMain sets it")
}
}
// novox/hq issue 497: a clear the operator's channel refused keeps its notice as told, and is sent again at the next
// step until it is taken.
func TestAClearThatFailsIsSentAgain(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
n := proposed(t, w, "author@g14", false)
w.judge.reviewFail = true
for try := 1; try <= MostReviewTries; try++ {
w.reviewAll(t)
w.clock = w.clock.Add(time.Duration(try) * ReviewPause)
}
if len(w.toldKeys) != 1 {
t.Fatalf("after the last try: told %v", w.toldKeys)
}
prefix := fmt.Sprintf("knowledge.unreviewed.decision-%d.", n)
refused := 0
w.k.Clear = func(key, resolved string) error { refused++; return errors.New("the channel is down") }
if _, err := w.k.Propose("author@g14", valid(n)); err != nil {
t.Fatal(err)
}
w.reviewAll(t)
if told, err := s.ToldOf(context.Background(), prefix); err != nil || len(told) != 1 || refused == 0 {
t.Fatalf("a refused clear: told %v (%v), refused %d", told, err, refused)
}
w.k.Clear = func(key, resolved string) error { w.cleared = append(w.cleared, [2]string{key, resolved}); return nil }
w.clock = w.clock.Add(time.Second)
w.reviewAll(t)
if told, err := s.ToldOf(context.Background(), prefix); err != nil || len(told) != 0 || len(w.cleared) != 1 ||
w.cleared[0][0] != w.toldKeys[0] {
t.Fatalf("the next step: told %v (%v), cleared %v", told, err, w.cleared)
}
})
}
@@ -546,6 +546,36 @@ func JudgeSet(records []*Record) []Finding {
return out
}
// UncitedOnAcceptance is the whole-set rule `cited` judged ahead, on a decision's working version as it would
// stand once accepted, over the records as JudgeSet reads them: "" when it names a home or a record accepted (or
// superseded) links it, else what the rule would say. hq held this rule at the merge of the record's own pull
// request, which carried the design citing it; the knowledge base holds it in the review, before the judgement, so
// an acceptance never leaves a decision cited by nothing (novox/hq issue 504). It is not refused at propose: no
// record can link a number only reserved, so a record that will cite a new one is proposed after it, accepted, and
// the new one proposed again. A citing record changed by a correction after the acceptance is what the rule, run
// after every acceptance, still says. An imported record was held to hq's checks, and its correction is not judged
// here, as JudgeSet does not judge it.
func UncitedOnAcceptance(r *Record, all []*Entry) string {
if r.Kind != KindDecision || len(r.Homes) > 0 || (r.Pending != nil && r.AcceptedAt.IsZero()) {
return ""
}
k := r.Key()
for _, e := range all {
from := e.Current
if from.Key() == k || !(from.Accepted() || from.Status == StatusSuperseded) {
continue
}
for _, l := range from.Links {
if to, ok := l.To.Record(); ok && to == k {
return ""
}
}
}
return fmt.Sprintf("%s would be accepted with nothing citing it: it names no home and no accepted record links it. Name "+
"its home (homes: the hq design or playbook that cites it), or have the record that links it accepted first; then "+
"propose this one again", k)
}
// RuleFrom is the day each dated rule began to hold a record, as hq's checks hold it: a record dated earlier was
// never held to it, and a dated correction of it (a progressive insight, which keeps the record's date) is not.
// words.py held the retired words from 2026-10-07 and the daemons' lines from 2026-10-09; cycle.py the deferrals
@@ -63,8 +63,12 @@ type Store interface {
// SetFindings replaces the whole-set findings; Findings reads them.
SetFindings(ctx context.Context, f []Finding) error
Findings(ctx context.Context) ([]Finding, error)
// Told records that the operator was told about a topic's clauses, once; false when it was already.
// Told records that the operator was told of something, once, by its key: a topic's clauses drafted, or a
// version that ran out of review tries (novox/hq issue 497); false when it was already.
Told(ctx context.Context, topic string) (bool, error)
// ToldOf is every key told of that starts with prefix; Untold takes one away once what it told of is over.
ToldOf(ctx context.Context, prefix string) ([]string, error)
Untold(ctx context.Context, key string) error
}
// memStore is the Store in memory.
@@ -271,6 +275,26 @@ func (s *memStore) Findings(context.Context) ([]Finding, error) {
return append([]Finding(nil), s.findings...), nil
}
func (s *memStore) ToldOf(_ context.Context, prefix string) ([]string, error) {
s.mu.Lock()
defer s.mu.Unlock()
var out []string
for k := range s.told {
if strings.HasPrefix(k, prefix) {
out = append(out, k)
}
}
sort.Strings(out)
return out, nil
}
func (s *memStore) Untold(_ context.Context, key string) error {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.told, key)
return nil
}
func (s *memStore) Told(_ context.Context, topic string) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -1,38 +1,119 @@
package main
import (
"context"
"errors"
"strings"
"testing"
)
// to-be 55 §10: "A whole-set rule that fails is loud — the module's test: an acceptance that leaves a decision
// cited by nothing raises the condition." The condition is the mesh's, raised when the module's health answers
// unhealthy; the test reads the health answer the node-engine reads.
func TestAnAcceptanceThatLeavesADecisionCitedByNothingMakesTheModuleUnhealthy(t *testing.T) {
// A decision nothing would cite is not accepted: its review asks changes itself, without a judgement, naming the
// rule cited, and the module stays healthy (novox/hq issue 504). Before, the review approved it, the acceptance
// broke the rule, and the module turned unhealthy for a record its author could only repair after the fact.
func TestADecisionNothingWouldCiteIsNotAccepted(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy after the import of real records: %s", h.Why)
}
n := w.reserved(t, "author@g14")
w.accepted(t, "author@g14", valid(n))
h := w.k.Health(nil)
if h.Healthy || h.State != "unhealthy" || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
t.Fatalf("an accepted decision nothing cites left the module %+v", h)
p := valid(n)
p.Homes = nil
if _, err := w.k.Propose("author@g14", p); err != nil {
t.Fatal(err)
}
// The acceptance stands: the condition says it, and undoes nothing.
judged := len(w.judge.reviews)
w.reviewAll(t)
e := mustGet(t, s, Key{KindDecision, n})
if e.Current.Status != StatusProposed {
t.Fatalf("a decision nothing cites was %s by its review", e.Current.Status)
}
rv := e.Current.Reviews
if len(rv) == 0 || rv[len(rv)-1].Verdict != VerdictChanges || !strings.Contains(rv[len(rv)-1].Findings, "the rule cited") ||
rv[len(rv)-1].By != ReviewBy {
t.Fatalf("its review asked no changes for the rule cited: %+v", rv)
}
if len(w.judge.reviews) != judged {
t.Fatal("a judgement was spent on a record the rule cited refuses")
}
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy for a record not accepted: %s", h.Why)
}
// Named with its home, it is accepted, and the module stays healthy.
p.Homes = []string{testHome}
w.accepted(t, "author@g14", p)
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy after a decision with its home was accepted: %s", h.Why)
}
})
}
// A new decision is cited by a record proposed after it: no record can link a number only reserved. Once the
// citing record is accepted, the new one proposed again is accepted. A correction of the citing record that drops
// the link still leaves the decision cited by nothing, and that is loud: told to the operator once, as a warning
// naming the rule and the record, while the module stays healthy and its rollouts are not held (the operator's
// decision of 2026-10-12 on novox/hq issue 504; to-be 55 §10). A record that cites it clears the notice.
func TestARecordAcceptedCitingANewDecisionLetsItBeAcceptedAndDroppingTheLinkIsLoud(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
n := w.reserved(t, "author@g14")
a := valid(n)
a.Homes = nil
if _, err := w.k.Propose("author@g14", a); err != nil {
t.Fatal(err)
}
w.reviewAll(t)
m := w.reserved(t, "author@g14")
b := valid(m)
b.Links = append(b.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", b)
w.accepted(t, "author@g14", a) // its revision: an accepted record links it now
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy with both accepted, one citing the other: %s", h.Why)
}
w.k.TellFindings()
if len(w.told) != 0 {
t.Fatalf("told with every rule holding: %v", w.told)
}
// A correction of the citing record drops its link: the rule, run after the acceptance, says so.
b.Links = b.Links[:len(b.Links)-1]
b.Insight = "it cited " + itoaKey(n)
w.accepted(t, "author@g14", b)
h := w.k.Health(nil)
if !h.Healthy || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
t.Fatalf("a decision left cited by nothing: the health should stay healthy and name it, not %+v", h)
}
w.k.TellFindings()
w.k.TellFindings()
if len(w.told) != 1 || !strings.Contains(w.told[0], "rule cited") || !strings.Contains(w.told[0], itoaKey(n)) ||
!strings.HasPrefix(w.toldKeys[0], findingPrefix) {
t.Fatalf("the operator is not told once, naming the rule and the record: %v %v", w.told, w.toldKeys)
}
// A restart tells nothing again: what was told is kept in the store.
told := w.toldKeys[0]
restarted := newWorld(t, s, fixture())
restarted.clock = w.clock
restarted.k.TellFindings()
if len(restarted.told) != 0 {
t.Fatalf("told again after a restart: %v", restarted.told)
}
w.told = nil
if mustGet(t, s, Key{KindDecision, n}).Current.Status != StatusAccepted {
t.Fatal("the acceptance was undone")
}
// A record that cites it heals it.
m := w.reserved(t, "author@g14")
p := valid(m)
p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", p)
h = w.k.Health(nil)
if strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
k := w.reserved(t, "author@g14")
c := valid(k)
c.Links = append(c.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", c)
if h := w.k.Health(nil); strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
t.Fatalf("still said after a record cited it: %s", h.Why)
}
w.k.TellFindings()
if len(w.cleared) != 1 || w.cleared[0][0] != told {
t.Fatalf("the notice is not cleared once the rule holds again: %v", w.cleared)
}
w.k.TellFindings()
if len(w.cleared) != 1 || len(w.told) != 0 {
t.Fatalf("after the clear: cleared %v, told %v", w.cleared, w.told)
}
})
}
@@ -83,3 +164,29 @@ func TestEachWholeSetRuleFails(t *testing.T) {
t.Errorf("a consistent set fails: %+v", f)
}
}
// A finding's notice the operator's channel refused is not kept as told: it is sent again at the next step.
func TestAFindingsNoticeThatFailsIsSentAgain(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := newWorld(t, s, fixture())
f := Finding{Rule: "cited", Records: []string{"ADR 0300"}, Says: "ADR 0300 is accepted and nothing cites it"}
if err := s.SetFindings(context.Background(), []Finding{f}); err != nil {
t.Fatal(err)
}
tries := 0
w.k.Notify = func(key, summary, headline, explanation, resolved string) error {
tries++
if tries == 1 {
return errors.New("nothing answers operator-channel.notify")
}
w.told = append(w.told, summary)
return nil
}
w.k.TellFindings()
w.k.TellFindings()
w.k.TellFindings()
if tries != 2 || len(w.told) != 1 {
t.Fatalf("a refused notice: %d tries, told %v", tries, w.told)
}
})
}
+1
View File
@@ -71,6 +71,7 @@
"invokes": [
"seat:issue-tracker.reserve",
"claude-code.claude_code_judge",
"claude-code.claude_code_judged",
"records.records_status",
"records.records_list",
"records.records_read",
+52
View File
@@ -0,0 +1,52 @@
# mesh-reads
A node's files and accounts, read through the mesh (novox/hq ADR 0310, extending ADR 0268 §5; hq to-be 41
WP8; issue 418). It holds the node seat `node-reads` on every node, the control-node included, so reading a
file, searching one, listing a directory, the owner and mode of each component of a path and looking up an
account or a group never go through the login shell's `execute`, and an approval is asked for an act, never
for a read.
## Who it reads as
The account `mesh-reads`, which this module makes on every node (`runs-as`, ADR 0259 §8): no login shell, a
home of its own under `/var/lib`, and no group beside its own. The tool runner serves it in a runtime of its
own, on its own bus credential, never the machine's runtime, which runs as the operator's account. So the
node's file modes bound what it sees: a file only the operator, root or a module's own account may read is
refused by its mode, and the answer says so. It refuses every read when it finds itself running as root.
## What it refuses by name
Before any read, whatever the file modes would allow (ADR 0310 §5):
- **The places the mesh keeps a secret on this node**, which the controller composes from the manifests
and gives it as `MESH_SECRET_PLACES`: the directory of every module's own secret (its bus credential
among them), every credential file and grant directory, the vault's directories, and where the
operator-sealed secrets are kept. Without that list the module does not start.
- **The known credential files of a machine's programs** (`refuse.go`): an account's `.ssh`, `.gnupg`,
`.aws`, `.kube`; the agent's `.credentials.json`, `.netrc`, `.pgpass`, `.git-credentials`; private key
names (`id_ed25519`, `*.key`, `ssh_host_*_key`); `/etc/shadow`; `/etc/ssl/private`, `/etc/wireguard`
and the network manager's connections.
A link is judged by where it leads as well as by its name. `stat` and `readlink` say a secret place's owner
and mode, and mark it, because those are not its content; `list`, `read` and `search` refuse it.
## The content rule
Every answer passes the content rule's secret class, from `mesh-sdk`'s `outward` package (ADR 0310 §4): a
line of a file or a matching line shaped like a secret is withheld and its number said, as is a link target
or an account's words. A path and an entry's name are not judged: they are what the caller named or what the
file system calls a thing, and the rule would take a temporary directory's or a container layer's name for a
token.
## Its verbs and their bounds
| verb | replaces | bounds |
|---|---|---|
| `stat` | `stat`, `ls -ld`, `namei -l`, `findmnt -T` | 200 paths, 64 components each; links followed only with `follow` |
| `readlink` | `readlink`, `readlink -f`, `realpath` | one path |
| `list` | `ls`, `ls -la` | 2000 entries, cut and said |
| `read` | `cat`, `head`, `tail`, `less` | 64 KiB, or up to 256 KiB when asked; cut at a line and said; text only |
| `search` | `grep`, `grep -r`, `grep -F` | a fixed string; 100 matches (500 asked), 4 deep (8 asked), 5000 files, 4 MiB a file, 20 seconds; the bound met is said |
| `account` | `getent passwd`, `getent group`, `id`, `groups` | 2000 accounts and groups; never a password field |
No answer is larger than 1 MiB. Nothing here writes a file.
+204
View File
@@ -0,0 +1,204 @@
// mesh-reads (novox/hq ADR 0310, to-be 41 WP8): the holder of node-reads. A node's files and accounts read
// through the mesh — the owner and mode of each component of a path, a link, a directory, a text file, a search
// by fixed string, an account or a group — so these reads never go through the login shell's execute.
//
// It runs as an account of its own, mesh-reads, served by the tool runner in a runtime of its own (ADR 0259
// §8): in no group that holds a secret, with no login shell and no home of the operator's, so the node's file
// modes bound what it sees. On top, it refuses by name the places a secret is kept (refuse.go) and withholds
// any piece of an answer shaped like a secret (mesh-sdk's outward). Read-only: nothing here writes a file.
//
// stdout is the MCP channel; what this module says, it says on stderr.
package main
import (
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Seat is the role this module holds.
const Seat = "node-reads"
// PlacesWord is where the controller names this node's secret places (novox/hq ADR 0310 §5).
const PlacesWord = "MESH_SECRET_PLACES"
func main() {
places, err := placesFrom(os.Getenv(PlacesWord))
if err != nil {
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
os.Exit(1)
}
if err := stdio.Serve("", tools(newReader(places))); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
os.Exit(1)
}
}
// placesFrom reads the controller's list. Absent is refused: without it, a module's own-secret directory
// would be read wherever the account's modes allow, and a refusal the decision promises would be silent.
func placesFrom(raw string) ([]string, error) {
if strings.TrimSpace(raw) == "" {
return nil, fmt.Errorf("%s is not given, so this node's secret places are unknown and nothing would be "+
"refused by them (novox/hq ADR 0310 §5): a controller that knows node-reads composes it", PlacesWord)
}
var places []string
if err := json.Unmarshal([]byte(raw), &places); err != nil {
return nil, fmt.Errorf("%s is not a list of paths: %v", PlacesWord, err)
}
return places, nil
}
func str(description string) map[string]any {
return map[string]any{"type": "string", "description": description}
}
func arg(a map[string]any, k string) string {
switch v := a[k].(type) {
case string:
return strings.TrimSpace(v)
case float64:
return strconv.FormatFloat(v, 'f', -1, 64)
case bool:
return strconv.FormatBool(v)
}
return ""
}
func number(a map[string]any, k string) (int, error) {
s := arg(a, k)
if s == "" {
return 0, nil
}
n, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("%s is a whole number, and %q was given", k, s)
}
return n, nil
}
// pathsArg is the paths given, one per line — or, from a caller that sends a list, the list.
func pathsArg(a map[string]any) []string {
var raw []string
switch v := a["paths"].(type) {
case string:
raw = strings.Split(v, "\n")
case []any:
for _, p := range v {
if s, ok := p.(string); ok {
raw = append(raw, s)
}
}
}
var out []string
for _, p := range raw {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
// bounded is the last bound (novox/hq ADR 0310): whatever a verb made, no answer is larger than MaxAnswer.
func bounded(answer any, err error) (any, error) {
if err != nil {
return nil, err
}
body, err := json.Marshal(answer)
if err != nil {
return nil, err
}
if len(body) > MaxAnswer {
return nil, fmt.Errorf("the answer is %d bytes, and an answer of node-reads is at most %d: ask for less", len(body), MaxAnswer)
}
return answer, nil
}
// verb is one of the seat's verbs: listed as `<seat>.<verb>`, so the runtime serves it on the seat's subject.
// The descriptions are the controller's (internal/catalogue/reads.go), said again here for a caller of the
// module's own address.
func verb(name, description string, input map[string]any, run func(a map[string]any) (any, error)) stdio.Tool {
return stdio.Tool{Name: Seat + "." + name, Description: description, Input: input,
Run: func(a map[string]any) (any, error) { return bounded(run(a)) }}
}
func object(props map[string]any, required ...string) map[string]any {
if required == nil {
required = []string{}
}
return map[string]any{"type": "object", "properties": props, "required": required}
}
func tools(r Reader) []stdio.Tool {
return []stdio.Tool{
verb("stat", "Each component of one path or several, from / down: its type, owner and group (name and "+
"number), mode, whether it is a link and to what, and whether it is a mount point and of which file "+
"system. A link is not followed unless follow is \"true\". Read as the account mesh-reads, which holds no "+
"secret: a component it may not reach says so. At most 200 paths in one call.",
object(map[string]any{"paths": str("one absolute path, or several separated by line breaks"),
"follow": str("\"true\": also say where the path leads once every link is followed")}, "paths"),
func(a map[string]any) (any, error) { return r.Stat(pathsArg(a), arg(a, "follow") == "true") }),
verb("readlink", "What one link points to, as written, and where the path leads once every link is "+
"followed, and whether that exists.",
object(map[string]any{"path": str("an absolute path")}, "path"),
func(a map[string]any) (any, error) { return r.Readlink(arg(a, "path")) }),
verb("list", "Every entry of one directory with its type, owner, group, mode, size and, for a link, its "+
"target. At most 2000 entries; a longer directory is cut and says so. A place a secret is kept is "+
"refused by name.",
object(map[string]any{"path": str("an absolute path to a directory")}, "path"),
func(a map[string]any) (any, error) { return r.List(arg(a, "path")) }),
verb("read", "One text file, or a part of it by line: at most 256 KiB in one answer (64 KiB unless more "+
"is asked), cut at a line and said so. A negative from counts from the end, as tail does. A binary file "+
"is refused; a place a secret is kept, or a known credential file, is refused by name; a line shaped "+
"like a secret is withheld and its number said. /proc and /sys are read as far as mesh-reads may.",
object(map[string]any{"path": str("an absolute path to a file"),
"from": str("the first line to answer, counted from 1; negative: that many lines from the end (optional)"),
"lines": str("how many lines to answer (optional)"),
"bytes": str("the most bytes to answer, up to 262144 (optional, 65536 unless given)")}, "path"),
func(a map[string]any) (any, error) {
var ask ReadAsk
var err error
if ask.From, err = number(a, "from"); err != nil {
return nil, err
}
if ask.Lines, err = number(a, "lines"); err != nil {
return nil, err
}
if ask.Bytes, err = number(a, "bytes"); err != nil {
return nil, err
}
return r.Read(arg(a, "path"), ask)
}),
verb("search", "Find a fixed string (never a pattern) in one file or under one directory: every matching "+
"line with its file and number. Bounded: at most 500 matches (100 unless more is asked), 8 directories "+
"deep (4 unless more is asked), 5000 files, 4 MiB per file and 20 seconds; links are not followed and "+
"binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like "+
"a secret is withheld. The answer says which bound it met.",
object(map[string]any{"text": str("the fixed string to find"),
"path": str("an absolute path to a file or a directory"),
"depth": str("how many directories deep, up to 8 (optional)"),
"matches": str("the most matches to answer, up to 500 (optional)"),
"case": str("\"false\": match whatever the case (optional)")}, "text", "path"),
func(a map[string]any) (any, error) {
text, _ := a["text"].(string) // as given: a space at either end is part of what is found
ask := SearchAsk{Text: text, Path: arg(a, "path"), CaseInsensitive: arg(a, "case") == "false"}
var err error
if ask.Depth, err = number(a, "depth"); err != nil {
return nil, err
}
if ask.Matches, err = number(a, "matches"); err != nil {
return nil, err
}
return r.Search(ask)
}),
verb("account", "One account or group by name or number, as this node's account files say: an account's "+
"number, group, home, shell and every group it is in; a group's number and members. With neither, every "+
"account and group. Never a password field.",
object(map[string]any{"name": str("an account's name or number (optional)"),
"group": str("a group's name or number (optional)")}),
func(a map[string]any) (any, error) { return r.Account(arg(a, "name"), arg(a, "group")) }),
}
}
@@ -0,0 +1,59 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
)
// ADR 0310 §3: the account the reads run as is the module's own, made by the mesh, with no login shell, no
// home of the operator's and no group beside its own, and the module runs as it.
func TestTheAccountHoldsNoSecretsGroupAndNoLogin(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
RunsAs string `json:"runs-as"`
SecretsOwner string `json:"secrets-owner"`
Claims []map[string]any `json:"claims"`
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if m.RunsAs != "mesh-reads" || m.SecretsOwner != "mesh-reads" {
t.Errorf("runs as %q, secrets owned by %q", m.RunsAs, m.SecretsOwner)
}
if len(m.Claims) != 1 || m.Claims[0]["name"] != Seat {
t.Errorf("claims %v", m.Claims)
}
found := false
for _, r := range m.Resources {
if r["type"] != "user" {
continue
}
found = true
if r["name"] != m.RunsAs {
t.Errorf("a user other than its own: %v", r)
}
if r["shell"] != "/usr/bin/nologin" {
t.Errorf("the account has a login shell: %v", r["shell"])
}
if home, _ := r["home"].(string); !strings.HasPrefix(home, "/var/lib/") {
t.Errorf("the account's home is %q, not one of the mesh's", home)
}
if _, grouped := r["groups"]; grouped {
t.Errorf("the account is put in groups: %v", r["groups"])
}
}
if !found {
t.Error("the module makes no account of its own")
}
for _, r := range m.Resources {
if r["type"] != "user" && r["type"] != "directory" {
t.Errorf("a read-only holder declares a %v: %v", r["type"], r)
}
}
}
+926
View File
@@ -0,0 +1,926 @@
package main
import (
"bufio"
"bytes"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"syscall"
"time"
)
// The bounds (novox/hq ADR 0310, Consequences: set by the build and stated in each verb's description). Every
// answer stays under MaxAnswer, which the tools check once more after an answer is made.
const (
MaxAnswer = 1 << 20 // no answer of this module is larger, whatever the bounds below allow
MaxPaths = 200 // paths one stat answers
MaxComponents = 64 // components of one path
MaxEntries = 2000 // entries one list answers
DefaultReadSize = 64 << 10 // bytes a read answers unless more is asked
MaxReadSize = 256 << 10 // bytes a read answers at most
MaxScan = 64 << 20 // bytes a read passes over to reach the lines asked
DefaultMatches = 100
MaxMatches = 500
DefaultDepth = 4
MaxDepth = 8
MaxFiles = 5000 // files one search opens
MaxFileScan = 4 << 20 // bytes one search reads of one file
MaxLine = 300 // characters of one matching line answered
SearchTime = 20 * time.Second
MaxAccounts = 2000
)
// Reader reads one node's files and accounts as the account it runs as.
type Reader struct {
Refuser
// Passwd, Group and MountInfo are the files accounts, groups and mount points are read from.
Passwd, Group, MountInfo string
// Euid is the account this runs as; never root (ADR 0310 §3).
Euid func() int
// Now is the clock a search is bounded by.
Now func() time.Time
}
func newReader(places []string) Reader {
return Reader{Refuser: Refuser{Places: places}, Passwd: "/etc/passwd", Group: "/etc/group",
MountInfo: "/proc/self/mountinfo", Euid: os.Geteuid, Now: time.Now}
}
// asItsOwn refuses every read when this runs as root: what it may see is bounded by its account's file
// modes, and root's are none (novox/hq ADR 0310 §3). Loud, never a silent widening.
func (r Reader) asItsOwn() error {
if r.Euid() == 0 {
return errors.New("mesh-reads runs as root here, and its reads are bounded by an account of its own that " +
"holds no secret (novox/hq ADR 0310 §3): it answers nothing until it runs as mesh-reads")
}
return nil
}
func absolute(path string) (string, error) {
path = strings.TrimSpace(path)
if !strings.HasPrefix(path, "/") {
return "", fmt.Errorf("%q is not an absolute path: name it from /", path)
}
return filepath.Clean(path), nil
}
// why turns a file error into the words an answer says: what stopped the read, never a Go error's text.
func why(err error) string {
switch {
case errors.Is(err, fs.ErrNotExist):
return "it does not exist"
case errors.Is(err, fs.ErrPermission):
return "the account mesh-reads may not reach it (its file modes refuse it)"
case errors.Is(err, syscall.ELOOP):
return "its links loop"
case errors.Is(err, syscall.ENOTDIR):
return "a component of it is not a directory"
}
var pe *fs.PathError
if errors.As(err, &pe) {
return pe.Err.Error()
}
return err.Error()
}
// ---- stat ----
// Component is one component of a path as lstat sees it.
type Component struct {
Path string `json:"path"`
Type string `json:"type,omitempty"`
Owner string `json:"owner,omitempty"`
UID *int `json:"uid,omitempty"`
Group string `json:"group,omitempty"`
GID *int `json:"gid,omitempty"`
Mode string `json:"mode,omitempty"` // octal, with the setuid, setgid and sticky bits
Perms string `json:"perms,omitempty"` // as ls shows them
Size *int64 `json:"size,omitempty"`
Link string `json:"link,omitempty"` // what a link points to, as written
Mount *Mount `json:"mount,omitempty"`
Secret string `json:"secret-place,omitempty"` // why a read under it is refused
Unreachd string `json:"unreachable,omitempty"`
}
// Mount is a mount point: its file system and source.
type Mount struct {
FS string `json:"fs"`
Source string `json:"source"`
}
// PathStat is one path's components, from / down, and where it leads once every link is followed.
type PathStat struct {
Path string `json:"path"`
Components []Component `json:"components"`
LeadsTo string `json:"leads-to,omitempty"`
Leads *Component `json:"there,omitempty"`
Stopped string `json:"stopped,omitempty"`
}
// StatAnswer is what stat answers.
type StatAnswer struct {
Paths []PathStat `json:"paths"`
Withheld []Withheld `json:"withheld,omitempty"`
}
func (r Reader) Stat(paths []string, follow bool) (StatAnswer, error) {
if err := r.asItsOwn(); err != nil {
return StatAnswer{}, err
}
if len(paths) == 0 {
return StatAnswer{}, errors.New("name a path to stat")
}
if len(paths) > MaxPaths {
return StatAnswer{}, fmt.Errorf("%d paths asked, and one stat answers at most %d: ask in parts", len(paths), MaxPaths)
}
names := r.names()
mounts := r.mounts()
var out StatAnswer
for _, raw := range paths {
path, err := absolute(raw)
if err != nil {
return StatAnswer{}, err
}
ps := PathStat{Path: path}
walk := []string{"/"}
acc := ""
for _, part := range strings.Split(strings.TrimPrefix(path, "/"), "/") {
if part == "" {
continue
}
acc += "/" + part
walk = append(walk, acc)
}
if len(walk) > MaxComponents {
return StatAnswer{}, fmt.Errorf("%s has %d components, and a stat answers at most %d", path, len(walk), MaxComponents)
}
for _, p := range walk {
c := r.component(p, names, mounts)
ps.Components = append(ps.Components, c)
if c.Unreachd != "" {
ps.Stopped = c.Path + ": " + c.Unreachd
break
}
}
if follow && ps.Stopped == "" {
if leads, err := filepath.EvalSymlinks(path); err != nil {
ps.Stopped = "following its links: " + why(err)
} else {
ps.LeadsTo = leads
if leads != path {
c := r.component(leads, names, mounts)
ps.Leads = &c
}
}
}
out.Paths = append(out.Paths, ps)
}
out.Withheld = cleanStat(&out)
return out, nil
}
func (r Reader) component(p string, names accounts, mounts map[string]Mount) Component {
c := Component{Path: p}
info, err := os.Lstat(p)
if err != nil {
c.Unreachd = why(err)
return c
}
c.Type = typeOf(info.Mode())
c.Perms = info.Mode().String()
if st, ok := info.Sys().(*syscall.Stat_t); ok {
uid, gid := int(st.Uid), int(st.Gid)
c.UID, c.GID = &uid, &gid
c.Owner, c.Group = names.user(uid), names.group(gid)
c.Mode = fmt.Sprintf("%04o", st.Mode&07777)
}
if info.Mode().IsRegular() {
size := info.Size()
c.Size = &size
}
if info.Mode()&fs.ModeSymlink != 0 {
if target, err := os.Readlink(p); err == nil {
c.Link = target
}
}
if m, ok := mounts[p]; ok {
c.Mount = &m
}
if why := r.Refused(p); why != "" {
c.Secret = why
}
return c
}
func typeOf(m fs.FileMode) string {
switch {
case m.IsDir():
return "directory"
case m&fs.ModeSymlink != 0:
return "link"
case m.IsRegular():
return "file"
case m&fs.ModeNamedPipe != 0:
return "fifo"
case m&fs.ModeSocket != 0:
return "socket"
case m&fs.ModeCharDevice != 0:
return "character device"
case m&fs.ModeDevice != 0:
return "block device"
}
return "other"
}
func cleanStat(a *StatAnswer) []Withheld {
var w []Withheld
cleanC := func(c *Component) {
for _, f := range []*string{&c.Link} {
if s, what := clean(*f); what != "" {
w = append(w, Withheld{At: "a link's target", Class: what})
*f = s
}
}
if c.Mount != nil {
if s, what := clean(c.Mount.Source); what != "" {
w = append(w, Withheld{At: "a mount's source", Class: what})
c.Mount.Source = s
}
}
}
for i := range a.Paths {
p := &a.Paths[i]
for j := range p.Components {
cleanC(&p.Components[j])
}
if p.Leads != nil {
cleanC(p.Leads)
}
}
return w
}
// ---- readlink ----
// LinkAnswer is what readlink answers.
type LinkAnswer struct {
Path string `json:"path"`
Link string `json:"link,omitempty"` // as written; empty when the path is not a link
IsLink bool `json:"is-link"`
LeadsTo string `json:"leads-to,omitempty"`
Exists bool `json:"exists"`
Stopped string `json:"stopped,omitempty"`
Withheld []Withheld `json:"withheld,omitempty"`
}
func (r Reader) Readlink(raw string) (LinkAnswer, error) {
if err := r.asItsOwn(); err != nil {
return LinkAnswer{}, err
}
path, err := absolute(raw)
if err != nil {
return LinkAnswer{}, err
}
a := LinkAnswer{Path: path}
info, err := os.Lstat(path)
if err != nil {
a.Stopped = why(err)
return a, nil
}
if info.Mode()&fs.ModeSymlink != 0 {
a.IsLink = true
if a.Link, err = os.Readlink(path); err != nil {
a.Stopped = why(err)
}
}
if leads, err := filepath.EvalSymlinks(path); err == nil {
a.LeadsTo, a.Exists = leads, true
} else if a.Stopped == "" {
a.Stopped = "following its links: " + why(err)
}
for _, f := range []*string{&a.Link} {
if s, what := clean(*f); what != "" {
a.Withheld = append(a.Withheld, Withheld{At: "a link's target", Class: what})
*f = s
}
}
return a, nil
}
// ---- list ----
// Entry is one entry of a directory.
type Entry struct {
Name string `json:"name"`
Type string `json:"type"`
Owner string `json:"owner,omitempty"`
Group string `json:"group,omitempty"`
Mode string `json:"mode,omitempty"`
Size *int64 `json:"size,omitempty"`
Link string `json:"link,omitempty"`
}
// ListAnswer is what list answers.
type ListAnswer struct {
Path string `json:"path"`
Entries []Entry `json:"entries"`
Count int `json:"count"`
Cut string `json:"cut,omitempty"`
Withheld []Withheld `json:"withheld,omitempty"`
}
// refusedRead is a refusal by name, before any read: the path asked and where it leads are both judged, so a
// link to a secret place is refused as the place is.
func (r Reader) refusedRead(path string) error {
if why := r.Refused(path); why != "" {
return fmt.Errorf("%s is refused by name: %s (novox/hq ADR 0310 §5)", path, why)
}
if leads, err := filepath.EvalSymlinks(path); err == nil && leads != path {
if why := r.Refused(leads); why != "" {
return fmt.Errorf("%s leads to %s, which is refused by name: %s (novox/hq ADR 0310 §5)", path, leads, why)
}
}
return nil
}
func (r Reader) List(raw string) (ListAnswer, error) {
if err := r.asItsOwn(); err != nil {
return ListAnswer{}, err
}
path, err := absolute(raw)
if err != nil {
return ListAnswer{}, err
}
if err := r.refusedRead(path); err != nil {
return ListAnswer{}, err
}
entries, err := os.ReadDir(path)
if err != nil {
return ListAnswer{}, fmt.Errorf("%s cannot be listed: %s", path, why(err))
}
names := r.names()
a := ListAnswer{Path: path, Count: len(entries), Entries: []Entry{}}
if len(entries) > MaxEntries {
a.Cut = fmt.Sprintf("cut at %d of %d entries (the bound of one list), in name order", MaxEntries, len(entries))
entries = entries[:MaxEntries]
}
for _, e := range entries {
en := Entry{Name: e.Name(), Type: typeOf(e.Type())}
if info, err := e.Info(); err == nil {
en.Type = typeOf(info.Mode())
if st, ok := info.Sys().(*syscall.Stat_t); ok {
en.Owner, en.Group = names.user(int(st.Uid)), names.group(int(st.Gid))
en.Mode = fmt.Sprintf("%04o", st.Mode&07777)
}
if info.Mode().IsRegular() {
size := info.Size()
en.Size = &size
}
if info.Mode()&fs.ModeSymlink != 0 {
en.Link, _ = os.Readlink(filepath.Join(path, e.Name()))
}
}
for _, f := range []*string{&en.Link} {
if s, what := clean(*f); what != "" {
a.Withheld = append(a.Withheld, Withheld{At: "a link's target", Class: what})
*f = s
}
}
a.Entries = append(a.Entries, en)
}
return a, nil
}
// ---- read ----
// ReadAnswer is what read answers.
type ReadAnswer struct {
Path string `json:"path"`
Size int64 `json:"size"` // as the file says; 0 for most of /proc and /sys
From int `json:"from"` // the first line answered, counted from 1
Lines int `json:"lines"`
Text string `json:"text"`
Cut string `json:"cut,omitempty"`
Withheld []int `json:"withheld-lines,omitempty"`
Why string `json:"withheld-because,omitempty"`
}
// ReadAsk is what a read asks: the lines from From (1 is the first; a negative From counts from the end, as
// tail does), at most Lines of them (0: as many as fit), and at most Bytes bytes.
type ReadAsk struct {
From, Lines, Bytes int
}
func (r Reader) Read(raw string, ask ReadAsk) (ReadAnswer, error) {
if err := r.asItsOwn(); err != nil {
return ReadAnswer{}, err
}
path, err := absolute(raw)
if err != nil {
return ReadAnswer{}, err
}
if err := r.refusedRead(path); err != nil {
return ReadAnswer{}, err
}
bound := ask.Bytes
switch {
case bound <= 0:
bound = DefaultReadSize
case bound > MaxReadSize:
return ReadAnswer{}, fmt.Errorf("%d bytes asked, and one read answers at most %d", bound, MaxReadSize)
}
if ask.Lines < 0 {
return ReadAnswer{}, errors.New("lines is a count: name how many, or none for as many as fit")
}
// Non-blocking, so a fifo answers rather than waits; and judged by what was opened, not by its name.
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
if !info.Mode().IsRegular() {
return ReadAnswer{}, fmt.Errorf("%s is a %s, not a file: list a directory, stat anything else", path, typeOf(info.Mode()))
}
a := ReadAnswer{Path: path, Size: info.Size()}
lines, cut, err := pick(io.LimitReader(f, MaxScan), ask, bound)
if err != nil {
return ReadAnswer{}, fmt.Errorf("%s: %v", path, err)
}
a.Cut = cut
if len(lines) > 0 {
a.From = lines[0].n
}
var text strings.Builder
for _, l := range lines {
s, what := clean(l.text)
if what != "" {
a.Withheld = append(a.Withheld, l.n)
a.Why = "a line shaped like a secret is withheld by the content rule (novox/hq ADR 0310 §4)"
}
text.WriteString(s)
if l.newline {
text.WriteByte('\n')
}
}
a.Text, a.Lines = text.String(), len(lines)
return a, nil
}
type line struct {
n int
text string
newline bool
size int
}
// pick reads lines and keeps the ones asked, within bound bytes. A NUL byte is a binary file, refused.
func pick(in io.Reader, ask ReadAsk, bound int) ([]line, string, error) {
br := bufio.NewReaderSize(in, 64<<10)
var kept []line
size, n := 0, 0
from := ask.From
if from == 0 {
from = 1
}
for {
raw, err := br.ReadString('\n')
if raw == "" && err != nil {
if err == io.EOF {
break
}
return nil, "", err
}
n++
if strings.IndexByte(raw, 0) >= 0 {
return nil, "", errors.New("it is a binary file, and read answers text")
}
l := line{n: n, text: strings.TrimSuffix(raw, "\n"), newline: strings.HasSuffix(raw, "\n"), size: len(raw)}
if from > 0 {
if n < from {
continue
}
if ask.Lines > 0 && len(kept) == ask.Lines {
break
}
if size+len(raw) > bound {
return kept, fmt.Sprintf("cut at line %d: the answer holds at most %d bytes", n, bound), nil
}
kept = append(kept, l)
size += len(raw)
} else {
// From the end, as tail does: the last -from lines, kept as they pass.
kept = append(kept, l)
size += len(raw)
for len(kept) > -from || size > bound {
size -= kept[0].size
kept = kept[1:]
}
}
if err == io.EOF {
break
}
}
if from > 0 && n < from && n > 0 {
return nil, "", fmt.Errorf("it has %d lines, and line %d was asked", n, from)
}
if from < 0 && ask.Lines > 0 && len(kept) > ask.Lines {
kept = kept[:ask.Lines]
}
return kept, "", nil
}
// ---- search ----
// Match is one matching line.
type Match struct {
File string `json:"file"`
Line int `json:"line"`
Text string `json:"text"`
}
// SearchAnswer is what search answers.
type SearchAnswer struct {
Text string `json:"text"`
Path string `json:"path"`
Matches []Match `json:"matches"`
Files int `json:"files-read"`
Bound string `json:"bound-met,omitempty"`
Skipped []string `json:"refused-by-name,omitempty"`
Unreadable int `json:"unreadable,omitempty"`
Binary int `json:"binary-skipped,omitempty"`
Withheld []Withheld `json:"withheld,omitempty"`
}
// SearchAsk is what a search asks.
type SearchAsk struct {
Text, Path string
Depth, Matches int
CaseInsensitive bool
}
var errBound = errors.New("bound met")
func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
if err := r.asItsOwn(); err != nil {
return SearchAnswer{}, err
}
if ask.Text == "" {
return SearchAnswer{}, errors.New("name the text to find")
}
if strings.ContainsAny(ask.Text, "\n\x00") {
return SearchAnswer{}, errors.New("the text is found within one line: it may not hold a line break")
}
root, err := absolute(ask.Path)
if err != nil {
return SearchAnswer{}, err
}
if err := r.refusedRead(root); err != nil {
return SearchAnswer{}, err
}
depth, matches := ask.Depth, ask.Matches
switch {
case depth == 0:
depth = DefaultDepth
case depth < 0 || depth > MaxDepth:
return SearchAnswer{}, fmt.Errorf("depth %d asked; a search goes at most %d directories deep", depth, MaxDepth)
}
switch {
case matches == 0:
matches = DefaultMatches
case matches < 0 || matches > MaxMatches:
return SearchAnswer{}, fmt.Errorf("%d matches asked; a search answers at most %d", matches, MaxMatches)
}
needle := ask.Text
if ask.CaseInsensitive {
needle = strings.ToLower(needle)
}
a := SearchAnswer{Text: ask.Text, Path: root, Matches: []Match{}}
deadline := r.Now().Add(SearchTime)
base := strings.Count(root, "/")
if root == "/" {
base = 0
}
err = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if r.Now().After(deadline) {
a.Bound = fmt.Sprintf("time: a search runs at most %s", SearchTime)
return errBound
}
if err != nil {
a.Unreadable++
if d != nil && d.IsDir() {
return fs.SkipDir
}
return nil
}
if p != root {
if why := r.Refused(p); why != "" {
a.Skipped = append(a.Skipped, p)
if d.IsDir() {
return fs.SkipDir
}
return nil
}
}
if d.IsDir() {
if p != root && strings.Count(p, "/")-base > depth {
return fs.SkipDir
}
return nil
}
if !d.Type().IsRegular() {
return nil // links are not followed; a device, a fifo or a socket is not text
}
if a.Files >= MaxFiles {
a.Bound = fmt.Sprintf("files: a search reads at most %d files", MaxFiles)
return errBound
}
a.Files++
return r.searchFile(p, needle, ask.CaseInsensitive, matches, &a)
})
if err != nil && !errors.Is(err, errBound) {
return SearchAnswer{}, fmt.Errorf("%s cannot be searched: %s", root, why(err))
}
return a, nil
}
func (r Reader) searchFile(p, needle string, fold bool, bound int, a *SearchAnswer) error {
f, err := os.OpenFile(p, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
a.Unreadable++
return nil
}
defer f.Close()
br := bufio.NewReaderSize(io.LimitReader(f, MaxFileScan), 64<<10)
if head, _ := br.Peek(8 << 10); bytes.IndexByte(head, 0) >= 0 {
a.Binary++
return nil
}
n := 0
for {
raw, err := br.ReadString('\n')
if raw == "" && err != nil {
return nil
}
n++
hay := raw
if fold {
hay = strings.ToLower(hay)
}
if strings.Contains(hay, needle) {
if len(a.Matches) >= bound {
a.Bound = fmt.Sprintf("matches: %d answered, as asked or the default", bound)
return errBound
}
text := strings.TrimRight(raw, "\r\n")
if r := []rune(text); len(r) > MaxLine {
text = string(r[:MaxLine]) + "…"
}
file := p
for _, f := range []*string{&text} {
if s, what := clean(*f); what != "" {
a.Withheld = append(a.Withheld, Withheld{At: "a matching line", Class: what})
*f = s
}
}
a.Matches = append(a.Matches, Match{File: file, Line: n, Text: text})
}
if err != nil {
return nil
}
}
}
// ---- accounts ----
// Account is one account as the account file says it, never its password field.
type Account struct {
Name string `json:"name"`
UID int `json:"uid"`
GID int `json:"gid"`
Group string `json:"group,omitempty"`
Gecos string `json:"gecos,omitempty"`
Home string `json:"home"`
Shell string `json:"shell"`
Groups []string `json:"groups,omitempty"`
}
// GroupEntry is one group.
type GroupEntry struct {
Name string `json:"name"`
GID int `json:"gid"`
Members []string `json:"members"`
}
// AccountAnswer is what account answers.
type AccountAnswer struct {
Accounts []Account `json:"accounts,omitempty"`
Groups []GroupEntry `json:"groups,omitempty"`
Source string `json:"source"`
Cut string `json:"cut,omitempty"`
Withheld []Withheld `json:"withheld,omitempty"`
}
type accounts struct {
users []Account
groups []GroupEntry
}
func (a accounts) user(uid int) string {
for _, u := range a.users {
if u.UID == uid {
return u.Name
}
}
return ""
}
func (a accounts) group(gid int) string {
for _, g := range a.groups {
if g.GID == gid {
return g.Name
}
}
return ""
}
// names reads the account files; a file it cannot read gives no names, and a component says only numbers.
func (r Reader) names() accounts {
var a accounts
if raw, err := os.ReadFile(r.Passwd); err == nil {
for _, l := range strings.Split(string(raw), "\n") {
f := strings.Split(l, ":")
if len(f) < 7 || strings.HasPrefix(l, "#") {
continue
}
uid, e1 := strconv.Atoi(f[2])
gid, e2 := strconv.Atoi(f[3])
if e1 != nil || e2 != nil {
continue
}
a.users = append(a.users, Account{Name: f[0], UID: uid, GID: gid, Gecos: f[4], Home: f[5], Shell: f[6]})
}
}
if raw, err := os.ReadFile(r.Group); err == nil {
for _, l := range strings.Split(string(raw), "\n") {
f := strings.Split(l, ":")
if len(f) < 4 || strings.HasPrefix(l, "#") {
continue
}
gid, err := strconv.Atoi(f[2])
if err != nil {
continue
}
members := []string{}
for _, m := range strings.Split(f[3], ",") {
if m = strings.TrimSpace(m); m != "" {
members = append(members, m)
}
}
a.groups = append(a.groups, GroupEntry{Name: f[0], GID: gid, Members: members})
}
}
return a
}
func (r Reader) Account(name, group string) (AccountAnswer, error) {
if err := r.asItsOwn(); err != nil {
return AccountAnswer{}, err
}
all := r.names()
if len(all.users) == 0 {
return AccountAnswer{}, fmt.Errorf("%s gave no account: it is unreadable or empty", r.Passwd)
}
out := AccountAnswer{Source: r.Passwd + ", " + r.Group + " (an account kept elsewhere, such as a directory " +
"service or systemd's dynamic accounts, is not here)"}
complete := func(u Account) Account {
u.Group = all.group(u.GID)
for _, g := range all.groups {
if g.GID == u.GID {
u.Groups = append(u.Groups, g.Name)
continue
}
for _, m := range g.Members {
if m == u.Name {
u.Groups = append(u.Groups, g.Name)
}
}
}
return u
}
name, group = strings.TrimSpace(name), strings.TrimSpace(group)
switch {
case name != "":
for _, u := range all.users {
if u.Name == name || strconv.Itoa(u.UID) == name {
out.Accounts = append(out.Accounts, complete(u))
}
}
if len(out.Accounts) == 0 {
return AccountAnswer{}, fmt.Errorf("no account %q on this node", name)
}
case group != "":
for _, g := range all.groups {
if g.Name == group || strconv.Itoa(g.GID) == group {
// Its members as getent says them, and every account whose own group it is.
for _, u := range all.users {
if u.GID == g.GID && !contains(g.Members, u.Name) {
g.Members = append(g.Members, u.Name)
}
}
out.Groups = append(out.Groups, g)
}
}
if len(out.Groups) == 0 {
return AccountAnswer{}, fmt.Errorf("no group %q on this node", group)
}
default:
for _, u := range all.users {
out.Accounts = append(out.Accounts, complete(u))
}
out.Groups = all.groups
if len(out.Accounts) > MaxAccounts {
out.Cut = fmt.Sprintf("cut at %d of %d accounts", MaxAccounts, len(out.Accounts))
out.Accounts = out.Accounts[:MaxAccounts]
}
if len(out.Groups) > MaxAccounts {
out.Cut = strings.TrimPrefix(out.Cut+"; ", "; ") + fmt.Sprintf("cut at %d of %d groups", MaxAccounts, len(out.Groups))
out.Groups = out.Groups[:MaxAccounts]
}
}
for i := range out.Accounts {
u := &out.Accounts[i]
for _, f := range []*string{&u.Gecos, &u.Home, &u.Shell} {
if s, what := clean(*f); what != "" {
out.Withheld = append(out.Withheld, Withheld{At: "account " + u.Name, Class: what})
*f = s
}
}
}
sort.SliceStable(out.Accounts, func(i, j int) bool { return out.Accounts[i].UID < out.Accounts[j].UID })
return out, nil
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// ---- mounts ----
// mounts reads the mount points this process sees, by path. /proc/self/mountinfo is readable by every account.
func (r Reader) mounts() map[string]Mount {
out := map[string]Mount{}
raw, err := os.ReadFile(r.MountInfo)
if err != nil {
return out
}
for _, l := range strings.Split(string(raw), "\n") {
f := strings.Fields(l)
dash := -1
for i, x := range f {
if x == "-" {
dash = i
break
}
}
if len(f) < 5 || dash < 0 || dash+2 >= len(f) {
continue
}
out[unescape(f[4])] = Mount{FS: f[dash+1], Source: unescape(f[dash+2])}
}
return out
}
// unescape reads mountinfo's octal escapes (\040 for a space).
func unescape(s string) string {
if !strings.Contains(s, `\`) {
return s
}
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) {
if v, err := strconv.ParseUint(s[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(s[i])
}
return b.String()
}
@@ -0,0 +1,452 @@
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"testing"
"time"
)
// A token shaped as a secret, written into files the tests read.
const aToken = "ghp_abcdefghijklmnopqrstuvwxyz0123456789"
func testReader(t *testing.T, places ...string) (Reader, string) {
t.Helper()
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
group := filepath.Join(dir, "group")
mountinfo := filepath.Join(dir, "mountinfo")
write(t, passwd, "root:x:0:0::/root:/usr/bin/bash\nops:x:1000:1000:The Operator:/home/ops:/usr/bin/zsh\n"+
"mesh-reads:x:961:961::/var/lib/mesh-reads:/usr/bin/nologin\n")
write(t, group, "root:x:0:\nwheel:x:998:ops\nops:x:1000:\nmesh-reads:x:961:\ndocker:x:970:ops,builder\n")
data := filepath.Join(dir, "data")
write(t, mountinfo, "22 1 0:21 / / rw,relatime - btrfs /dev/nvme0n1p2 rw\n"+
"40 22 0:40 / "+strings.ReplaceAll(data, " ", `\040`)+" rw - nfs4 home-server:/tank/data rw\n")
r := Reader{Refuser: Refuser{Places: places}, Passwd: passwd, Group: group, MountInfo: mountinfo,
Euid: func() int { return 961 }, Now: time.Now}
return r, dir
}
func write(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// ADR 0310 §3: run as root, every read is refused, loudly — its bound is an account of its own.
func TestEveryReadIsRefusedAsRoot(t *testing.T) {
r, dir := testReader(t)
r.Euid = func() int { return 0 }
write(t, filepath.Join(dir, "f"), "x\n")
if _, err := r.Read(filepath.Join(dir, "f"), ReadAsk{}); err == nil || !strings.Contains(err.Error(), "root") {
t.Errorf("a read as root: %v", err)
}
if _, err := r.Stat([]string{dir}, false); err == nil {
t.Error("a stat as root was answered")
}
if _, err := r.Account("ops", ""); err == nil {
t.Error("an account lookup as root was answered")
}
}
// ADR 0310 §5: without the controller's list of secret places it does not start: a refusal it promises would be
// silent.
func TestItDoesNotStartWithoutTheNodesSecretPlaces(t *testing.T) {
if _, err := placesFrom(""); err == nil || !strings.Contains(err.Error(), PlacesWord) {
t.Errorf("no places: %v", err)
}
if p, err := placesFrom(`["/var/lib/mesh-vault"]`); err != nil || len(p) != 1 {
t.Errorf("a list: %v %v", p, err)
}
if p, err := placesFrom(`[]`); err != nil || len(p) != 0 {
t.Errorf("an empty list is a node with no secret place: %v %v", p, err)
}
}
// ADR 0310 §5: a path under the vault's directory, a module's own-secret directory and a known credential file
// are each refused by name before any read — though the account's modes would allow it — and a link to one is
// refused as the place is.
func TestSecretPlacesAreRefusedByNameBeforeAnyRead(t *testing.T) {
r0, dir := testReader(t)
vault := filepath.Join(dir, "var/lib/mesh-vault")
own := filepath.Join(dir, "var/lib/desk-channel")
r, _ := r0, dir
r.Places = []string{vault, own}
for _, p := range []string{
filepath.Join(vault, "ledger/entries.json"),
filepath.Join(own, "broker"),
filepath.Join(own, "settings.json"),
filepath.Join(dir, "home/ops/.claude/.credentials.json"),
filepath.Join(dir, "home/ops/.ssh/config"),
filepath.Join(dir, "home/ops/.pgpass"),
filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key"),
filepath.Join(dir, "etc/shadow"),
filepath.Join(dir, "srv/tls/server.key"),
} {
write(t, p, "harmless words\n")
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "refused by name") {
t.Errorf("%s was read: %v", p, err)
}
}
if _, err := r.List(vault); err == nil || !strings.Contains(err.Error(), "refused by name") {
t.Errorf("the vault's directory was listed: %v", err)
}
link := filepath.Join(dir, "tmp/innocent")
write(t, filepath.Join(dir, "tmp/x"), "")
if err := os.Symlink(filepath.Join(own, "broker"), link); err != nil {
t.Fatal(err)
}
if _, err := r.Read(link, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "leads to") {
t.Errorf("a link to a secret place was read: %v", err)
}
// A search under a directory holding a secret place skips it and names it.
write(t, filepath.Join(dir, "home/ops/notes.txt"), "harmless words\n")
a, err := r.Search(SearchAsk{Text: "harmless", Path: filepath.Join(dir, "home"), Depth: 8})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "notes.txt") || len(a.Skipped) != 3 {
t.Errorf("search: matches %v, skipped %v", a.Matches, a.Skipped)
}
// What a public key or a certificate is, is not refused.
for _, p := range []string{filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key.pub"), filepath.Join(dir, "etc/ssl/certs/ca.pem")} {
write(t, p, "harmless words\n")
if _, err := r.Read(p, ReadAsk{}); err != nil {
t.Errorf("%s was refused: %v", p, err)
}
}
}
// ADR 0310 §3: what the account's file modes refuse is refused — the first bound, below the names.
func TestAFileTheAccountMayNotReadIsRefusedByItsMode(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("run as root, a file's mode refuses nothing")
}
r, dir := testReader(t)
p := filepath.Join(dir, "operators-only")
write(t, p, "the operator's words\n")
if err := os.Chmod(p, 0o000); err != nil {
t.Fatal(err)
}
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "may not reach it") {
t.Errorf("a file of mode 0000: %v", err)
}
closed := filepath.Join(dir, "closed")
write(t, filepath.Join(closed, "inside"), "x")
if err := os.Chmod(closed, 0o700&^0o100); err != nil {
t.Fatal(err)
}
defer os.Chmod(closed, 0o755)
a, err := r.Stat([]string{filepath.Join(closed, "inside")}, false)
if err != nil {
t.Fatal(err)
}
if got := a.Paths[0]; !strings.Contains(got.Stopped, "may not reach") {
t.Errorf("a path through a directory it may not search: %+v", got)
}
}
// ADR 0310 §2 and the bounds: a read over the size bound is cut at a line and says so; more than the most is
// refused; lines and a tail are answered as asked.
func TestAReadIsBounded(t *testing.T) {
r, dir := testReader(t)
p := filepath.Join(dir, "big")
var b strings.Builder
for i := 1; i <= 20000; i++ {
fmt.Fprintf(&b, "line %d of the file\n", i)
}
write(t, p, b.String())
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
if len(a.Text) > DefaultReadSize || !strings.Contains(a.Cut, fmt.Sprint(DefaultReadSize)) || a.From != 1 {
t.Errorf("a default read: %d bytes, cut %q", len(a.Text), a.Cut)
}
if _, err := r.Read(p, ReadAsk{Bytes: MaxReadSize + 1}); err == nil {
t.Error("more than the most a read answers was answered")
}
a, err = r.Read(p, ReadAsk{From: 10, Lines: 2})
if err != nil || a.Text != "line 10 of the file\nline 11 of the file\n" || a.From != 10 || a.Cut != "" {
t.Errorf("lines 10 and 11: %+v %v", a, err)
}
a, err = r.Read(p, ReadAsk{From: -2})
if err != nil || a.Text != "line 19999 of the file\nline 20000 of the file\n" || a.From != 19999 {
t.Errorf("the last two lines: %q from %d, %v", a.Text, a.From, err)
}
if _, err := r.Read(p, ReadAsk{From: 30000}); err == nil || !strings.Contains(err.Error(), "20000 lines") {
t.Errorf("a line past the end: %v", err)
}
}
// A binary file, a directory and a fifo are refused, and the fifo without waiting for a writer.
func TestOnlyATextFileIsRead(t *testing.T) {
r, dir := testReader(t)
bin := filepath.Join(dir, "bin")
write(t, bin, "ELF\x00\x01\x02")
if _, err := r.Read(bin, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "binary") {
t.Errorf("a binary file: %v", err)
}
if _, err := r.Read(dir, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "directory") {
t.Errorf("a directory: %v", err)
}
fifo := filepath.Join(dir, "fifo")
if err := syscall.Mkfifo(fifo, 0o644); err != nil {
t.Fatal(err)
}
done := make(chan error, 1)
go func() { _, err := r.Read(fifo, ReadAsk{}); done <- err }()
select {
case err := <-done:
if err == nil || !strings.Contains(err.Error(), "fifo") {
t.Errorf("a fifo: %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("a read of a fifo waits for a writer")
}
}
// ADR 0310 §4: a line shaped like a secret is withheld, its number said, and the rest is answered.
func TestALineShapedLikeASecretIsWithheld(t *testing.T) {
r, dir := testReader(t)
p := filepath.Join(dir, "app.conf")
write(t, p, "listen = /run/app.sock\ntoken = "+aToken+"\nuser = app\n")
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
if strings.Contains(a.Text, aToken) || !strings.Contains(a.Text, "user = app") ||
!strings.Contains(a.Text, "listen = /run/app.sock") || fmt.Sprint(a.Withheld) != "[2]" {
t.Errorf("read: %q withheld %v", a.Text, a.Withheld)
}
m, err := r.Search(SearchAsk{Text: "token", Path: dir})
if err != nil {
t.Fatal(err)
}
body, _ := json.Marshal(m)
if strings.Contains(string(body), aToken) || len(m.Matches) != 1 || len(m.Withheld) != 1 {
t.Errorf("search: %s", body)
}
}
// The bounds of a search: matches, depth, the most asked, and time — each met is said.
func TestASearchIsBounded(t *testing.T) {
r, dir := testReader(t)
for i := 0; i < 30; i++ {
write(t, filepath.Join(dir, fmt.Sprintf("f%02d", i)), "needle\nneedle\n")
}
a, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: 7})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 7 || !strings.HasPrefix(a.Bound, "matches") {
t.Errorf("seven matches asked: %d, bound %q", len(a.Matches), a.Bound)
}
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: MaxMatches + 1}); err == nil {
t.Error("more matches than the most were asked and answered")
}
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Depth: MaxDepth + 1}); err == nil {
t.Error("deeper than the most was asked and answered")
}
deep := t.TempDir()
write(t, filepath.Join(deep, "a/b/c/shallow"), "needle\n")
write(t, filepath.Join(deep, "a/b/c/d/e/deep"), "needle\n")
a, err = r.Search(SearchAsk{Text: "needle", Path: deep, Depth: 3})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "shallow") {
t.Errorf("three deep: %+v", a.Matches)
}
// Case: as asked.
write(t, filepath.Join(deep, "case"), "NEEDLE\n")
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case")}); len(a.Matches) != 0 {
t.Error("a match whatever the case, unasked")
}
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case"), CaseInsensitive: true}); len(a.Matches) != 1 {
t.Error("no match whatever the case, asked")
}
// Time: a clock past the bound stops the walk and says so.
start := time.Now()
calls := 0
r.Now = func() time.Time { calls++; return start.Add(time.Duration(calls) * SearchTime / 3) }
a, err = r.Search(SearchAsk{Text: "needle", Path: dir})
if err != nil || !strings.HasPrefix(a.Bound, "time") {
t.Errorf("the time bound: %q %v", a.Bound, err)
}
// A fixed string: a pattern's characters are themselves.
write(t, filepath.Join(deep, "pattern"), "a.c\nabc\n")
if a, _ := r.Search(SearchAsk{Text: "a.c", Path: filepath.Join(deep, "pattern")}); len(a.Matches) != 1 || a.Matches[0].Line != 1 {
t.Errorf("a.c as a fixed string: %+v", a.Matches)
}
}
// The survey issue 418 was asked for: each component of a path with its owner, group, mode, whether it is a
// link and to what, and whether it is a mount point — the link not followed unless asked.
func TestStatSaysEachComponent(t *testing.T) {
r, dir := testReader(t)
data := filepath.Join(dir, "data")
write(t, filepath.Join(data, "real/file"), "x")
if err := os.Chmod(filepath.Join(data, "real"), os.ModeSetgid|0o750); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(data, "link")); err != nil {
t.Fatal(err)
}
a, err := r.Stat([]string{filepath.Join(data, "link"), filepath.Join(data, "real/file")}, false)
if err != nil {
t.Fatal(err)
}
link := a.Paths[0]
last := link.Components[len(link.Components)-1]
if link.Components[0].Path != "/" || last.Type != "link" || last.Link != "real" || link.LeadsTo != "" {
t.Errorf("the link, not followed: %+v", link)
}
var dataC, realC Component
for _, c := range a.Paths[1].Components {
switch c.Path {
case data:
dataC = c
case filepath.Join(data, "real"):
realC = c
}
}
if dataC.Mount == nil || dataC.Mount.FS != "nfs4" {
t.Errorf("the mount point is not said: %+v", dataC)
}
if realC.Mode != "2750" || realC.Type != "directory" || realC.UID == nil || realC.Perms == "" {
t.Errorf("a component's mode: %+v", realC)
}
if uid := *realC.UID; uid == 1000 && realC.Owner != "ops" {
t.Errorf("owner %d named %q", uid, realC.Owner)
}
a, err = r.Stat([]string{filepath.Join(data, "link")}, true)
if err != nil {
t.Fatal(err)
}
if a.Paths[0].LeadsTo != filepath.Join(data, "real") || a.Paths[0].Leads == nil || a.Paths[0].Leads.Type != "directory" {
t.Errorf("followed: %+v", a.Paths[0])
}
a, err = r.Stat([]string{filepath.Join(dir, "missing/deeper")}, false)
if err != nil || !strings.Contains(a.Paths[0].Stopped, "does not exist") {
t.Errorf("a missing path: %+v %v", a, err)
}
if _, err := r.Stat([]string{"relative/path"}, false); err == nil {
t.Error("a relative path was answered")
}
many := make([]string, MaxPaths+1)
for i := range many {
many[i] = "/"
}
if _, err := r.Stat(many, false); err == nil {
t.Error("more paths than the most were answered")
}
}
func TestReadlinkSaysTheTargetAndWhereItLeads(t *testing.T) {
r, dir := testReader(t)
write(t, filepath.Join(dir, "target"), "")
if err := os.Symlink("target", filepath.Join(dir, "l1")); err != nil {
t.Fatal(err)
}
if err := os.Symlink("l1", filepath.Join(dir, "l2")); err != nil {
t.Fatal(err)
}
if err := os.Symlink("nowhere", filepath.Join(dir, "dangling")); err != nil {
t.Fatal(err)
}
a, err := r.Readlink(filepath.Join(dir, "l2"))
if err != nil || !a.IsLink || a.Link != "l1" || a.LeadsTo != filepath.Join(dir, "target") || !a.Exists {
t.Errorf("l2: %+v %v", a, err)
}
a, err = r.Readlink(filepath.Join(dir, "dangling"))
if err != nil || !a.IsLink || a.Exists || !strings.Contains(a.Stopped, "does not exist") {
t.Errorf("dangling: %+v %v", a, err)
}
}
func TestListSaysEveryEntry(t *testing.T) {
r, dir := testReader(t)
write(t, filepath.Join(dir, "d/a"), "abc")
if err := os.Symlink("a", filepath.Join(dir, "d/b")); err != nil {
t.Fatal(err)
}
a, err := r.List(filepath.Join(dir, "d"))
if err != nil {
t.Fatal(err)
}
if a.Count != 2 || a.Entries[0].Name != "a" || *a.Entries[0].Size != 3 || a.Entries[1].Type != "link" || a.Entries[1].Link != "a" {
t.Errorf("list: %+v", a)
}
for i := 0; i < MaxEntries+5; i++ {
write(t, filepath.Join(dir, "many", fmt.Sprintf("e%05d", i)), "")
}
a, err = r.List(filepath.Join(dir, "many"))
if err != nil || len(a.Entries) != MaxEntries || a.Count != MaxEntries+5 || a.Cut == "" {
t.Errorf("a long directory: %d of %d, cut %q, %v", len(a.Entries), a.Count, a.Cut, err)
}
}
// getent passwd, getent group and id: an account with every group it is in, a group with its members, never a
// password field.
func TestAccountAndGroupLookup(t *testing.T) {
r, _ := testReader(t)
a, err := r.Account("ops", "")
if err != nil {
t.Fatal(err)
}
u := a.Accounts[0]
sort.Strings(u.Groups)
if u.UID != 1000 || u.Group != "ops" || u.Shell != "/usr/bin/zsh" || fmt.Sprint(u.Groups) != "[docker ops wheel]" {
t.Errorf("ops: %+v", u)
}
if a, err := r.Account("961", ""); err != nil || a.Accounts[0].Name != "mesh-reads" {
t.Errorf("by number: %+v %v", a, err)
}
g, err := r.Account("", "docker")
if err != nil || g.Groups[0].GID != 970 || fmt.Sprint(g.Groups[0].Members) != "[ops builder]" {
t.Errorf("docker: %+v %v", g, err)
}
if _, err := r.Account("nobody-here", ""); err == nil {
t.Error("a missing account was answered")
}
all, err := r.Account("", "")
if err != nil || len(all.Accounts) != 3 || len(all.Groups) != 5 {
t.Errorf("all: %+v %v", all, err)
}
body, _ := json.Marshal(all)
if strings.Contains(string(body), `"x"`) || strings.Contains(string(body), "password") {
t.Errorf("a password field is answered: %s", body)
}
}
// No answer is larger than the bound, whatever a verb made.
func TestNoAnswerIsLargerThanTheBound(t *testing.T) {
if _, err := bounded(strings.Repeat("x", MaxAnswer+1), nil); err == nil {
t.Error("an answer over the bound was given")
}
if _, err := bounded("small", nil); err != nil {
t.Error(err)
}
}
// The seat's six verbs, as the controller defines them (mesh-controller internal/catalogue/reads.go).
func TestItServesTheSeatsVerbs(t *testing.T) {
r, _ := testReader(t)
var names []string
for _, tool := range tools(r) {
names = append(names, tool.Name)
}
if fmt.Sprint(names) != "[node-reads.stat node-reads.readlink node-reads.list node-reads.read node-reads.search node-reads.account]" {
t.Errorf("verbs: %v", names)
}
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/outward"
)
// What is refused by name, before any read (novox/hq ADR 0310 §5), and what an answer withholds (§4).
//
// **Two lists of places.** The places the mesh knows a secret is kept on this node — every module's own-secret
// directory, every credential file and grant directory, the vault's directories — come from the controller,
// which composes them from the manifests (MESH_SECRET_PLACES). The known credential files of the programs
// on a machine are this module's list, below: the mesh does not write them, so nothing else names them. Both
// hold whatever the account's file modes would allow: the file modes are the first bound, the names the second.
// knownDirs are directories that hold nothing but credentials, wherever they are: an account's keys.
var knownDirs = map[string]bool{
".ssh": true, ".gnupg": true, ".password-store": true, ".aws": true, ".kube": true, ".azure": true,
}
// knownNames are file names that are a credential whatever directory they are in.
var knownNames = map[string]bool{
".credentials.json": true, // the agent's credentials (Claude Code)
".netrc": true, ".pgpass": true, ".git-credentials": true, ".npmrc": true, ".pypirc": true,
".my.cnf": true, ".vault-token": true, "credentials": true, "credentials.json": true,
"id_rsa": true, "id_dsa": true, "id_ecdsa": true, "id_ed25519": true, "id_ecdsa_sk": true, "id_ed25519_sk": true,
"shadow": true, "gshadow": true, "shadow-": true, "gshadow-": true, "opasswd": true,
"broker": true, // the mesh's name for a module's bus credential
}
// knownSuffixes are file name endings a credential has.
var knownSuffixes = []string{".key", ".p12", ".pfx", ".jks", ".keystore", ".creds", ".nk", ".kdbx", "_key"}
// knownPaths are the places on a machine that hold credentials and are not a module's.
var knownPaths = []string{
"/etc/ssl/private", "/etc/wireguard", "/etc/NetworkManager/system-connections", "/etc/ipsec.d/private",
"/var/lib/iwd", "/etc/docker/key.json", "/root/.docker/config.json", "/var/lib/sss/secrets",
}
// Refuser says whether a path is a place a secret is kept.
type Refuser struct {
// Places are the mesh's, from the controller: refused with everything beneath them.
Places []string
}
// Refused says why a path may not be read, or "" when it may. The path is clean and absolute.
func (r Refuser) Refused(path string) string {
path = filepath.Clean(path)
for _, p := range r.Places {
if within(path, p) {
return "it is a place the mesh keeps a secret (" + p + ")"
}
}
for _, p := range knownPaths {
if within(path, p) {
return "it is a place a machine keeps credentials (" + p + ")"
}
}
parts := strings.Split(strings.TrimPrefix(path, "/"), "/")
for i, part := range parts {
if knownDirs[part] {
return "it is in " + part + ", which holds an account's credentials"
}
if i == len(parts)-1 {
if knownNames[part] {
return part + " is a known credential file"
}
for _, s := range knownSuffixes {
if strings.HasSuffix(part, s) && len(part) > len(s) {
return part + " is named as a key or a credential (" + s + ")"
}
}
if strings.HasSuffix(part, ".pem") && (strings.Contains(part, "key") || strings.Contains(part, "private")) {
return part + " is named as a private key"
}
if part == "config.json" && i > 0 && parts[i-1] == ".docker" {
return "a container registry login is kept there"
}
}
}
return ""
}
func within(path, place string) bool {
return path == place || strings.HasPrefix(path, strings.TrimSuffix(place, "/")+"/")
}
// Withheld is one piece of an answer the content rule kept back: where it was, and the class it carried.
type Withheld struct {
At string `json:"at"`
Class string `json:"why"`
}
// clean is the content rule's secret class (mesh-sdk outward, novox/hq ADR 0310 §4) over one piece of an
// answer: the piece, or a mark in its place when it is shaped like a secret.
//
// **What a file holds, not what the file system names.** It judges a file's lines, a matching line, a link's
// target, a mount's source and an account's words. A path and an entry's name are not judged: they are what
// the caller named or the file system calls a thing, and the rule's random-run test takes a temporary
// directory's or a container layer's name for a token, which would withhold the very survey asked for. A
// place where a secret sits by name is refused before any read instead (Refuser).
func clean(s string) (string, string) {
if refusal, ok := outward.CheckSecret(s); !ok {
return "‹withheld: " + refusal.What + "›", refusal.What
}
return s, ""
}
+5
View File
@@ -0,0 +1,5 @@
module mesh-reads
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23
+2
View File
@@ -0,0 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23 h1:/F4NwABPAuFZlBx1yK4rdj6sVL5K1XvRlOnZZfNnlsQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+54
View File
@@ -0,0 +1,54 @@
{
"module": "mesh-reads",
"version": "1",
"runs-as": "mesh-reads",
"claims": [
{
"name": "node-reads",
"scope": "node",
"serves": [
"stat",
"readlink",
"list",
"read",
"search",
"account"
]
}
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-reads",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-reads",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-reads"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-reads",
"place": "."
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-reads",
"binary": "mesh-reads",
"loads": [
"mesh-reads"
]
}
]
}
}
+37 -6
View File
@@ -16,15 +16,20 @@ channel's client and no channel's secret.
its sender) and `desk-channel` (kind `desktop`, the machines' notifiers, proves nothing). Which kinds exist
and what each promises is the controller's word, in this module's membership; whether each can send now
is the channel's own, said on `intake.standing`.
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages), `asks` (every ask, closed ones
for 30 days), `identities` (the operator's linked accounts, one per kind, and a link asked for).
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages, and the restart passes made),
`asks` (every ask, closed ones for 30 days), `identities` (the operator's linked accounts, one per kind, and a
link asked for), `walks` (the delivery walks open now, each with what it holds).
- **Lost answers are said again from here, never from a channel's memory** (issue 384,
`cmd/messenger/restart.go`): a channel whose messages' answers were lost — the desk's notifier started
again — says when in its standing (`answers-lost-at`). The router says again on that kind, directly and not
through the outbox, each message that is still open, asked there and last said before that time: one per
problem, under its handle, with the words and answers of now, so it replaces the message in place. What is
silenced is edited to say so, with no answers and no urgency; what ended never comes back. A kind that cannot
send waits; it never goes to another channel. Telegram's answers are never lost and it says no time.
through the outbox, each message that is still open, asked there, never answered and last said before that
time: one per problem, under its handle, with the words and answers of now, so it replaces the message in
place. **A restart says nothing it has said** (novox/hq ADR 0308 §6, issue 531): a silenced message, one
withheld for its content, one answered and one a walk holds are left as they are, and what ended never comes
back. The standings of one restart (a notifier on each machine, seconds apart) are one pass: a loss is acted
on once it stood 10 s, and the pass made is kept in the state (`sent`, key `passes`), so a router that starts
again does not make it again. A kind that cannot send waits; it never goes to another channel. Telegram's
answers are never lost and it says no time.
## Asks, answers and warrants (novox/hq ADR 0259)
@@ -165,10 +170,36 @@ ends takes them away everywhere; its expiry is said with its day when it is not
history. An event that cannot be read
is refused by name, counted, and told to the operator once an hour.
## While a delivery walks (novox/hq ADR 0308)
The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`started`, `ended`), `touches`
(`[{machine, modules}]`: what it moves and what a moved tool runner or node-engine carries), `outcome`
(`delivered`, `failed`, `stopped`), `bound_seconds` and `budget_class` (`cmd/messenger/walks.go`).
- **Held:** while a walk is open and within its bound, a message whose subject is a machine it touches, or a
module it touches (on a machine it touches, when the subject names one), is held: kept on the walk with when
and what, and the history says `held`. A message several open walks touch is told when the last ends or passes
its bound. At a walk's end, a later open walk that touches a message takes it over only as a first hold would:
still open, not silenced, not said since, and not one that is never held.
- **Never held:** an ask waiting for the operator (and a condition an open ask is about), a secret given or
replaced, a subject no open walk touches, and a walk's own failure (a plan, walk, delivery or build).
- **The end is one message:** the outcome, the modules and machines reached, how long; each held message still
open, one line each; the ones that cleared counted. Delivered with nothing held and nothing open: one line —
and nothing on a kind listed in the setting `only-what-needs-me`. What it carries is marked told on a kind
only once the message was sent there. Until then each is kept owed in the state (`owed`): a message lost with a
restart before it was sent leaves them untold and owed, and the restarted router says each on its own.
- **The bound:** at its start plus `bound_seconds` (900 when not given), what is held and still open is said,
with "The walk … is still running, since HH:MM", and the cleared counted; the walk holds nothing more. A walk
whose end never comes is forgotten a day after it started. A start heard after its end opens nothing: the
controller's moments are compared on its own clock, never with one capped at the router's now.
- **Kept in the state** (`walks`): a router that starts again mid-walk holds and tells as before.
## What the operator gives
1. **The time zone**, as a setting: `{"time-zone": "<an IANA zone name>"}`. Every time a message says is
said in it; not given, the zone of the machine this runs on.
**The kinds that say only what needs the operator**, as a setting: `{"only-what-needs-me": ["<kind>"]}`. A
walk that delivered with nothing held and nothing open is not told on them (ADR 0308 §4); empty by default.
2. **The channels**: assign `telegram` and `desk-channel` (each says in its README what it needs). Linking
an account as the operator is done on the channel, never by a setting.
+54 -12
View File
@@ -154,6 +154,11 @@ type Record struct {
Subject string `json:"subject"`
Severity string `json:"severity"`
Summary string `json:"summary"`
// Scope, SubjectID and Machine are the condition's subject as the controller gave it, which a walk's touches
// are matched against (ADR 0308 §2); empty for a record kept before, whose key names them.
Scope string `json:"scope,omitempty"`
SubjectID string `json:"subject_id,omitempty"`
Machine string `json:"machine,omitempty"`
// Headline, Explanation and Resolved are the plain words the operator reads (novox/hq ADR 0253):
// the controller's, or a module's through notify; empty, the message is said from the summary.
Headline string `json:"headline,omitempty"`
@@ -195,14 +200,17 @@ type Record struct {
// ClearHeld is what was waiting to be said of it when it cleared (a reminder, an escalation): said after
// all if it comes back before its clearing was said. Blips counts those comebacks, never said, so a
// condition that flaps is known as one and its message's "again, Nth time" counts only what was said.
ClearHeld string `json:"clear_held,omitempty"`
Blips int `json:"blips,omitempty"`
Reminded bool `json:"reminded,omitempty"`
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
Folded bool `json:"folded,omitempty"` // held by the cap
Cleared time.Time `json:"cleared,omitempty"`
Count int `json:"count"`
Refused string `json:"refused,omitempty"`
ClearHeld string `json:"clear_held,omitempty"`
Blips int `json:"blips,omitempty"`
Reminded bool `json:"reminded,omitempty"`
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
// Owed is what a walk held of it and has not told yet (ADR 0308 §4, §5): kept in the state until a message
// carrying it was sent, so a router that restarts before says it on its own (issue 538), never history.
Owed string `json:"owed,omitempty"`
Folded bool `json:"folded,omitempty"` // held by the cap
Cleared time.Time `json:"cleared,omitempty"`
Count int `json:"count"`
Refused string `json:"refused,omitempty"`
// History: known from the state or from an old event, and never said — not news.
History bool `json:"history,omitempty"`
// Seen is when this holder last took word of it; the open set read after it is authoritative.
@@ -211,7 +219,8 @@ type Record struct {
// recordOf is a condition as this holder keeps it.
func recordOf(c Condition) Record {
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Severity: c.Severity, Summary: c.Summary,
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Scope: c.Scope, SubjectID: c.ID, Machine: c.Machine,
Severity: c.Severity, Summary: c.Summary,
Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved, Needs: c.Needs, Actions: c.Actions,
Origin: "condition",
More: "conditions show " + c.Key, Raised: c.Raised, SilencedTill: c.SilencedTill, SilencedWhy: c.SilencedWhy}
@@ -232,6 +241,13 @@ func (r *Record) takeWords(from Record) {
}
}
// takeSubject keeps the subject as the controller says it now, when it says one.
func (r *Record) takeSubject(from Record) {
if from.Scope != "" || from.SubjectID != "" || from.Machine != "" {
r.Scope, r.SubjectID, r.Machine = from.Scope, from.SubjectID, from.Machine
}
}
func (r *Record) silenced(now time.Time) bool {
return !r.SilencedTill.IsZero() && now.Before(r.SilencedTill)
}
@@ -292,6 +308,9 @@ type Holder struct {
Zone func() *time.Location
// Tracking asks the issue tracker which issues track which conditions; nil asks it through Ask (tracking.go).
Tracking func() (map[string][]TrackedIssue, error)
// NeedsOnly says whether a kind is set to say only what needs the operator (the setting
// `only-what-needs-me`, ADR 0308 §4); nil, none is.
NeedsOnly func(kind string) bool
work sync.Mutex // one event, call or tick at a time
mu sync.Mutex // what the status reads
@@ -329,8 +348,14 @@ type Holder struct {
lastUnlink map[string]Unlinked
// rootWord is the controller's last word on whether an agent can become root, by the machines asked.
rootWord map[string]rootWord
// shows is, by kind, the AnswersLostAt this router last acted on in full (restart.go).
shows map[string]time.Time
// passes is, by kind, the restart pass this router last made in full, kept in its state (restart.go); lost
// the loss of answers it has yet to act on, by kind.
passes map[string]Pass
lost map[string]time.Time
// walks are the walks open now, by id, each with what it holds (walks.go, ADR 0308).
walks map[string]*Walk
// ended are the walks whose end was heard, by id: what a start heard after its end is matched against.
ended map[string]*Walk
// track is the issue tracker's last word on which issues track which conditions (tracking.go).
track trackWord
}
@@ -348,7 +373,10 @@ func (h *Holder) init() {
h.chanOK = map[string]time.Time{}
h.saidOnce = map[string]time.Time{}
h.heard = map[string]int{}
h.shows = map[string]time.Time{}
h.passes = map[string]Pass{}
h.lost = map[string]time.Time{}
h.walks = map[string]*Walk{}
h.ended = map[string]*Walk{}
}
if h.Now == nil {
h.Now = time.Now
@@ -382,11 +410,20 @@ func (h *Holder) Load() error {
r.History = true
}
}
if r.Owed != "" && r.Cleared.IsZero() {
// Held from the operator on purpose and not told yet: said on its own, as the next tick tries what is
// unsent (issue 538). What the operator could have heard is history; this they could not.
r.Pending, r.History = r.Owed, false
}
h.open[r.Key] = &r
}
if sent, err := h.Store.Recent(); err == nil {
h.recent = sent
}
if err := h.loadWalks(); err != nil {
h.storeErr = err.Error()
return err
}
h.mu.Unlock()
err = h.loadAsks()
h.mu.Lock()
@@ -485,6 +522,7 @@ func (h *Holder) raised(rec Record, quiet bool) {
wasQuiet := needsNothing(old)
wasSilenced := old.silenced(now)
old.Summary, old.Subject, old.Kind = rec.Summary, rec.Subject, rec.Kind
old.takeSubject(rec)
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
old.takeWords(rec)
if rec.Severity != "" {
@@ -584,6 +622,7 @@ func (h *Holder) changed(rec Record, quiet bool) {
wasQuiet := needsNothing(old)
wasSilenced := old.silenced(h.Now())
old.Summary, old.Subject = rec.Summary, rec.Subject
old.takeSubject(rec)
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
old.takeWords(rec)
if rec.Severity != "" {
@@ -710,6 +749,8 @@ func (h *Holder) Tick() {
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
h.sweepAsks(now)
h.breakGlass()
h.boundWalks(now)
h.duePasses(now)
for _, r := range recs {
switch {
case !r.Cleared.IsZero():
@@ -775,6 +816,7 @@ func (h *Holder) Flush() {
h.init()
now := h.Now()
h.mu.Unlock()
h.duePasses(now)
h.flush(now, false)
}
+21 -1
View File
@@ -66,6 +66,9 @@ func (f *fakeChannel) Edit(id string, m Message) error {
type memStore struct {
recs map[string]Record
recent []Sent
// walks and passes are what ADR 0308 keeps: the open walks and the restart passes.
walks map[string]Walk
passes map[string]Pass
// puts counts the writes, so a write of bytes identical to what is stored is still seen (issue 422's review).
puts int
}
@@ -87,7 +90,24 @@ func (m *memStore) All() ([]Record, error) {
return out, nil
}
func (m *memStore) PutRecent(s []Sent) error { m.recent = s; return nil }
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
func (m *memStore) PutWalk(w Walk) error {
if m.walks == nil {
m.walks = map[string]Walk{}
}
m.walks[w.ID] = w
return nil
}
func (m *memStore) DeleteWalk(id string) error { delete(m.walks, id); return nil }
func (m *memStore) Walks() ([]Walk, error) {
var out []Walk
for _, w := range m.walks {
out = append(out, w)
}
return out, nil
}
func (m *memStore) PutPasses(p map[string]Pass) error { m.passes = p; return nil }
func (m *memStore) Passes() (map[string]Pass, error) { return m.passes, nil }
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
type clock struct{ t time.Time }
+80
View File
@@ -29,6 +29,9 @@ type Settings struct {
// TimeZone is the operator's, an IANA name such as Europe/Brussels: every time a message says is said
// in it. Empty is this machine's own zone.
TimeZone string `json:"time-zone"`
// OnlyWhatNeedsMe are the channel kinds set to say only what needs the operator: a walk that delivered
// with nothing held and nothing open is not told there at all (novox/hq ADR 0308 §4).
OnlyWhatNeedsMe []string `json:"only-what-needs-me"`
}
// zoneOf is the location a time-zone setting names, or this machine's own when it names none or one
@@ -60,6 +63,19 @@ func readSettings(path string) (Settings, error) {
if tz, ok := loose["time-zone"].(string); ok {
s.TimeZone = strings.TrimSpace(tz)
}
switch v := loose["only-what-needs-me"].(type) {
case nil:
case []any:
for _, k := range v {
kind, ok := k.(string)
if !ok || strings.TrimSpace(kind) == "" {
return s, fmt.Errorf("only-what-needs-me lists something that is not a channel kind: %v", k)
}
s.OnlyWhatNeedsMe = append(s.OnlyWhatNeedsMe, strings.TrimSpace(kind))
}
default:
return s, fmt.Errorf("only-what-needs-me is not a list of channel kinds")
}
return s, nil
}
@@ -109,6 +125,50 @@ func (stateStore) All() ([]Record, error) {
}
return out, nil
}
// The walks open now, one key each in the state `walks`, and the restart passes under one key of `sent`
// (novox/hq ADR 0308): a router that starts again holds, tells and passes as if it had not.
func (stateStore) PutWalk(w Walk) error {
_, err := stdio.State("walks").Put(kvKey(w.ID), w)
return err
}
func (stateStore) DeleteWalk(id string) error { return stdio.State("walks").Delete(kvKey(id)) }
func (stateStore) Walks() ([]Walk, error) {
keys, err := stdio.State("walks").Keys()
if err != nil {
return nil, err
}
var out []Walk
for _, k := range keys {
e, err := stdio.State("walks").Get(k)
if err != nil {
return nil, err
}
if e == nil {
continue
}
var w Walk
if err := json.Unmarshal(e.Value, &w); err != nil {
logf("[messenger] the walk kept as %s cannot be read (%v); left as it is", k, err)
continue
}
out = append(out, w)
}
return out, nil
}
func (stateStore) PutPasses(p map[string]Pass) error {
_, err := stdio.State("sent").Put("passes", p)
return err
}
func (stateStore) Passes() (map[string]Pass, error) {
e, err := stdio.State("sent").Get("passes")
if err != nil || e == nil {
return nil, err
}
var p map[string]Pass
return p, json.Unmarshal(e.Value, &p)
}
func (stateStore) PutRecent(s []Sent) error {
_, err := stdio.State("sent").Put("recent", s)
return err
@@ -271,6 +331,14 @@ func main() {
return loc
},
Emit: func(event string, body any) error { return stdio.Emit(event, body) },
NeedsOnly: func(kind string) bool {
for _, k := range settings().OnlyWhatNeedsMe {
if k == kind {
return true
}
}
return false
},
}
bench.To = func(kind string) string {
if ids := h.identitiesOf(kind); len(ids) > 0 {
@@ -326,6 +394,18 @@ func run(h *Holder, l *listening) {
}
}()
handle := func(e stdio.Envelope) error {
if e.Key == ControllerSeat+"."+EventPlanMoved || e.Key == ControllerSeat+"."+EventWalk {
// A walk's start or end (novox/hq ADR 0308 §1); any other move of a plan is not read here.
ev, isWalk, err := DecodeWalk(e.Body)
if err != nil {
h.Unreadable(e.Key, err)
return nil
}
if isWalk {
h.Walked(ev)
}
return nil
}
event, ok := eventOf(e.Key)
if !ok {
return nil
@@ -81,7 +81,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
}
if !reflect.DeepEqual(m.Consumes, []string{
ControllerSeat + "." + EventRaised, ControllerSeat + "." + EventChanged, ControllerSeat + "." + EventCleared,
"intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
ControllerSeat + "." + EventPlanMoved, "intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
t.Fatalf("consumes %v", m.Consumes)
}
// It performs nothing: the verbs it calls read the open conditions, whether root is free, and which issues
@@ -100,7 +100,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
}
state = append(state, name)
}
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities"}) {
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities", "walks"}) {
t.Fatalf("state %v", state)
}
// It holds no channel's secret: only its own bus account, read by its own account alone (ADR 0259 §8).
+65 -2
View File
@@ -18,6 +18,9 @@ type item struct {
Severity string
At time.Time
Msg *Message
// Carries are the records a walk's end or bound message tells (ADR 0308 §4): each is marked told on a
// channel only once that message was sent there (issue 538 A), never when it is held to be sent.
Carries []string
}
func (i item) urgent() bool { return i.Severity == Urgent && i.What != "cleared" }
@@ -32,6 +35,10 @@ func (h *Holder) deliver(r *Record, what string) {
h.record(Sent{At: h.Now(), Channel: "-", Key: r.Key, What: what, Outcome: "quiet: nothing for the operator to do"})
return
}
if h.hold(r, what) {
// An open walk touches it: told when the walk ends or passes its bound (ADR 0308 §2).
return
}
var to []Channel
if kinds := h.askKinds(r); len(kinds) > 0 {
// An ask goes to every channel that carries one of its answers (novox/hq ADR 0259): the desk for an
@@ -118,6 +125,8 @@ func (h *Holder) queuedWhat(key string) string {
return ""
}
// queued says whether something is held to be said of a record: its own message, or a walk's message that
// carries it.
func (h *Holder) queued(key string) bool {
h.mu.Lock()
defer h.mu.Unlock()
@@ -126,6 +135,11 @@ func (h *Holder) queued(key string) bool {
if it.Msg == nil && it.Key == key {
return true
}
for _, k := range it.Carries {
if k == key {
return true
}
}
}
}
return false
@@ -135,6 +149,9 @@ func (h *Holder) queued(key string) bool {
// edit fails, or that said it inside a digest, is given a new message instead — held like any other.
// silentOnly: only where an edit notifies nobody; elsewhere nothing.
func (h *Holder) edit(r *Record, what string, silentOnly bool) {
if h.hold(r, what) {
return // an open walk touches it (ADR 0308 §2); a clearing or a silence is never held
}
h.mu.Lock()
sent := map[string]string{}
for k, v := range r.Sent {
@@ -225,12 +242,23 @@ func (h *Holder) due(ch Channel, list []item, now time.Time) bool {
if now.Sub(oldest) < BurstWindow {
return false
}
if ch == h.Desk && !lastWarn.IsZero() && now.Sub(lastWarn) < DesktopWarningEvery {
if ch == h.Desk && !lastWarn.IsZero() && now.Sub(lastWarn) < DesktopWarningEvery && !carriesWalk(list) {
// A walk's message is what the rhythm would have spared the operator many of: not held to it.
return false
}
return true
}
// carriesWalk says whether held items carry a walk's end or bound message.
func carriesWalk(list []item) bool {
for _, it := range list {
if it.What == WalkEndWhat {
return true
}
}
return false
}
// line is what an item says, resolved against the record now: nil when it is no longer to be said.
func (h *Holder) resolve(it item, now time.Time, kind string) (*Record, *Message) {
if it.Msg != nil {
@@ -310,7 +338,9 @@ func (h *Holder) flushChannel(ch Channel, now time.Time, force bool) {
for _, e := range entries {
// A message said again about a record shown here goes alone too: it replaces that record's message,
// so a reminder is never a digest line beside the message it follows (issue 373).
if len(e.msg.Actions) > 1 || (len(e.msg.Actions) > 0 && e.r != nil && e.r.Ask != "") || h.replacing(ch, e) != "" {
// A walk's end or bound message goes alone too: its lines are what it says (ADR 0308 §4).
if len(e.msg.Actions) > 1 || (len(e.msg.Actions) > 0 && e.r != nil && e.r.Ask != "") || h.replacing(ch, e) != "" ||
e.it.What == WalkEndWhat {
alone = append(alone, e)
} else {
rest = append(rest, e)
@@ -400,6 +430,7 @@ func (h *Holder) sendEntries(ch Channel, entries []entry, now time.Time) bool {
}
r.Pending, r.Folded, r.History = "", false, false
if e.it.What != "cleared" {
r.Owed = ""
if r.Sent == nil {
r.Sent = map[string]string{}
}
@@ -415,7 +446,39 @@ func (h *Holder) sendEntries(ch Channel, entries []entry, now time.Time) bool {
r.LastSaid = now
}
}
type carried struct {
r *Record
walk string
}
var told []carried
for _, e := range entries {
for _, key := range e.it.Carries {
r := h.open[key]
if r == nil {
continue
}
if r.Sent == nil {
r.Sent = map[string]string{}
}
if _, has := r.Sent[name]; !has {
r.Sent[name] = ""
}
if r.FirstSent.IsZero() {
r.FirstSent = now
}
r.LastSaid, r.Pending, r.History, r.Owed = now, "", false, ""
if !r.Cleared.IsZero() && !r.ClearUnsaid {
// It cleared while the message saying it open waited: its clearing is said too.
r.ClearUnsaid = true
}
told = append(told, carried{r, strings.TrimPrefix(e.it.Key, "walk.")})
}
}
h.mu.Unlock()
for _, c := range told {
h.record(Sent{At: now, Channel: name, Key: c.r.Key, What: "released", Outcome: "told in the message of walk " + c.walk})
h.persist(c.r)
}
key, what := "(digest)", fmt.Sprintf("digest of %d", len(entries))
if len(entries) == 1 {
key, what = entries[0].it.Key, entries[0].it.What
+101 -52
View File
@@ -18,6 +18,22 @@ package main
// Whether a record is said again is its own: last said before the answers were lost. So a router that started
// after the loss still says it, and a standing that repeats the time says nothing more (the record was said
// since). A kind that cannot send waits: what is open there is said once it can.
//
// **A restart says nothing it has said** (novox/hq ADR 0308 §6, issue 531). On 2026-10-12 every delivery of the
// tool runner made this pass twice, a second apart, each time editing the four silenced messages on the desk,
// which showed them again — one as "A mesh message was withheld". Three causes, each closed here:
//
// - **Twice:** the desk's standing says the latest start of any machine's notifier, and the notifiers of two
// machines started a second apart, so two standings with two times came, and each was acted on at once.
// Now a loss is acted on once it stands for LostSettle: the standings of one restart are one pass.
// - **Again after the router's own restart:** which loss was acted on was kept in memory, so a router that
// started again acted on the standing it had already acted on. Now it is kept in the module's state
// (`sent`, key `passes`), read back on start.
// - **What it said:** a silenced message was edited to its silenced words, and a message withheld for its
// content was said again as withheld. Now the pass says again only what is open, asked there and never
// answered, whose words may leave; a silenced, a withheld or an answered message is left as it is. Being
// held by a walk is no reason: a message with an open ask is never held (ADR 0308 §3), even one a walk took
// before its ask came, so its answers are made to work again like any other's.
import (
"fmt"
@@ -25,7 +41,17 @@ import (
"time"
)
// Stand takes what a kind said of itself, and acts on answers lost there since this router last acted.
// LostSettle: a loss of answers is acted on once no later loss came for this long — the standings of one
// restart, a notifier on each machine, are one pass.
const LostSettle = 10 * time.Second
// Pass is the restart pass made in full on a kind: the loss it answered and when it was made.
type Pass struct {
Lost time.Time `json:"lost"`
At time.Time `json:"at"`
}
// Stand takes what a kind said of itself, and notes answers lost there since this router last acted.
func (h *Holder) Stand(kind string, s Standing) {
if h.Bench != nil {
h.Bench.Stand(kind, s)
@@ -33,28 +59,73 @@ func (h *Holder) Stand(kind string, s Standing) {
if s.AnswersLostAt.IsZero() {
return
}
h.work.Lock()
defer h.work.Unlock()
h.mu.Lock()
defer h.mu.Unlock()
h.init()
done := h.shows[kind].Equal(s.AnswersLostAt)
if !s.AnswersLostAt.After(h.passes[kind].Lost) || !s.AnswersLostAt.After(h.lost[kind]) {
return // acted on, or noted already
}
h.lost[kind] = s.AnswersLostAt
}
// duePasses makes the pass of each kind whose loss of answers stood for LostSettle, once per loss.
func (h *Holder) duePasses(now time.Time) {
h.mu.Lock()
due := map[string]time.Time{}
for kind, lost := range h.lost {
if now.Sub(lost) >= LostSettle {
due[kind] = lost
}
}
h.mu.Unlock()
if done {
return
}
ch := h.channelOf(kind)
if ch == nil {
return // the controller's word on the kinds is not read yet: acted on at a later standing
}
if err := ch.Ready(); err != nil {
h.Logf("[messenger] %s lost its answers at %s and cannot send (%v): what is open there is said again once it can",
kind, s.AnswersLostAt.UTC().Format(time.RFC3339), err)
return
}
if h.reoffer(ch, s.AnswersLostAt) {
for kind, lost := range due {
ch := h.channelOf(kind)
if ch == nil {
continue // the controller's word on the kinds is not read yet: acted on later
}
if err := ch.Ready(); err != nil {
h.mu.Lock()
said := h.saidOnce["lost."+kind]
h.mu.Unlock()
if said.IsZero() || now.Sub(said) >= 10*time.Minute {
h.Logf("[messenger] %s lost its answers at %s and cannot send (%v): what is open there is said again once it can",
kind, lost.UTC().Format(time.RFC3339), err)
h.mu.Lock()
h.saidOnce["lost."+kind] = now
h.mu.Unlock()
}
continue
}
h.mu.Lock()
h.shows[kind] = s.AnswersLostAt
tried := h.saidOnce["lost-try."+kind]
h.mu.Unlock()
if !tried.IsZero() && now.Sub(tried) < RetryEvery {
continue
}
if !h.reoffer(ch, lost) {
h.mu.Lock()
h.saidOnce["lost-try."+kind] = now // what failed is tried again in a minute
h.mu.Unlock()
continue
}
h.mu.Lock()
delete(h.saidOnce, "lost-try."+kind)
h.mu.Unlock()
h.mu.Lock()
h.passes[kind] = Pass{Lost: lost, At: now}
if h.lost[kind].Equal(lost) {
delete(h.lost, kind)
}
passes := map[string]Pass{}
for k, p := range h.passes {
passes[k] = p
}
h.mu.Unlock()
if k, ok := h.Store.(walkKeeper); ok {
if err := k.PutPasses(passes); err != nil {
h.noteStore(err)
}
}
}
}
@@ -68,9 +139,9 @@ func (h *Holder) channelOf(kind string) Channel {
return nil
}
// reoffer says again on one channel, whose answers were lost at `lost`, what is still open and asked there,
// edits what is silenced to say so, and leaves what ended alone. It answers whether every message to be
// said again was.
// reoffer says again on one channel, whose answers were lost at `lost`, what is still open, asked there and
// never answered, and leaves everything else as it is: what ended, what is silenced, what was withheld for its
// content, and what was answered (ADR 0308 §6). It answers whether every message to be said again was.
func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
kind := ch.Name()
rp, canReplace := ch.(replacer)
@@ -82,13 +153,14 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
}
h.mu.Unlock()
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
again, silenced, ended, failed := 0, 0, 0, 0
again, left, ended, failed := 0, 0, 0, 0
for _, r := range recs {
h.mu.Lock()
id, shown := r.Sent[kind]
open := r.Cleared.IsZero()
before := lastSaid(r).Before(lost)
reminded := r.Reminded
withheld, answered := r.Refused != "", r.Answered != ""
h.mu.Unlock()
if !shown || id == "" {
continue // never a message of its own there: nothing to say again under
@@ -96,13 +168,8 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
switch {
case !open:
ended++ // its end is said where its message is; nothing comes back
case r.silenced(now):
h.mu.Lock()
r.SilenceSaid = true
h.mu.Unlock()
h.editOn(ch, r, "silenced")
h.persist(r)
silenced++
case r.silenced(now), withheld, answered:
left++ // a silence, a withheld message or an answer is never said again for a restart
case h.askKinds(r)[kind] && before && canReplace:
what := "raised"
if reminded {
@@ -125,28 +192,10 @@ func (h *Holder) reoffer(ch Channel, lost time.Time) bool {
}
}
h.record(Sent{At: now, Channel: kind, Key: "(answers lost)", What: "at " + lost.UTC().Format(time.RFC3339),
Outcome: fmt.Sprintf("%d open and asked said again, %d silenced said so, %d ended left ended, %d failed",
again, silenced, ended, failed)})
h.Logf("[messenger] %s lost its answers at %s: %d open and asked said again in place, %d silenced said so, %d ended "+
"left ended, %d failed; nothing shown from the channel's memory", kind, lost.UTC().Format(time.RFC3339), again,
silenced, ended, failed)
Outcome: fmt.Sprintf("%d open and asked said again, %d silenced, withheld or answered left as they are, "+
"%d ended left ended, %d failed", again, left, ended, failed)})
h.Logf("[messenger] %s lost its answers at %s: %d open and asked said again in place, %d silenced, withheld or "+
"answered left as they are, %d ended left ended, %d failed; nothing shown from the channel's memory", kind,
lost.UTC().Format(time.RFC3339), again, left, ended, failed)
return failed == 0
}
// editOn edits a record's message on one channel, where it is shown as a message of its own.
func (h *Holder) editOn(ch Channel, r *Record, what string) {
kind := ch.Name()
h.mu.Lock()
id := r.Sent[kind]
h.mu.Unlock()
if id == "" || !ch.CanEdit() {
return
}
err := ch.Edit(id, h.sayOn(r, what, kind))
h.noteChannel(kind, err)
if err != nil {
h.Logf("[messenger] could not edit the message for %s on %s (%v)", r.Key, kind, err)
return
}
h.record(Sent{At: h.Now(), Channel: kind, Key: r.Key, What: what, Outcome: "edited"})
}
+16 -13
View File
@@ -12,8 +12,8 @@ import (
// resolved, silenced and cancelled ones among them. Now the desk says in its standing when the answers were
// lost, and this router says again only what is still open and unanswered: with the desk showing one open
// message with an answer, one silenced, one open asking nothing and one resolved, one is said again under its
// handle (in place of its message), with its answers; one is edited to the silenced words, with no answers
// and no urgency; nothing is said for the others. A standing that repeats the time says nothing more; its
// handle (in place of its message), with its answers; nothing is said for the others — the silenced one
// included (novox/hq ADR 0308 §6, issue 531: its edit showed it on the desk again). A standing that repeats the time says nothing more; its
// ticket is one-time still, and the channel that reaches the operator away hears nothing of it.
func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
h, tg, dt, c, _, w := newRouter(t)
@@ -77,8 +77,7 @@ func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
if e := dt.edits["g14=4"]; !strings.HasPrefix(e.Title, "Resolved: ") {
t.Fatalf("the clearing was not said: %+v", e)
}
silencedBefore := dt.edits["g14=2"]
delete(dt.edits, "g14=2") // so the edit said again is seen as such
delete(dt.edits, "g14=2") // so an edit said again would be seen as such
sends, edits, replaced, tgSaid := len(dt.sends), len(dt.edits), len(dt.replaced), saidOn(tg)
// The desk's notifier started again: its standing says when the answers were lost.
c.pass(time.Minute)
@@ -95,12 +94,11 @@ func TestLostAnswersGetOnlyWhatIsOpenAndUnansweredSaidAgain(t *testing.T) {
if ticketFor(again, "Silence for a week") == "" || !strings.Contains(again.Title, headlineOf(open.Key)) {
t.Fatalf("said again without its answer or its words: %+v", again)
}
if e, said := dt.edits["g14=2"]; !said || !strings.HasPrefix(e.Title, "Silenced until") || len(e.Actions) != 0 || e.Urgent ||
!e.Quiet || e.Title != silencedBefore.Title {
t.Fatalf("the silenced one was not edited to the silenced words again: %+v (said %v)", e, said)
if e, said := dt.edits["g14=2"]; said {
t.Fatalf("the silenced one was said again for the restart: %+v", e)
}
if len(dt.edits) != edits+1 {
t.Fatalf("an edit reached a message that is not silenced: %+v", dt.edits)
if len(dt.edits) != edits {
t.Fatalf("an edit reached the desk for the restart: %+v", dt.edits)
}
if e := dt.edits["g14=4"]; !strings.HasPrefix(e.Title, "Resolved: ") {
t.Fatalf("the resolved one was changed: %+v", e)
@@ -179,8 +177,13 @@ func TestLostAnswersAreJudgedPerMessageAndWaitForTheDesk(t *testing.T) {
if len(dt.replaced) != replaced+1 || saidOn(tg) != tgSaid {
t.Fatalf("once the desk can send: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
}
// The router started again, remembering no standing it acted on: the message was said since, so nothing.
h.shows = map[string]time.Time{}
// The router started again, reading back the pass it made from its state: nothing.
h.mu.Lock()
h.passes, h.lost = map[string]Pass{}, map[string]time.Time{}
if err := h.loadWalks(); err != nil {
t.Fatal(err)
}
h.mu.Unlock()
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
runFor(h, c, time.Minute)
if len(dt.replaced) != replaced+1 {
@@ -191,12 +194,12 @@ func TestLostAnswersAreJudgedPerMessageAndWaitForTheDesk(t *testing.T) {
lost = c.now()
dt.fail = errorf("no session answers")
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
runFor(h, c, 30*time.Second)
dt.fail = nil
runFor(h, c, time.Minute)
if len(dt.replaced) != replaced+1 || saidOn(tg) != tgSaid {
t.Fatalf("after a failed send: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
}
h.Stand("desktop", Standing{Ready: true, At: c.now(), AnswersLostAt: lost})
runFor(h, c, time.Minute)
if len(dt.replaced) != replaced+2 || saidOn(tg) != tgSaid {
t.Fatalf("the failed send was not tried again on the desk: desk %+v, telegram %+v", dt.replaced[replaced:], tg.sends)
}
+9 -3
View File
@@ -113,6 +113,9 @@ func (h *Holder) Status(listening string) Status {
"deduplicated by the condition's key; reminded once after 1 h (urgent) or 12 h (warning); cleared by editing the first message",
"at most 20 messages an hour per channel, a last line of defence: reached, it is said once and the rest held for one digest",
"refused: anything carrying an address, a path or a secret; channel-refused is sent with the words withheld",
"while a delivery walks, a message about a machine or module it touches is held, and the walk's end is one message: its outcome, what it reached, how long, what is still open; past the walk's bound (twice its largest delivery budget, 15 min with none) what is held and open is said (novox/hq ADR 0308)",
"an ask, a secret given or replaced, and a walk's own failure are never held",
"a channel that lost its answers is said again only what is open, asked there and never answered — never a silenced or withheld message — once per restart, and the pass is kept in this module's state",
"a silenced condition's message says it is silenced, until when and why, and nothing more is said of it while the silence lasts; a silence is the controller's, asked of the operator or set through the mesh",
}
anyCan := false
@@ -152,9 +155,11 @@ type OpenMessage struct {
Reminded bool `json:"reminded"`
Unsent string `json:"unsent,omitempty"`
Held bool `json:"held,omitempty"`
History bool `json:"never_said,omitempty"`
Refused string `json:"refused,omitempty"`
Count int `json:"times_opened"`
// HeldBy are the walks holding it until they end or pass their bound (novox/hq ADR 0308).
HeldBy []string `json:"held_by_walks,omitempty"`
History bool `json:"never_said,omitempty"`
Refused string `json:"refused,omitempty"`
Count int `json:"times_opened"`
}
// Open is what is unresolved now, urgent first, oldest first.
@@ -180,6 +185,7 @@ func (h *Holder) Open() []OpenMessage {
o.SentOn = append(o.SentOn, ch)
}
sort.Strings(o.SentOn)
o.HeldBy = h.heldBy(r.Key, now)
if r.silenced(now) {
o.Silenced = r.SilencedTill.UTC().Format(time.RFC3339)
}

Some files were not shown because too many files have changed in this diff Show More