Compare commits

Author SHA1 Message Date
jochen aed7e0c968 Read the mesh's comment verdicts as approvals, and warn on no fix timed from the log (issue 403, review 69)
mesh/merge-gate pass: builds mesh-delivery, mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery ready: it delivers once merged
mesh/delivery-group group fix/403-fix-time ready: every member ready, and composed together they pass
The forge refuses an approval of one's own pull request, so the independent
reviews are comment reviews opening with "Verdict: approve at <sha>"; read
only APPROVED, the review phase was never measured. A fix whose start was
read back from the log was fixed before fix times were measured: it is kept
and shown, never warned on. A carried value at fixing after one at diagnosing
starts at its worker's active (the move), not its since (the diagnosis).
2026-10-12 16:16:04 +02:00
jochen 43df7ebc5a Measure each fix from the move to fixing to delivered, and say one over its budget (issue 403)
mesh/merge-gate pass: builds mesh-delivery, mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery-group group fix/403-fix-time ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
hq ADR 0313: the operator's five minutes start when an agent moves the issue
to fixing, not at the merge. mesh-delivery says on each transition, and in
checks, when the pull request opened, when it merged and the class of what it
moves. mesh-issues keeps the first move to fixing, the fix time with its four
phases once the last fix is delivered, serves fix-times, and says
issue-tracker.fix.<class>.over-budget on the operator's channel, cleared by the
next fix of that class within budget.
2026-10-12 15:50:12 +02:00
mesh-admin c5e1bc7bb4 Merge pull request 'mesh-issues: a resolved issue ends its fix-stuck warning, and a closed issue's fix is never said stuck (hq issue 547)' (#239) from fix/547-resolved-ends-stuck into main 2026-10-12 13:21:49 +00:00
mesh-admin 6190b40175 Merge pull request 'artifact-store-tools: an image a definition pins, under any name the store has had, is named by a definition (hq issue 323)' (#249) from fix/323-named-by-a-definition-2 into main 2026-10-12 13:08:31 +00:00
mesh-admin edf39a3b74 Merge pull request 'betterbird: Betterbird from its pinned release, Owl by policy, accounts declared without credentials and adopted, and a reminder test that is visibly a test (hq issue 345, ADR 0316, ADR 0317)' (#240) from fix/345-betterbird-module into main 2026-10-12 12:54:53 +00:00
jochen 15983e2f96 betterbird: an account's own aliases and identities that do not send by SMTP share no outgoing server, and the pins name their vendors on purpose (review 36, review 35 of mesh-controller #251)
mesh/merge-gate pass: builds betterbird, sent nowhere; no bus step; 1 wait(s) for a person; every machine composes with the change as it did without (4 of …
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
The shared-server check refused an adopted account for its own alias, and
counted Owl's, local folders' and feeds' identities as users of the default
server, though they never send by SMTP. The controller now judges an
upstream pin's host like any name, so each pin declares its vendor.
2026-10-12 14:49:21 +02:00
jochen 263c86d2e5 betterbird: withhold account values in the user.js answer, never write a shared outgoing server, and say a newer Owl as a fact (review 34, hq issue 345, ADR 0317)
The user.js answer carried declared addresses and hosts (ADR 0316 §16). An
adopted identity's outgoing server may be shared with an account the module
does not own, whose sending the declared values would change at every start:
refused with the remedy, since ADR 0317 §2 leaves the identity's server and
the lists as they are; a recorded adoption follows the identity's current
server. Owl installed by policy cannot be removed, so a newer Owl than the
pin is a fact whose remedy is moving the pin.
2026-10-12 14:49:21 +02:00
jochen ec4ab7afe7 betterbird: install Betterbird from its pinned release, ship Owl by policy, and declare the accounts without credentials (hq issue 345, ADR 0316, ADR 0317)
A module that cannot set Betterbird up is of little use (the operator,
2026-10-12). The module now declares the community package absent and
unpacks the pinned upstream release where that package had it, so the same
profile opens; places the enterprise policy that turns Betterbird's updater
off and force-installs Owl from the xpi the mesh pinned by its vendor's
digest (never in a repository); and takes the operator's accounts from the
trusted setting accounts, without any password or token: an account the
profile already holds is adopted under its own keys, a new one is created
under the module's mesh_ keys, an ambiguous one is refused with the remedy,
and the account lists are only written into. betterbird_accounts says the
profile's accounts by count and proposes the setting built from them for the
operator's Approve. Every new build waits for the operator's push. The
reminder test is visibly a test (MESH TEST band, event name) and its
description says it opens a window on the operator's screen.
2026-10-12 14:49:21 +02:00
jochen 0e6f382bb7 betterbird: keep the reminder test's update check off so no About window takes the screen (hq issue 345)
A fresh profile has never checked for updates, so Betterbird checked at the
test's start, found a newer build published and opened its About window on the
operator's screen instead of leaving the reminder to show. The test's own
preferences now also come last in its user.js, so nothing declared or carried
overrides them.
2026-10-12 14:49:21 +02:00
jochen d745a4d31e artifact-store-tools: say an image a definition pins is named by a definition, under any name the store has had (issue 323)
mesh/merge-gate pass: builds artifact-store-tools → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
No record names an image the mesh did not make, so one a definition pins under an
older name of the store read as unrecorded, the same word as an image nothing
uses. The tool now reads every definition through the catalogue and says
named-by-a-definition with the module. The store's older names are the mesh's
setting older-store-names, since a catalogue names no installation.
2026-10-12 14:47:06 +02:00
mesh-admin 2772120758 Merge pull request 'mesh-delivery: say how many owed effects were tried since the start (hq issue 407)' (#246) from fix/407-owed-effects-are-counted into main 2026-10-12 12:23:36 +00:00
mesh-admin 13a7d40ac5 Merge pull request 'mesh-knowledge: a record may change a drafted clause, and a record whose clauses others link by number is never redrafted (hq issue 494, ADR 0321)' (#245) from fix/494-change-a-drafted-clause into main 2026-10-12 12:18:34 +00:00
jschoubben a3ad3c5061 Count the owed effects mesh-delivery tried, so a window's writes can be judged against them
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
delivery_status said only what is owed now; issue 326's rule needs the tries made in a window.
It now says the tries since this owner started, by how each ended, and the last try's time.
(novox/hq issue 407)
2026-10-12 14:16:58 +02:00
mesh-admin 5b75499ea6 Merge pull request 'mesh-delivery: say an unanswered line's delivery state as data (hq issue 451)' (#243) from fix/451-a-recheck-is-the-step into main 2026-10-12 12:14:45 +00:00
mesh-admin 0366208274 Merge pull request 'nats: say whether a user's grant is in force on the running server (review of hq issue 353)' (#166) from feat/353-the-bus-says-a-grant-is-in-force into main 2026-10-12 12:13:25 +00:00
jochen 9f893b6305 mesh-knowledge: let a record change a drafted clause, and never redraft a record whose clauses others link by number
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A changes link onto a drafted clause was refused, so records that really
change such a rule could only cite it (ADR 0294, 0301, 0320); and return
renumbered ADR 0281's clauses, pointing other records' links at other
clauses. The changed drafted clause keeps its number and stays unserved;
return refuses a record whose clauses another record changed or links by
number. Issue 494, ADR 0321.
2026-10-12 14:05:37 +02:00
mesh-admin d1baca9b94 Merge pull request 'mesh-delivery: a stop a later merge covered reads covered, and stays in deliveries while stalled lists it (hq issue 480)' (#242) from fix/480-a-covered-stop-reads-covered into main 2026-10-12 11:56:09 +00:00
mesh-admin 3a5c86b5d6 Merge pull request 'messenger: a replaced ask's own message ends with it, and a cleared record owes nothing (hq issue 548)' (#241) from fix/548-replaced-ask-message into main 2026-10-12 11:50:29 +00:00
jschoubben befe451f6d mesh-delivery: say an unanswered line's delivery state as data (hq issue 451)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
mesh/delivery-group group fix/451-a-recheck-is-the-step delivered: every member is delivered
The controller advises a recheck for a ready or rejected delivery whose
check answered once, and must read that state from the line, not its words.
2026-10-12 13:47:52 +02:00
jschoubben 7ad54356fd Say in the deliveries verb's description that a stop stalled lists stays listed
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
2026-10-12 13:46:55 +02:00
jschoubben 69f76a53e7 Read a stop a later merge covered as covered, and keep it in deliveries while stalled lists it
A stop for a walk that never built its modules (issue 474) kept saying "error: stopped" on the forge and in
show after a later delivered merge carried its change to every machine. The stop stays final, as a covered
superseded member stays superseded (issue 442); its status, view, note and show now say covered and by which
merge, told once and from what is kept, so a restart or a stop covered before this tells it too. deliveries
dropped such a stop after a day while stalled still listed it; it now lists it as long as stalled does.

novox/hq issue 480, points 1 and 3.
2026-10-12 13:46:23 +02:00
mesh-admin b82aa1691c Merge pull request 'mesh-knowledge: the review asks changes for a decision nothing would cite, so its acceptance never turns the module unhealthy (hq issue 504)' (#237) from fix/504-propose-refuses-uncited-record into main 2026-10-12 11:44:58 +00:00
jochen 7e27259a64 messenger: end a replaced ask's own message, and drop what a cleared record owed (issue 548)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A replaced ask's message was left open as if the newer ask took it over, but
the newer ask opens a message of its own, so the old one kept offering answers
nobody takes. It now ends on every channel with 'Replaced by a newer question',
and the minute's sweep ends any such message left from before. A record that
clears no longer keeps what a walk owed of it (the review of PR 238).
2026-10-12 13:44:08 +02:00
mesh-admin 3fe19c9c0e Merge pull request 'messenger: a walk's held messages are told only once its message was sent, handed on only as a first hold would take them, and its start and end compared on the controller's clock (hq issue 538)' (#238) from fix/538-walk-hold-told-handover-clock into main 2026-10-12 11:41:47 +00:00
jochen 424fc5a281 messenger: keep what a walk held owed in the state until its message was sent, so a restart says a held warning too (issue 538, review 39)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A warning held on purpose and lost with the walk's message at a restart was turned
into history and never told. Name the walk in the told line from its own item.
2026-10-12 13:35:48 +02:00
jochen f067fab9db mesh-issues: end a fix-stuck warning when its issue is resolved, and never say a closed issue's fix stuck (issue 547)
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A resolved issue's worker proved the acceptance live, so a fix whose own walk failed
and that a later merge carried is not waited for. Only wontfix ended the warning, so
16 warnings stayed open on the operator's channel and the sweep raised new ones for
resolved issues.
2026-10-12 13:31:51 +02:00
jochen 56a9a1b082 mesh-knowledge README: say a failed whole-set rule is a warning to the operator and leaves the module healthy (hq issue 504)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
2026-10-12 13:27:06 +02:00
jochen df94be0e60 messenger: mark a walk's held messages told only once its message was sent, hand on only what a first hold would take, compare a walk's start and end on the controller's clock (issue 538)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A message lost with a restart before it was sent left what it carried marked told,
so it reached the operator only as its reminder hours later. A later walk took over
silenced, already said or asked messages that ADR 0308 section 3 never holds. A start
capped at the router's now was compared with the controller's moments.
2026-10-12 13:26:41 +02:00
jochen 8f5e8bcb6e mesh-knowledge: tell a failed whole-set rule to the operator as a warning naming the records, and leave the module healthy (hq issue 504)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The operator decided on 2026-10-12 that a rule of the record set that fails
is a fault of the records, which their authors repair, not the module
failing: as issue 508 decided for a proposal never reviewed, health says
only whether the module works, and a finding must not hold the module's
own rollouts. The findings go through the notice the knowledge base
already sends (operator-channel.notify, a warning), once per finding,
kept as told across restarts, and cleared when the rule holds again.
2026-10-12 13:25:09 +02:00
jochen ca3c3e6dfd mesh-knowledge tests: set the local zone once before any goroutine, not under a running bus server (race found by the repository check on #237)
inZone wrote time.Local while the bus test's nats server read it from a
goroutine that outlives its test; the race detector the repository check
now runs failed TestAReviewOwedOutlivesARestartAndItsLastTryIsSaid.
2026-10-12 13:25:09 +02:00
jochen 2051d8b80b mesh-knowledge: ask changes in the review for a decision nothing would cite, so its acceptance never turns the module unhealthy (hq issue 504)
The whole-set rule cited ran only after acceptance: the review approved a
decision with no home and no record linking it, the rule then failed, and
the module's health turned red for a fault its author could only repair
after the fact. hq held the rule at the record's own merge. The review now
judges it ahead, before the judgement: such a decision stays proposed with
a changes finding naming the rule. It is not refused at propose, because
no record can link a number only reserved.
2026-10-12 13:25:09 +02:00
mesh-admin 3e84581927 Merge pull request 'messenger: hold what a delivery walk touches, tell its end once, and say nothing again for a restart (hq ADR 0308, issues 532, 531)' (#230) from adr-0308-hold-during-walks into main 2026-10-12 11:21:16 +00:00
mesh-admin 4d9cb0827f Merge pull request 'gitea forge bundle: the parent the forge used, protection rules as the forge matches them, size and time bounds, one HTTP client (issue 539)' (#236) from fix/539-forge-commit-hardening into main 2026-10-12 10:58:07 +00:00
jochen ac90b35b49 guard.ts: cover every branch for a rule Go's regexp reads otherwise, and hold both matchers to one table (issue 539)
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
Review 30 found guard.ts missing branches the forge protects: Go reads \d, \w,
\pL, a class opening with ] or ^, and [[:digit:]] otherwise than JavaScript.
A rule using any of them now covers every branch, so the guard refuses more
than the forge, never less. test/rule-cases.json holds the forge's answers,
read by the Go and TypeScript tests alike.
2026-10-12 12:51:22 +02:00
jochen f2d765320a gitea forge bundle: name the parent the forge used, match protection rules as the forge does, bound a commit's size and a search's time (issue 539)
The forge's contents and diffpatch endpoints take a branch, never a commit to
build on, so the parent is read from the commit the forge made, and a branch
that moved under the commit is said. Blobs are read at the commit read first,
not at the branch name. Protection rules match as Gitea 1.27 matches them
([...], {a,b}, escapes, a plain name in any case), in the Go bundle and in
guard.ts alike. One commit carries at most 4 MiB per file or patch and 16 MiB
together; code search stops at its deadline and says so. The HTTP client the
two Go bundles copied is one package, internal/forgehttp.
2026-10-12 12:51:22 +02:00
mesh-admin c5abdf35ed Merge pull request 'A module whose tests could not run fails the repository check (issue 385)' (#231) from fix/385-an-untestable-module-fails-the-check into main 2026-10-12 10:48:11 +00:00
mesh-admin 9d6bd2d475 Merge pull request 'gitea: a recheck's pending goes through the one writer of mesh/merge-gate (issue 456)' (#234) from fix/456-a-recheck-writes-through-the-one-writer into main 2026-10-12 10:32:00 +00:00
mesh-admin d06733c490 Merge pull request 'mesh-knowledge: the bus test passes on any day (issue 540)' (#235) from fix/540-knowledge-test-clock into main 2026-10-12 10:31:18 +00:00
jochen 7628b0e5b9 mesh-knowledge: give the bus test's bundle a fixed clock, so it passes on any day (issue 540)
mesh/delivery delivered
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The bundle dated a reservation from the real day while the proposal kept the
fixture's 2026-10-11, so the date rule refused it on every later day.
2026-10-12 12:23:38 +02:00
jschoubben 5053e96d4e Route a recheck's pending through the gitea module's one writer of mesh/merge-gate
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
The recheck's pending was written from the tools' bundle, its own process, past the
writer's per-head chain: the first run's verdict mark stayed set, so a rechecked head
never said where its new run builds, and an old run's verdict in flight or redelivered
could land over the pending. The status tool is now served from the watcher's bundle
and writes the merge check's pending through the writer, and a verdict whose ask
(checked.id) predates the check now pending is dropped, weighed after a restart against
the forge's own history of the gate (novox/hq issue 456).
2026-10-12 12:20:23 +02:00
mesh-admin b707f27911 Merge pull request 'gitea: an agent's commit on a branch of its own, and the reads around it (issue 375)' (#233) from fix/375-a-gitea-commit-tool into main 2026-10-12 10:14:05 +00:00
jschoubben 4fdb047cc8 nats: say whether a user's grant is in force on the running server, not only written (hq issue 353)
mesh/merge-gate pass: builds nats, sent nowhere; merging this needs a person's bus upgrade; 1 wait(s) for a person; every machine composes with the change …
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
nats_user_can read the user list on the machine. A list written is not a list the bus enforces until
the server reloads it: on 2026-10-09 the controller asked between the write and the reload's effect.
The answer now carries in_force — allowed in the file, and the server's config_load_time after the
file's write — with why when it is not, and anything not known is not in force.
2026-10-12 12:13:03 +02:00
jochen a92c08fd5b messenger: keep what a walk held until it is said, and open no walk whose end came first (review of #230)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
- No channel ready at a walk's end or bound: what it held is left unsent and tried
  again every minute, as any other message, instead of lost with the walk.
- At a walk's end or bound, a message another open walk touches is held on by
  that walk, even one started after it was held (hq ADR 0308 §2).
- A walk read back past its forget is released at its bound first, then forgotten.
- A walk's end is kept a day: a start heard after it (same started_at, or not
  later than the end) opens nothing, also after the router restarts; a later
  start under the same id still opens.
- A walk's plain end, when its words cannot be said, leaves what it held to be
  said on its own instead of marking it told.
- The restart pass no longer skips a held message: one with an open ask is never
  held (ADR 0308 §3), so its answers are made to work again.
2026-10-12 12:07:38 +02:00
jochen 9eb2ea8d69 gitea: let an agent put a commit on a branch of its own, and read history, compares, reviews, directories and code (issue 375)
mesh/merge-gate pass: builds gitea → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
An agent could not get a finished change onto the forge without someone
pushing by hand, so the branch / pull request / approval rule could not be
followed. A Go bundle serves gitea_commit, held to gitea_api's rule (never the
default branch, never a protected one), and the summarised reads agents
otherwise made through gitea_api as raw JSON.
2026-10-12 12:06:23 +02:00
jochen de8a866189 Fail the check when a module's tests could not run, and take a pinned SDK from its clone beside (issue 385)
mesh/delivery delivered
mesh/delivery-group group fix/385-an-untestable-module-fails-the-check delivered: every member is delivered
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh, merge-check-typescript.sh): main's version judged it; the change's judges the pull r…
A module whose dependencies could not be fetched was said NOT TESTED and the
repository check passed, so a merge on it would have landed untested code.
The build seat clones the SDK beside every check already: a forge requirement
the module cache lacks is now taken from that clone through a go.mod of its
own, and a module that still cannot be built fails the check by name and why.
A TypeScript module whose dependencies the toolchain lacks fails the same way.
2026-10-12 12:03:50 +02:00
jschoubben 991ffcfda0 messenger: run a walk's bound from its started_at, as the controller's walk events say it (hq ADR 0308 §5)
mesh/merge-gate pass: builds messenger → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 11:56:30 +02:00
jschoubben 2927cb2ceb messenger: hold what a delivery walk touches and tell its end once, and say nothing again for a restart (hq ADR 0308, issues 532, 531)
A delivery's own restarts reached the operator as notifications while it ran, and every restart of the
tool runner popped the four silenced messages up again on the desk, twice. The router now reads the
controller's walk start and end on plan-moved, holds what an open walk touches until its end or bound,
tells the end as one message, and makes the restart pass once per restart from its own state, leaving
silenced, withheld, answered and held messages as they are.
2026-10-12 11:55:35 +02:00
mesh-admin da296cacc6 Merge pull request 'mesh-delivery: give a waiting walk its word or say what holds it (issues 529, 362)' (#229) from fix/529-a-waiting-walk-is-answered-or-named into main 2026-10-12 09:55:20 +00:00
jschoubben 1de89cd702 Name a walk of nothing-to-move merges instead of letting it go (issue 529)
mesh/delivery delivered
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named delivered: every member is delivered
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
The review showed the give-the-word rule skipped a failed or missing check
and group order, and let go moves nothing judged; to-be 47 has no such path.
Such a walk is now said, by stalled with walks, as held by nothing, why.
2026-10-12 11:33:51 +02:00
jschoubben 2bee70a478 Give a waiting walk its word or say what holds it (issues 529, 362)
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group fix/529-a-waiting-walk-is-answered-or-named checking: 0 of 2 member(s) ready
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
A merge whose plan sends nothing is delivered at once, yet the controller
can still open a walk for it; no live delivery then held the walk and it
waited for a person (the nzbgeek case). Give such a walk its word.

Name each held delivery in the status, and let stalled say, for every
waiting walk, what holds it, so the controller's plans line points at the
person's decision instead of a hand start.
2026-10-12 11:24:30 +02:00
mesh-admin 6546c05f6e Merge pull request 'mesh-issues: the issues are kept in the log changes, and the bucket is their fold (hq ADR 0300, issues 501, 502)' (#228) from adr-0300-issues-on-a-log into main 2026-10-12 08:46:10 +00:00
jochen d413da203c mesh-issues: resume an interrupted carry by revision and fault one issue instead of the whole start, so no kept value is lost and the tracker always starts (review of PR 228)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
An open's lost answer is judged by the issue's first entry; an entry that does not
fold faults its issue on start; a value written beside the log is never folded over;
one issue is kept in at most 252 KiB so its whole operation fits a log entry.
2026-10-12 10:35:57 +02:00
jochen d33f3e3f71 mesh-issues: pin mesh-sdk at main 9747f3e, where the log and the history are both merged
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 10:22:19 +02:00
jochen dcc7510cd8 mesh-issues: carry each value the issues bucket kept, read through the runtime's history, so the log begins with all the bucket still knew (hq ADR 0300 §6, ADR 0307)
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 10:20:51 +02:00
jochen 39da2cd793 mesh-issues: keep the issues bucket at 8 values until the carry can read them, so the bus drops nothing before it is carried (hq ADR 0300 §6)
mesh/merge-gate fail: builds mesh-issues → novox; no bus step; a manifest the change touches fails the module check: modules/mesh-issues/module.json: this …
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 02:43:58 +02:00
jochen 983d434623 mesh-issues: keep every issue in the log changes and the bucket as its fold, so the record outlives the bucket's cap and a week of events (hq ADR 0300, issues 501, 502)
mesh/merge-gate fail: builds mesh-issues → novox; no bus step; a manifest the change touches fails the module check: modules/mesh-issues/module.json: this …
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery superseded: a newer head of the same pull request
Every write appends one operation at the issue's last sequence before the
bucket is written; a refused append writes nothing. The start carries what
the bucket holds, refolds what is behind and indexes every issue in memory,
so list answers from memory with a ranked search and a before cursor.
2026-10-12 02:41:54 +02:00
mesh-admin 8a92a47bf7 Merge pull request 'Hold a leaf delivery to three minutes in times (hq ADR 0304 decision 4, issue 525)' (#227) from feat/525-leaf-budget-three-minutes into main 2026-10-12 00:02:25 +00:00
jschoubben 2711e812c8 Hold a leaf delivery to three minutes in times (hq ADR 0304 decision 4, issue 525)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
mesh/delivery-group group feat/525-leaf-budget-three-minutes delivered: every member is delivered
ADR 0304 lowered a leaf module's delivery budget from five minutes to
three; times says each class against the controller's same budget.
2026-10-12 01:45:14 +02:00
mesh-admin a0fb555f79 Merge pull request 'Read the tunnel through sudo where the tool runner is not root (hq issue 522)' (#226) from fix/522-tunnel-escalates into main 2026-10-11 22:07:54 +00:00
jschoubben b1280f81a2 Read the tunnel through sudo where the tool runner is not root (hq issue 522)
mesh/merge-gate pass: builds netcheck → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
On novox the tool runner runs as the operator's account, and wg show
answered 'Operation not permitted'. Root is the module's concern: the
command runs as given when the process is root, else through sudo -n,
which the sudo module allows the runtime's account, as nftables does.
2026-10-12 00:05:04 +02:00
mesh-admin d2dff99bcf Merge pull request 'Show the tunnel's peers as each machine sees them, and say on the bus when they move (hq issue 522)' (#225) from feat/522-tunnel-peers into main 2026-10-11 21:55:14 +00:00
jschoubben c31e29f9a6 Keep watching the tunnel after a failed look, and remember peers' names
mesh/merge-gate pass: builds netcheck → ace, g14; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
2026-10-11 23:51:40 +02:00
jschoubben 4fc769ed29 Show the tunnel's peers as each machine sees them, and say on the bus when they move (hq issue 522)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds netcheck → ace, g14; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
ace's tunnel leg lost most packets while its own check called the tunnel
healthy, and nothing on the mesh could say which address and port the
hub saw for each peer, which is what tells a home router's port mix-up
from anything else. netcheck_tunnel answers each peer's endpoint, last
handshake and traffic, never a key, and netcheck now says when a
peer's endpoint changes or its handshake goes stale or fresh again.
2026-10-11 23:50:17 +02:00
mesh-admin 11be8a548a Merge pull request 'mesh-delivery: a stop ends only its own change on a shared walk (hq issue 459, ADR 0299)' (#221) from feat/459-a-stop-re-walks-the-others into main 2026-10-11 19:49:16 +00:00
mesh-admin f7fdb5e87b Merge pull request 'restic: nothing declared has no overdue night; a count's age is judged by awake time (issue 489)' (#224) from fix/489-nothing-declared-is-healthy into main 2026-10-11 19:31:05 +00:00
jschoubben 7f05fad58a restic: judge no night overdue where nothing is declared, and a count's age by the time the machine was awake
mesh/merge-gate pass: builds restic → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A machine that backs up nothing read as missing its night after 26 hours up, and a laptop's count, last
written before it slept, read as stale on waking until the next look (novox/hq issue 489).
2026-10-11 21:10:11 +02:00
mesh-admin fca2f92216 Merge pull request 'mesh-issues: reserve's description says it asks the knowledge base (issue 493)' (#223) from fix/493-reserve-says-what-it-reads into main 2026-10-11 19:07:27 +00:00
jschoubben c0710e2db7 Say that reserve asks the knowledge base for the highest number, not hq (hq issue 493)
mesh/merge-gate pass: builds mesh-issues → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
Since #214 reserve reads knowledge_highest; its description still named hq's folders.
2026-10-11 20:53:21 +02:00
mesh-admin 3fa076829f Merge pull request 'mesh-knowledge, claude-code: a long record is reviewed within a bound sized to it, and a proposal waiting on its author leaves the module healthy (hq issue 508)' (#222) from fix/508-a-long-record-is-reviewed into main 2026-10-11 18:41:53 +00:00
jschoubben fcab25604e Review a long record within a bound sized to it, and let a proposal waiting on its author leave the knowledge base healthy (hq issue 508)
mesh/merge-gate pass: builds claude-code, mesh-knowledge → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (…
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
A judgement over one bus call's 30 seconds is collected by its ticket from
claude-code, so an eleven-clause record can be reviewed; a proposal out of
review tries is told to the operator, not raised as the module's fault, so
it no longer holds the module's own rollouts.
2026-10-11 20:32:33 +02:00
jschoubben 3d7f9a3e66 mesh-delivery: say a shared-module leave-out truly, hold back only what was stopped by an act, and say a held walk where it is read (review of #221)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
mesh/delivery-group group feat/459-a-stop-re-walks-the-others delivered: every member is delivered
A merge left out for sharing a module read as containing the stopped merge; the
safety net also held back merges stranded by their walk, which nobody stopped;
a held walk said why only in the log; and a stop of a merge in a batch did not
ask the batch.
2026-10-11 20:27:55 +02:00
jschoubben 8d7f60c9e2 mesh-delivery: never ship a stopped merge walked again, and stop only once the walk is (review of #221, hq issue 459)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery-group group feat/459-a-stop-re-walks-the-others checking: 2 of 3 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
A merge walked again into a batch and then stopped by its group stayed in that
batch, which was later let go. Any walk or batch that names a stopped
delivery's merge is now asked to leave it out and is never let go; a refused
leave-out stops the whole walk. A person's stop marks the delivery stopped only
after its walk is, and the answer says what stays on the machines and what
will still carry the change.
2026-10-11 20:16:37 +02:00
jschoubben aeab5b80b4 mesh-delivery: a stop ends only its own change on a shared walk (hq issue 459, ADR 0299)
mesh/merge-gate pass: builds mesh-delivery → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery-group group feat/459-a-stop-re-walks-the-others checking: 2 of 3 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
A group stop ended the stopped member's walk, and with it every other delivery
the batch put on that walk. It now asks delivery-stop to leave the stopped
merges out, so the others are walked again; a person's stop on a shared walk
first says what ending the walk would take down and acts only on a way.
2026-10-11 19:59:43 +02:00
mesh-admin 977e7a6fd2 Merge pull request 'mesh-knowledge: a review owed says when it is due, survives a restart, and its last try is told to the operator (hq issue 497)' (#219) from fix/497-a-review-owed-outlives-a-restart into main 2026-10-11 17:49:34 +00:00
jschoubben c394ecd9fd mesh-knowledge: give a never-reviewed notice its resolved line when raised, and test a refused clear and local time in a set zone (review of #219)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
The operator's channel edits a cleared message with the resolved line it was raised with, so one sent only with
the clear was never shown.
2026-10-11 19:41:45 +02:00
jschoubben d45761678a mesh-knowledge: clear a never-reviewed notice once the record moves on, key it by version and run of tries, say times locally (review of #219)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A notice left open kept telling the operator a record was proposed and not in force after it was proposed again
or accepted, and a later version's notice merged into it.
2026-10-11 19:36:14 +02:00
jschoubben 9cfb5f9edb mesh-knowledge: keep when an owed review is due, say it in the history, and tell the operator once after its last try (hq issue 497)
mesh/merge-gate pass: builds mesh-knowledge → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery superseded: a newer head of the same pull request
A failed review's try count was already kept with the version, but not when it was asked again, so a 10-minute
pause read as a review lost to a restart; and after the last try only the history and the health said so.
2026-10-11 19:24:07 +02:00
145 changed files with 17515 additions and 871 deletions
+3
View File
@@ -0,0 +1,3 @@
module git.novox.be/novox/mesh-catalog/checks/deps
go 1.22
+255
View File
@@ -0,0 +1,255 @@
// Command deps makes a module's dependencies available to the catalogue's check, or says why it cannot
// (novox/hq issue 385).
//
// A module's tests run only when its dependencies can be fetched. The build seat clones what a check
// needs with the forge credential and forgets that credential before the check runs, so a module pinning
// an SDK commit the seat's module cache does not hold cannot fetch it from the forge. The SDK is cloned
// beside the check all the same (MESH_CHECK_BESIDE, mesh-controller's besideRefs): every requirement of the
// mesh's own forge is taken from the clone of its repository there, at the commit or the tag its version
// names, through a go.mod of its own (-modfile) — the module's own files are not changed, and no forge
// credential is added.
//
// Usage: deps -beside <dir> -work <dir> <module dir>. It prints the go.mod to build the module with, or
// nothing when the module fetches what it needs; when it cannot, it prints why and exits 1, and the check
// fails: a module whose tests could not run is a failure, never a note.
package main
import (
"archive/tar"
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
)
// forge is the prefix of the Go modules of the mesh's own forge: <forge><repository>[/<directory>].
const forge = "git.novox.be/novox/"
func main() {
beside := flag.String("beside", "", "where the repositories beside the check are cloned (MESH_CHECK_BESIDE)")
work := flag.String("work", os.TempDir(), "where the go.mod and the sources taken from beside are written")
flag.Parse()
if flag.NArg() != 1 {
fmt.Println("usage: deps -beside <dir> -work <dir> <module dir>")
os.Exit(2)
}
modfile, err := resolve(flag.Arg(0), *beside, *work)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
if modfile != "" {
fmt.Println(modfile)
}
}
// resolve is the go.mod the module at dir builds with: "" when its own fetches everything, else one
// beside it in work that takes every requirement of the mesh's forge from its clone beside the check.
func resolve(dir, beside, work string) (string, error) {
out, err := goIn(dir, "mod", "download")
if err == nil {
return "", nil
}
fetching := lastLine(out)
required, err := forgeRequirements(dir)
if err != nil {
return "", err
}
if len(required) == 0 {
return "", fmt.Errorf("its dependencies cannot be fetched: %s", fetching)
}
if beside == "" {
return "", fmt.Errorf("its dependencies cannot be fetched (%s), and nothing was cloned beside the check to "+
"take %s from", fetching, repositories(required))
}
into, err := os.MkdirTemp(work, filepath.Base(dir)+"-")
if err != nil {
return "", err
}
modfile := filepath.Join(into, "go.mod")
if err := copyFile(filepath.Join(dir, "go.mod"), modfile); err != nil {
return "", err
}
if err := copyFile(filepath.Join(dir, "go.sum"), filepath.Join(into, "go.sum")); err != nil &&
!errors.Is(err, os.ErrNotExist) {
return "", err
}
for _, r := range required {
source, err := export(beside, r, into)
if err != nil {
return "", fmt.Errorf("its dependencies cannot be fetched (%s), and %s@%s cannot be taken from the "+
"clone beside the check: %w", fetching, r.Path, r.Version, err)
}
if out, err := goIn(dir, "mod", "edit", "-replace", r.Path+"@"+r.Version+"="+source, modfile); err != nil {
return "", fmt.Errorf("%s@%s cannot be taken from the clone beside the check: %s", r.Path, r.Version,
lastLine(out))
}
}
if out, err := goIn(dir, "mod", "download", "-modfile="+modfile); err != nil {
return "", fmt.Errorf("its dependencies cannot be fetched even with %s taken from the clone beside the "+
"check: %s", repositories(required), lastLine(out))
}
return modfile, nil
}
// requirement is one module the module requires.
type requirement struct {
Path string
Version string
}
// forgeRequirements is every requirement of the module at dir that comes from the mesh's own forge.
func forgeRequirements(dir string) ([]requirement, error) {
out, err := goIn(dir, "mod", "edit", "-json")
if err != nil {
return nil, fmt.Errorf("its go.mod cannot be read: %s", lastLine(out))
}
var mod struct{ Require []requirement }
if err := json.Unmarshal(out, &mod); err != nil {
return nil, fmt.Errorf("its go.mod cannot be read: %w", err)
}
var on []requirement
for _, r := range mod.Require {
if strings.HasPrefix(r.Path, forge) {
on = append(on, r)
}
}
return on, nil
}
// repositoryOf splits a module path of the forge into its repository and the directory in it.
func repositoryOf(path string) (string, string) {
repo, sub, _ := strings.Cut(strings.TrimPrefix(path, forge), "/")
return repo, sub
}
func repositories(rs []requirement) string {
var names []string
for _, r := range rs {
repo, _ := repositoryOf(r.Path)
names = append(names, repo)
}
return strings.Join(names, ", ")
}
// pseudo is a pseudo-version's commit: v0.1.14-0.20261012082113-9747f3e1b1ac → 9747f3e1b1ac.
var pseudo = regexp.MustCompile(`[-.]\d{14}-([0-9a-f]{12})$`)
// refOf is the ref a version names in its repository: a pseudo-version's commit, else the tag a module in
// a directory is released under (<directory>/<version>).
func refOf(sub, version string) string {
version = strings.TrimSuffix(version, "+incompatible")
if m := pseudo.FindStringSubmatch(version); m != nil {
return m[1]
}
if sub != "" {
return sub + "/" + version
}
return version
}
// export writes the module r names, at its version, from its repository's clone beside the check into
// into, and answers its directory.
func export(beside string, r requirement, into string) (string, error) {
repo, sub := repositoryOf(r.Path)
clone := filepath.Join(beside, repo)
if _, err := os.Stat(filepath.Join(clone, ".git")); err != nil {
return "", fmt.Errorf("no clone of %s is beside the check", repo)
}
ref := refOf(sub, r.Version)
commit, err := exec.Command("git", "-C", clone, "rev-parse", "--verify", "--quiet", ref+"^{commit}").Output()
if err != nil {
return "", fmt.Errorf("the clone of %s holds no %s (a commit or tag not pushed to the forge?)", repo, ref)
}
at := strings.TrimSpace(string(commit))
args := []string{"-C", clone, "archive", "--format=tar", at}
if sub != "" {
args = append(args, "--", sub)
}
var stderr bytes.Buffer
archive := exec.Command("git", args...)
archive.Stderr = &stderr
raw, err := archive.Output()
if err != nil {
return "", fmt.Errorf("git archive of %s at %s: %v %s", repo, ref, err, strings.TrimSpace(stderr.String()))
}
root := filepath.Join(into, repo+"@"+at[:12])
if err := untar(raw, root); err != nil {
return "", err
}
source := filepath.Join(root, filepath.FromSlash(sub))
if _, err := os.Stat(filepath.Join(source, "go.mod")); err != nil {
return "", fmt.Errorf("%s at %s has no go.mod in %q", repo, ref, sub)
}
return source, nil
}
// untar writes a tar's directories and regular files under root.
func untar(raw []byte, root string) error {
r := tar.NewReader(bytes.NewReader(raw))
for {
h, err := r.Next()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
name := filepath.Clean(filepath.FromSlash(h.Name))
if name == "." || strings.HasPrefix(name, "..") || filepath.IsAbs(name) {
continue
}
path := filepath.Join(root, name)
switch h.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(path, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
body, err := io.ReadAll(r)
if err != nil {
return err
}
if err := os.WriteFile(path, body, 0o644); err != nil {
return err
}
}
}
}
func copyFile(from, to string) error {
raw, err := os.ReadFile(from)
if err != nil {
return err
}
return os.WriteFile(to, raw, 0o644)
}
// goIn runs the go command in dir.
func goIn(dir string, args ...string) ([]byte, error) {
cmd := exec.Command("go", args...)
cmd.Dir = dir
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.Output()
if err != nil {
return append(out, stderr.Bytes()...), err
}
return out, nil
}
// lastLine is the last line a command said, which is where go says what failed.
func lastLine(out []byte) string {
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
+177
View File
@@ -0,0 +1,177 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// offline is a Go that fetches nothing: what the build seat is once the forge credential is gone, for a pin
// its module cache does not hold (novox/hq issue 385).
func offline(t *testing.T) {
t.Helper()
t.Setenv("GOPROXY", "off")
t.Setenv("GOFLAGS", "")
t.Setenv("GOWORK", "off")
// Not GOPRIVATE: a private module is fetched directly by git, around GOPROXY, with whatever credential
// the machine running the test holds.
t.Setenv("GOPRIVATE", "")
t.Setenv("GONOPROXY", "")
t.Setenv("GOSUMDB", "off")
t.Setenv("GOMODCACHE", t.TempDir())
}
func git(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t",
"GIT_COMMITTER_EMAIL=t@t", "GIT_AUTHOR_DATE=2026-10-12T08:21:13Z", "GIT_COMMITTER_DATE=2026-10-12T08:21:13Z")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
}
return strings.TrimSpace(string(out))
}
func write(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// sdkBeside is the SDK's repository cloned beside a check, its Go module under go/, one commit tagged
// go/v0.1.7. It answers the beside root and the commit.
func sdkBeside(t *testing.T) (string, string) {
t.Helper()
beside := t.TempDir()
repo := filepath.Join(beside, "mesh-sdk")
write(t, filepath.Join(repo, "go", "go.mod"), "module git.novox.be/novox/mesh-sdk/go\n\ngo 1.22\n")
write(t, filepath.Join(repo, "go", "sdk", "sdk.go"), "package sdk\n\nfunc Hello() string { return \"hello\" }\n")
write(t, filepath.Join(repo, "README.md"), "the SDK\n")
git(t, repo, "init", "--quiet")
git(t, repo, "add", ".")
git(t, repo, "commit", "--quiet", "-m", "sdk")
git(t, repo, "tag", "go/v0.1.7")
return beside, git(t, repo, "rev-parse", "HEAD")
}
// moduleRequiring is a catalogue module whose code and test use the SDK at the version given.
func moduleRequiring(t *testing.T, version string) string {
t.Helper()
dir := filepath.Join(t.TempDir(), "messenger")
write(t, filepath.Join(dir, "go.mod"), "module git.novox.be/novox/mesh-catalog/modules/messenger\n\ngo 1.22\n\n"+
"require git.novox.be/novox/mesh-sdk/go "+version+"\n")
write(t, filepath.Join(dir, "main.go"), "package main\n\nimport \"git.novox.be/novox/mesh-sdk/go/sdk\"\n\n"+
"func main() { println(sdk.Hello()) }\n")
return dir
}
func builds(t *testing.T, dir, modfile string) {
t.Helper()
args := []string{"vet"}
if modfile != "" {
args = append(args, "-modfile="+modfile)
}
cmd := exec.Command("go", append(args, "./...")...)
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("the module does not build with %q: %v\n%s", modfile, err, out)
}
}
// A module that needs nothing it cannot fetch is left as it is: no other go.mod.
func TestAModuleThatFetchesItsDependenciesNeedsNothingBeside(t *testing.T) {
offline(t)
dir := filepath.Join(t.TempDir(), "plain")
write(t, filepath.Join(dir, "go.mod"), "module example.com/plain\n\ngo 1.22\n")
write(t, filepath.Join(dir, "main.go"), "package main\n\nfunc main() {}\n")
modfile, err := resolve(dir, "", t.TempDir())
if err != nil || modfile != "" {
t.Fatalf("resolve = %q, %v; want nothing", modfile, err)
}
}
// The case of issue 385: the module pins an SDK commit the build seat cannot fetch. It is taken from the
// SDK's clone beside the check, and the module builds against it.
func TestAPinnedSDKCommitIsTakenFromTheCloneBeside(t *testing.T) {
offline(t)
beside, commit := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261012082113-"+commit[:12])
modfile, err := resolve(dir, beside, t.TempDir())
if err != nil {
t.Fatalf("resolve: %v", err)
}
if modfile == "" {
t.Fatal("resolve said the module fetches its dependencies, with nothing to fetch them from")
}
builds(t, dir, modfile)
if raw, _ := os.ReadFile(filepath.Join(dir, "go.mod")); strings.Contains(string(raw), "replace") {
t.Fatalf("the module's own go.mod was changed:\n%s", raw)
}
}
// A released SDK is found by the tag it is released under: go/<version>.
func TestATaggedSDKIsTakenFromTheCloneBeside(t *testing.T) {
offline(t)
beside, _ := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.7")
modfile, err := resolve(dir, beside, t.TempDir())
if err != nil {
t.Fatalf("resolve: %v", err)
}
builds(t, dir, modfile)
}
// A commit the clone beside does not hold (a branch never pushed) cannot be tested, and says which.
func TestACommitTheCloneBesideDoesNotHoldIsSaid(t *testing.T) {
offline(t)
beside, _ := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261010132341-4f3efc98e0c0")
_, err := resolve(dir, beside, t.TempDir())
if err == nil {
t.Fatal("resolve found a commit nothing holds")
}
for _, want := range []string{"git.novox.be/novox/mesh-sdk/go@v0.1.14-0.20261010132341-4f3efc98e0c0", "4f3efc98e0c0"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the reason %q does not name %s", err, want)
}
}
}
// With nothing cloned beside (a check by hand without the forge), the reason says that.
func TestNoCloneBesideIsSaid(t *testing.T) {
offline(t)
_, commit := sdkBeside(t)
dir := moduleRequiring(t, "v0.1.14-0.20261012082113-"+commit[:12])
for _, beside := range []string{"", t.TempDir()} {
_, err := resolve(dir, beside, t.TempDir())
if err == nil {
t.Fatalf("resolve with %q beside found what is not there", beside)
}
if !strings.Contains(err.Error(), "mesh-sdk") {
t.Errorf("the reason %q does not name the repository it needed", err)
}
}
}
func TestTheRefOfAVersion(t *testing.T) {
cases := []struct{ sub, version, want string }{
{"go", "v0.1.14-0.20261012082113-9747f3e1b1ac", "9747f3e1b1ac"},
{"go", "v0.0.0-20261012082113-9747f3e1b1ac", "9747f3e1b1ac"},
{"go", "v0.1.7", "go/v0.1.7"},
{"", "v1.2.0", "v1.2.0"},
{"", "v2.0.0+incompatible", "v2.0.0"},
}
for _, c := range cases {
if got := refOf(c.sub, c.version); got != c.want {
t.Errorf("refOf(%q, %q) = %q, want %q", c.sub, c.version, got, c.want)
}
}
}
+7 -3
View File
@@ -9,14 +9,17 @@
#
# 1. its dependencies are the toolchain's own (/app/node_modules: the mesh's SDK, Node's types and what the
# runtime carries), copied beside it, never fetched: the check holds no credential for the package
# registry. A module that depends on a package the toolchain does not hold is said NOT CHECKED, by name;
# registry. A module that depends on a package the toolchain does not hold **fails the check**, by name:
# it could not be type-checked or tested, and what could not run is a failure, never a note (novox/hq
# issue 385);
# 2. every source that is not a test is type-checked with the toolchain's compiler and the flags a build
# compiles with — the same check the build makes, before the merge rather than in the walk;
# 3. its tests (test/*.test.ts) run under Node's own runner with types stripped, as its own `npm test` does
# after its build: the sources are compiled to dist/ and beside themselves first, so a test that imports
# the compiled module, or a source that imports its sibling by its .js name, finds it.
#
# Said, never passed silently: a module with no tests is said NOT TESTED on every run that touches it.
# Said, never passed silently: a module with no tests is said NOT TESTED on every run that touches it —
# nothing of it could fail. A module whose tests exist and could not run fails (novox/hq issue 385).
#
# By hand: MESH_CHECK_MODULES=gitea sh merge-check-typescript.sh, in the toolchain image (it reads
# /app/node_modules), or with TOOLCHAIN_MODULES pointing at a node_modules that holds the same.
@@ -54,7 +57,8 @@ for m in $touched; do
require("fs").existsSync(process.argv[2] + "/" + n));
console.log(names.filter((n) => n !== "typescript" && !held(n)).join(" "));' "$copy/package.json" "$deps")
if [ -n "$missing" ]; then
echo "NOT CHECKED: $d depends on $missing, which the toolchain does not hold and the check cannot fetch"
echo "FAILED: $d could not be checked — it depends on $missing, which the toolchain does not hold and the check cannot fetch (novox/hq issue 385)"
failed="$failed $m"
continue
fi
mkdir -p "$copy/node_modules"
+23 -6
View File
@@ -14,8 +14,11 @@
# it shares a rule with refused (MESH_GATE is the controller the mesh runs) — and the count of
# long-running resources without `health` held to the number in health-undeclared, which only goes down;
# 2. the Go tests of every module the change touches that has them, under the race detector when the
# toolchain has a C compiler — and a module whose dependencies cannot be fetched here is said as not
# tested, never passed silently. A module written in TypeScript is type-checked and tested by
# toolchain has a C compiler. A requirement of the mesh's own forge the module cache does not hold (a
# pinned SDK commit) is taken from its repository's clone beside the check (checks/deps); a module whose
# dependencies still cannot be had **fails the check**, by name and why — its tests could not run, and a
# test that could not run is a failure, never a note (novox/hq issue 385). A module written in
# TypeScript is type-checked and tested by
# merge-check-typescript.sh, declared above, in the TypeScript toolchain (novox/hq issues 302, 311);
# 3. the words (novox/hq ADR 0244): no word the glossary retired for the tools, as copied in
# retired-words, in any text a module's tools can show an agent — every string in its own code that is
@@ -77,16 +80,26 @@ if [ -n "${MESH_CHECK_MODULES:-}" ]; then
else
touched=$(printf '%s\n' "${MESH_CHECK_CHANGED:-}" | tr ',' '\n' | sed -n 's#^modules/\([^/]*\)/.*#\1#p' | sort -u)
fi
# A module's dependencies: fetched, or a forge requirement taken from its clone beside the check
# (MESH_CHECK_BESIDE); its tests run against them (checks/deps, novox/hq issue 385). The helper's own test
# runs first.
(cd checks/deps && go test -count=1 .)
deps=$(mktemp -d)
trap 'rm -rf "$deps"' EXIT
(cd checks/deps && go build -o "$deps/deps" .)
untestable=""
for m in $touched; do
[ -d "modules/$m" ] || continue
if [ ! -f "modules/$m/go.mod" ]; then
[ -f "modules/$m/package.json" ] && echo "modules/$m is TypeScript: merge-check-typescript.sh type-checks and tests it"
continue
fi
if ! (cd "modules/$m" && GOPRIVATE=git.novox.be go mod download >/dev/null 2>&1); then
echo "NOT TESTED: modules/$m — its dependencies cannot be fetched by the build seat"
if ! modfile=$(GOPRIVATE=git.novox.be "$deps/deps" -beside "${MESH_CHECK_BESIDE:-}" -work "$deps" "modules/$m"); then
echo "FAILED: modules/$m could not be tested — $modfile"
untestable="$untestable modules/$m"
continue
fi
[ -n "$modfile" ] && echo "modules/$m takes its forge dependencies from the clones beside the check"
echo "testing modules/$m"
unformatted=$(cd "modules/$m" && gofmt -l .)
if [ -n "$unformatted" ]; then
@@ -95,6 +108,10 @@ for m in $touched; do
fi
cgo=0
[ -n "$race" ] && cgo=1
(cd "modules/$m" && CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go vet ./... &&
CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test $race -count=1 ./...)
(cd "modules/$m" && CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go vet ${modfile:+"-modfile=$modfile"} ./... &&
CGO_ENABLED=$cgo GOPRIVATE=git.novox.be go test ${modfile:+"-modfile=$modfile"} $race -count=1 ./...)
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
+22 -4
View File
@@ -1,7 +1,8 @@
# artifact-store-tools
Tools that say what the mesh's artifact store holds, and what the controller's records say of it
(novox/hq ADR 0251 §1–3, to-be 51). It declares no resources and changes nothing on its machine.
(novox/hq ADR 0251 §1–3, to-be 51). It changes nothing on its machine; its only resources are its state
directory and the file the mesh writes there from its setting `older-store-names`.
## Why a module beside the store
@@ -18,7 +19,7 @@ on that machine.
|---|---|---|
| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size |
| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) |
| `artifact_store_references` | r | what the controller's records say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_references` | r | what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` |
### How the store is read
@@ -43,6 +44,7 @@ it lists. The store's collector removes every blob no manifest marks, so these n
| state | what | removable |
|---|---|---|
| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no |
| `named-by-a-definition` | no record names it, and a definition in the catalogue pins it by digest, under the store's current name or an older one; `why` names the module (hq issue 323) | no |
| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no |
| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` |
| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive |
@@ -56,6 +58,20 @@ its own right; the controller lets an index go with the platform manifests only
counted as the index's. Two indexes naming one platform give it the more keeping state. An index no
record names leaves its platform manifests unrecorded with it.
**What the definitions pin** (hq issue 323). A definition may pin an image of the store the mesh did not
make: pushed before the mesh built it, and pinned under the store's current name or a name it had before.
No record of the controller names it, yet a machine runs it, so `artifact_store_references` reads every
definition the catalogue holds (`mesh-catalog.catalog_modules`, then `catalog_module` for each, eight at
a time and within twenty seconds) and finds every `<name>/<repository>@sha256:…` in any of its strings,
under each of the store's names: `artifact-store://`, the store's address as the controller answers it,
and the names it had before. Those are the mesh's setting `older-store-names` (space-separated, `none` by
default, set with the controller's `settings`), which the mesh writes to the module's `store-names` file: a
catalogue names no installation. When they cannot be read, the answer's first line says so. A manifest no record
names that a definition pins is `named-by-a-definition`; one the records name keeps the records' state. A
definition read at its commit is kept, so the next call asks only for the list. Definitions that could
not be read are listed under `definitions.unread`, and the answer's first line says that what they pin
reads as unrecorded until they are.
**Unrecorded copies of upstream images** are listed in `unrecorded_mirrors`: each manifest at the top of
a repository only the build's mirror writes (`upstream/<host>/<path>`, or a module's `<module>/on-<argument>`
from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the
@@ -88,6 +104,8 @@ with slashes, a cut listing refused), manifests read with one absent or unreadab
digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks),
every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept
manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked,
the fall-back when the references let go of cannot be carried, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs.
the fall-back when the references let go of cannot be carried, a manifest a definition pins under the
store's current or an older name, a definition not read said first, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs and the
catalogue's two tools.
Any command other than the two read-only scripts fails the test.
@@ -7,12 +7,15 @@ import (
// The states a manifest the store holds can be in, as the records see it (novox/hq to-be 51).
const (
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
// StateNamedByDefinition is a manifest no record names that a definition pins by digest, under the
// store's current name or an older one (novox/hq issue 323): a machine may run it, so it is kept.
StateNamedByDefinition = "named-by-a-definition"
StateUnrecorded = "unrecorded"
collectionRemovesThese = "eligible and holder-of-eligible-archive, through the controller's collect; nothing else"
)
@@ -45,8 +48,8 @@ var mirrorRepository = regexp.MustCompile(`^(upstream/[a-z0-9][a-z0-9._/-]*|[a-z
// keeping orders the states a platform manifest may take from the indexes naming it: the most keeping
// wins, so a platform two indexes name is as kept as the more kept of them.
var keeping = map[string]int{
StateKept: 0, StateNamedDocument: 1, StateHolderKept: 2, StateEligible: 3, StateHolderEligible: 4,
StateCollectedPresent: 5,
StateKept: 0, StateNamedByDefinition: 1, StateNamedDocument: 2, StateHolderKept: 3, StateEligible: 4,
StateHolderEligible: 5, StateCollectedPresent: 6,
}
// recordIndex finds a record by repository and digest.
@@ -64,8 +67,9 @@ func indexRecords(r *Records) recordIndex {
return idx
}
// Classify gives every manifest the store holds one state.
func Classify(v *View, recs *Records) []Entry {
// Classify gives every manifest the store holds one state. A manifest no record names that a definition
// pins is named by a definition; defs may be nil, when the definitions were not read.
func Classify(v *View, recs *Records, defs *Definitions) []Entry {
idx := indexRecords(recs)
var out []Entry
for _, name := range v.L.RepoNames() {
@@ -99,6 +103,12 @@ func Classify(v *View, recs *Records) []Entry {
e.State = StateNamedDocument
}
}
if e.State == StateUnrecorded && defs != nil {
for _, p := range defs.Pins[Key(name, d)] {
e.State = StateNamedByDefinition
e.Why = append(e.Why, p.why())
}
}
out = append(out, e)
}
}
@@ -0,0 +1,343 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"regexp"
"sort"
"strings"
"sync"
"time"
)
// What the definitions pin in the store (novox/hq issue 323). A definition may pin an image of the store by
// digest that the mesh did not make: one pushed before the mesh built it, pinned under the store's current
// name or under a name the store had before. No record of the controller names such a manifest, and yet a
// machine runs it, so it is said to be named by a definition and never unrecorded.
// NamesFileEnv names the file the mesh writes from the setting older-store-names: a line `older=` with the
// names the store had before, as a definition may still write them — host names, or host:port, separated by
// spaces — or `none`. A catalogue names no installation, so the names are the mesh's setting, not the
// manifest's.
const NamesFileEnv = "MESH_ARTIFACT_STORE_NAMES_FILE"
// CatalogueModules and CatalogueModule are the catalogue's tools these definitions are read through.
const (
CatalogueModules = "mesh-catalog.catalog_modules"
CatalogueModule = "mesh-catalog.catalog_module"
)
// DefinitionsTimeout bounds reading every definition, below the runtime's thirty-second call limit; what
// was read is kept, so a call that runs out reads only the rest the next time.
const DefinitionsTimeout = 20 * time.Second
// definitionReaders is how many definitions are asked at once.
const definitionReaders = 8
// StoreNames are the names a definition may pin an image of the store under: the scheme the mesh writes
// (artifact-store://), the store's address now, and the names it had before.
type StoreNames struct {
Current []string
Older []string
}
// NamesOf is the store's names: its address as the controller answers it, and the older ones given.
func NamesOf(address, older string) StoreNames {
n := StoreNames{Current: []string{"artifact-store://"}}
if a := strings.Trim(strings.TrimSpace(address), "/"); a != "" {
n.Current = append(n.Current, a+"/")
}
for _, o := range strings.FieldsFunc(older, func(r rune) bool { return r == ',' || r == ' ' || r == '\n' || r == '\t' }) {
if o = strings.Trim(o, "/"); o != "" && o != "none" {
n.Older = append(n.Older, o+"/")
}
}
return n
}
// OlderNames reads the older names from the file the mesh writes, and says why when it cannot.
func OlderNames() (string, string) {
path := os.Getenv(NamesFileEnv)
if path == "" {
return "", NamesFileEnv + " is not set, so no older name of the store was read"
}
raw, err := os.ReadFile(path)
if err != nil {
return "", "the store's older names could not be read: " + err.Error()
}
for _, line := range strings.Split(string(raw), "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "older="); ok {
return v, ""
}
}
return "", path + " holds no line older=, so no older name of the store was read"
}
// Pin is one definition pinning a manifest of the store by digest.
type Pin struct {
Module string `json:"module"`
// Name is the store's name it is pinned under, as written: "artifact-store://", an address, or an older name.
Name string `json:"name"`
Older bool `json:"older,omitempty"`
}
// pinned is one pin found in a manifest: where in the store, and under which name.
type pinned struct {
Repository, Digest, Name string
Older bool
}
var pinShapes sync.Map // name -> *regexp.Regexp
// pinShape is `<name><repository>@sha256:<hex>`, the name standing on its own: not the tail of a longer host.
func pinShape(name string) *regexp.Regexp {
if re, ok := pinShapes.Load(name); ok {
return re.(*regexp.Regexp)
}
re := regexp.MustCompile(`(?:^|[^A-Za-z0-9.-])` + regexp.QuoteMeta(name) +
`([a-z0-9][a-z0-9._/-]*)@(sha256:[0-9a-f]{64})`)
pinShapes.Store(name, re)
return re
}
// PinsIn finds every image of the store a manifest pins by digest, in any string it holds: a resource's
// image, a build's base, a file's content.
func PinsIn(manifest json.RawMessage, names StoreNames) []pinned {
var doc any
if json.Unmarshal(manifest, &doc) != nil {
return nil
}
seen := map[string]bool{}
var out []pinned
var walk func(any)
walk = func(x any) {
switch v := x.(type) {
case map[string]any:
for _, e := range v {
walk(e)
}
case []any:
for _, e := range v {
walk(e)
}
case string:
for _, group := range []struct {
names []string
older bool
}{{names.Current, false}, {names.Older, true}} {
for _, name := range group.names {
for _, m := range pinShape(name).FindAllStringSubmatch(v, -1) {
p := pinned{Repository: strings.Trim(m[1], "/"), Digest: m[2], Name: name, Older: group.older}
if k := p.Repository + "@" + p.Digest + " " + p.Name; !seen[k] {
seen[k] = true
out = append(out, p)
}
}
}
}
}
}
walk(doc)
return out
}
// Definitions is what the catalogue's definitions pin in the store, and what of them could not be read.
type Definitions struct {
// Pins are by Key(repository, digest).
Pins map[string][]Pin
// Read is how many definitions were read; Unread says, per module, why its definition was not.
Read int
Unread map[string]string
// NotAsked says why the catalogue could not be asked at all, when it could not.
NotAsked string
// NamesNotRead says why the store's older names could not be read, when they could not.
NamesNotRead string
Names StoreNames
}
// Catalogue reads the definitions through the catalogue's tools, and keeps each definition it read by its
// module and commit, which never change together.
type Catalogue struct {
Ask Asking
mu sync.Mutex
kept map[string][]pinned // "<module>@<commit>|<names>" -> its pins
}
type catalogueEntry struct {
Module string `json:"module"`
Commit string `json:"commit"`
}
// Read asks the catalogue for every module and reads each definition, bounded by DefinitionsTimeout.
func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
defs := &Definitions{Pins: map[string][]Pin{}, Unread: map[string]string{}, Names: names}
if c == nil || c.Ask == nil {
defs.NotAsked = "this bundle has no way to ask the catalogue"
return defs
}
raw, err := c.Ask(CatalogueModules, map[string]any{})
if err != nil {
defs.NotAsked = "the catalogue's modules could not be asked: " + err.Error()
return defs
}
answer, output, ok := answerOf(raw)
if !ok {
defs.NotAsked = "the catalogue refused its modules: " + lastLine(output)
return defs
}
var list struct {
Modules []catalogueEntry `json:"modules"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &list); err != nil || list.Error != "" || list.Modules == nil {
defs.NotAsked = "the catalogue's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
return defs
}
c.mu.Lock()
if c.kept == nil {
c.kept = map[string][]pinned{}
}
c.mu.Unlock()
nameKey := strings.Join(names.Current, ",") + ";" + strings.Join(names.Older, ",")
ctx, cancel := context.WithTimeout(ctx, DefinitionsTimeout)
defer cancel()
type result struct {
module string
pins []pinned
why string
}
results := make(chan result, len(list.Modules))
work := make(chan catalogueEntry)
var wg sync.WaitGroup
for i := 0; i < definitionReaders; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for m := range work {
pins, why := c.definition(m, names, nameKey)
results <- result{m.Module, pins, why}
}
}()
}
go func() {
defer close(work)
for _, m := range list.Modules {
select {
case work <- m:
case <-ctx.Done():
return
}
}
}()
go func() { wg.Wait(); close(results) }()
got := map[string]bool{}
collect:
for {
select {
case r, open := <-results:
if !open {
break collect
}
got[r.module] = true
if r.why != "" {
defs.Unread[r.module] = r.why
continue
}
defs.Read++
for _, p := range r.pins {
k := Key(p.Repository, p.Digest)
defs.Pins[k] = append(defs.Pins[k], Pin{Module: r.module, Name: p.Name, Older: p.Older})
}
case <-ctx.Done():
break collect
}
}
for _, m := range list.Modules {
if !got[m.Module] {
defs.Unread[m.Module] = fmt.Sprintf("not read within %d s", int(DefinitionsTimeout/time.Second))
}
}
for k := range defs.Pins {
sort.Slice(defs.Pins[k], func(i, j int) bool { return defs.Pins[k][i].Module < defs.Pins[k][j].Module })
}
return defs
}
// definition reads one module's definition, or answers it from what was read before.
func (c *Catalogue) definition(m catalogueEntry, names StoreNames, nameKey string) ([]pinned, string) {
key := m.Module + "@" + m.Commit + "|" + nameKey
if m.Commit != "" {
c.mu.Lock()
pins, had := c.kept[key]
c.mu.Unlock()
if had {
return pins, ""
}
}
args := map[string]any{"module": m.Module}
if m.Commit != "" {
args["commit"] = m.Commit
}
raw, err := c.Ask(CatalogueModule, args)
if err != nil {
return nil, "could not be asked: " + err.Error()
}
answer, output, ok := answerOf(raw)
if !ok {
return nil, "refused: " + lastLine(output)
}
var one struct {
Module *struct {
Manifest json.RawMessage `json:"manifest"`
} `json:"module"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &one); err != nil || one.Error != "" || one.Module == nil || len(one.Module.Manifest) == 0 {
return nil, "not readable: " + firstOf(one.Error, errText(err), "no manifest in the answer")
}
pins := PinsIn(one.Module.Manifest, names)
if m.Commit != "" {
c.mu.Lock()
c.kept[key] = pins
c.mu.Unlock()
}
return pins, ""
}
// why says which definitions name a manifest, and under which of the store's names.
func (p Pin) why() string {
if p.Older {
return fmt.Sprintf("a definition names it: %s, under %s, an older name of the store", p.Module, shownName(p.Name))
}
return "a definition names it: " + p.Module
}
func firstOf(s ...string) string {
for _, x := range s {
if strings.TrimSpace(x) != "" {
return x
}
}
return ""
}
func errText(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// shownName is a name of the store as a person reads it: a host without the slash a pin puts after it, the
// scheme as written.
func shownName(name string) string {
if strings.HasSuffix(name, "://") {
return name
}
return strings.TrimSuffix(name, "/")
}
@@ -0,0 +1,259 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// A definition may pin an image of the store the mesh did not make, under a name the store had before
// (novox/hq issue 323): three apps the anchor runs are pinned so. No record names them; a definition does.
const oldName = "registry-api.example.org"
// container is a definition whose one container runs image.
func container(image string) string {
return fmt.Sprintf(`{"module":"x","resources":[{"id":"server","type":"container","image":%q}]}`, image)
}
func TestAnImageADefinitionPinsUnderAnOlderNameOfTheStoreIsNamedByADefinition(t *testing.T) {
v := view(t, world())
names := NamesOf("store:5000", oldName)
defs := &Definitions{Pins: map[string][]Pin{}}
for module, manifest := range map[string]string{
// m-stranger: no record names it; the invoicing app pins it under the store's older name.
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
// m-old: the records say eligible; a definition pinning it does not change what the records say.
"website": container("store:5000/app/server@" + d("m-old")),
} {
for _, p := range PinsIn(json.RawMessage(manifest), names) {
defs.Pins[Key(p.Repository, p.Digest)] = append(defs.Pins[Key(p.Repository, p.Digest)], Pin{Module: module, Name: p.Name, Older: p.Older})
}
}
got := stateByName(Classify(v, withFill(records()), defs))
stranger := got["app/server "+d("m-stranger")]
if stranger.State != StateNamedByDefinition {
t.Fatalf("a manifest a definition pins under an older name reads %q, want %q", stranger.State, StateNamedByDefinition)
}
if want := []string{"a definition names it: invoicing, under " + oldName + ", an older name of the store"}; !reflect.DeepEqual(stranger.Why, want) {
t.Errorf("why %q, want %q", stranger.Why, want)
}
if s := got["app/server "+d("m-old")].State; s != StateEligible {
t.Errorf("a recorded manifest a definition pins reads %q: the records decide what the mesh made", s)
}
// Without the definitions it is what it was: unrecorded.
if s := stateByName(Classify(v, withFill(records()), nil))["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("without definitions %q", s)
}
}
func TestPinsAreFoundInEveryStringAndOnlyUnderTheStoresNames(t *testing.T) {
names := NamesOf("store:5000", oldName+", older.example/")
manifest := `{"resources":[
{"type":"container","image":"` + oldName + `/team/www@` + d("www") + `"},
{"type":"file","content":"IMAGE=older.example/team/tool@` + d("tool") + `\n"},
{"type":"container","image":"artifact-store://mod/art@` + d("built") + `"},
{"type":"container","image":"store:5000/mod/art@` + d("addressed") + `"},
{"type":"container","image":"not` + oldName + `/team/www@` + d("lookalike") + `"},
{"type":"container","image":"docker.io/library/nginx@` + d("upstream") + `"},
{"type":"container","image":"` + oldName + `/team/www:latest"}],
"build":{"on":{"base":"` + oldName + `/team/www@` + d("www") + `"}}}`
var got []string
for _, p := range PinsIn(json.RawMessage(manifest), names) {
got = append(got, fmt.Sprintf("%s@%s %s %v", p.Repository, p.Digest, p.Name, p.Older))
}
want := map[string]bool{
"team/www@" + d("www") + " " + oldName + "/ true": true,
"team/tool@" + d("tool") + " older.example/ true": true,
"mod/art@" + d("built") + " artifact-store:// false": true,
"mod/art@" + d("addressed") + " store:5000/ false": true,
}
if len(got) != len(want) {
t.Fatalf("found %d pins, want %d (one per image and name; not a longer host's tail, not another registry, not a tag): %v", len(got), len(want), got)
}
for _, g := range got {
if !want[g] {
t.Errorf("found %s", g)
}
}
}
func TestAPlatformTakesTheStateOfAnIndexADefinitionNames(t *testing.T) {
v := mirrorWorld()
defs := &Definitions{Pins: map[string][]Pin{
Key("route-proxy/on-go_base", d("i-pre")): {{Module: "route-proxy", Name: oldName + "/", Older: true}},
}}
got := stateByName(Classify(v, withFill(mirrorRecords()), defs))
for _, name := range []string{"i-pre", "p-pre"} {
if s := got["route-proxy/on-go_base "+d(name)].State; s != StateNamedByDefinition {
t.Errorf("%s reads %q, want %q", name, s, StateNamedByDefinition)
}
}
if e := got["route-proxy/on-go_base "+d("p-pre")]; e.Index != d("i-pre") {
t.Errorf("p-pre takes its state from %q", e.Index)
}
if s := got["route-proxy/on-go_base "+d("p-lost")].State; s != StateUnrecorded {
t.Errorf("a platform nothing names reads %q", s)
}
for _, um := range UnrecordedMirrors(v, Classify(v, withFill(mirrorRecords()), defs)) {
if strings.Contains(um.Reference, hexOf(d("i-pre"))) {
t.Errorf("a copy a definition names is offered to the mirrors verb: %v", um)
}
}
}
// olderNames writes the file the mesh writes from the setting older-store-names.
func olderNames(t *testing.T, names string) {
t.Helper()
path := filepath.Join(t.TempDir(), "store-names")
if err := os.WriteFile(path, []byte("# written by the mesh\nolder="+names+"\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(NamesFileEnv, path)
}
func TestTheOlderNamesAreReadFromTheFileTheMeshWrites(t *testing.T) {
olderNames(t, oldName+" older.example")
got, why := OlderNames()
if why != "" || !reflect.DeepEqual(NamesOf("", got).Older, []string{oldName + "/", "older.example/"}) {
t.Errorf("read %q (%s)", got, why)
}
olderNames(t, "none")
if got, why := OlderNames(); why != "" || len(NamesOf("", got).Older) != 0 {
t.Errorf("none read as %q (%s)", got, why)
}
t.Setenv(NamesFileEnv, filepath.Join(t.TempDir(), "absent"))
if _, why := OlderNames(); why == "" {
t.Error("an absent file was not said")
}
}
func TestOlderNamesNotReadAreSaidFirst(t *testing.T) {
t.Setenv(NamesFileEnv, "")
a := &asked{definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger"))}}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, NamesFileEnv) {
t.Errorf("first said %q", w)
}
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q without the older names", s)
}
}
// references is one artifact_store_references tool, kept across calls as the runtime keeps the bundle.
func references(t *testing.T, a *asked) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
if tl.Name == "artifact_store_references" {
return tl.Run
}
}
t.Fatal("no artifact_store_references")
return nil
}
func manifestsOf(t *testing.T, out any) map[string]Entry {
t.Helper()
entries, ok := out.(map[string]any)["manifests"].([]Entry)
if !ok {
t.Fatalf("no manifests in %v", out)
}
return stateByName(entries)
}
func TestReferencesReadTheDefinitionsUnderTheStoresOlderNames(t *testing.T) {
olderNames(t, oldName)
a := &asked{definitions: map[string]string{
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
"grafana": container("grafana/grafana@" + d("upstream")),
}}
run := references(t, a)
out, err := run(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
e := manifestsOf(t, out)["app/server "+d("m-stranger")]
if e.State != StateNamedByDefinition || len(e.Why) != 1 || !strings.Contains(e.Why[0], "invoicing") || !strings.Contains(e.Why[0], oldName) {
t.Errorf("m-stranger: %+v", e)
}
said := out.(map[string]any)["definitions"].(map[string]any)
if said["read"] != 2 || !reflect.DeepEqual(said["older_store_names"], []string{oldName}) ||
!reflect.DeepEqual(said["store_names"], []string{"artifact-store://", "store:5000"}) {
t.Errorf("definitions said %v", said)
}
if w := out.(map[string]any)["said"].([]string)[0]; strings.Contains(w, "could not be read") {
t.Errorf("a warning with every definition read: %s", w)
}
// A definition read at its commit is kept: the next call asks only for the list.
before := 0
for _, c := range a.calls {
if c == CatalogueModule {
before++
}
}
if _, err := run(map[string]any{"repository": "app/server"}); err != nil {
t.Fatal(err)
}
after := 0
for _, c := range a.calls {
if c == CatalogueModule {
after++
}
}
if before != 2 || after != 2 {
t.Errorf("asked catalog_module %d times, then %d: each definition at its commit is read once", before, after)
}
}
func TestADefinitionNotReadIsSaidFirstAndLeavesWhatItPinsUnrecorded(t *testing.T) {
olderNames(t, oldName)
a := &asked{
definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger")), "grafana": container("x")},
failing: map[string]bool{"invoicing": true},
}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q with its definition unread", s)
}
if w := m["said"].([]string)[0]; !strings.Contains(w, "1 definitions could not be read") {
t.Errorf("first said %q", w)
}
if u := m["definitions"].(map[string]any)["unread"].(map[string]string); u["invoicing"] == "" || len(u) != 1 {
t.Errorf("unread %v", u)
}
// The catalogue not answering at all: the records are still answered, and the warning stands first.
a = &asked{noCatalogue: true}
out, err = references(t, a)(map[string]any{})
if err != nil {
t.Fatal(err)
}
m = out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, "the definitions could not be read") || !strings.Contains(w, "no responders") {
t.Errorf("first said %q", w)
}
if m["definitions"].(map[string]any)["not_asked"] == nil {
t.Errorf("definitions %v", m["definitions"])
}
}
func TestNoCatalogueIsSaidAndNothingIsAsked(t *testing.T) {
defs := (&Catalogue{}).Read(context.Background(), NamesOf("", ""))
if defs.NotAsked == "" || defs.Read != 0 {
t.Errorf("%+v", defs)
}
}
@@ -24,7 +24,7 @@ func main() {
UID: os.Getuid(),
}
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools.
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask})); err != nil {
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask}, &Catalogue{Ask: stdio.Ask})); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -65,7 +65,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
for i := range recs.References {
fill(&recs.References[i])
}
got := stateByName(Classify(v, recs))
got := stateByName(Classify(v, recs, nil))
up := "upstream/docker.io/library/golang "
for name, want := range map[string]string{
up + d("p-amd"): StateEligible, // only the eligible index names it
@@ -92,7 +92,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
if len(mirrors) != 2 || !got["route-proxy/on-go_base "+d("i-pre")].Mirror || !got["route-proxy/on-go_base "+d("p-lost")].Mirror {
t.Errorf("offered as unrecorded copies: %v", mirrors)
}
um := UnrecordedMirrors(v, Classify(v, recs))
um := UnrecordedMirrors(v, Classify(v, recs, nil))
if len(um) != 2 || um[0].Reference != "artifact-store://route-proxy/on-go_base@"+d("i-pre") || um[0].OnlyTheir != 1+1+1000 {
t.Errorf("unrecorded copies: %+v", um)
}
@@ -104,7 +104,7 @@ func TestLettingAnIndexGoTakesItsOwnPlatformsAndSaysWhatStaysAndNamesOne(t *test
for i := range recs.References {
fill(&recs.References[i])
}
entries := Classify(v, recs)
entries := Classify(v, recs, nil)
old := "artifact-store://upstream/docker.io/library/golang@" + d("i-old")
gone := LetGoKeys(v, []string{old}, Spared(v, entries))
up := "upstream/docker.io/library/golang"
@@ -215,7 +215,7 @@ func TestTheMarkIsTheCollectorsIncludingAnIndexAndSharedBlobs(t *testing.T) {
func TestEveryStateIsGiven(t *testing.T) {
v := view(t, world())
got := map[string]string{}
for _, e := range Classify(v, withFill(records())) {
for _, e := range Classify(v, withFill(records()), nil) {
got[e.Repository+" "+e.Digest] = e.State
}
want := map[string]string{
@@ -239,7 +239,7 @@ func TestALetGoReferenceStillPresentIsSaidSo(t *testing.T) {
v := view(t, world())
r := records()
r.References = append(r.References, Record{Reference: "artifact-store://app/server@" + d("m-stranger"), State: "collected"})
for _, e := range Classify(v, withFill(r)) {
for _, e := range Classify(v, withFill(r), nil) {
if e.Digest == d("m-stranger") && e.State != StateCollectedPresent {
t.Errorf("state %s", e.State)
}
@@ -270,7 +270,7 @@ func TestBytesFreedCountASharedBlobAsKept(t *testing.T) {
func TestOnlyTheirBytesLeaveOutWhatAnotherStateMarks(t *testing.T) {
v := view(t, world())
entries := Classify(v, withFill(records()))
entries := Classify(v, withFill(records()), nil)
sums := Summarise(v, entries, entries)
el := sums[StateEligible]
if el.OnlyTheirBytes != 500+v.L.Blobs[d("m-old")] {
@@ -284,6 +284,11 @@ type asked struct {
calls []string
args []map[string]any
fail map[string]bool
// definitions are the catalogue's modules and their manifests; failing, the modules whose
// definition cannot be read; noCatalogue, the catalogue not answering at all.
definitions map[string]string
failing map[string]bool
noCatalogue bool
}
func (a *asked) ask(key string, body any) (json.RawMessage, error) {
@@ -299,6 +304,31 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
switch key {
case "seat:mesh-controller.artifacts":
answer = records()
case CatalogueModules:
if a.noCatalogue {
return nil, fmt.Errorf("no responders")
}
ms := []map[string]any{}
for name := range a.definitions {
ms = append(ms, map[string]any{"module": name, "commit": "c-" + name, "path": "modules/" + name})
}
answer = map[string]any{"modules": ms}
case CatalogueModule:
name, _ := args["module"].(string)
if a.failing[name] {
return nil, fmt.Errorf("no answer")
}
if args["commit"] != "c-"+name {
return nil, fmt.Errorf("asked %s at %v", name, args["commit"])
}
manifest, has := a.definitions[name]
if !has {
answer = map[string]any{"error": "the catalogue holds no " + name}
break
}
// The catalogue's tool answers through the runtime as the protocol's content list.
raw, _ := json.Marshal(map[string]any{"module": map[string]any{"module": name, "manifest": json.RawMessage(manifest)}})
return json.Marshal(map[string]any{"content": []map[string]any{{"type": "text", "text": string(raw)}}})
case "seat:mesh-controller.collect":
if args["confirm"] == "true" {
answer = CollectAnswer{LetGo: []string{"artifact-store://app/server@" + d("m-old")}}
@@ -315,7 +345,7 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}) {
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
if tl.Name == name {
return tl.Run
}
@@ -386,7 +416,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
t.Fatal(err)
}
var served []string
for _, tl := range Tools(&Store{}, Controller{}) {
for _, tl := range Tools(&Store{}, Controller{}, &Catalogue{}) {
if !strings.HasPrefix(tl.Name, "artifact_store_") || tl.Description == "" || tl.Run == nil {
t.Errorf("tool %q", tl.Name)
}
@@ -398,7 +428,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
if !reflect.DeepEqual(served, listed) {
t.Fatalf("served %v, manifest %v", served, listed)
}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect"}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect", CatalogueModules, CatalogueModule}
if !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes %v", m.Invokes)
}
@@ -13,7 +13,7 @@ import (
// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect
// only reads too, unless it is a real run, and then it asks the controller, which decides and deletes.
func Tools(s *Store, c Controller) []stdio.Tool {
func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
ctx := context.Background()
repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (<module>/<artifact>)"}
return []stdio.Tool{
@@ -53,7 +53,9 @@ func Tools(s *Store, c Controller) []stdio.Tool {
{
Name: "artifact_store_references",
Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " +
"names it, or one of the five most recent builds of a module the mesh holds), the holder of a kept archive, " +
"names it, or one of the five most recent builds of a module the mesh holds), named by a definition (no " +
"record names it, and a definition in the catalogue pins it by digest, under the store's current name or " +
"an older one: the module is said), the holder of a kept archive, " +
"eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " +
"a named document the controller keeps under a tag, or unrecorded — no record names it. A platform manifest " +
"takes the state of the recorded index naming it. Counted and sized by " +
@@ -67,11 +69,19 @@ func Tools(s *Store, c Controller) []stdio.Tool {
if err != nil {
return nil, err
}
// The definitions are read while the store is: each bounded below the runtime's call limit.
read := make(chan *Definitions, 1)
go func() {
older, why := OlderNames()
defs := cat.Read(ctx, NamesOf(recs.Store, older))
defs.NamesNotRead = why
read <- defs
}()
v, err := s.View(ctx)
if err != nil {
return nil, err
}
return References(v, recs, repo)
return References(v, recs, <-read, repo)
},
},
{
@@ -116,7 +126,7 @@ func Tools(s *Store, c Controller) []stdio.Tool {
if err != nil {
return map[string]any{"controller": answer, "bytes_not_worked_out": "the records could not be read: " + err.Error()}, nil
}
return Collected(v, answer, dry, Classify(v, recs)), nil
return Collected(v, answer, dry, Classify(v, recs, nil)), nil
},
},
}
@@ -225,11 +235,11 @@ func Usage(v *View, top int) map[string]any {
}
// References answers artifact_store_references.
func References(v *View, recs *Records, only string) (map[string]any, error) {
func References(v *View, recs *Records, defs *Definitions, only string) (map[string]any, error) {
if only != "" && v.L.Repos[only] == nil {
return nil, fmt.Errorf("the store holds no repository %q", only)
}
entries := Classify(v, recs)
entries := Classify(v, recs, defs)
byRepo := map[string][]Entry{}
for _, e := range entries {
byRepo[e.Repository] = append(byRepo[e.Repository], e)
@@ -243,7 +253,7 @@ func References(v *View, recs *Records, only string) (map[string]any, error) {
}
whole := Summarise(v, entries, entries)
said := []string{}
for _, state := range []string{StateKept, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
for _, state := range []string{StateKept, StateNamedByDefinition, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
if s := whole[state]; s != nil {
said = append(said, fmt.Sprintf("%s: %d manifests marking %s, %s of it marked by nothing else", state, s.Manifests, human(s.Bytes), human(s.OnlyTheirBytes)))
}
@@ -276,9 +286,57 @@ func References(v *View, recs *Records, only string) (map[string]any, error) {
if u := unreadNote(v); u != nil {
out["unread"] = u
}
out["definitions"] = definitionsSaid(defs)
if warn := definitionsWarning(defs); warn != "" {
// Said first: what it leaves out is a manifest a machine may run, read as unrecorded.
out["said"] = append([]string{warn}, said...)
}
return out, nil
}
// definitionsSaid is how the definitions were read: how many, under which names, and what was not read.
func definitionsSaid(defs *Definitions) map[string]any {
if defs == nil {
return map[string]any{"read": 0, "not_asked": "the definitions were not read"}
}
var current, older []string
for _, n := range defs.Names.Current {
current = append(current, shownName(n))
}
for _, n := range defs.Names.Older {
older = append(older, shownName(n))
}
out := map[string]any{"read": defs.Read, "store_names": nonNil(current), "older_store_names": nonNil(older)}
if defs.NotAsked != "" {
out["not_asked"] = defs.NotAsked
}
if defs.NamesNotRead != "" {
out["older_store_names_not_read"] = defs.NamesNotRead
}
if len(defs.Unread) > 0 {
out["unread"] = defs.Unread
}
return out
}
// definitionsWarning says, when any definition was not read, that a manifest it pins reads as unrecorded.
func definitionsWarning(defs *Definitions) string {
switch {
case defs == nil:
return ""
case defs.NotAsked != "":
return "the definitions could not be read (" + defs.NotAsked + "), so a manifest a definition pins " +
"and no record names reads as unrecorded here: ask again before reading unrecorded as unused"
case defs.NamesNotRead != "":
return defs.NamesNotRead + ", so a manifest a definition pins under an older name of the store and no " +
"record names reads as unrecorded here"
case len(defs.Unread) > 0:
return fmt.Sprintf("%d definitions could not be read (listed in definitions.unread), so a manifest one "+
"of them pins and no record names reads as unrecorded here: ask again — what was read is kept", len(defs.Unread))
}
return ""
}
// Collected answers artifact_store_collect: the controller's answer, and what the store's collector frees.
func Collected(v *View, a *CollectAnswer, dry bool, entries []Entry) map[string]any {
refs := a.LetGo
+28 -2
View File
@@ -3,7 +3,9 @@
"version": "1",
"invokes": [
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect"
"seat:mesh-controller.collect",
"mesh-catalog.catalog_modules",
"mesh-catalog.catalog_module"
],
"tools": [
"artifact_store_repositories",
@@ -11,6 +13,29 @@
"artifact_store_references",
"artifact_store_collect"
],
"settings": {
"older-store-names": {
"kind": "preference",
"default": "none",
"why": "the names the artifact store had before, as a definition may still pin an image under them: host names or host:port, separated by spaces; none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323)"
}
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "store-names",
"type": "file",
"path": "${dir:state}/store-names",
"mode": "0600",
"trusted": false,
"content": "# The names the artifact store had before, from the setting older-store-names, written by the mesh.\n# artifact_store_references reads every definition under these too (novox/hq issue 323).\n# Replaced on every push; change the setting, never this file.\nolder=${setting:older-store-names}\n"
}
],
"build": {
"artifacts": [
{
@@ -24,7 +49,8 @@
"artifact-store-tools"
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry"
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry",
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names"
}
}
]
+147 -32
View File
@@ -1,8 +1,10 @@
# betterbird
Betterbird, the mail and calendar client of g14 and shanks (a Thunderbird fork, with the operator's
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208). It requires
`x11-display`, so it is assigned only where a display server is held on the same machine.
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208, ADR 0316, ADR 0317,
to-be 56). It requires `x11-display`, so it is assigned only where a display server is held on the same
machine. It installs Betterbird from a pinned upstream release, ships Owl with it through Betterbird's
enterprise policy, and declares the operator's mail accounts without their credentials.
It exists because a symptom had no tools: on a workstation, a calendar reminder appeared as a
transparent window with nothing in it (2026-10-09). Nothing in the mesh could say which window that
@@ -11,24 +13,57 @@ issue 345.
## Owns
| what | where |
|---|---|
| the module's directory | `~/.config/betterbird/` |
| the module's `user.js`, filled from the setting `prefs` | `~/.config/betterbird/user.js` |
| what | where | class |
|---|---|---|
| Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole |
| the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole |
| Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole |
| the spelling dictionaries' default (`install/defaults-pref/vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its one file |
| the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole |
| the start | `/usr/local/bin/betterbird` | owned, whole |
| the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole |
| the module's directory, with its `user.js` (setting `prefs`), `accounts.json` (setting `accounts`) and the record of adopted and created accounts | `~/.config/betterbird/` | owned, whole |
| the profile's `user.js`, placed by the tools | the profile | the module's, or never touched |
| the account lists in the profile's `prefs.js` | the profile | written into: the module's keys only |
| the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) |
Nothing else. It holds no seat and adds no start.
**The community package `betterbird-bin` is declared absent** (ADR 0180), before anything is placed in
`/opt/betterbird`: the node-engine removes it with `pacman -R`, then unpacks the release where the package
had it. Betterbird chooses its profile by its installation's path, so the same profile opens after the
switch. A package left in place (its removal failed) holds the path, and the release is refused there by
the node-engine rather than mixed in. The module creates no link: the start, a launcher entry naming the
icon in the installation, and the dictionaries as a default preference replace the package's links.
**Every new build waits for the operator** (`upgrade: record`): the first push to a node removes the
package and replaces what a running Betterbird has open, so the operator quits Betterbird, then pushes.
An upgrade is a change to the pins, reviewed and delivered like any other (ADR 0316 §5).
**The pins.** The release is `betterbird-153.4.0esr-bb10.en-US.linux-x86_64.tar.xz` (81,620,684 bytes) and
Owl is 1.5.6.1 (430,201 bytes), each pinned by sha256 in `module.json`. The build seat fetches each once,
refuses it when the digest differs, keeps it under its sha256, and packs it into the artifact store; a node
receives the mesh's archive by digest and never asks the vendor. **Owl is never in this repository**: its
licence forbids copying it to third parties, and its vendor serves only its newest version, at one
address. Keeping the pinned copy in the mesh's private artifact store for the operator's own machines was
the operator's call (2026-10-12, ADR 0316 §6).
**Owl's range.** Owl declares which Betterbird versions it supports (`strict_min_version` to
`strict_max_version`: 115 to 156 for 1.5.6.1). `owlSupports` in `manifest_test.go` holds the range read
from each pinned xpi, and the module's tests refuse a Betterbird pin outside it: moving Owl's pin means
reading the new range there (ADR 0316 §8).
**The policy** turns Betterbird's own updater off (`DisableAppUpdate`, which Betterbird's update check
honours) and installs Owl into every profile from `file:///opt/betterbird/owl/owl.xpi` as
`force_installed`, with `updates_disabled`. A profile that already runs another Owl keeps it: the policy
installs the pinned one where Owl is missing. A newer Owl in the profile (g14 runs 1.5.7) is said as a
fact by `betterbird_status`; Owl installed by policy cannot be removed in Betterbird, so the remedy is
moving the pin. An older one is a finding of `betterbird_check`, with the same remedy. **Owl's licence stays the person's**: the mesh holds no licence, key or payment (ADR 0316 §9).
**No start of its own.** Betterbird is started by the operator from the launcher. The start
(`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the
account running, places the user.js and writes the created accounts into the account lists; then it
starts Betterbird either way. `betterbird_restart` starts it as the transient user unit
`mesh-betterbird`, so its output then goes to the journal.
- **No package.** g14 and shanks both run `betterbird-bin` 153.1.0esr-1, installed explicitly from the
AUR: `pacman` counts it as foreign. The node-engine installs packages from the official
repositories only, so the module does not declare it, and **installing and upgrading Betterbird stay
the operator's** (an AUR helper). `betterbird_status` says the version and that it is from the AUR;
`betterbird_check` says when it is missing.
- **No start.** Betterbird is started by the operator from the launcher (on the workstation measured,
it runs in the login session's scope, started by hand, with no autostart entry and no line in i3's
configuration). `betterbird_restart` starts it as the transient user unit `mesh-betterbird`, so its
output then goes to the journal.
- **The profile is Betterbird's** (ADR 0182): `~/.thunderbird/` holds the mail, the address books,
the calendars' caches, the logins and the keys. The module declares none of it.
## The user.js: how a preference is declared
@@ -52,33 +87,104 @@ default, the one in use, is the other.
what to do. The tools refuse a declared file that holds anything but comments and `user_pref`
statements, or that sets a preference whose name can hold a credential.
The file is **trusted** (it carries no `"trusted": false`): a preference can weaken how the mail client
checks a server, so the setting is changed only at the controller's terminal, never through a verb
(novox/hq issue 339).
The file is **trusted** (`"trusted": true`): a preference can weaken how the mail client checks a server,
so the setting is changed only at the controller's terminal or proposed and set on the operator's warrant
(novox/hq issue 339, ADR 0277). An account's preferences (`mail.account.`, `mail.identity.`, `mail.server.`,
`mail.smtpserver.` and the account lists) are refused in `prefs`: an account is declared in `accounts`, so
it has one owner (ADR 0316 §11).
## The accounts: declared without credentials (ADR 0316 §10–§16, ADR 0317)
The setting `accounts` reaches the machine as `~/.config/betterbird/accounts.json`, a trusted mergeable
file: a map of accounts by name, each with `address`, `from-name` and `label` (optional), `incoming` (`kind`
imap or pop3, `host`, `port`, `security` tls, starttls or none, `sign-in` password, encrypted-password,
oauth, kerberos, ntlm or tls-certificate, `user`) and `outgoing` (the same, without a kind; its sign-in may
be none), and, only in a node's own layer, `key`. **Never a password or a token**: the whole file is
refused when any key in it is shaped like one, and an account signs in at its first use, in Betterbird's
own window, where Betterbird keeps the result in its login store. The work account through Owl is never
declared: the person creates it in Owl's setup (ADR 0316 §13).
At every apply (the start from the launcher, `betterbird_user_js` with apply, `betterbird_restart`), each
declared account, in its place in the setting, is (ADR 0317 §1):
1. **kept by its recorded keys**, when the module recorded it on this node for this profile;
2. **its creation**, when it is under the module's own keys (`account_mesh_<name>`, `id_mesh_<name>`,
`server_mesh_<name>`, `smtp_mesh_<name>`);
3. **adopted by the key a node's layer names** (`key`), when that account is in the list, its default
identity has the address and it is not Owl's;
4. **adopted by its address**, when exactly one account's default identity has it: its keys (account,
default identity, incoming server, and the outgoing server that identity names) become the module's, and
the account lists are left as they are;
5. **created** under the module's own keys when no account has it: its keys are added at the end of
`mail.accountmanager.accounts` and `mail.smtpservers` in `prefs.js`, every other key kept in its place,
only while Betterbird is not running.
**Refused, for that account alone and with the remedy** (§7): two accounts with the address (remove the
duplicate in Betterbird, or name the `key`), Owl's account at the address, a named key that does not
qualify, an account of another kind than declared (never converted), an identity that names no outgoing
server, and an outgoing server that another identity, neither adopted nor the module's, also uses
(Betterbird lets identities share one, and the declared values would change that other account's sending;
ADR 0317 §2 leaves the identity's server and the lists as they are, so the module refuses rather than give
it a server of its own). A recorded adoption follows the identity's current outgoing server at every apply,
under the same rule. The other accounts go ahead; the start and `betterbird_check` say it.
`betterbird_user_js` answers an account's preferences by name with their values withheld.
The declared fields are written in the user.js under the account's keys, so they win at every start; only
the fields the setting names, never the account's directory, folders, filters, caches, mail or kept
sign-in, so an adopted account keeps its sign-in (§5). The apply names, by preference name only, each one
whose value it changed (§4). What was adopted or created is recorded in
`~/.config/betterbird/accounts-record.json`. **Leaving the setting** releases an adopted account (its
lines and its record go, the account stays the person's) and takes a created account's keys out of the
lists (its mail stays in the profile).
**The value is derived, not retyped** (§9): `betterbird_accounts` with `propose` builds `accounts` from the
profile (every account but Owl's, no credential) and proposes it through the controller's settings verb;
the operator reads it whole at their desk (`settings proposals <id>`) and approves on a channel that proves
who answers. Its answer to its caller holds counts and kinds only.
**A node with no profile yet.** The preparation needs the profile Betterbird makes at its first start: on a
node set up from nothing, the declared accounts are written at the second start from the launcher (or at
`betterbird_restart`).
## Tools
They are served by the node's runtime as the operator account (ADR 0175). **No answer carries mail, a
calendar entry, an account, an address or a credential.**
calendar entry, an address, a host, an account's name or a credential**: accounts are said by count, place
and key.
| tool | does |
|---|---|
| `betterbird_status` (r) | whether it runs (pid, since, scope, children), the package and version and that it is from the AUR, where its output goes, the profile and how it was found, the user.js (declared, placed, in effect), the add-ons the operator installed (ids), and the preferences that decide how its windows and notifications look |
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where (the mesh's release, or the community package before the switch), its version and build beside the newest published build (Betterbird's own `versions.txt`); the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect); the declared accounts by count and, per place, adopted, created or refused with their keys; the add-ons in the profile (ids, versions); the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (user.js, prefs.js or the default). Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, …); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked |
| `betterbird_user_js` (r; a with `apply`) | the declared user.js against the profile's; `apply` places it |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked (the value after `password=`, `token:` and the like, and the token after `Bearer`/`Basic`). A Betterbird the module did not start prints elsewhere, and the answer says where |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js where allowed, starts it as `mesh-betterbird`. **Refused while a composer window is open** (X instance `Msgcompose`, whatever its class), because the unsent message would be lost, unless `discard_unsent` is true, which only the operator's yes justifies. It answers within the runtime's 30 s: a start not yet seen after 4 s is said in `note`, and `betterbird_status` shows it |
| `betterbird_check` (r) | installed; one running; profile found; the user.js on the machine, valid, placed, in effect; its output read |
| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts by count and kind (declarable, Owl's, others); with `propose`, the setting `accounts` built from them and proposed to the operator (ADR 0317 §9) |
| `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) |
### A reminder on demand
**It opens windows on the operator's screen**, and its description says so first, so a caller tells the
operator before calling it. On 2026-10-12 at 12:05 a test Betterbird with an empty profile carrying the
operator's theme looked like the operator's configuration lost; so the test is **visibly a test**: its
profile carries a `userChrome.css` that frames every window of the test Betterbird in red and runs a red
band reading MESH TEST across its top, and its event is called "MESH TEST reminder (a throw-away
Betterbird of the mesh, not your calendar)". The reminder window itself is left unstyled, because how
Betterbird draws it is what the test looks at; its title is the event's. A window title cannot be set by a
preference, so the marking is the band, the frame and the event's name.
`betterbird_reminder_test` starts a **second** Betterbird with `--no-remote` in a throw-away profile
under `/run/user/<uid>/mesh-betterbird-reminder-test/`, as the user unit
`mesh-betterbird-reminder-test` (ended after 10 minutes at most). The profile holds the module's
user.js and one calendar: a local `.ics` file with one event, "Mesh reminder test", whose reminder is
user.js (the setting `prefs`; never a declared account) and one calendar: a local `.ics` file with one
event, "MESH TEST reminder", whose reminder is
due `after` seconds from the call (default 40). `carry` copies named preferences of the operator's
profile into it (a theme, a colour switch), under the same limits as `betterbird_prefs`. Then:
profile into it (a theme, a colour switch), under the same limits as `betterbird_prefs`. The test's own
preferences come last in its user.js, so nothing declared or carried overrides them; one of them turns
Betterbird's update check off (`betterbird.update.checkIntervalDays` 0), because a fresh profile checks
at its start and, when a newer build is published, opens the About window on the operator's screen
(seen 2026-10-12). Then:
- `xorg_windows` with class `Calendar` answers the reminder window: role `AlarmWindow`, type `Dialog`,
its depth (32: a visual with alpha, which the compositor blends), its map state;
@@ -106,13 +212,19 @@ A window of depth 32 that its program has not drawn is transparent under a compo
that is why the symptom looks the way it does. Why Betterbird did not draw it is hq issue 345's
question.
## What changes when it is assigned
## What changes when it is pushed
| | g14 | shanks |
|---|---|---|
| files | `~/.config/betterbird/` and its `user.js` (no preference) | the same |
| the profile | untouched until `betterbird_user_js` with `apply` or `betterbird_restart` | the same; the profile has no user.js today, so `apply` may place one |
| Betterbird | keeps running as started | keeps running as started (since 2026-10-04, its output on the session's pipe nobody reads: `betterbird_check` names it until `betterbird_restart`) |
The module is assigned to shanks today (look-only, from issue 345's first part); g14 runs Betterbird
outside the mesh. With this version, a push to a node (each waits for the operator: `upgrade: record`):
| | on a node with the community package (g14, shanks) |
|---|---|
| the package | `betterbird-bin` removed (quit Betterbird first) |
| `/opt/betterbird` | the mesh's 153.4.0esr-bb10, in the same place, so the same profile opens |
| the policy, Owl | placed; Owl installed into the profile by Betterbird at its next start where it is missing |
| `/usr/local` | the start, the launcher entry, the tools' program |
| `~/.config/betterbird/` | `user.js`, `accounts.json` (no account until the setting `accounts` is approved) |
| the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` |
## Relies on
@@ -120,3 +232,6 @@ question.
- **`xorg`** for `xorg_windows` and `xorg_screenshot`, which look at Betterbird's windows. This module
carries no window code of its own.
- `systemd-run` and `journalctl` of the account's user manager.
- The build seat's archive pinned by URL and sha256 (mesh-controller, ADR 0316 §2): a controller that does
not know it refuses this manifest.
- The controller's `settings` verb, for `betterbird_accounts` with `propose` (`invokes`).
@@ -0,0 +1,811 @@
package main
// The operator's mail accounts, declared without credentials (novox/hq ADR 0316 §10–§14, to-be 56 §4).
//
// The setting `accounts` reaches the machine as ~/.config/betterbird/accounts.json, a file the node-engine
// writes from the module's settings layers. Per account it names the address, the incoming server and the
// outgoing server; never a password or a token, and the whole file is refused when any key in it is shaped
// like one. Betterbird's accounts are preferences, so a declared account becomes:
//
// - its declared fields as preferences in the user.js the module places, where they win at every start
// (§11): under the keys of the account the profile already holds at that address, which the module adopts
// (novox/hq ADR 0317), or, where the profile holds none, under keys of the module's own that end in
// mesh_<name> (account_mesh_<name>, id_mesh_<name>, server_mesh_<name>, smtp_mesh_<name>);
// - for an account the module created, its key in the account list (mail.accountmanager.accounts) and in the
// list of outgoing servers (mail.smtpservers) of the profile's prefs.js: written into, never owned, and
// only while Betterbird is not running, because it rewrites prefs.js when it quits (§12). Every other key
// keeps its place, and an adopted account's keys are in the lists already.
//
// Which account is adopted, created or refused is adoption.go's.
//
// Nobody's password passes through here: an account signs in at its first use, in Betterbird's own window,
// and Betterbird keeps the result in its login store (§14). Of that store the module reads only the server
// names each entry is kept for, which Betterbird writes in the clear, to count the declared accounts that
// have not signed in yet (§16); never a user name or a password, which are encrypted there anyway.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// accountsFile is the setting accounts on the machine, written by the node-engine.
const accountsFile = ".config/betterbird/accounts.json"
// Account is one declared mail account, in the shape of the setting accounts: keyed by its name.
type Account struct {
Address string `json:"address"`
// FromName is the name the account's mail is sent under. Absent: Betterbird's, none.
FromName string `json:"from-name,omitempty"`
// Label is what Betterbird shows the account as. Absent: its address.
Label string `json:"label,omitempty"`
Incoming Server `json:"incoming"`
Outgoing Server `json:"outgoing"`
// Key is the profile's account key to adopt, given only in a node's own layer to settle an address the
// profile holds more than once (novox/hq ADR 0317 §8).
Key string `json:"key,omitempty"`
}
// Server is an account's incoming or outgoing server: never a password or a token.
type Server struct {
// Kind is imap or pop3, for the incoming server; the outgoing one is smtp and says nothing.
Kind string `json:"kind,omitempty"`
Host string `json:"host"`
Port int `json:"port,omitempty"`
Security string `json:"security"`
SignIn string `json:"sign-in"`
User string `json:"user,omitempty"`
}
// The words a server's connection security and sign-in method are said in, and Betterbird's numbers for
// them (socketType / try_ssl, authMethod).
var (
securities = map[string]int{"none": 0, "starttls": 2, "tls": 3}
signIns = map[string]int{"none": 1, "password": 3, "encrypted-password": 4, "kerberos": 5, "ntlm": 6,
"tls-certificate": 7, "oauth": 10}
)
func wordFor(words map[string]int, n int) string {
for w, v := range words {
if v == n {
return w
}
}
return ""
}
var (
accountName = regexp.MustCompile(`^[a-z][a-z0-9]{0,30}$`)
addressShape = regexp.MustCompile(`^[^\s@"\\]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$`)
hostShape = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9.\-]*[A-Za-z0-9])?$`)
profileKey = regexp.MustCompile(`^[A-Za-z0-9_]{1,64}$`)
// plainText is a value written into a preference: no quote, backslash or line break.
plainText = regexp.MustCompile(`^[^"\\\n\r]*$`)
// secretKey is a key the accounts file may never carry, at any depth: the setting holds no credential.
secretKey = regexp.MustCompile(`(?i)(pass|secret|token|oauth.?refresh|refresh|credential|api.?key|private.?key|cookie|bearer)`)
)
// keysOf are the module's own keys for a declared account: each ends in mesh_<name>.
func keysOf(name string) (account, identity, server, smtp string) {
return "account_mesh_" + name, "id_mesh_" + name, "server_mesh_" + name, "smtp_mesh_" + name
}
// ownKey is a key of an account list the module wrote, and the name it was declared under.
var ownKey = regexp.MustCompile(`^(?:account|smtp)_mesh_([a-z][a-z0-9]{0,30})$`)
// readAccounts reads the declared accounts. A missing file is no account (the module is older here); a file
// that carries a key shaped like a credential, an unknown field or a value that cannot be written is refused
// whole, naming the key and never a value.
func (m *Machine) readAccounts() (map[string]Account, bool, error) {
raw, err := readBounded(m.home(accountsFile))
if errors.Is(err, os.ErrNotExist) {
return map[string]Account{}, false, nil
}
if err != nil {
return nil, false, err
}
accounts, err := parseAccounts(raw)
return accounts, true, err
}
func parseAccounts(raw []byte) (map[string]Account, error) {
var tree any
if err := json.Unmarshal(raw, &tree); err != nil {
return nil, fmt.Errorf("~/%s is not JSON: %v", accountsFile, err)
}
if where := secretKeyIn(tree, ""); where != "" {
// An account's name is said by nothing: its segment of the path is withheld.
if parts := strings.SplitN(where, ".", 3); len(parts) == 3 && parts[0] == "accounts" {
where = "accounts.<an account>." + parts[2]
}
return nil, fmt.Errorf("the setting accounts carries %s, a key shaped like a credential: an account is declared "+
"without its password or token, which it signs in with at its first use (novox/hq ADR 0316 §10, §14). "+
"Take it out at the controller's terminal", where)
}
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
return nil, fmt.Errorf("~/%s is not an object: %v", accountsFile, err)
}
accounts := map[string]Account{}
if body, ok := top["accounts"]; ok && string(body) != "null" {
dec := json.NewDecoder(bytes.NewReader(body))
dec.DisallowUnknownFields()
if err := dec.Decode(&accounts); err != nil {
return nil, fmt.Errorf("the setting accounts is not a map of accounts by name: %v", err)
}
}
var problems []string
names := sortedNames(accounts)
for i, name := range names {
problems = append(problems, accountProblems(fmt.Sprintf("declared account %d of %d", i+1, len(names)),
name, accounts[name])...)
}
if len(problems) > 0 {
return nil, fmt.Errorf("the setting accounts is refused: %s", strings.Join(problems, "; "))
}
return accounts, nil
}
// secretKeyIn is the path of the first key shaped like a credential in a JSON tree, or "".
func secretKeyIn(node any, at string) string {
switch v := node.(type) {
case map[string]any:
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
here := k
if at != "" {
here = at + "." + k
}
if secretKey.MatchString(k) {
return here
}
if w := secretKeyIn(v[k], here); w != "" {
return w
}
}
case []any:
for i, x := range v {
if w := secretKeyIn(x, fmt.Sprintf("%s[%d]", at, i)); w != "" {
return w
}
}
}
return ""
}
func sortedNames(accounts map[string]Account) []string {
names := make([]string, 0, len(accounts))
for n := range accounts {
names = append(names, n)
}
sort.Strings(names)
return names
}
// accountProblems is what is wrong with one declared account, said of it by its place (label) and never by a
// value of it, its name included (novox/hq ADR 0316 §16).
func accountProblems(label, name string, a Account) []string {
var out []string
say := func(format string, args ...any) { out = append(out, label+": "+fmt.Sprintf(format, args...)) }
if !accountName.MatchString(name) {
say("an account's name is lower-case letters and digits, starting with a letter (it names the module's keys, mesh_<name>)")
}
if !addressShape.MatchString(a.Address) {
say("the address is not a mail address")
}
if a.Key != "" && !profileKey.MatchString(a.Key) {
say("key is an account key of the profile, such as account2")
}
for field, v := range map[string]string{"from-name": a.FromName, "label": a.Label} {
if !plainText.MatchString(v) {
say("%s holds a quote, a backslash or a line break", field)
}
}
for side, s := range map[string]Server{"incoming": a.Incoming, "outgoing": a.Outgoing} {
switch {
case side == "incoming" && s.Kind != "imap" && s.Kind != "pop3":
say("the incoming server's kind is imap or pop3")
case side == "outgoing" && s.Kind != "" && s.Kind != "smtp":
say("the outgoing server is smtp")
}
if !hostShape.MatchString(s.Host) {
say("the %s server's host is not a host name", side)
}
if s.Port < 0 || s.Port > 65535 {
say("the %s server's port is not a port", side)
}
if _, ok := securities[s.Security]; !ok {
say("the %s server's security is tls, starttls or none", side)
}
if _, ok := signIns[s.SignIn]; !ok || (side == "incoming" && s.SignIn == "none") {
say("the %s server's sign-in is password, encrypted-password, oauth, kerberos, ntlm or tls-certificate%s",
side, map[bool]string{true: ", or none", false: ""}[side == "outgoing"])
}
if !plainText.MatchString(s.User) {
say("the %s server's user holds a quote, a backslash or a line break", side)
}
}
return out
}
// quoted is a preference's string value as user.js writes it.
func quoted(s string) string { return strconv.Quote(s) }
// declaredPrefs are a declared account's fields as preferences under the keys it resolved to, name to value
// as user.js writes it, in a fixed order. Only the fields the setting names (novox/hq ADR 0317 §5): never
// the account's directory, folders, filters or caches. An adopted account keeps its kind, the outgoing server
// its identity names, and its own name and sender's name where the declaration says none; a created one
// gets every field.
func declaredPrefs(k keySet, a Account, adopted bool) [][2]string {
label := a.Label
if label == "" && !adopted {
label = a.Address
}
inPort := a.Incoming.Port
if inPort == 0 {
inPort = defaultPort(a.Incoming.Kind, a.Incoming.Security)
}
outPort := a.Outgoing.Port
if outPort == 0 {
outPort = defaultPort("smtp", a.Outgoing.Security)
}
var out [][2]string
add := func(key, value string) { out = append(out, [2]string{key, value}) }
if !adopted {
add("mail.account."+k.Account+".identities", quoted(k.Identity))
add("mail.account."+k.Account+".server", quoted(k.Server))
}
if a.FromName != "" || !adopted {
add("mail.identity."+k.Identity+".fullName", quoted(a.FromName))
}
add("mail.identity."+k.Identity+".useremail", quoted(a.Address))
if !adopted {
add("mail.identity."+k.Identity+".smtpServer", quoted(k.SMTP))
add("mail.identity."+k.Identity+".valid", "true")
add("mail.server."+k.Server+".type", quoted(a.Incoming.Kind))
}
add("mail.server."+k.Server+".hostname", quoted(a.Incoming.Host))
add("mail.server."+k.Server+".port", strconv.Itoa(inPort))
add("mail.server."+k.Server+".socketType", strconv.Itoa(securities[a.Incoming.Security]))
add("mail.server."+k.Server+".authMethod", strconv.Itoa(signIns[a.Incoming.SignIn]))
add("mail.server."+k.Server+".userName", quoted(a.Incoming.User))
if label != "" {
add("mail.server."+k.Server+".name", quoted(label))
}
add("mail.smtpserver."+k.SMTP+".hostname", quoted(a.Outgoing.Host))
add("mail.smtpserver."+k.SMTP+".port", strconv.Itoa(outPort))
add("mail.smtpserver."+k.SMTP+".try_ssl", strconv.Itoa(securities[a.Outgoing.Security]))
add("mail.smtpserver."+k.SMTP+".authMethod", strconv.Itoa(signIns[a.Outgoing.SignIn]))
add("mail.smtpserver."+k.SMTP+".username", quoted(a.Outgoing.User))
if label != "" {
add("mail.smtpserver."+k.SMTP+".description", quoted(label))
}
return out
}
// defaultPort is the port a server listens on when the declaration says none.
func defaultPort(kind, security string) int {
switch {
case kind == "imap" && security == "tls":
return 993
case kind == "imap":
return 143
case kind == "pop3" && security == "tls":
return 995
case kind == "pop3":
return 110
case security == "tls":
return 465
case security == "starttls":
return 587
}
return 25
}
// composeUserJS is the user.js the module places in the profile: the declared file (the setting prefs), then
// each adopted or created account's declared fields under its keys. A refused account has none.
func composeUserJS(declaredText string, accounts map[string]Account, resolved []resolvedAccount) string {
var lines []string
for _, r := range resolved {
if r.How == refused {
continue
}
lines = append(lines, fmt.Sprintf("// declared account %d: %s under %s", r.Place, r.How, r.Keys.Account))
for _, p := range declaredPrefs(r.Keys, accounts[r.name], r.How == adopted) {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(p[0]), p[1]))
}
}
if len(lines) == 0 {
return declaredText
}
return strings.TrimRight(declaredText, "\n") + "\n\n" +
"// The accounts the setting accounts declares (novox/hq ADR 0316 §11, ADR 0317): the fields it names, under\n" +
"// the keys of the account adopted or created. No password or token: an account signs in at its first use.\n" +
strings.Join(lines, "\n") + "\n"
}
// profileAccount is one account the profile holds, as its preferences say it.
type profileAccount struct {
Key, Server, Kind string
// Identities are its identities, the default first, and Addresses their addresses in that order.
Identities []string
Addresses []string
Owl, Ours bool
}
// profileAccounts reads the accounts the profile's list names, from prefs.js and user.js (user.js wins).
func profileAccounts(prefs map[string]string) []profileAccount {
var out []profileAccount
for _, key := range listOf(prefs["mail.accountmanager.accounts"]) {
a := profileAccount{Key: key, Server: unquote(prefs["mail.account."+key+".server"])}
a.Kind = unquote(prefs["mail.server."+a.Server+".type"])
a.Owl = strings.HasPrefix(a.Kind, "owl")
a.Ours = ownKey.MatchString(key)
for _, id := range listOf(prefs["mail.account."+key+".identities"]) {
a.Identities = append(a.Identities, id)
a.Addresses = append(a.Addresses, unquote(prefs["mail.identity."+id+".useremail"]))
}
out = append(out, a)
}
return out
}
// mergedPrefs are a profile's preferences as Betterbird starts with them: prefs.js, then user.js over it.
func mergedPrefs(profile string) map[string]string {
prefsJS, _ := readBounded(filepath.Join(profile, "prefs.js"))
userJS, _ := readBounded(filepath.Join(profile, "user.js"))
out := readPrefs(prefsJS)
for k, v := range readPrefs(userJS) {
out[k] = v
}
return out
}
// listOf reads a comma-separated list preference's value.
func listOf(v string) []string {
var out []string
for _, k := range strings.Split(unquote(v), ",") {
if k = strings.TrimSpace(k); k != "" {
out = append(out, k)
}
}
return out
}
// unquote is a preference's string value without its quotes; any other value as written.
func unquote(v string) string {
if s, err := strconv.Unquote(v); err == nil {
return s
}
return v
}
// writeAccountLists writes the module's keys into the profile's account list and list of outgoing servers
// (§12): the declared accounts' keys added at the end, the module's keys of accounts no longer declared taken
// out, every other key and line kept as it was. Only while Betterbird is not running. Says what changed.
func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) (string, error) {
if ps := m.mainProcs(); len(ps) > 0 {
return "", fmt.Errorf("Betterbird is running (pid %d), and it rewrites prefs.js when it quits: the account "+
"list is written at its next start from the launcher, or by betterbird_restart", ps[0].PID)
}
path := filepath.Join(profile, "prefs.js")
raw, err := readBounded(path)
if err != nil && !errors.Is(err, os.ErrNotExist) {
return "", err
}
var wantAccounts, wantSMTP []string
for _, r := range resolved {
if r.How == created {
wantAccounts, wantSMTP = append(wantAccounts, r.Keys.Account), append(wantSMTP, r.Keys.SMTP)
}
}
out, changed := withListed(string(raw), map[string][]string{
"mail.accountmanager.accounts": wantAccounts, "mail.smtpservers": wantSMTP})
if !changed {
return "unchanged", nil
}
mode := os.FileMode(0o600)
if st, err := os.Stat(path); err == nil {
mode = st.Mode().Perm()
}
tmp := path + ".mesh-new"
if err := os.WriteFile(tmp, []byte(out), mode); err != nil {
return "", err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return "", err
}
return fmt.Sprintf("written: %d account(s) the module created in the account list", len(wantAccounts)), nil
}
// withListed is prefs.js with each list preference holding the module's wanted keys: its own keys that are
// not wanted taken out, the wanted ones not there added at its end, everything else as it was. A list the
// file does not hold yet is added as a line, when anything is wanted in it.
func withListed(text string, want map[string][]string) (string, bool) {
changed := false
done := map[string]bool{}
var lines []string
s := bufio.NewScanner(strings.NewReader(text))
s.Buffer(make([]byte, 64<<10), 4<<20)
for s.Scan() {
line := s.Text()
if mm := prefLine.FindStringSubmatch(line); mm != nil {
if keys, ok := want[mm[1]]; ok && !done[mm[1]] {
done[mm[1]] = true
next := listedWith(listOf(mm[2]), keys)
if next != strings.Join(listOf(mm[2]), ",") {
line = fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), quoted(next))
changed = true
}
}
}
lines = append(lines, line)
}
names := make([]string, 0, len(want))
for n := range want {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
if !done[n] && len(want[n]) > 0 {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(n), quoted(strings.Join(want[n], ","))))
changed = true
}
}
out := strings.Join(lines, "\n")
if text == "" || strings.HasSuffix(text, "\n") {
out += "\n"
}
return out, changed
}
// listedWith is a list with the module's own keys made the wanted ones, every other key in its place.
func listedWith(have, wanted []string) string {
want := map[string]bool{}
for _, k := range wanted {
want[k] = true
}
var out []string
there := map[string]bool{}
for _, k := range have {
if ownKey.MatchString(k) && !want[k] {
continue
}
out = append(out, k)
there[k] = true
}
for _, k := range wanted {
if !there[k] {
out = append(out, k)
}
}
return strings.Join(out, ",")
}
// AccountCount is what status and check say of the accounts: numbers, places and keys, never an address, a
// host or a name (novox/hq ADR 0316 §16, ADR 0317 §3).
type AccountCount struct {
Declared int `json:"declared"`
// InList are the declared accounts the profile's account list names, adopted or created.
InList int `json:"declared_in_the_list"`
// Others are the accounts the profile holds that no declaration names (the work account in Owl among them).
Others int `json:"other_accounts"`
// WithoutSignIn are the declared accounts with no sign-in kept in Betterbird's login store yet.
WithoutSignIn int `json:"declared_without_a_sign_in,omitempty"`
// Each is every declared account by its place in the setting: adopted, created or refused, and its keys.
Each []resolvedAccount `json:"each,omitempty"`
Problem string `json:"problem,omitempty"`
}
// countAccounts counts the declared accounts against the profile's.
func (m *Machine) countAccounts(profile, profileDir string) AccountCount {
var c AccountCount
accounts, _, err := m.readAccounts()
if err != nil {
c.Problem = err.Error()
return c
}
c.Declared = len(accounts)
prefs := mergedPrefs(profile)
have := profileAccounts(prefs)
c.Each = resolveAccounts(accounts, prefs, m.readRecord(profileDir))
mine := map[string]bool{}
refusedN := 0
for _, r := range c.Each {
if r.How == refused {
refusedN++
continue
}
mine[r.Keys.Account] = true
}
for _, a := range have {
switch {
case mine[a.Key]:
c.InList++
case a.Kind != "none":
c.Others++
}
}
if refusedN > 0 {
c.Problem = fmt.Sprintf("%d declared account(s) are refused: each says why and what to do", refusedN)
}
c.WithoutSignIn = len(withoutSignIn(accounts, keptSignIns(profile)))
return c
}
// keptSignIns are the server names Betterbird's login store keeps a sign-in for: the `hostname` of each
// entry of logins.json, which Betterbird writes in the clear. Nothing else of the file is read.
func keptSignIns(profile string) map[string]bool {
raw, err := readBounded(filepath.Join(profile, "logins.json"))
out := map[string]bool{}
if err != nil {
return out
}
var store struct {
Logins []struct {
Hostname string `json:"hostname"`
} `json:"logins"`
}
if json.Unmarshal(raw, &store) != nil {
return out
}
for _, l := range store.Logins {
out[strings.ToLower(l.Hostname)] = true
}
return out
}
// oauthIssuers are the sign-in services of the providers whose servers sign in through OAuth, by a suffix of
// the server's host: their kept sign-in is under the service's name, not the server's.
var oauthIssuers = map[string]string{"gmail.com": "oauth://accounts.google.com",
"googlemail.com": "oauth://accounts.google.com", "office365.com": "oauth://login.microsoftonline.com",
"outlook.com": "oauth://login.microsoftonline.com"}
// withoutSignIn are the declared accounts whose incoming server has no sign-in kept yet. A server signed in
// to without a password (none, kerberos, ntlm, a certificate) keeps nothing and is never counted.
func withoutSignIn(accounts map[string]Account, kept map[string]bool) []string {
var out []string
for _, name := range sortedNames(accounts) {
in := accounts[name].Incoming
switch in.SignIn {
case "password", "encrypted-password":
if !kept[strings.ToLower(in.Kind+"://"+in.Host)] {
out = append(out, name)
}
case "oauth":
issuer := ""
for suffix, i := range oauthIssuers {
if strings.HasSuffix(strings.ToLower(in.Host), suffix) {
issuer = i
}
}
found := false
for k := range kept {
if (issuer != "" && k == issuer) || (issuer == "" && strings.HasPrefix(k, "oauth://")) {
found = true
}
}
if !found {
out = append(out, name)
}
}
}
return out
}
// AccountsAnswer is what betterbird_accounts answers its caller: counts and kinds, never an address, a host or
// a name (novox/hq ADR 0317 §9). The value itself goes only to the operator, in a proposal they read at
// their own desk.
type AccountsAnswer struct {
Profile string `json:"profile"`
// Declarable are the profile's mail accounts a declaration can hold (IMAP and POP3), by kind.
Declarable map[string]int `json:"declarable"`
// KeptByOwl are the accounts Owl keeps (the work account through Exchange): never declared (ADR 0316 §13).
KeptByOwl int `json:"kept_by_owl"`
// NotDeclarable are accounts of another kind (local folders, feeds, news, chat), by kind.
NotDeclarable map[string]int `json:"not_declarable,omitempty"`
// AlreadyDeclared are the declarable accounts under the module's own keys.
AlreadyDeclared int `json:"under_the_modules_keys"`
// Proposed is the controller's answer to the proposal, with every address and host masked.
Proposed string `json:"proposed,omitempty"`
Note string `json:"note"`
}
// profileDeclarable reads the profile's accounts: the value a declaration of them would be, by name, and
// what is answered of them.
func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error) {
p, err := m.profile()
if err != nil {
return nil, AccountsAnswer{}, err
}
prefs := mergedPrefs(p.path)
a := AccountsAnswer{Profile: p.Dir, Declarable: map[string]int{}, NotDeclarable: map[string]int{}}
value := map[string]Account{}
used := map[string]bool{}
for _, pa := range profileAccounts(prefs) {
switch {
case pa.Owl:
a.KeptByOwl++
continue
case pa.Kind != "imap" && pa.Kind != "pop3":
kind := pa.Kind
if kind == "none" {
kind = "local folders"
}
a.NotDeclarable[kind]++
continue
}
a.Declarable[pa.Kind]++
if pa.Ours {
a.AlreadyDeclared++
}
address := ""
if len(pa.Addresses) > 0 {
address = pa.Addresses[0]
}
name := nameFor(pa, address, used)
used[name] = true
value[name] = readAccount(prefs, pa)
}
if len(a.NotDeclarable) == 0 {
a.NotDeclarable = nil
}
return value, a, nil
}
// asker calls a verb of the mesh as this module, through the runtime: a variable so a test answers.
var asker = stdio.Ask
// ProfileAccounts says, by count and kind, which accounts the profile in use holds; with propose it builds
// the setting accounts from them (every account but Owl's, without any credential) and proposes it through
// the controller's settings verb for the operator's Approve (novox/hq ADR 0317 §9, ADR 0277). The value
// never comes back to the caller: the operator reads it whole at their desk (`settings proposals <id>`) and
// checks the fingerprint the ask shows.
func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) {
value, a, err := m.profileDeclarable()
if err != nil {
return a, err
}
a.Note = "counts and kinds only: no address, host or name leaves through this answer. With propose, the " +
"setting accounts is built from these accounts (never Owl's, never a password or token) and proposed to " +
"the operator, who reads it whole at their desk before approving (novox/hq ADR 0317 §9)"
if !propose {
return a, nil
}
if len(value) == 0 {
return a, errors.New("the profile holds no IMAP or POP3 account to declare: nothing was proposed")
}
var problems []string
for i, name := range sortedNames(value) {
if p := accountProblems("", name, value[name]); len(p) > 0 {
problems = append(problems, fmt.Sprintf("account %d of %d: %d field(s) cannot be declared as the profile "+
"holds them", i+1, len(value), len(p)))
}
}
if len(problems) > 0 {
return a, fmt.Errorf("nothing was proposed: %s; betterbird_prefs shows the account's preferences by name",
strings.Join(problems, "; "))
}
body, err := json.Marshal(map[string]any{"accounts": value})
if err != nil {
return a, err
}
raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "values": string(body),
"propose": "true"})
if err != nil {
return a, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
a.Proposed = m.maskValue(said)
if !ok {
return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed)
}
return a, nil
}
// verbSaid is a verb's words and whether it went ahead, whatever wraps them: the controller's {ok, output},
// or the protocol's content list.
func verbSaid(raw json.RawMessage) (string, bool) {
var s string
if json.Unmarshal(raw, &s) == nil {
return verbSaid(json.RawMessage(s))
}
var m map[string]json.RawMessage
if json.Unmarshal(raw, &m) != nil {
return string(raw), true
}
if content, has := m["content"]; has {
var items []struct {
Text string `json:"text"`
}
var e struct {
IsError bool `json:"isError"`
}
_ = json.Unmarshal(raw, &e)
if json.Unmarshal(content, &items) == nil && len(items) > 0 {
out, ok := verbSaid(json.RawMessage(items[0].Text))
return out, ok && !e.IsError
}
}
if okRaw, has := m["ok"]; has {
var ok bool
_ = json.Unmarshal(okRaw, &ok)
var output string
_ = json.Unmarshal(m["output"], &output)
return output, ok
}
return string(raw), true
}
// readAccount reads one profile account's preferences into the shape of the setting.
func readAccount(prefs map[string]string, pa profileAccount) Account {
num := func(key string, def int) int {
if n, err := strconv.Atoi(prefs[key]); err == nil {
return n
}
return def
}
text := func(key string) string { return unquote(prefs[key]) }
srv := "mail.server." + pa.Server + "."
acc := Account{Incoming: Server{Kind: pa.Kind, Host: text(srv + "hostname"), User: text(srv + "userName"),
Security: wordFor(securities, num(srv+"socketType", 0)), SignIn: wordFor(signIns, num(srv+"authMethod", 3))}}
acc.Incoming.Port = num(srv+"port", defaultPort(pa.Kind, acc.Incoming.Security))
ids := listOf(prefs["mail.account."+pa.Key+".identities"])
if len(ids) > 0 {
id := "mail.identity." + ids[0] + "."
acc.Address, acc.FromName = text(id+"useremail"), text(id+"fullName")
smtp := text(id + "smtpServer")
if smtp == "" {
smtp = text("mail.smtp.defaultserver")
}
if smtp != "" {
s := "mail.smtpserver." + smtp + "."
acc.Outgoing = Server{Host: text(s + "hostname"), User: text(s + "username"),
Security: wordFor(securities, num(s+"try_ssl", 0)), SignIn: wordFor(signIns, num(s+"authMethod", 3))}
acc.Outgoing.Port = num(s+"port", defaultPort("smtp", acc.Outgoing.Security))
}
}
if label := text(srv + "name"); label != "" && label != acc.Address {
acc.Label = label
}
return acc
}
// nameFor is the name a profile account is answered under: its own name when the module declared it, else
// the first label of its address's domain, made unique.
func nameFor(pa profileAccount, address string, used map[string]bool) string {
if mm := ownKey.FindStringSubmatch(pa.Key); mm != nil {
return mm[1]
}
base := "account"
if _, domain, ok := strings.Cut(address, "@"); ok {
label, _, _ := strings.Cut(strings.ToLower(domain), ".")
label = regexp.MustCompile(`[^a-z0-9]`).ReplaceAllString(label, "")
if label != "" && label[0] >= 'a' && label[0] <= 'z' {
base = label
}
}
if len(base) > 28 {
base = base[:28]
}
name := base
for i := 2; used[name]; i++ {
name = base + strconv.Itoa(i)
}
return name
}
@@ -0,0 +1,518 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// The setting accounts (novox/hq ADR 0316 §10–§16, ADR 0317 §9 for betterbird_accounts): declared without
// credentials, written under the module's own keys, the account list written into and never owned, refused
// rather than duplicated, and said by count.
const personal = `{"accounts": {
"personal": {"address": "me@mail.example.org", "from-name": "Me",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password", "user": "me"},
"outgoing": {"host": "smtp.example.org", "port": 587, "security": "starttls", "sign-in": "password", "user": "me"}}
}}`
// profileWithAccounts is a profile holding an account of the person's at another address, Owl's work account,
// local folders, and an account list in that order.
func (f *fake) profileWithAccounts() {
f.betterbird()
f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+
`user_pref("mail.account.account1.server", "server1");`+"\n"+
`user_pref("mail.account.account1.identities", "id1");`+"\n"+
`user_pref("mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account3.server", "server3");`+"\n"+
`user_pref("mail.account.account3.identities", "id3");`+"\n"+
`user_pref("mail.accountmanager.accounts", "account1,account3,account2");`+"\n"+
`user_pref("mail.identity.id1.useremail", "other@mail.example.org");`+"\n"+
`user_pref("mail.identity.id1.fullName", "Other");`+"\n"+
`user_pref("mail.identity.id1.smtpServer", "smtp1");`+"\n"+
`user_pref("mail.identity.id3.useremail", "work@corp.example.com");`+"\n"+
`user_pref("mail.server.server1.type", "imap");`+"\n"+
`user_pref("mail.server.server1.hostname", "imap.example.org");`+"\n"+
`user_pref("mail.server.server1.socketType", 3);`+"\n"+
`user_pref("mail.server.server1.authMethod", 10);`+"\n"+
`user_pref("mail.server.server1.userName", "other");`+"\n"+
`user_pref("mail.server.server1.directory", "/home/operator/.thunderbird/x/ImapMail/imap.example.org");`+"\n"+
`user_pref("mail.server.server2.type", "none");`+"\n"+
`user_pref("mail.server.server3.type", "owl-ews");`+"\n"+
`user_pref("mail.smtpserver.smtp1.hostname", "smtp.example.org");`+"\n"+
`user_pref("mail.smtpserver.smtp1.try_ssl", 3);`+"\n"+
`user_pref("mail.smtpserver.smtp1.authMethod", 10);`+"\n"+
`user_pref("mail.smtpserver.smtp1.username", "other");`+"\n"+
`user_pref("mail.smtpservers", "smtp1");`+"\n"+
`user_pref("mailnews.tags.version", 2);`+"\n")
}
func (f *fake) declareAccounts(body string) {
f.write(testHome+"/"+accountsFile, body)
}
func (f *fake) read(path string) string {
f.t.Helper()
raw, err := os.ReadFile(filepath.Join(f.Root, path))
if err != nil {
f.t.Fatal(err)
}
return string(raw)
}
func TestAnAccountsValueCarryingAPasswordLikeFieldIsRefusedWithoutItsValue(t *testing.T) {
for _, body := range []string{
`{"accounts": {"personal": {"address": "me@mail.example.org", "password": "hunter2"}}}`,
`{"accounts": {"personal": {"incoming": {"oauth_refresh": "1//hunter2"}}}}`,
`{"token": "hunter2", "accounts": {}}`,
} {
_, err := parseAccounts([]byte(body))
if err == nil || !strings.Contains(err.Error(), "shaped like a credential") || strings.Contains(err.Error(), "hunter2") {
t.Errorf("%s: %v", body, err)
}
}
// An unknown field is refused too, and a value that cannot be declared, said by the account's place.
if _, err := parseAccounts([]byte(`{"accounts": {"personal": {"address": "me@mail.example.org", "keychain": 1}}}`)); err == nil {
t.Error("an unknown field was taken")
}
_, err := parseAccounts([]byte(strings.Replace(personal, `"security": "tls"`, `"security": "maybe"`, 1)))
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 1") || strings.Contains(err.Error(), "personal") ||
strings.Contains(err.Error(), "example.org") {
t.Errorf("a bad value is said by the account's place, never its name or a value: %v", err)
}
}
func TestTheSettingPrefsRefusesAnAccountsPreferences(t *testing.T) {
for _, name := range []string{"mail.account.account1.server", "mail.identity.id1.useremail", "mail.server.server1.hostname",
"mail.smtpserver.smtp1.hostname", "mail.accountmanager.accounts", "mail.smtpservers"} {
err := validUserJS(mark + ")\nuser_pref(\"" + name + "\", \"x\");\n")
if err == nil || !strings.Contains(err.Error(), "setting accounts") {
t.Errorf("%s was taken in prefs: %v", name, err)
}
}
if err := validUserJS(mark + ")\nuser_pref(\"calendar.alarms.show\", true);\n"); err != nil {
t.Fatal(err)
}
}
// noPackage answers pacman as a machine without the community package.
func (f *fake) noPackage() {
f.answer = func(name string, args []string) Output {
if name == "pacman" {
return Output{Code: 1}
}
return Output{}
}
}
// userPrefLines are the user_pref lines of a user.js, by preference name.
func userPrefLines(t *testing.T, text string) map[string]string {
t.Helper()
out := map[string]string{}
for _, line := range strings.Split(text, "\n") {
if !strings.HasPrefix(line, "user_pref(") {
continue
}
if !statement.MatchString(line) {
t.Errorf("not a statement Betterbird reads as one: %s", line)
}
out[line[len(`user_pref("`):strings.Index(line, `",`)]] = line
}
return out
}
func TestADeclaredAccountTheProfileDoesNotHoldIsCreatedUnderTheModulesKeysAndListed(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
// Betterbird is running: the user.js is placed, the list waits for its next start.
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "running") {
t.Fatalf("the list is never written while Betterbird runs: %+v", u)
}
if len(u.Each) != 1 || u.Each[0].How != created || u.Each[0].Keys != (keySet{"account_mesh_personal", "id_mesh_personal",
"server_mesh_personal", "smtp_mesh_personal"}) {
t.Fatalf("%+v", u.Each)
}
placed := f.read(profileDir + "/user.js")
for name := range userPrefLines(t, placed) {
if key := strings.Split(name, ".")[2]; !strings.HasSuffix(key, "mesh_personal") {
t.Errorf("a created account's preference outside the module's keys: %s", name)
}
if strings.Contains(strings.ToLower(name), "password") || strings.Contains(name, "directory") {
t.Errorf("the module writes no credential and none of Betterbird's own preferences: %s", name)
}
}
for _, want := range []string{`user_pref("mail.server.server_mesh_personal.port", 993);`,
`user_pref("mail.server.server_mesh_personal.socketType", 3);`,
`user_pref("mail.smtpserver.smtp_mesh_personal.try_ssl", 2);`,
`user_pref("mail.identity.id_mesh_personal.useremail", "me@mail.example.org");`} {
if !strings.Contains(placed, want) {
t.Errorf("the user.js lacks %s:\n%s", want, placed)
}
}
// Quit: the list gains the module's keys at its end, every other line as it was.
before := f.read(profileDir + "/prefs.js")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
after := f.read(profileDir + "/prefs.js")
if !strings.Contains(after, `user_pref("mail.accountmanager.accounts", "account1,account3,account2,account_mesh_personal");`) ||
!strings.Contains(after, `user_pref("mail.smtpservers", "smtp1,smtp_mesh_personal");`) {
t.Fatalf("the lists:\n%s", after)
}
bl, al := strings.Split(before, "\n"), strings.Split(after, "\n")
if len(bl) != len(al) {
t.Fatalf("lines were added or lost: %d then %d", len(bl), len(al))
}
for i := range bl {
if bl[i] != al[i] && !strings.Contains(bl[i], "mail.accountmanager.accounts") && !strings.Contains(bl[i], "mail.smtpservers") {
t.Errorf("line %d changed: %q to %q", i+1, bl[i], al[i])
}
}
// The record lost: an account under the module's own keys is still its creation, never adopted.
os.Remove(filepath.Join(f.Root, testHome, recordFile))
u, err = f.userJS(true)
if err != nil || u.Each[0].How != created {
t.Fatalf("without the record, the module's keys still say created: %+v %v", u.Each, err)
}
// The account leaves the setting: only the module's keys go, the mail stays in the profile.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if got := f.read(profileDir + "/prefs.js"); got != before {
t.Fatalf("removing the module's keys left the file otherwise as it was:\n%s", got)
}
}
// adoptable is a declaration of the profile's account1 (other@mail.example.org): equal but for its outgoing
// port, said explicitly, and its incoming user.
const adoptable = `{"accounts": {
"mail": {"address": "Other@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "oauth", "user": "other2"},
"outgoing": {"host": "smtp.example.org", "port": 465, "security": "tls", "sign-in": "oauth", "user": "other"}}}}`
func TestADeclaredAccountTheProfileHoldsIsAdoptedUnderItsOwnKeysAndTheListIsLeftAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
before := f.read(profileDir + "/prefs.js")
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if len(u.Each) != 1 || u.Each[0].How != adopted || u.Each[0].Keys != (keySet{"account1", "id1", "server1", "smtp1"}) {
t.Fatalf("adopted under the profile's keys: %+v", u.Each)
}
// The address's case and the user name differ; the default port declared explicitly is no change. Names
// only, no value.
if strings.Join(u.Each[0].Changed, ",") != "mail.identity.id1.useremail,mail.server.server1.userName" {
t.Fatalf("the changed preferences, by name: %v", u.Each[0].Changed)
}
if f.read(profileDir+"/prefs.js") != before || u.Lists != "unchanged" {
t.Fatalf("an adopted account's keys are in the lists already: nothing written to prefs.js (%s)", u.Lists)
}
lines := userPrefLines(t, f.read(profileDir+"/user.js"))
for name := range lines {
key := strings.Split(name, ".")[2]
if key != "account1" && key != "id1" && key != "server1" && key != "smtp1" {
t.Errorf("an adopted account's preference outside its keys: %s", name)
}
for _, never := range []string{"directory", ".type", "smtpServer", "identities", ".server"} {
if strings.HasSuffix(name, never) {
t.Errorf("the module wrote %s, which is Betterbird's or the account's own", name)
}
}
}
if !strings.Contains(lines["mail.server.server1.userName"], `"other2"`) {
t.Fatalf("the declared value wins at every start: %v", lines)
}
// The status says it by place and keys, never by address, host or name.
s, err := f.Status()
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(s.Accounts)
if s.Accounts.InList != 1 || s.Accounts.Others != 1 || strings.Contains(string(raw), "example") ||
strings.Contains(string(raw), `"mail"`) || !strings.Contains(string(raw), `"account1"`) {
t.Fatalf("%s", raw)
}
// Leaving the setting releases it: its lines and its record go, the account and the lists stay.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if f.read(profileDir+"/prefs.js") != before || strings.Contains(f.read(profileDir+"/user.js"), "server1") ||
strings.Contains(f.read(testHome+"/"+recordFile), "account1") {
t.Fatal("a released account stays the person's, with no line or record of the module's")
}
}
func TestAnAmbiguousOwlsOrOtherKindOfMatchIsRefusedForThatAccountAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account with account1's address.
prefs := f.read(profileDir + "/prefs.js")
prefs = strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account4"`, 1) +
`user_pref("mail.account.account4.server", "server4");` + "\n" +
`user_pref("mail.account.account4.identities", "id4");` + "\n" +
`user_pref("mail.identity.id4.useremail", "other@mail.example.org");` + "\n" +
`user_pref("mail.identity.id4.smtpServer", "smtp4");` + "\n" +
`user_pref("mail.smtpserver.smtp4.hostname", "smtp.example.org");` + "\n" +
`user_pref("mail.server.server4.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
both := strings.Replace(adoptable, `{"accounts": {`, `{"accounts": {
"zz": {"address": "new@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},`, 1)
f.declareAccounts(both)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 2: 2 accounts in the profile have its address (account1, account4)") ||
!strings.Contains(err.Error(), "key") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused by place, with the matches' keys and the remedy, never the address: %v", err)
}
u, _ := f.userJS(false)
if u.Each[0].How != refused || u.Each[1].How != created {
t.Fatalf("the other declared account goes ahead: %+v", u.Each)
}
if strings.Contains(f.read(profileDir+"/user.js"), "server1") || strings.Contains(f.read(profileDir+"/user.js"), "server4") {
t.Fatal("a refused account gets no line")
}
// The node's layer names the key: that one is adopted.
f.declareAccounts(strings.Replace(both, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err != nil {
t.Fatalf("%v", err)
}
if u, _ := f.userJS(false); u.Each[0].How != adopted || u.Each[0].Keys.Account != "account4" {
t.Fatalf("%+v", u.Each)
}
os.Remove(filepath.Join(f.Root, testHome, recordFile))
for key, want := range map[string]string{"account9": "not in the profile's account list", "account3": "Owl's",
"account2": "another address"} {
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "`+key+`",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("key %s: %v", key, err)
}
}
// Owl's account at the address, and an account of another kind than declared.
f.declareAccounts(`{"accounts": {"work": {"address": "work@corp.example.com",
"incoming": {"kind": "imap", "host": "outlook.example.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.example.com", "security": "tls", "sign-in": "oauth"}}}}`)
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "Owl's") {
t.Errorf("Owl's account is never adopted: %v", err)
}
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.server.server4.type", "imap"`, `"mail.server.server4.type", "pop3"`, 1))
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "never converts") {
t.Errorf("an account of another kind is refused, not converted: %v", err)
}
}
func TestDeclaredAccountsWithoutASignInAreCountedFromTheServerNamesAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
links(t, nil)
f.declareAccounts(`{"accounts": {
"personal": {"address": "me@mail.example.org", "incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},
"google": {"address": "me@gmail.com", "incoming": {"kind": "imap", "host": "imap.gmail.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.gmail.com", "security": "tls", "sign-in": "oauth"}},
"lan": {"address": "me@lan.example.org", "incoming": {"kind": "pop3", "host": "pop.example.org", "security": "tls", "sign-in": "kerberos"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "none"}}}}`)
// The username and password fields are encrypted and never read; a fixture holding junk there proves it.
f.write(profileDir+"/logins.json", `{"logins": [
{"hostname": "imap://imap.example.org", "httpRealm": "imap://imap.example.org", "encryptedUsername": "MEIE...", "encryptedPassword": "MDoE..."},
{"hostname": "smtp://smtp.example.org", "encryptedUsername": "x", "encryptedPassword": "y"}]}`)
c := f.countAccounts(filepath.Join(f.Root, profileDir), "~/.thunderbird/60016jsp.default-default")
if c.Declared != 3 || c.WithoutSignIn != 1 || c.Others != 2 || c.InList != 0 {
t.Fatalf("three declared, the OAuth one not signed in, the person's and Owl's other (local folders not counted): %+v", c)
}
raw, _ := json.Marshal(c)
if strings.Contains(string(raw), "example") || strings.Contains(string(raw), "gmail") {
t.Fatalf("a count names nothing: %s", raw)
}
}
func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredential(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
links(t, nil)
a, err := f.ProfileAccounts(false)
if err != nil {
t.Fatal(err)
}
if a.Declarable["imap"] != 1 || a.KeptByOwl != 1 || a.NotDeclarable["local folders"] != 1 || a.Proposed != "" {
t.Fatalf("%+v", a)
}
raw, _ := json.Marshal(a)
for _, never := range []string{"example", "other", "Other", "work@", "smtp1"} {
if strings.Contains(string(raw), never) {
t.Errorf("the answer to the caller names %q: %s", never, raw)
}
}
var asked map[string]any
asker = func(key string, body any) (json.RawMessage, error) {
if key != "seat:mesh-controller.settings" {
t.Fatalf("asked %s", key)
}
asked = body.(map[string]any)
return json.RawMessage(`{"ok": true, "output": "proposed p-17 for betterbird on the whole mesh: accounts (other@mail.example.org); fingerprint ab12"}`), nil
}
t.Cleanup(func() { asker = nil })
a, err = f.ProfileAccounts(true)
if err != nil {
t.Fatal(err)
}
if asked["module"] != "betterbird" || asked["propose"] != "true" || asked["node"] != nil {
t.Fatalf("proposed for the whole mesh, never set: %v", asked)
}
value, err := parseAccounts([]byte(asked["values"].(string)))
if err != nil {
t.Fatalf("the proposed value is one the module takes: %v", err)
}
other, ok := value["mail"]
if len(value) != 1 || !ok || other.Address != "other@mail.example.org" || other.Incoming.SignIn != "oauth" ||
other.Incoming.Port != 993 || other.Outgoing.Security != "tls" || other.Outgoing.Port != 465 {
t.Fatalf("the value holds the person's account and not Owl's: %+v", value)
}
if strings.Contains(a.Proposed, "other@") || !strings.Contains(a.Proposed, "p-17") {
t.Fatalf("the controller's words come back with every address masked: %s", a.Proposed)
}
}
func TestAnAdoptedAccountNeverWritesIntoAnOutgoingServerAnotherIdentityUses(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account of the person's, not declared, sends through smtp1 too.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"account1,account3,account2"`, `"account1,account3,account2,account5"`, 1) +
`user_pref("mail.account.account5.server", "server5");` + "\n" +
`user_pref("mail.account.account5.identities", "id5");` + "\n" +
`user_pref("mail.identity.id5.useremail", "fifth@mail.example.org");` + "\n" +
`user_pref("mail.identity.id5.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.server.server5.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "also used by identity id5") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused with the remedy, by keys: %v", err)
}
if user, _ := os.ReadFile(filepath.Join(f.Root, profileDir, "user.js")); strings.Contains(string(user), "smtpserver.smtp1") {
t.Fatal("the shared outgoing server was written")
}
// Recorded adoption, then the person moves the identity to a server of its own: the module follows it.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.identity.id5.smtpServer", "smtp1"`, `"mail.identity.id5.smtpServer", "smtp5"`, 1))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
moved := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.identity.id1.smtpServer", "smtp1"`, `"mail.identity.id1.smtpServer", "smtp9"`, 1) +
`user_pref("mail.smtpserver.smtp9.hostname", "smtp.example.org");` + "\n"
f.write(profileDir+"/prefs.js", moved)
u, err := f.userJS(true)
if err != nil || u.Each[0].How != adopted || u.Each[0].Keys.SMTP != "smtp9" {
t.Fatalf("the identity's current outgoing server is followed: %+v %v", u.Each, err)
}
user := f.read(profileDir + "/user.js")
if strings.Contains(user, "smtpserver.smtp1.") || !strings.Contains(user, "smtpserver.smtp9.hostname") {
t.Fatalf("never the old one:\n%s", user)
}
// And the move back onto a shared server is refused again.
f.write(profileDir+"/prefs.js", strings.Replace(moved, `"mail.identity.id5.smtpServer", "smtp5"`, `"mail.identity.id5.smtpServer", "smtp9"`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id5") {
t.Fatalf("%v", err)
}
}
func TestTheUserJSAnswerWithholdsTheAccountsValues(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare(`user_pref("mail.compose.default_to", "x");`)
f.declareAccounts(personal)
u, err := f.userJS(false)
if err != nil {
t.Fatal(err)
}
for _, never := range []string{"example.org", `"me"`, `"Me"`} {
if strings.Contains(u.Content, never) {
t.Errorf("the answer carries %s:\n%s", never, u.Content)
}
}
if !strings.Contains(u.Content, `user_pref("mail.server.server_mesh_personal.hostname", "<withheld>");`) {
t.Fatalf("an account's preference is named and its value withheld:\n%s", u.Content)
}
// What is placed in the profile keeps the values: only the answer withholds them.
if f.read(testHome+"/"+accountsFile) == "" {
t.Fatal("no declaration")
}
}
func TestACredentialRefusalNamesNoAccount(t *testing.T) {
_, err := parseAccounts([]byte(`{"accounts": {"personal": {"incoming": {"password": "hunter2"}}}}`))
if err == nil || strings.Contains(err.Error(), "personal") || !strings.Contains(err.Error(), "accounts.<an account>.incoming.password") {
t.Fatalf("%v", err)
}
}
func TestAnAccountsOwnAliasesAndNonMailIdentitiesDoNotShareItsOutgoingServer(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// account1 has an alias identity on smtp1; smtp1 is the default server; Owl's account and local folders
// have identities that name no outgoing server.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.account.account1.identities", "id1"`, `"mail.account.account1.identities", "id1,id1b"`, 1)
prefs = strings.Replace(prefs, `"mail.account.account2.server", "server2");`, `"mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account2.identities", "id2");`, 1) +
`user_pref("mail.identity.id1b.useremail", "alias@mail.example.org");` + "\n" +
`user_pref("mail.identity.id1b.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.identity.id2.useremail", "local@mail.example.org");` + "\n" +
`user_pref("mail.smtp.defaultserver", "smtp1");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
if err := f.Prepare(); err != nil {
t.Fatalf("an account's own alias, and identities that do not send by SMTP, are no sharing: %v", err)
}
// An identity of another mail account naming no server does use the default one.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account6"`, 1)+
`user_pref("mail.account.account6.server", "server6");`+"\n"+
`user_pref("mail.account.account6.identities", "id6");`+"\n"+
`user_pref("mail.identity.id6.useremail", "six@mail.example.org");`+"\n"+
`user_pref("mail.server.server6.type", "pop3");`+"\n")
os.Remove(filepath.Join(f.Root, testHome, recordFile))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id6") {
t.Fatalf("%v", err)
}
}
@@ -0,0 +1,330 @@
package main
// Which account of the profile a declared account is (novox/hq ADR 0317): adopted, created or refused.
//
// At every apply, per declared account in its place in the setting (§1):
//
// 1. an account the module recorded on this node for this profile, adopted or created, is kept by its
// recorded keys without matching again, while its account key is still in the profile's account list;
// 2. otherwise an account under the module's own mesh_<name> keys is the module's creation;
// 3. otherwise a node's layer may name the key to adopt (§8): adopted only when it is in the account list,
// its default identity has the declared address and it is not Owl's;
// 4. otherwise the profile's accounts whose default identity has the declared address: one is adopted, none
// means the account is created under mesh_<name> keys (ADR 0316 §11).
//
// Refused, with the remedy, for that account alone (§7): two or more accounts with the address, the one that
// has it being Owl's, a named key that does not qualify, an account of another kind than the declaration says
// (the module never converts one: the review of ADR 0317), and an identity that names no outgoing server. The
// other declared accounts go ahead.
//
// Adopting takes the account's keys as the module's (§2): its account key, its default identity, its incoming
// server and the outgoing server that identity names. The account lists are left as they are, because the keys
// are in them already; the declared fields are written under those keys (§4, §5). What was adopted or created
// is recorded in the module's directory in the home, by the tools and never by a setting (§3); leaving the
// setting releases an adopted account (its lines and its record go, the account stays the person's, §6).
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// recordFile is where the module records, per profile, what it adopted or created (ADR 0317 §3).
const recordFile = ".config/betterbird/accounts-record.json"
// How a declared account resolved.
const (
adopted = "adopted"
created = "created"
refused = "refused"
)
// keySet are an account's four keys.
type keySet struct {
Account string `json:"account"`
Identity string `json:"identity"`
Server string `json:"server"`
SMTP string `json:"smtp"`
}
// resolvedAccount is one declared account as it resolved, said by its place and keys, never its name, address
// or host (ADR 0316 §16, ADR 0317 §3).
type resolvedAccount struct {
Place int `json:"place"`
How string `json:"how"`
Keys keySet `json:"keys,omitempty"`
// Why is a refusal's reason and its remedy.
Why string `json:"why,omitempty"`
// Changed are the preferences an apply changed for it, by name only (ADR 0317 §4).
Changed []string `json:"changed,omitempty"`
name string
}
// record is what the module adopted or created, per profile (its directory as the tools say it), per
// declared account's name.
type record map[string]map[string]recorded
type recorded struct {
How string `json:"how"`
Keys keySet `json:"keys"`
}
func (m *Machine) readRecord(profileDir string) map[string]recorded {
raw, err := readBounded(m.home(recordFile))
if err != nil {
return map[string]recorded{}
}
var r record
if json.Unmarshal(raw, &r) != nil || r[profileDir] == nil {
return map[string]recorded{}
}
return r[profileDir]
}
// writeRecord keeps what this apply resolved for the profile: a refused account is not recorded, and an
// account no longer declared is dropped (§6).
func (m *Machine) writeRecord(profileDir string, resolved []resolvedAccount) error {
r := record{}
if raw, err := readBounded(m.home(recordFile)); err == nil {
_ = json.Unmarshal(raw, &r)
}
mine := map[string]recorded{}
for _, a := range resolved {
if a.How != refused {
mine[a.name] = recorded{How: a.How, Keys: a.Keys}
}
}
if len(mine) == 0 {
delete(r, profileDir)
} else {
r[profileDir] = mine
}
body, err := json.MarshalIndent(r, "", " ")
if err != nil {
return err
}
path := m.home(recordFile)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
tmp := path + ".mesh-new"
if err := os.WriteFile(tmp, append(body, '\n'), 0o600); err != nil {
return err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return err
}
return nil
}
// resolveAccounts resolves every declared account, in its place in the setting (the file's order, by name).
func resolveAccounts(accounts map[string]Account, prefs map[string]string, rec map[string]recorded) []resolvedAccount {
have := profileAccounts(prefs)
inList := map[string]profileAccount{}
for _, a := range have {
inList[a.Key] = a
}
names := sortedNames(accounts)
taken := map[string]bool{}
var out []resolvedAccount
for i, name := range names {
r := resolveOne(i+1, len(names), name, accounts[name], prefs, have, inList, rec, taken)
r.name = name
if r.How != refused {
taken[r.Keys.Account] = true
}
out = append(out, r)
}
return refuseSharedOutgoing(out, prefs, have)
}
// refuseSharedOutgoing refuses an adopted account whose outgoing server another identity uses that is neither
// adopted nor the module's (review 34 of mesh-catalog #232): Betterbird lets identities share an outgoing
// server, and the declared values written into it would change that other account's sending at every start.
// ADR 0317 §2 adopts the outgoing server the identity names and leaves the lists as they are, so the module
// does not give the identity a server of its own; it refuses, with the remedy.
func refuseSharedOutgoing(resolved []resolvedAccount, prefs map[string]string, have []profileAccount) []resolvedAccount {
// Every identity of an adopted or created account is its own, its aliases too (review 36).
accountOf := map[string]profileAccount{}
kindOf := map[string]string{}
for _, a := range have {
accountOf[a.Key] = a
for _, id := range a.Identities {
kindOf[id] = a.Kind
}
}
ours := map[string]bool{}
for _, r := range resolved {
if r.How == refused {
continue
}
ours[r.Keys.Identity] = true
for _, id := range accountOf[r.Keys.Account].Identities {
ours[id] = true
}
}
defaultServer := unquote(prefs["mail.smtp.defaultserver"])
users := map[string][]string{}
seen := map[string]bool{}
use := func(id string) {
if seen[id] || ours[id] {
return
}
seen[id] = true
server := unquote(prefs["mail.identity."+id+".smtpServer"])
// Only a mail account's identity sends through the default server when it names none: Owl's, local
// folders and feeds do not send by SMTP (review 36).
if server == "" && (kindOf[id] == "imap" || kindOf[id] == "pop3") {
server = defaultServer
}
if server != "" {
users[server] = append(users[server], id)
}
}
for _, a := range have {
for _, id := range a.Identities {
use(id)
}
}
for k := range prefs {
if strings.HasPrefix(k, "mail.identity.") && strings.HasSuffix(k, ".smtpServer") {
use(strings.TrimSuffix(strings.TrimPrefix(k, "mail.identity."), ".smtpServer"))
}
}
for i, r := range resolved {
if r.How != adopted {
continue
}
if others := users[r.Keys.SMTP]; len(others) > 0 {
sort.Strings(others)
resolved[i] = resolvedAccount{Place: r.Place, How: refused, name: r.name, Why: fmt.Sprintf(
"declared account %d of %d: the outgoing server %s of the profile's account %s is also used by "+
"identity %s, whose sending the declared values would change: give the account an outgoing "+
"server of its own in Betterbird's account settings, then start Betterbird again",
r.Place, len(resolved), r.Keys.SMTP, r.Keys.Account, strings.Join(others, ", "))}
}
}
return resolved
}
func resolveOne(place, of int, name string, a Account, prefs map[string]string, have []profileAccount,
inList map[string]profileAccount, rec map[string]recorded, taken map[string]bool) resolvedAccount {
r := resolvedAccount{Place: place}
refuse := func(format string, args ...any) resolvedAccount {
r.How, r.Keys = refused, keySet{}
r.Why = fmt.Sprintf("declared account %d of %d: ", place, of) + fmt.Sprintf(format, args...)
return r
}
// 1. Recorded, while its account is still in the list.
if was, ok := rec[name]; ok {
if _, there := inList[was.Keys.Account]; there || was.How == created {
r.How, r.Keys = was.How, was.Keys
if was.How == adopted {
// The identity's outgoing server as it is now: a person may have moved it since.
smtpKey := unquote(prefs["mail.identity."+was.Keys.Identity+".smtpServer"])
if smtpKey == "" {
return refuse("the profile's account %s has an identity that names no outgoing server: choose one "+
"for it in Betterbird's account settings, then start Betterbird again", was.Keys.Account)
}
r.Keys.SMTP = smtpKey
}
return r
}
}
// 2. Under the module's own keys: its creation.
account, identity, server, smtp := keysOf(name)
own := keySet{account, identity, server, smtp}
if _, there := inList[account]; there {
r.How, r.Keys = created, own
return r
}
adopt := func(pa profileAccount) resolvedAccount {
if pa.Kind != a.Incoming.Kind {
return refuse("the profile's account %s is of kind %s and the declaration says %s: the module never "+
"converts an account. Declare the kind the account has, or remove the account in Betterbird",
pa.Key, pa.Kind, a.Incoming.Kind)
}
smtpKey := unquote(prefs["mail.identity."+pa.Identities[0]+".smtpServer"])
if smtpKey == "" {
return refuse("the profile's account %s has an identity that names no outgoing server: choose one for it "+
"in Betterbird's account settings, then start Betterbird again", pa.Key)
}
r.How, r.Keys = adopted, keySet{pa.Key, pa.Identities[0], pa.Server, smtpKey}
return r
}
matchesDefault := func(pa profileAccount) bool {
return len(pa.Addresses) > 0 && strings.EqualFold(pa.Addresses[0], a.Address)
}
// 3. The key a node's layer names.
if a.Key != "" {
pa, there := inList[a.Key]
switch {
case !there:
return refuse("the key %s this node's layer names is not in the profile's account list", a.Key)
case pa.Owl:
return refuse("the key %s this node's layer names is Owl's account, which is never declared "+
"(novox/hq ADR 0316 §13)", a.Key)
case !matchesDefault(pa):
return refuse("the key %s this node's layer names is an account whose default identity has another "+
"address", a.Key)
case taken[pa.Key]:
return refuse("the key %s is already another declared account's", a.Key)
}
return adopt(pa)
}
// 4. By the default identity's address.
var matches []profileAccount
for _, pa := range have {
if !pa.Ours && !taken[pa.Key] && matchesDefault(pa) {
matches = append(matches, pa)
}
}
switch {
case len(matches) == 0:
r.How, r.Keys = created, own
return r
case len(matches) > 1:
var keys []string
for _, pa := range matches {
keys = append(keys, pa.Key)
}
return refuse("%d accounts in the profile have its address (%s): remove the duplicate in Betterbird, or "+
"give the one to adopt as key in this node's layer of the setting accounts (novox/hq ADR 0317 §7, §8)",
len(matches), strings.Join(keys, ", "))
case matches[0].Owl:
return refuse("the profile's account with its address, %s, is Owl's, which is never declared (novox/hq "+
"ADR 0316 §13): take it out of the setting accounts", matches[0].Key)
}
return adopt(matches[0])
}
// changedPrefs are the preferences of a resolved account whose declared value differs from the profile's
// saved one (prefs.js), by name only (ADR 0317 §4). A preference prefs.js does not hold is changed unless the
// declared value is what Betterbird reads in its place.
func changedPrefs(r resolvedAccount, a Account, saved map[string]string) []string {
defaults := map[string]string{}
if r.How == adopted {
// What Betterbird reads when prefs.js holds no value: the same defaults betterbird_accounts reads with.
pa := profileAccount{Key: r.Keys.Account, Server: r.Keys.Server, Kind: a.Incoming.Kind,
Identities: []string{r.Keys.Identity}}
view := readAccount(saved, pa)
for _, p := range declaredPrefs(r.Keys, view, true) {
defaults[p[0]] = p[1]
}
}
var out []string
for _, p := range declaredPrefs(r.Keys, a, r.How == adopted) {
have, ok := saved[p[0]]
if !ok {
have, ok = defaults[p[0]]
}
if !ok || have != p[1] {
out = append(out, p[0])
}
}
return out
}
@@ -10,19 +10,23 @@ package main
// (passwordSecret, accountPrefix);
// - the profile's extensions.json, for the add-ons' ids and whether each is on;
// - the module's declared user.js (~/.config/betterbird/user.js), which the node-engine writes from
// the module's setting `prefs`.
// the module's setting `prefs`, and its declared accounts (~/.config/betterbird/accounts.json, the
// setting `accounts`, accounts.go);
// - of logins.json, only the server name each kept sign-in is for, which Betterbird keeps in the clear,
// to count the declared accounts that have not signed in yet (novox/hq ADR 0316 §16).
//
// Never read: the mail store, the address books, logins.json, key4.db, cert9.db, cookies, the
// calendar's caches. Nothing of the operator's mail, calendar or accounts is answered.
// Never read: the mail store, the address books, a user name or password in logins.json, key4.db,
// cert9.db, cookies, the calendar's caches. Nothing of the operator's mail or calendar is answered, and of
// their accounts only counts and kinds.
//
// The one file a tool writes in the profile is user.js, and only the module's own: the declared file,
// copied whole (betterbird_user_js with apply, betterbird_restart). A user.js that is not the module's
// is never overwritten.
// The files a tool writes in the profile are user.js, and only the module's own (the declared file and
// the declared accounts' preferences, betterbird_user_js with apply, betterbird_restart, and the start
// from the launcher), and the module's own keys in the account lists of prefs.js, only while Betterbird is
// not running (accounts.go). A user.js that is not the module's is never overwritten.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
@@ -35,8 +39,12 @@ import (
)
const (
mainComm = "betterbird-bin"
startBin = "/usr/bin/betterbird"
mainComm = "betterbird-bin"
// startBin is Betterbird itself, where the mesh's release and the community package both put it. The
// module's start (/usr/local/bin/betterbird) runs the preparation first; a tool that starts Betterbird
// prepares itself and starts this.
startBin = installDir + "/betterbird"
// packageFor is the community package the mesh's release replaces (novox/hq ADR 0316 §4).
packageFor = "betterbird-bin"
appIni = "/opt/betterbird/application.ini"
restartAs = "mesh-betterbird"
@@ -262,6 +270,11 @@ var (
allowedExtensions = map[string]bool{"extensions.activeThemeID": true}
)
// accountOwned are the preferences of an account and of the account lists: the setting accounts' alone, so
// the setting prefs refuses them and an account has one owner (novox/hq ADR 0316 §11).
var accountOwned = []string{"mail.account.", "mail.identity.", "mail.server.", "mail.smtpserver.",
"mail.accountmanager.", "mail.smtpservers", "mail.smtp.defaultserver"}
// withheld says why a preference's value is never answered, or "" when it may be.
func withheld(name string) string {
if strings.HasPrefix(name, "extensions.") && !allowedExtensions[name] {
@@ -392,6 +405,12 @@ func validUserJS(text string) error {
if passwordSecret.MatchString(name) {
return fmt.Errorf("line %d sets %s, a preference that can hold a credential: not through the mesh", i+1, name)
}
for _, p := range accountOwned {
if strings.HasPrefix(name, p) {
return fmt.Errorf("line %d sets %s, an account's preference: an account is declared in the setting "+
"accounts, never in prefs (novox/hq ADR 0316 §11)", i+1, name)
}
}
}
}
return nil
@@ -433,13 +452,35 @@ type UserJS struct {
Effective string `json:"effective,omitempty"`
Content string `json:"content,omitempty"`
Applied bool `json:"applied,omitempty"`
// Accounts are how many accounts the setting accounts declares, and what is wrong with them.
Accounts int `json:"declared_accounts"`
AccountsProblem string `json:"accounts_problem,omitempty"`
// Each is every declared account by its place: adopted, created or refused (with why), its keys, and the
// preferences an apply changed for it, by name (novox/hq ADR 0317 §3, §4).
Each []resolvedAccount `json:"accounts,omitempty"`
// Lists says whether the keys of the accounts the module created were written into the account lists.
Lists string `json:"account_lists,omitempty"`
}
// userJS compares the declared user.js with the profile's; apply places the declared one when the
// profile has none or the module's.
// refusedAccounts are the reasons of the declared accounts that were refused.
func (u UserJS) refusedAccounts() []string {
var out []string
for _, r := range u.Each {
if r.How == refused {
out = append(out, r.Why)
}
}
return out
}
// userJS compares the user.js the module places — the declared file, then the declared accounts' fields under
// the keys each was adopted or created under (adoption.go) — with the profile's; apply places it when the
// profile has none or the module's, records what was adopted or created, and writes the keys of the accounts
// it created into the profile's account lists when Betterbird is not running. A refused account gets nothing;
// the others go ahead (novox/hq ADR 0317 §7).
func (m *Machine) userJS(apply bool) (UserJS, error) {
a := UserJS{Declared: "~/" + declared}
want, err := readBounded(m.home(declared))
declaredText, err := readBounded(m.home(declared))
a.Present = err == nil
p, perr := m.profile()
if perr != nil {
@@ -447,11 +488,21 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
}
target := filepath.Join(p.path, "user.js")
a.Profile = m.tilde(strings.TrimPrefix(target, m.Root))
accounts, _, aerr := m.readAccounts()
if aerr != nil {
a.AccountsProblem = aerr.Error()
accounts = nil
} else {
a.Accounts = len(accounts)
a.Each = resolveAccounts(accounts, mergedPrefs(p.path), m.readRecord(p.Dir))
}
var want []byte
if a.Present {
if verr := validUserJS(string(want)); verr != nil {
if verr := validUserJS(string(declaredText)); verr != nil {
a.Valid = verr.Error()
}
a.Content = string(want)
want = []byte(composeUserJS(string(declaredText), accounts, a.Each))
a.Content = m.answeredUserJS(string(want))
for _, line := range strings.Split(string(want), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
if statement.MatchString(st) {
@@ -477,6 +528,17 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
return a, fmt.Errorf("the module's user.js is not on this machine (~/%s): the module is not pushed here yet", declared)
case a.Valid != "":
return a, fmt.Errorf("the module's user.js is refused: %s; change the setting prefs", a.Valid)
case a.AccountsProblem != "":
return a, fmt.Errorf("nothing was placed: %s", a.AccountsProblem)
}
saved, _ := readBounded(filepath.Join(p.path, "prefs.js"))
savedPrefs := readPrefs(saved)
for i, r := range a.Each {
if r.How != refused {
a.Each[i].Changed = changedPrefs(r, accounts[r.name], savedPrefs)
}
}
switch {
case herr == nil && !bytes.HasPrefix(have, []byte(mark)):
return a, fmt.Errorf("%s is not the module's: move it aside (its preferences belong in the setting prefs), then apply again", a.Profile)
}
@@ -491,6 +553,14 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
}
}
a.Applied, a.Placed = true, "the module's, as declared"
if err := m.writeRecord(p.Dir, a.Each); err != nil {
return a, fmt.Errorf("the user.js is placed, and what was adopted or created could not be recorded: %v", err)
}
if lists, err := m.writeAccountLists(p.path, a.Each); err != nil {
a.Lists = "not written: " + err.Error()
} else {
a.Lists = lists
}
}
if ps := m.mainProcs(); len(ps) > 0 {
if st, err := os.Stat(target); err == nil {
@@ -504,27 +574,74 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
return a, nil
}
// answeredUserJS is the user.js as a tool answers it (novox/hq ADR 0316 §16, ADR 0317 §9): an account's
// preferences with their values withheld, every other value masked as betterbird_prefs masks it.
func (m *Machine) answeredUserJS(text string) string {
lines := strings.Split(text, "\n")
for i, line := range lines {
for _, st := range splitStatements(strings.TrimSpace(line)) {
mm := prefLine.FindStringSubmatch(st)
if mm == nil {
continue
}
value := m.maskValue(mm[2])
if why := withheld(mm[1]); why != "" {
value = `"<withheld>"`
}
lines[i] = fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), value)
}
}
return strings.Join(lines, "\n")
}
// Prepare is what the module's start runs before Betterbird (`betterbird-tools prepare`): with no Betterbird
// of this account running, the user.js placed and the declared accounts written into the account lists. With
// one running it does nothing, since the start then only hands over to it. Its error is said by the start,
// which starts Betterbird either way.
func (m *Machine) Prepare() error {
if len(m.mainProcs()) > 0 {
return nil
}
u, err := m.userJS(true)
if err != nil {
return err
}
var problems []string
if strings.HasPrefix(u.Lists, "not written") {
problems = append(problems, "the account lists were "+u.Lists)
}
problems = append(problems, u.refusedAccounts()...)
if len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
return nil
}
// Status is what betterbird_status answers.
type Status struct {
Running []Proc `json:"running"`
Children int `json:"children"`
Installed string `json:"installed,omitempty"`
Running []Proc `json:"running"`
Children int `json:"children"`
// Installation is what is installed and from where, the policy and Owl (novox/hq ADR 0316 §16).
Installation Installation `json:"installation"`
// Installed is the community package's version, while it is still there.
Installed string `json:"community_package,omitempty"`
Foreign bool `json:"from_the_aur,omitempty"`
Version string `json:"version,omitempty"`
Build string `json:"build,omitempty"`
Profile *Profile `json:"profile,omitempty"`
Problem string `json:"profile_problem,omitempty"`
Output []Stream `json:"output,omitempty"`
UserJS *UserJS `json:"user_js,omitempty"`
AddOns []AddOn `json:"add_ons,omitempty"`
// Accounts are counts: the declared, those in the profile's list, the others, those without a sign-in.
Accounts *AccountCount `json:"accounts,omitempty"`
AddOns []AddOn `json:"add_ons,omitempty"`
// Display are the preferences that decide how its windows and notifications look.
Display []Pref `json:"display_prefs,omitempty"`
}
// AddOn is one add-on the operator installed: its id, and whether it is on.
type AddOn struct {
ID string `json:"id"`
Active bool `json:"active"`
ID string `json:"id"`
Version string `json:"version,omitempty"`
Active bool `json:"active"`
}
// displayPrefs are the preferences betterbird_status answers: the reminder dialog, the theme, the
@@ -550,39 +667,24 @@ func (m *Machine) Status() (Status, error) {
if v, err := m.installed(packageFor); err == nil && v != "" {
s.Installed, s.Foreign = v, m.foreign(packageFor)
}
if raw, err := readBounded(m.path(appIni)); err == nil {
_, sec := iniSections(raw)
s.Version, s.Build = sec["App"]["Version"], sec["App"]["BuildID"]
}
if len(s.Running) > 0 {
s.Output = m.streams(s.Running[0].PID)
}
p, err := m.profile()
if err != nil {
s.Installation = m.installation("")
s.Problem = err.Error()
return s, nil
}
s.Profile = &p
s.Installation = m.installation(p.path)
if u, err := m.userJS(false); err == nil {
u.Content = ""
s.UserJS = &u
}
if raw, err := readBounded(filepath.Join(p.path, "extensions.json")); err == nil {
var ext struct {
Addons []struct {
ID string `json:"id"`
Active bool `json:"active"`
Location string `json:"location"`
} `json:"addons"`
}
if json.Unmarshal(raw, &ext) == nil {
for _, a := range ext.Addons {
if a.Location == "app-profile" {
s.AddOns = append(s.AddOns, AddOn{a.ID, a.Active})
}
}
}
}
c := m.countAccounts(p.path, p.Dir)
s.Accounts = &c
s.AddOns = addOns(p.path)
if got, err := m.Prefs(displayPrefs); err == nil {
s.Display = got["prefs"].([]Pref)
}
@@ -711,7 +813,8 @@ func (m *Machine) Restart(discard bool) (RestartAnswer, error) {
if u, err := m.userJS(true); err != nil {
a.UserJS = "not placed: " + err.Error()
} else {
a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s", u.Statements, declared)
a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s, %d declared account(s); account lists %s",
u.Statements, declared, u.Accounts, u.Lists)
}
// As detach does, with shorter bounds: the worst case of the whole call (3 s for the window tree,
// 8 s to quit, 3 + 5 s to start, 4 s to see it) stays under the runtime's 30 s.
@@ -750,12 +853,56 @@ type CheckAnswer struct {
func (m *Machine) Check() (CheckAnswer, error) {
a := CheckAnswer{Findings: []Finding{}}
add := func(what, do string) { a.Findings = append(a.Findings, Finding{what, do}) }
v, err := m.installed(packageFor)
if err != nil {
if _, err := m.installed(packageFor); err != nil {
return a, err
}
if v == "" {
add("Betterbird ("+packageFor+") is not installed", "install it from the AUR: the node-engine installs packages from the official repositories only, so the module does not")
p, perr := m.profile()
profilePath := ""
if perr == nil {
profilePath = p.path
}
in := m.installation(profilePath)
switch {
case in.From == "not installed":
add("Betterbird is not installed in "+installDir, "the module's push places the mesh's release; it waits for the operator's push (upgrade policy record)")
case in.From != "the mesh":
add("Betterbird is "+in.From, "quit Betterbird, then push the module to this node: the push removes the community package and places the mesh's release")
}
if in.Behind {
add("Betterbird "+in.Version+" (build "+in.Build+") is behind the newest published build "+in.Newest,
"move the module's pin in a reviewed change (novox/hq ADR 0316 §5)")
}
if in.From == "the mesh" && !strings.HasPrefix(in.Policy, "placed:") {
add("the enterprise policy is "+in.Policy, "push the module to this node")
}
switch {
case in.From != "the mesh":
case in.Owl == nil || in.Owl.Pinned == "":
add("Owl's xpi is not placed ("+owlFile+")", "push the module to this node")
case in.Owl.InProfile == "":
add("Owl is not installed in the profile yet", "Betterbird installs it from the policy at its next start")
case !in.Owl.AtThePin && !newerVersion(in.Owl.InProfile, in.Owl.Pinned):
add("Owl "+in.Owl.InProfile+" runs in the profile, older than the mesh's pin "+in.Owl.Pinned,
"Owl is installed by policy and cannot be removed in Betterbird: move the module's Owl pin in a reviewed change (novox/hq ADR 0316 §6)")
case !in.Owl.Active:
add("Owl is installed and not active", "turn it on in Betterbird's add-ons")
}
if perr == nil {
c := m.countAccounts(p.path, p.Dir)
for _, r := range c.Each {
if r.How == refused {
add("refused: "+r.Why, "as it says; the other declared accounts go ahead")
}
}
switch {
case c.InList < c.Declared:
add(fmt.Sprintf("%d of %d declared account(s) are not in the profile's account list yet", c.Declared-c.InList, c.Declared),
"quit Betterbird and start it from the launcher, or betterbird_restart")
}
if c.WithoutSignIn > 0 {
add(fmt.Sprintf("%d declared account(s) have not signed in yet", c.WithoutSignIn),
"a person signs in once, in Betterbird's own window, when it asks (novox/hq ADR 0316 §14)")
}
}
ps := m.mainProcs()
switch {
@@ -772,6 +919,8 @@ func (m *Machine) Check() (CheckAnswer, error) {
add("the module's user.js is not on this machine (~/"+declared+")", "the module is not pushed here: the controller's push")
case u.Valid != "":
add("the module's user.js is refused: "+u.Valid, "change the module's setting prefs")
case u.AccountsProblem != "":
add("the module's user.js is not placed: "+u.AccountsProblem, "change the module's setting accounts at the controller's terminal, or propose it (betterbird_accounts)")
case !strings.HasPrefix(u.Placed, "the module's, as declared"):
add("the profile's user.js is "+u.Placed, "betterbird_user_js with apply, then betterbird_restart")
case strings.HasPrefix(u.Effective, "not yet"):
@@ -161,7 +161,7 @@ func TestStatusSaysWhatRunsTheProfileTheAddOnsAndTheDisplayPrefs(t *testing.T) {
t.Fatal(err)
}
if len(s.Running) != 1 || s.Running[0].PID != 4000 || s.Children != 1 || s.Installed != "153.1.0esr-1" || !s.Foreign ||
s.Version != "153.1.0" || s.Profile == nil || len(s.AddOns) != 1 || s.AddOns[0].ID != "owl@beonex.com" {
s.Installation.Version != "153.1.0" || s.Profile == nil || len(s.AddOns) != 1 || s.AddOns[0].ID != "owl@beonex.com" {
t.Fatalf("%+v", s)
}
if !s.Output[0].Dead {
@@ -238,6 +238,13 @@ func TestTheReminderTestTouchesNothingOfTheOperatorsProfile(t *testing.T) {
!strings.Contains(string(user), "file:///run/user/1000/"+testUnit+"/test.ics") || strings.Contains(string(user), "someone@") {
t.Fatalf("%s", user)
}
// Betterbird's update check stays off, so no About window opens on the operator's screen, and the
// test's own preferences come last, so nothing carried or declared overrides them.
if !strings.Contains(string(user), `"betterbird.update.checkIntervalDays", 0`) ||
strings.Index(string(user), "extensions.activeThemeID") > strings.Index(string(user), "calendar.registry.meshtest.uri") ||
strings.Index(string(user), `"calendar.alarms.show"`) > strings.Index(string(user), "calendar.registry.meshtest.uri") {
t.Fatalf("%s", user)
}
// The event starts 10 minutes after the reminder, which is due 40 s from now.
if !strings.Contains(string(cal), "DTSTART:20270115T081040Z") || !strings.Contains(string(cal), "TRIGGER:-PT10M") ||
a.Due != "2027-01-15T08:00:40Z" {
@@ -0,0 +1,253 @@
package main
// What the mesh installs (novox/hq ADR 0316 §1–§9, to-be 56 §1–§3): Betterbird from its pinned upstream release
// in /opt/betterbird, where the community package betterbird-bin put it, so Betterbird keeps choosing the same
// profile; the enterprise policy in its distribution directory, which turns Betterbird's own updater off and
// installs Owl into every profile from the xpi the mesh placed beside it; and a defaults file naming the
// system's spelling dictionaries. The tools read these files and say what they find: the version against the
// pin and the newest published build, whether the policy is placed, Owl's version against its pin.
//
// **The pin on the machine is what the mesh placed.** The manifest pins the release and the xpi by sha256; the
// node-engine places exactly those bytes, so the version in /opt/betterbird/application.ini (when the mesh
// placed it) and the version in the placed xpi are the pins. The community package's own copy is said as
// such, and the switch from it waits for the operator (ADR 0316 §4).
import (
"archive/zip"
"encoding/json"
"fmt"
"io"
"net/http"
"path/filepath"
"strings"
"time"
)
const (
installDir = "/opt/betterbird"
// policyFile is the enterprise policy the module places (ADR 0316 §7).
policyFile = installDir + "/distribution/policies.json"
// owlFile is Owl's xpi as the mesh placed it, and the URL the policy installs it from.
owlFile = installDir + "/owl/owl.xpi"
owlID = "owl@beonex.com"
// preparation is the tools' program as the module places it for its start (betterbird-tools prepare).
preparation = "/usr/local/lib/mesh-betterbird/betterbird-tools"
// versionsURL is where Betterbird publishes its newest build per major version: what its own updater
// reads (MailGlue's BetterbirdUpdateCheck), and what the status compares the installed build with.
versionsURL = "https://www.betterbird.eu/start/versions.txt"
)
// publishedBuilds fetches Betterbird's list of newest builds: a variable so a test answers.
var publishedBuilds = func() (string, error) {
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Get(versionsURL)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("%s answered %s", versionsURL, resp.Status)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
return string(body), err
}
// Installation is what is installed, and from where.
type Installation struct {
// From is the mesh, the community package (betterbird-bin, before the switch), or nothing.
From string `json:"from"`
Version string `json:"version,omitempty"`
Build string `json:"build,omitempty"`
// Newest is the newest build Betterbird published for this major version, and whether this one is behind.
Newest string `json:"newest_published_build,omitempty"`
Behind bool `json:"behind,omitempty"`
// Policy says whether the module's enterprise policy is placed, and what it does.
Policy string `json:"policy"`
// Owl is the add-on's version as the mesh placed it (the pin), and as the profile runs it.
Owl *OwlState `json:"owl,omitempty"`
}
// OwlState is Owl against its pin.
type OwlState struct {
Pinned string `json:"pinned,omitempty"`
InProfile string `json:"in_profile,omitempty"`
Active bool `json:"active"`
AtThePin bool `json:"at_the_pin"`
// Note says, as a fact, a profile running an Owl newer than the pin: the policy's install keeps a newer
// version, and Owl installed by policy cannot be removed in Betterbird.
Note string `json:"note,omitempty"`
}
// newerVersion says whether version a is newer than b, comparing dotted numbers.
func newerVersion(a, b string) bool {
if a == "" || b == "" {
return false
}
pa, pb := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; i < len(pa) || i < len(pb); i++ {
var x, y int
if i < len(pa) {
fmt.Sscanf(pa[i], "%d", &x)
}
if i < len(pb) {
fmt.Sscanf(pb[i], "%d", &y)
}
if x != y {
return x > y
}
}
return false
}
// installation reads what is installed. profile may be empty when none is found yet.
func (m *Machine) installation(profile string) Installation {
var in Installation
community, _ := m.installed(packageFor)
raw, err := readBounded(m.path(appIni))
switch {
case community != "":
in.From = "the community package " + packageFor + " " + community + ": the switch to the mesh's release waits for the operator's push, with Betterbird quit (novox/hq ADR 0316 §4)"
case err == nil:
in.From = "the mesh"
default:
in.From = "not installed"
}
if err == nil {
_, sec := iniSections(raw)
in.Version, in.Build = sec["App"]["Version"], sec["App"]["BuildID"]
in.Newest, in.Behind = newestBuild(in.Version, in.Build)
}
in.Policy = m.policyState()
pinned := xpiVersion(m.path(owlFile))
owl := &OwlState{Pinned: pinned}
if profile != "" {
for _, a := range addOns(profile) {
if a.ID == owlID {
owl.InProfile, owl.Active = a.Version, a.Active
}
}
}
owl.AtThePin = owl.Pinned != "" && owl.Pinned == owl.InProfile
if newerVersion(owl.InProfile, owl.Pinned) {
owl.Note = "the profile runs Owl " + owl.InProfile + ", newer than the mesh's pin " + owl.Pinned + ": the policy " +
"keeps a newer version and installs the pin only where Owl is missing. To run the same Owl everywhere, " +
"move the module's pin in a reviewed change"
}
if owl.Pinned != "" || owl.InProfile != "" {
in.Owl = owl
}
return in
}
// newestBuild is the newest build Betterbird published for a version's major, and whether build is older.
func newestBuild(version, build string) (string, bool) {
major, _, _ := strings.Cut(version, ".")
if major == "" || build == "" {
return "", false
}
list, err := publishedBuilds()
if err != nil {
return "unknown: " + err.Error(), false
}
for _, l := range strings.Split(list, "\n") {
f := strings.Split(strings.TrimSpace(l), ",")
if len(f) == 3 && f[0] == "Linux" && f[1] == major {
return f[2], f[2] > build
}
}
return "unknown: " + versionsURL + " names no build for " + major, false
}
// policyState says whether the module's policy is placed and does what ADR 0316 §5 and §7 say.
func (m *Machine) policyState() string {
raw, err := readBounded(m.path(policyFile))
if err != nil {
return "not placed: " + policyFile + " is missing"
}
if why := policyProblem(raw); why != "" {
return "placed, and " + why
}
return "placed: Betterbird's updater off, Owl installed into every profile from the mesh's xpi, its updates off"
}
// policyProblem is what the policy fails to say of what ADR 0316 asks, or "".
func policyProblem(raw []byte) string {
var p struct {
Policies struct {
DisableAppUpdate bool `json:"DisableAppUpdate"`
ExtensionSettings map[string]struct {
Mode string `json:"installation_mode"`
URL string `json:"install_url"`
Disabled bool `json:"updates_disabled"`
} `json:"ExtensionSettings"`
} `json:"policies"`
}
if err := json.Unmarshal(raw, &p); err != nil {
return "it is not JSON: " + err.Error()
}
owl, ok := p.Policies.ExtensionSettings[owlID]
switch {
case !p.Policies.DisableAppUpdate:
return "it does not turn Betterbird's updater off (DisableAppUpdate)"
case !ok || owl.Mode != "force_installed":
return "it does not install Owl into every profile (force_installed)"
case owl.URL != "file://"+owlFile:
return "it installs Owl from " + owl.URL + ", not from the mesh's xpi"
case !owl.Disabled:
return "it leaves Owl's updates on"
}
return ""
}
// xpiVersion is the version an add-on's manifest.json says, or "" when it cannot be read.
func xpiVersion(path string) string {
r, err := zip.OpenReader(path)
if err != nil {
return ""
}
defer r.Close()
for _, f := range r.File {
if f.Name != "manifest.json" {
continue
}
rc, err := f.Open()
if err != nil {
return ""
}
defer rc.Close()
var mf struct {
Version string `json:"version"`
}
if json.NewDecoder(io.LimitReader(rc, 1<<20)).Decode(&mf) != nil {
return ""
}
return mf.Version
}
return ""
}
// addOns reads a profile's extensions.json: each add-on's id, version, whether it is on, and where from.
func addOns(profile string) []AddOn {
raw, err := readBounded(filepath.Join(profile, "extensions.json"))
if err != nil {
return nil
}
var ext struct {
Addons []struct {
ID string `json:"id"`
Version string `json:"version"`
Active bool `json:"active"`
Location string `json:"location"`
} `json:"addons"`
}
if json.Unmarshal(raw, &ext) != nil {
return nil
}
var out []AddOn
for _, a := range ext.Addons {
if a.Location == "app-profile" {
out = append(out, AddOn{ID: a.ID, Version: a.Version, Active: a.Active})
}
}
return out
}
@@ -0,0 +1,174 @@
package main
import (
"archive/zip"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// What the mesh installs, as the tools say it (novox/hq ADR 0316 §16): the version against the pin and the
// newest published build, the policy, Owl against its pin; and the reminder test, visibly a test.
func (f *fake) xpi(path, version string) {
f.t.Helper()
full := filepath.Join(f.Root, path)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
f.t.Fatal(err)
}
out, err := os.Create(full)
if err != nil {
f.t.Fatal(err)
}
w := zip.NewWriter(out)
m, _ := w.Create("manifest.json")
m.Write([]byte(`{"version": "` + version + `", "browser_specific_settings": {"gecko": {"id": "owl@beonex.com"}}}`))
w.Close()
out.Close()
}
// placedByTheMesh is a node after the switch: the release, the policy and Owl's xpi in /opt/betterbird, and
// the profile running Owl at the pin.
func (f *fake) placedByTheMesh() {
f.noPackage()
f.write(appIni, "[App]\nVendor=Betterbird Project\nVersion=153.4.0\nBuildID=20260929222610\n")
raw, err := os.ReadFile(filepath.Join("..", "..", "install", "distribution", "policies.json"))
if err != nil {
f.t.Fatal(err)
}
f.write(policyFile, string(raw))
f.xpi(owlFile, "1.5.6.1")
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.6.1","active":true,"location":"app-profile"}]}`)
}
func TestTheStatusSaysTheInstallationAgainstThePinTheNewestBuildThePolicyAndOwl(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
publishedBuilds = func() (string, error) {
return "blocklist,{}\nLinux,153,20261101000000\nLinux,140,20260829111240\n", nil
}
t.Cleanup(func() { publishedBuilds = func() (string, error) { return "", os.ErrDeadlineExceeded } })
s, err := f.Status()
if err != nil {
t.Fatal(err)
}
in := s.Installation
if in.From != "the mesh" || in.Version != "153.4.0" || in.Newest != "20261101000000" || !in.Behind ||
!strings.HasPrefix(in.Policy, "placed:") || in.Owl == nil || in.Owl.Pinned != "1.5.6.1" || !in.Owl.AtThePin || !in.Owl.Active {
t.Fatalf("%+v %+v", in, in.Owl)
}
c, err := f.Check()
if err != nil {
t.Fatal(err)
}
found := false
for _, x := range c.Findings {
found = found || strings.Contains(x.What, "behind the newest published build 20261101000000")
}
if !found {
t.Fatalf("a release left behind is said: %+v", c.Findings)
}
}
func TestTheCommunityPackageAPolicyThatIsNotTheModulesAndAnOwlOffItsPinAreSaid(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
f.write(policyFile, `{"policies": {"ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "https://vendor.example/owl.xpi", "updates_disabled": true}}}}`)
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.7","active":true,"location":"app-profile"}]}`)
in := f.installation(filepath.Join(f.Root, profileDir))
if !strings.Contains(in.Policy, "DisableAppUpdate") || in.Owl.AtThePin || in.Owl.InProfile != "1.5.7" {
t.Fatalf("%+v %+v", in, in.Owl)
}
// A newer Owl than the pin is a fact with the real remedy, never a finding to remove it.
if !strings.Contains(in.Owl.Note, "move the module's pin") {
t.Fatalf("%+v", in.Owl)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`)
c, _ := f.Check()
for _, x := range c.Findings {
if strings.Contains(x.What, "Owl") {
t.Errorf("a newer Owl is not a finding: %+v", x)
}
}
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.5","active":true,"location":"app-profile"}]}`)
c, _ = f.Check()
found := false
for _, x := range c.Findings {
found = found || (strings.Contains(x.What, "older than the mesh's pin") && strings.Contains(x.Do, "move the module's Owl pin"))
}
if !found {
t.Fatalf("an older Owl is a finding whose remedy is the pin: %+v", c.Findings)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "https://vendor.example/owl.xpi", "updates_disabled": true}}}}`)
if in := f.installation(""); !strings.Contains(in.Policy, "not from the mesh's xpi") {
t.Fatalf("%s", in.Policy)
}
f.answer = func(name string, args []string) Output {
if name == "pacman" && args[0] == "-Q" {
return Output{Stdout: "betterbird-bin 153.1.0esr-1\n"}
}
return Output{}
}
in = f.installation("")
if !strings.HasPrefix(in.From, "the community package betterbird-bin 153.1.0esr-1") || !strings.Contains(in.From, "operator") {
t.Fatalf("%s", in.From)
}
raw, _ := json.Marshal(in)
if strings.Contains(string(raw), "example.org") {
t.Fatalf("%s", raw)
}
}
func TestTheReminderTestIsVisiblyATestAndSaysItOpensAWindow(t *testing.T) {
f := newFake(t)
f.desktopSession()
f.betterbird()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
a, err := f.ReminderTest(40*time.Second, nil, false)
if err != nil {
t.Fatal(err)
}
dir := filepath.Join("/run/user/1000", testUnit)
css := f.read(dir + "/profile/chrome/userChrome.css")
user := f.read(dir + "/profile/user.js")
cal := f.read(dir + "/test.ics")
if !strings.Contains(css, "MESH TEST") || !strings.Contains(css, `:root:not([windowtype="Calendar:AlarmWindow"])`) {
t.Fatalf("the test's windows but the reminder carry a MESH TEST band: %s", css)
}
if !strings.Contains(user, `user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);`) ||
strings.Contains(user, "mesh_personal") || strings.Contains(user, "example.org") {
t.Fatalf("the test profile reads its userChrome.css and holds no declared account: %s", user)
}
if !strings.Contains(cal, "SUMMARY:MESH TEST reminder") || !strings.Contains(a.OnScreen, "operator's screen") {
t.Fatalf("the event and the answer say it is a test on the operator's screen: %s %+v", cal, a)
}
if !f.called("systemd-run --user --collect --quiet --unit="+testUnit) || !strings.Contains(strings.Join(f.calls, "\n"), startBin+" --no-remote") {
t.Fatalf("the test starts Betterbird itself, never the module's start: %v", f.calls)
}
for _, tool := range tools() {
if tool.Name == "betterbird_reminder_test" && !strings.HasPrefix(tool.Description, "OPENS WINDOWS ON THE OPERATOR'S SCREEN") {
t.Fatalf("the description says first that a window opens on the operator's screen: %s", tool.Description)
}
}
}
func TestTheUserJSDescriptionSaysAdoption(t *testing.T) {
for _, tool := range tools() {
if tool.Name == "betterbird_user_js" && (!strings.Contains(tool.Description, "adopted under its own keys") ||
strings.Contains(tool.Description, "refuses the apply")) {
t.Fatalf("%s", tool.Description)
}
}
}
+58 -18
View File
@@ -16,6 +16,15 @@ import (
)
func main() {
// The module's start runs this before Betterbird (novox/hq ADR 0316 §11, §12): the user.js placed and the
// declared accounts written into the account lists, while no Betterbird of this account runs.
if len(os.Args) == 2 && os.Args[1] == "prepare" {
if err := machine.Prepare(); err != nil {
fmt.Fprintln(os.Stderr, "betterbird: "+err.Error())
os.Exit(1)
}
return
}
if err := stdio.Serve("", tools()); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
@@ -29,10 +38,13 @@ func tools() []stdio.Tool {
{
Name: "betterbird_status",
Description: "Betterbird: whether it runs (the main process's pid, since, and the scope it runs in; how " +
"many child processes), the installed package and version and whether it is from the AUR, where its " +
"output goes, the profile it uses and how that was found, the module's user.js (declared, placed, in " +
"effect), the add-ons the operator installed, and the preferences that decide how its windows and " +
"notifications look (reminder dialog, theme, renderer, system notifications). (r)",
"many child processes); what is installed in /opt/betterbird and from where (the mesh's pinned release, " +
"or the community package before the switch), its version and build beside the newest published build; " +
"whether the enterprise policy is placed; Owl's version against the mesh's pin and whether it is active; " +
"where its output goes; the profile it uses and how that was found; the module's user.js (declared, " +
"placed, in effect); the declared accounts by count (in the profile's list, other accounts, not signed " +
"in yet), never an address, host or name; the add-ons in the profile; and the preferences that decide " +
"how its windows and notifications look. (r)",
Run: func(map[string]any) (any, error) { return machine.Status() },
},
{
@@ -55,11 +67,15 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_user_js",
Description: "The module's user.js — the preferences the module declares, from its setting prefs, written by " +
"the mesh to ~/.config/betterbird/user.js — against the profile's user.js: whether it is placed, and " +
"whether the running Betterbird started after it was. With apply, place it in the profile (only where " +
"the profile has no user.js or the module's own; an operator's file is never overwritten). Betterbird " +
"reads it at its next start: betterbird_restart. (r; a with apply)",
Description: "The module's user.js — the preferences the module declares (setting prefs), then the declared " +
"accounts' preferences under the module's own keys (setting accounts) — against the profile's user.js: " +
"whether it is placed, and whether the running Betterbird started after it was. With apply, place it in " +
"the profile (only where the profile has no user.js or the module's own; an operator's file is never " +
"overwritten). A declared account the profile already holds at its address is adopted under its own keys; " +
"one it does not hold is created under the module's keys, which are added to the account lists when " +
"Betterbird is not running; an ambiguous one is refused alone, with the remedy (novox/hq ADR 0317). " +
"Accounts' values are withheld in the answer. Betterbird reads it at its next start: betterbird_restart. " +
"(r; a with apply)",
Input: map[string]any{
"apply": map[string]any{"type": "boolean", "description": "place the declared user.js in the profile (default false)"},
},
@@ -118,19 +134,43 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_check",
Description: "Check what the module promises: Betterbird is installed; one runs; its profile is found; the " +
"module's user.js is on the machine, valid, placed in the profile and in effect; its output is read " +
"(a pipe nobody reads loses what it prints). Answers ok and each finding with what to do. (r)",
Description: "Check what the module promises: Betterbird installed from the mesh's pinned release and not " +
"behind the newest published build; the enterprise policy placed; Owl at its pin and active; one " +
"Betterbird runs; its profile is found; the module's user.js on the machine, valid, placed and in " +
"effect; the declared accounts in the profile's list, none refused, each signed in once; its output " +
"read. Answers ok and each finding with what to do, never an address, host or name. (r)",
Run: func(map[string]any) (any, error) { return machine.Check() },
},
{
Name: "betterbird_accounts",
Description: "The accounts of the profile Betterbird uses, by count and kind: the IMAP and POP3 accounts a " +
"declaration can hold, the accounts Owl keeps (never declared), and the rest. Never an address, a host " +
"or a name, and never anything of the login store. With propose, it builds the module's setting " +
"accounts from them (every account but Owl's, without any password or token) and proposes it through " +
"the controller's settings verb for the operator's Approve on a channel that proves who answers " +
"(novox/hq ADR 0317, ADR 0277): nothing changes until then, and the operator reads the value whole at " +
"their desk. (r; a with propose)",
Input: map[string]any{
"propose": map[string]any{"type": "boolean", "description": "build the setting accounts from the profile and propose it to the operator (default false)"},
},
Run: func(args map[string]any) (any, error) {
propose, err := flag(args, "propose", false)
if err != nil {
return nil, err
}
return machine.ProfileAccounts(propose)
},
},
{
Name: "betterbird_reminder_test",
Description: "Show a calendar reminder without touching the operator's calendar: a second Betterbird in a " +
"throw-away profile under the runtime directory (--no-remote, unit mesh-betterbird-reminder-test, at most " +
"10 minutes), with the module's user.js and one local calendar file holding one event whose reminder is " +
"due after `after` seconds. carry copies named preferences of the operator's profile (a theme, a " +
"colour switch) into it, never a credential or an account. Then xorg_windows (class Calendar) and " +
"xorg_screenshot show the reminder window. stop ends it and removes the profile. (a)",
Description: "OPENS WINDOWS ON THE OPERATOR'S SCREEN: tell the operator before calling it. Shows a calendar " +
"reminder without touching the operator's calendar: a second Betterbird in a throw-away profile under " +
"the runtime directory (--no-remote, unit mesh-betterbird-reminder-test, at most 10 minutes), its main " +
"window marked MESH TEST in red, with the module's user.js (no accounts) and one local calendar file " +
"holding one event, \"MESH TEST reminder\", whose reminder is due after `after` seconds. carry copies " +
"named preferences of the operator's profile (a theme, a colour switch) into it, never a credential or " +
"an account. Then xorg_windows (class Calendar) and xorg_screenshot show the reminder window. stop ends " +
"it and removes the profile. (a)",
Input: map[string]any{
"after": map[string]any{"type": "integer", "description": "seconds until the reminder is due (default 40, 15 to 300)"},
"carry": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
@@ -2,25 +2,32 @@ package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"regexp"
"strconv"
"strings"
"testing"
)
// betterbird's shape (novox/hq issue 345, ADR 0208): no package (Betterbird comes from the AUR, and the
// node-engine installs from the official repositories only), no seat, no start of its own (the operator
// starts it from the launcher); the X display on its own machine; one directory and one file of its own,
// the user.js the setting prefs fills; and the Go bundle serving exactly the listed betterbird_ tools.
// betterbird's shape (novox/hq issue 345, ADR 0208, ADR 0316): the X display on its own machine; its directory
// in the home with the user.js the setting prefs fills and the accounts file the setting accounts fills, both
// trusted; the community package declared absent before anything is placed in /opt/betterbird; Betterbird from
// its pinned upstream release, Owl pinned by its vendor's digest, the policy and the dictionaries' default
// unpacked into the installation; the tools' program for the start's preparation, the start and the launcher;
// every new build waiting for the operator's push; and the Go bundle serving exactly the listed tools.
type manifest struct {
Module string `json:"module"`
Version string `json:"version"`
Requires []string `json:"requires"`
Settings map[string]any `json:"settings"`
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
Module string `json:"module"`
Version string `json:"version"`
Upgrade map[string]string `json:"upgrade"`
Requires []string `json:"requires"`
Invokes []string `json:"invokes"`
Settings map[string]any `json:"settings"`
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
Build struct {
Artifacts []map[string]any `json:"artifacts"`
} `json:"build"`
@@ -41,43 +48,216 @@ func readManifest(t *testing.T) (manifest, string) {
return m, string(raw)
}
func TestItOwnsTheUserJSAndNothingOfTheProfile(t *testing.T) {
func resource(t *testing.T, m manifest, id string) map[string]any {
t.Helper()
for _, r := range m.Resources {
if r["id"] == id {
return r
}
}
t.Fatalf("module.json has no resource %s", id)
return nil
}
func artifact(t *testing.T, m manifest, name string) map[string]any {
t.Helper()
for _, a := range m.Build.Artifacts {
if a["name"] == name {
return a
}
}
t.Fatalf("module.json builds no artifact %s", name)
return nil
}
func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *testing.T) {
m, raw := readManifest(t)
if m.Module != "betterbird" || !reflect.DeepEqual(m.Requires, []string{"x11-display"}) {
t.Fatalf("%+v", m)
}
// Every new build waits for a person's push: the switch removes a package and replaces what a running
// Betterbird has open (ADR 0316 §4).
if m.Upgrade["policy"] != "record" || strings.TrimSpace(m.Upgrade["why"]) == "" {
t.Fatalf("the module is marked for a person's approval, with why: %v", m.Upgrade)
}
var ids []string
for _, r := range m.Resources {
ids = append(ids, r["id"].(string))
if r["type"] == "package" {
t.Errorf("betterbird-bin is the AUR's: a package resource asks the node-engine for what it cannot install")
}
if p, _ := r["path"].(string); !strings.HasPrefix(p, "${machine:account-home}/.config/betterbird") {
t.Errorf("a path outside the module's directory: %s", p)
}
}
if strings.Join(ids, ",") != "configuration-dir,user-js" {
t.Fatalf("%v", ids)
want := "configuration-dir,user-js,accounts,community-package,installation,owl,policies,spelling,preparation,start,launcher"
if strings.Join(ids, ",") != want {
t.Fatalf("the resources, in the order the node-engine applies them: %v, want %s", ids, want)
}
pkg := resource(t, m, "community-package")
if pkg["type"] != "package" || pkg["package"] != packageFor || pkg["absent"] != true {
t.Fatalf("the community package is declared absent (ADR 0180): %v", pkg)
}
places := map[string]string{"installation": installDir, "owl": filepath.Dir(owlFile),
"policies": filepath.Dir(policyFile), "spelling": installDir + "/defaults/pref", "preparation": filepath.Dir(preparation)}
arts := map[string]string{"installation": "archive", "owl": "owl", "policies": "distribution", "spelling": "defaults", "preparation": "tools"}
for id, path := range places {
r := resource(t, m, id)
if r["type"] != "archive" || r["path"] != path || r["artifact"] != arts[id] || r["owner"] != nil {
t.Errorf("%s is the %s archive unpacked at %s, root's: %v", id, arts[id], path, r)
}
}
for _, never := range []string{".thunderbird", "prefs.js", "logins.json", "key4.db"} {
if strings.Contains(raw, never) {
t.Errorf("module.json names %q: the profile is Betterbird's", never)
}
}
s := strings.ToLower(raw)
for _, never := range []string{"/home/", "jochen", "g14", "shanks", "novox.be", "http://", "mediahuis"} {
if strings.Contains(s, never) {
t.Errorf("module.json names %q", never)
}
}
file := m.Resources[1]
}
func TestTheUserJSAndTheAccountsAreTrustedFilesOfTheModulesOwn(t *testing.T) {
m, _ := readManifest(t)
file := resource(t, m, "user-js")
content := file["content"].(string)
if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != nil {
t.Fatalf("the user.js begins with the mark, holds the setting, and stays trusted (only the terminal sets it): %v", file)
if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != true {
t.Fatalf("the user.js begins with the mark, holds the setting, and is trusted (only the terminal or the operator's warrant sets it): %v", file)
}
// The default is a comment, so a machine without the setting places a valid, empty user.js.
def := m.Settings["prefs"].(map[string]any)["default"].(string)
if err := validUserJS(strings.ReplaceAll(content, "${setting:prefs}", def)); err != nil {
t.Fatal(err)
}
acc := resource(t, m, "accounts")
if acc["path"] != "${machine:account-home}/"+accountsFile || acc["merge"] != "json" || acc["trusted"] != true ||
acc["mode"] != "0600" || acc["owner"] != "${machine:account}" {
t.Fatalf("the setting accounts reaches the machine as a trusted mergeable file of the account's (ADR 0316 §10, §15): %v", acc)
}
got, err := parseAccounts([]byte(acc["content"].(string)))
if err != nil || len(got) != 0 {
t.Fatalf("its default declares no account: %v %v", got, err)
}
if _, declared := m.Settings["accounts"]; declared {
t.Fatal("accounts is the operator's value and has no default in the definition (ADR 0262)")
}
}
func TestBothPinsAreAURLAVersionAndASha256AndOwlIsNeverInTheRepository(t *testing.T) {
m, _ := readManifest(t)
sha := regexp.MustCompile(`^[0-9a-f]{64}$`)
for _, name := range []string{"archive", "owl"} {
a := artifact(t, m, name)
u, _ := a["url"].(string)
if a["kind"] != "archive" || !strings.HasPrefix(u, "https://") || !sha.MatchString(fmt.Sprint(a["sha256"])) ||
strings.TrimSpace(fmt.Sprint(a["version"])) == "" || a["from"] != nil {
t.Errorf("%s is pinned by URL, version and sha256, and copied from nowhere in the repository: %v", name, a)
}
}
if a := artifact(t, m, "archive"); a["within"] != "betterbird" {
t.Errorf("the release unpacks into betterbird/, whose content is the installation: %v", a)
}
if a := artifact(t, m, "owl"); a["file"] != filepath.Base(owlFile) {
t.Errorf("Owl's xpi is kept whole as %s: %v", filepath.Base(owlFile), a)
}
// Owl's licence forbids copying it to third parties: no xpi anywhere in the module's tree.
filepath.Walk(filepath.Join("..", ".."), func(p string, info os.FileInfo, err error) error {
if err == nil && strings.HasSuffix(strings.ToLower(p), ".xpi") {
t.Errorf("%s is in the repository", p)
}
return nil
})
}
// owlSupports is the range of Betterbird versions each pinned Owl declares in its manifest.json
// (browser_specific_settings.gecko strict_min_version, strict_max_version), read from the vendor's xpi
// when its pin was taken. Moving Owl's pin means reading the new xpi's range into here.
var owlSupports = map[string][2]string{
"1.5.6.1": {"115.0", "156.*"},
}
// owlAccepts says why a Betterbird version is outside the range an Owl version declares, or nil.
func owlAccepts(owl, betterbird string) error {
r, known := owlSupports[owl]
if !known {
return fmt.Errorf("Owl %s's supported range is not recorded in owlSupports: read it from the xpi's manifest.json", owl)
}
major := func(v string) int {
head, _, _ := strings.Cut(v, ".")
n, err := strconv.Atoi(head)
if err != nil {
return -1
}
return n
}
b, lo, hi := major(betterbird), major(r[0]), major(r[1])
if b < 0 || b < lo || b > hi {
return fmt.Errorf("Betterbird %s is outside the range Owl %s supports (%s to %s): a Betterbird Owl does not support "+
"waits for Owl's pin to move (novox/hq ADR 0316 §8)", betterbird, owl, r[0], r[1])
}
return nil
}
func TestABetterbirdOwlDoesNotSupportIsRefused(t *testing.T) {
m, _ := readManifest(t)
owl, bb := fmt.Sprint(artifact(t, m, "owl")["version"]), fmt.Sprint(artifact(t, m, "archive")["version"])
if err := owlAccepts(owl, bb); err != nil {
t.Fatal(err)
}
for _, outside := range []string{"157.0esr-bb1", "102.15.1esr-bb4"} {
if err := owlAccepts(owl, outside); err == nil {
t.Errorf("Betterbird %s was accepted with Owl %s", outside, owl)
}
}
if err := owlAccepts("9.9", bb); err == nil {
t.Error("an Owl whose range nobody read was accepted")
}
}
func TestThePolicyTurnsTheUpdaterOffAndInstallsTheMeshsOwlEverywhere(t *testing.T) {
m, _ := readManifest(t)
if a := artifact(t, m, "distribution"); a["kind"] != "archive" || a["from"] != "install/distribution" {
t.Fatalf("%v", a)
}
raw, err := os.ReadFile(filepath.Join("..", "..", "install", "distribution", "policies.json"))
if err != nil {
t.Fatal(err)
}
if why := policyProblem(raw); why != "" {
t.Fatalf("the module's policies.json: %s", why)
}
entries, _ := os.ReadDir(filepath.Join("..", "..", "install", "distribution"))
if len(entries) != 1 {
t.Fatalf("the distribution directory holds the policy alone: %v", entries)
}
if a := artifact(t, m, "defaults"); a["kind"] != "archive" || a["from"] != "install/defaults-pref" {
t.Fatalf("%v", a)
}
prefs, err := os.ReadFile(filepath.Join("..", "..", "install", "defaults-pref", "vendor-prefs.js"))
if err != nil || !strings.Contains(string(prefs), `pref("spellchecker.dictionary_path", "/usr/share/hunspell");`) {
t.Fatalf("the dictionaries are named as a default, in place of a link: %s %v", prefs, err)
}
}
func TestTheStartPreparesThenStartsBetterbirdAndTheLauncherUsesIt(t *testing.T) {
m, _ := readManifest(t)
start := resource(t, m, "start")
content := start["content"].(string)
if start["path"] != "/usr/local/bin/betterbird" || start["mode"] != "0755" ||
!strings.Contains(content, "\n"+preparation+" prepare ||") || !strings.HasSuffix(content, "\nexec "+startBin+" \"$@\"\n") {
t.Fatalf("the start runs the preparation, says when it did not complete, and starts Betterbird either way: %v", start)
}
launcher := resource(t, m, "launcher")
lc := launcher["content"].(string)
if launcher["path"] != "/usr/local/share/applications/eu.betterbird.Betterbird.desktop" ||
!strings.Contains(lc, "\nExec=/usr/local/bin/betterbird %u\n") || !strings.Contains(lc, "\nIcon="+installDir+"/") ||
!strings.Contains(lc, "\nStartupWMClass=eu.betterbird.Betterbird\n") {
t.Fatalf("the launcher starts the module's start and names the icon in the installation: %v", launcher)
}
if b := artifact(t, m, "tools"); filepath.Base(preparation) != b["binary"] {
t.Fatalf("the preparation runs the tools' binary: %v", b)
}
}
func TestTheToolsAgreeWithTheManifest(t *testing.T) {
m, raw := readManifest(t)
m, _ := readManifest(t)
served := map[string]bool{}
for _, tool := range tools() {
served[tool.Name] = true
@@ -94,17 +274,12 @@ func TestTheToolsAgreeWithTheManifest(t *testing.T) {
for name := range served {
t.Errorf("the bundle serves %s, which module.json does not list", name)
}
if len(m.Build.Artifacts) != 1 {
t.Fatalf("%v", m.Build.Artifacts)
}
b := m.Build.Artifacts[0]
b := artifact(t, m, "tools")
if b["kind"] != "bundle" || b["language"] != "go" || b["from"] != "cmd/betterbird-tools" || b["binary"] != "betterbird-tools" {
t.Errorf("the Go tools bundle: %v", b)
}
s := strings.ToLower(raw)
for _, never := range []string{"/home/", "jochen", "g14", "shanks", "novox.be", "http"} {
if strings.Contains(s, never) {
t.Errorf("module.json names %q", never)
}
// betterbird_accounts proposes the setting accounts through the controller's settings verb (ADR 0317 §9).
if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings"}) {
t.Errorf("the module invokes the controller's settings verb and nothing else: %v", m.Invokes)
}
}
@@ -10,6 +10,13 @@ package main
// module declares. A theme or another preference of the operator's profile can be carried over by
// name (carry), within the same limits as betterbird_prefs: never a credential, never an account.
//
// **It is visibly a test** (the operator, 2026-10-12): on 2026-10-12 at 12:05 a test Betterbird with an empty
// profile carrying the operator's theme looked like the operator's configuration lost. So the test profile
// carries a userChrome.css that marks its main window with a red band reading MESH TEST, and the event is
// called "MESH TEST reminder". The reminder window itself is left as Betterbird draws it, because how it is
// drawn is what the test looks at. The tool's description says a window opens on the operator's screen, so
// a caller tells the operator first.
//
// It lives for at most ten minutes (the unit's RuntimeMaxSec), and stop ends it early and removes the
// profile. The reminder window is then found with xorg_windows (class Calendar, role AlarmWindow) and
// looked at with xorg_screenshot.
@@ -41,6 +48,7 @@ type ReminderAnswer struct {
Carried []string `json:"carried,omitempty"`
Withheld []string `json:"withheld,omitempty"`
Look string `json:"look"`
OnScreen string `json:"on_screen,omitempty"`
Stopped bool `json:"stopped,omitempty"`
}
@@ -50,11 +58,38 @@ func ics(now time.Time, after time.Duration, stamp string) string {
start := now.Add(after + 10*time.Minute)
return strings.Join([]string{"BEGIN:VCALENDAR", "VERSION:2.0", "PRODID:-//novox mesh//betterbird reminder test//EN",
"BEGIN:VEVENT", "UID:mesh-reminder-test-" + stamp + "@mesh", "DTSTAMP:" + f(now), "DTSTART:" + f(start),
"DTEND:" + f(start.Add(30*time.Minute)), "SUMMARY:Mesh reminder test", "BEGIN:VALARM", "ACTION:DISPLAY",
"DESCRIPTION:Mesh reminder test", "TRIGGER:-PT10M", "END:VALARM", "END:VEVENT", "END:VCALENDAR", ""}, "\r\n")
"DTEND:" + f(start.Add(30*time.Minute)), "SUMMARY:" + testTitle, "BEGIN:VALARM", "ACTION:DISPLAY",
"DESCRIPTION:" + testTitle, "TRIGGER:-PT10M", "END:VALARM", "END:VEVENT", "END:VCALENDAR", ""}, "\r\n")
}
// testTitle is the test event's title: what the reminder window shows.
const testTitle = "MESH TEST reminder (a throw-away Betterbird of the mesh, not your calendar)"
// testChrome is the test profile's userChrome.css: a red band reading MESH TEST across the top of every
// Betterbird window but the reminder (windowtype Calendar:AlarmWindow), and a red frame around it, so the test
// Betterbird is never taken for the operator's.
const testChrome = `/* The mesh's reminder test (module betterbird, novox/hq issue 345): a throw-away Betterbird, not yours. */
:root:not([windowtype="Calendar:AlarmWindow"]) {
outline: 6px solid #d00000 !important;
outline-offset: -6px !important;
}
:root:not([windowtype="Calendar:AlarmWindow"]) > body::before {
content: "MESH TEST \2014 a throw-away Betterbird started by the mesh for a reminder test; your mail and calendar are not here. It ends by itself within 10 minutes.";
display: block !important;
background: #d00000 !important;
color: #ffffff !important;
font: bold 15px sans-serif !important;
padding: 6px 12px !important;
text-align: center !important;
}
`
// testPrefs are the test profile's own preferences: its one calendar, and no first-run questions.
//
// betterbird.update.checkIntervalDays 0 is Betterbird's own switch for its update check (MailGlue): a
// fresh profile has never checked, so Betterbird checks at its start and, when betterbird.eu publishes a
// newer build than the installed one, opens the About window on the operator's screen and gives it the
// focus (seen on the workstation 2026-10-12, novox/hq issue 345). The test is about the reminder only.
func testPrefs(calendar string) string {
return strings.Join([]string{
`user_pref("calendar.registry.meshtest.type", "ics");`,
@@ -68,7 +103,9 @@ func testPrefs(calendar string) string {
`user_pref("datareporting.policy.dataSubmissionEnabled", false);`,
`user_pref("toolkit.telemetry.reportingpolicy.firstRun", false);`,
`user_pref("app.update.auto", false);`,
`user_pref("betterbird.update.checkIntervalDays", 0);`,
`user_pref("mail.rights.version", 1);`,
`user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);`,
"",
}, "\n")
}
@@ -125,14 +162,22 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) (
if err := os.WriteFile(calendar, []byte(ics(now, after, strconv.FormatInt(now.Unix(), 10))), 0o600); err != nil {
return a, err
}
user := testPrefs(strings.TrimPrefix(calendar, m.Root))
// The module's user.js first, then what is carried, then the test's own preferences: the last
// statement wins, so neither can take the test's calendar or its quiet start away.
user := ""
if mine, err := readBounded(m.home(declared)); err == nil && validUserJS(string(mine)) == nil {
user = string(mine) + "\n" + user
user = string(mine) + "\n"
}
user += strings.Join(carried, "\n") + "\n"
user += strings.Join(carried, "\n") + "\n" + testPrefs(strings.TrimPrefix(calendar, m.Root))
if err := os.WriteFile(filepath.Join(dir, "profile", "user.js"), []byte(user), 0o600); err != nil {
return a, err
}
if err := os.MkdirAll(filepath.Join(dir, "profile", "chrome"), 0o700); err != nil {
return a, err
}
if err := os.WriteFile(filepath.Join(dir, "profile", "chrome", "userChrome.css"), []byte(testChrome), 0o600); err != nil {
return a, err
}
// The unit removes the profile when it ends, however it ends: at its 10 minutes, on stop, or when the
// test Betterbird is closed.
call := []string{"--user", "--collect", "--quiet", "--unit=" + testUnit, "-p", "RuntimeMaxSec=600",
@@ -146,7 +191,9 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) (
if err := failed(m.cmd(8*time.Second, s.Env(), "systemd-run", call...), "systemd-run", call...); err != nil {
return a, err
}
a.Event = "Mesh reminder test, in a calendar file of the test profile only"
a.Event = testTitle + ", in a calendar file of the test profile only"
a.OnScreen = "a second Betterbird's window is now on the operator's screen, marked MESH TEST in red; its reminder " +
"window follows when the reminder is due. Both go when the test ends (stop, or 10 minutes)"
a.Due = now.Add(after).UTC().Format(time.RFC3339)
return a, nil
}
@@ -0,0 +1,16 @@
package main
import (
"encoding/json"
"errors"
"os"
"testing"
)
// No test reaches the network or the mesh: Betterbird's list of published builds and the controller's
// settings verb answer from here, and a test that needs them says what they answer.
func TestMain(m *testing.M) {
publishedBuilds = func() (string, error) { return "", errors.New("no network in tests") }
asker = func(string, any) (json.RawMessage, error) { return nil, errors.New("no mesh in tests") }
os.Exit(m.Run())
}
@@ -0,0 +1,3 @@
// Written by the mesh (module betterbird, novox/hq ADR 0316 §3): the system's spelling dictionaries, named
// as a default preference in place of the link the community package made.
pref("spellchecker.dictionary_path", "/usr/share/hunspell");
@@ -0,0 +1,12 @@
{
"policies": {
"DisableAppUpdate": true,
"ExtensionSettings": {
"owl@beonex.com": {
"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi",
"updates_disabled": true
}
}
}
}
+103 -2
View File
@@ -1,14 +1,21 @@
{
"module": "betterbird",
"version": "1",
"upgrade": {
"policy": "record",
"why": "a new build may remove the community package betterbird-bin and replaces the installation in /opt/betterbird, which a running Betterbird has open: the switch on each node and every upgrade wait for the operator, who quits Betterbird first and then pushes (novox/hq ADR 0316 §4, §5)"
},
"requires": [
"x11-display"
],
"invokes": [
"seat:mesh-controller.settings"
],
"settings": {
"prefs": {
"kind": "preference",
"default": "// none",
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345)"
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345). An account's preferences (mail.account., mail.identity., mail.server., mail.smtpserver. and the account lists) are refused here: an account is declared in the setting accounts, so it has one owner (novox/hq ADR 0316 §11)"
}
},
"tools": [
@@ -18,6 +25,7 @@
"betterbird_log",
"betterbird_restart",
"betterbird_check",
"betterbird_accounts",
"betterbird_reminder_test"
],
"resources": [
@@ -34,7 +42,68 @@
"path": "${machine:account-home}/.config/betterbird/user.js",
"owner": "${machine:account}",
"mode": "0644",
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n"
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n",
"trusted": true
},
{
"id": "accounts",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/accounts.json",
"owner": "${machine:account}",
"mode": "0600",
"merge": "json",
"content": "{\n \"accounts\": {}\n}\n",
"trusted": true
},
{
"id": "community-package",
"type": "package",
"package": "betterbird-bin",
"absent": true
},
{
"id": "installation",
"type": "archive",
"path": "/opt/betterbird",
"artifact": "archive"
},
{
"id": "owl",
"type": "archive",
"path": "/opt/betterbird/owl",
"artifact": "owl"
},
{
"id": "policies",
"type": "archive",
"path": "/opt/betterbird/distribution",
"artifact": "distribution"
},
{
"id": "spelling",
"type": "archive",
"path": "/opt/betterbird/defaults/pref",
"artifact": "defaults"
},
{
"id": "preparation",
"type": "archive",
"path": "/usr/local/lib/mesh-betterbird",
"artifact": "tools"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/betterbird",
"mode": "0755",
"content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the setting prefs, and the\n# accounts the setting accounts declares) and add the declared accounts to the profile's account list. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n"
},
{
"id": "launcher",
"type": "file",
"path": "/usr/local/share/applications/eu.betterbird.Betterbird.desktop",
"mode": "0644",
"content": "[Desktop Entry]\nVersion=1.0\nType=Application\nName=Betterbird\nGenericName=Mail Client\nComment=Send and receive mail with Betterbird\nExec=/usr/local/bin/betterbird %u\nIcon=/opt/betterbird/chrome/icons/default/default128.png\nTerminal=false\nCategories=Network;Email;\nMimeType=message/rfc822;x-scheme-handler/mailto;application/x-xpinstall;application/x-extension-ics;text/calendar;text/vcard;text/x-vcard;x-scheme-handler/webcal;x-scheme-handler/webcals;x-scheme-handler/mid;\nStartupNotify=true\nStartupWMClass=eu.betterbird.Betterbird\nKeywords=Email;E-mail;Calendar;Thunderbird;\nActions=ComposeMessage;OpenAddressBook;\n# Written by the mesh (module betterbird, novox/hq ADR 0316 §3), in place of the community package's entry:\n# it starts Betterbird through the module's start, and names the icon in the installation, with no link.\n\n[Desktop Action ComposeMessage]\nName=Write new message\nExec=/usr/local/bin/betterbird -compose\n\n[Desktop Action OpenAddressBook]\nName=Open address book\nExec=/usr/local/bin/betterbird -addressbook\n"
}
],
"build": {
@@ -49,6 +118,38 @@
"loads": [
"betterbird-tools"
]
},
{
"name": "archive",
"kind": "archive",
"url": "https://www.betterbird.eu/downloads/LinuxArchive/betterbird-153.4.0esr-bb10.en-US.linux-x86_64.tar.xz",
"sha256": "38dd24890a66b4cf9a45bc24e15192e00c26b929e1bb8df7974c16a4539129e2",
"version": "153.4.0esr-bb10",
"within": "betterbird",
"names-on-purpose": {
"www.betterbird.eu": "Betterbird's project publishes its Linux release there; the build seat fetches it once, by its pinned sha256 (novox/hq ADR 0316 §2)"
}
},
{
"name": "owl",
"kind": "archive",
"url": "https://www.beonex.com/owl/owl.xpi",
"sha256": "a54cc3d52113b34f97942ee19190947dee23561e2217f18db3d80ac023991221",
"version": "1.5.6.1",
"file": "owl.xpi",
"names-on-purpose": {
"www.beonex.com": "Owl's vendor publishes its xpi there; the build seat fetches it once, by its pinned sha256 (novox/hq ADR 0316 §6)"
}
},
{
"name": "distribution",
"kind": "archive",
"from": "install/distribution"
},
{
"name": "defaults",
"kind": "archive",
"from": "install/defaults-pref"
}
]
}
+4 -1
View File
@@ -199,7 +199,10 @@ only a language model can, for another module — the issue tracker's duplicate
its own), in an empty directory, the question on its input: no built-in tools, no tool servers, plugins, hooks or
instruction files, no session kept, so it can only answer. It answers the structured answer, the model, the node
and the seconds, within 25 seconds — the process is ended past that, under the bus's 30 — and one at a time on a
node. Refused: an empty question or one over 64 KiB, a schema that is not an object, a model not `sonnet`, `haiku`
node. A caller may ask a longer bound with `within` (seconds, at most 120), for a long question (hq issue 508): a
judgement not done in 20 seconds is then answered `pending` with a `ticket`, runs on to its bound, and
`claude_code_judged` (`ticket`) collects it, waiting up to 20 seconds a call. A ticket is answered once; an answer
nobody collects is dropped ten minutes after its bound, and none survives a restart of the module. Refused: an empty question or one over 64 KiB, a schema that is not an object, a model not `sonnet`, `haiku`
or `opus`, and a node holding no licence. The question is never logged.
## Settings
+148 -7
View File
@@ -12,12 +12,15 @@ package main
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -27,6 +30,14 @@ const (
// JudgeBound is how long one judgement may take, waiting for another included: under the bus's 30 seconds
// a tool call is answered in, so the caller hears this module's refusal rather than a timeout.
JudgeBound = 25 * time.Second
// MostJudgeBound is the longest bound a caller may ask with within: a long record's review takes longer than
// one bus call allows (novox/hq issue 508), and is then collected by its ticket.
MostJudgeBound = 120 * time.Second
// JudgeWait is how long one call waits for a judgement asked with a bound past JudgeBound before it answers
// the ticket: well under the bus's 30 seconds, so the call is always answered.
JudgeWait = 20 * time.Second
// JudgedKept is how long an answer nobody collected is kept past its judgement's bound.
JudgedKept = 10 * time.Minute
// MostQuestion is the largest question taken.
MostQuestion = 64 << 10
// MostSchema is the largest answer schema taken.
@@ -43,14 +54,29 @@ type JudgeAsk struct {
Question string
Schema map[string]any
Model string
// Within is the bound the caller asks; JudgeBound when zero. Past JudgeBound the call answers a ticket if the
// judgement is not done within JudgeWait, and claude_code_judged collects it (novox/hq issue 508).
Within time.Duration
}
// Judged is one answer.
type Judged struct {
Answer json.RawMessage `json:"answer"`
Answer json.RawMessage `json:"answer,omitempty"`
Model string `json:"model"`
Node string `json:"node"`
Seconds float64 `json:"seconds"`
// Ticket names a judgement asked with a bound past JudgeBound; Pending says it is not done yet, and
// claude_code_judged with the ticket collects it.
Ticket string `json:"ticket,omitempty"`
Pending bool `json:"pending,omitempty"`
}
// running is one judgement asked with a bound past JudgeBound, until it is collected.
type running struct {
done chan struct{}
out *Judged
err error
until time.Time
}
// Judge runs judgements on this node, one at a time.
@@ -63,7 +89,13 @@ type Judge struct {
Now func() time.Time
// Bound is how long one judgement may take; JudgeBound unless a test says less.
Bound time.Duration
turn chan struct{}
// Wait is how long a call waits for a long judgement before it answers its ticket; JudgeWait unless a test
// says less.
Wait time.Duration
turn chan struct{}
mu sync.Mutex
tickets map[string]*running
}
// NewJudge is the judge of this node.
@@ -118,11 +150,25 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
"bindings say which node holds which", j.P.Node)
}
start := j.Now()
bound := j.Bound
if bound <= 0 {
bound = JudgeBound
}
if a.Within > MostJudgeBound {
return nil, fmt.Errorf("within is %s; a judgement takes at most %s", a.Within, MostJudgeBound)
}
if a.Within > bound {
return j.start(string(schema), model, question, a.Within)
}
if a.Within > 0 {
bound = a.Within
}
return j.run(string(schema), model, question, bound)
}
// run is one judgement within its bound, waiting for its turn included.
func (j *Judge) run(schema, model, question string, bound time.Duration) (*Judged, error) {
start := j.Now()
ctx, cancel := context.WithTimeout(context.Background(), bound)
defer cancel()
select {
@@ -131,7 +177,7 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
case <-ctx.Done():
return nil, fmt.Errorf("another judgement held %s's agent for all of %s; ask again", j.P.Node, bound)
}
out, err := j.Exec(ctx, j.Argv(string(schema), model), []byte(question))
out, err := j.Exec(ctx, j.Argv(schema, model), []byte(question))
if ctx.Err() != nil {
return nil, fmt.Errorf("the agent on %s did not answer within %s; the judgement was ended", j.P.Node, bound)
}
@@ -145,6 +191,68 @@ func (j *Judge) Judge(a JudgeAsk) (*Judged, error) {
return &Judged{Answer: answer, Model: model, Node: j.P.Node, Seconds: j.Now().Sub(start).Round(100 * time.Millisecond).Seconds()}, nil
}
// start runs a judgement whose bound is past what one bus call allows, and waits for it as long as a call may: its
// answer when it is done by then, its ticket otherwise (novox/hq issue 508).
func (j *Judge) start(schema, model, question string, bound time.Duration) (*Judged, error) {
var b [12]byte
_, _ = rand.Read(b[:])
ticket := hex.EncodeToString(b[:])
r := &running{done: make(chan struct{}), until: j.Now().Add(bound + JudgedKept)}
j.mu.Lock()
j.forget()
if j.tickets == nil {
j.tickets = map[string]*running{}
}
j.tickets[ticket] = r
j.mu.Unlock()
go func() {
r.out, r.err = j.run(schema, model, question, bound)
close(r.done)
}()
return j.Collect(ticket)
}
// Collect waits as long as a call may for the judgement under a ticket: its answer, why it failed, or that it is
// still pending. A ticket is answered once; an answer nobody collects is kept JudgedKept past its bound.
func (j *Judge) Collect(ticket string) (*Judged, error) {
j.mu.Lock()
j.forget()
r := j.tickets[ticket]
j.mu.Unlock()
if r == nil {
return nil, fmt.Errorf("no judgement on %s under ticket %q: it was collected, kept past its time, or this module "+
"restarted since; ask the judgement again", j.P.Node, ticket)
}
wait := j.Wait
if wait <= 0 {
wait = JudgeWait
}
select {
case <-r.done:
case <-time.After(wait):
return &Judged{Node: j.P.Node, Ticket: ticket, Pending: true}, nil
}
j.mu.Lock()
delete(j.tickets, ticket)
j.mu.Unlock()
if r.err != nil {
return nil, r.err
}
out := *r.out
out.Ticket = ticket
return &out, nil
}
// forget drops the answers nobody collected in time. Called with mu held.
func (j *Judge) forget() {
now := j.Now()
for t, r := range j.tickets {
if now.After(r.until) {
delete(j.tickets, t)
}
}
}
// answerOf reads print mode's result: the structured answer, or why there is none.
func answerOf(out []byte) (json.RawMessage, error) {
var r struct {
@@ -213,17 +321,21 @@ func judgeTool(j *Judge) stdio.Tool {
"object); optionally the model (" + strings.Join(JudgeModels, ", ") + "; default " + JudgeModels[0] + "). The agent " +
"answers as the account agents run as here, under the licence this node is bound to, with no tools, " +
"no tool servers, no instructions and no session kept — it can only answer — within 25 seconds or " +
"not at all. One at a time on a node. Refused: an empty or larger than 64 KiB question, a schema that " +
"not at all, unless within asks longer (at most 120 seconds, for a long question): then, when it is not done in " +
"20 seconds, it answers pending with a ticket, and claude_code_judged collects the answer (novox/hq issue 508). " +
"One at a time on a node. Refused: an empty or larger than 64 KiB question, a schema that " +
"is not an object, an unknown model, a node holding no licence. The question is never logged.",
Input: map[string]any{
"question": str("the question, with everything the answer needs"),
"schema": map[string]any{"type": "object", "description": "the JSON schema the answer follows"},
"model": map[string]any{"type": "string", "enum": JudgeModels, "description": "the model (default " + JudgeModels[0] + ")"},
"within": map[string]any{"type": "number", "description": "the bound in seconds, at most 120 (default 25); " +
"past 25 a judgement not done in 20 seconds answers pending with a ticket for claude_code_judged"},
},
Run: func(a map[string]any) (any, error) {
for k := range a {
if k != "question" && k != "schema" && k != "model" {
return nil, fmt.Errorf("claude_code_judge takes question, schema and model; not %s", k)
if k != "question" && k != "schema" && k != "model" && k != "within" {
return nil, fmt.Errorf("claude_code_judge takes question, schema, model and within; not %s", k)
}
}
var ask JudgeAsk
@@ -241,6 +353,35 @@ func judgeTool(j *Judge) stdio.Tool {
return nil, errors.New("model is text")
}
}
if v, given := a["within"]; given && v != nil {
secs, ok := v.(float64)
if !ok || secs <= 0 {
return nil, errors.New("within is a number of seconds")
}
ask.Within = time.Duration(secs * float64(time.Second))
}
return j.Judge(ask)
}}
}
// judgedTool collects a judgement asked with a bound past one bus call's (novox/hq issue 508).
func judgedTool(j *Judge) stdio.Tool {
return stdio.Tool{Name: "claude_code_judged",
Description: "The answer of a judgement claude_code_judge answered pending with a ticket (novox/hq issue 508): " +
"waits up to 20 seconds, then answers the judgement's answer, why it failed, or pending again. A ticket is " +
"answered once; one nobody collects is dropped ten minutes after its bound, and none survives a restart of " +
"this module.",
Input: map[string]any{"ticket": str("the ticket claude_code_judge answered")},
Run: func(a map[string]any) (any, error) {
for k := range a {
if k != "ticket" {
return nil, fmt.Errorf("claude_code_judged takes ticket; not %s", k)
}
}
ticket, ok := a["ticket"].(string)
if !ok || ticket == "" {
return nil, errors.New("claude_code_judged needs ticket, as claude_code_judge answered it")
}
return j.Collect(ticket)
}}
}
@@ -166,3 +166,102 @@ func TestTheToolTakesOnlyItsArguments(t *testing.T) {
t.Fatalf("%s", raw)
}
}
// A long question may ask a longer bound than one bus call allows (novox/hq issue 508): the call answers a ticket
// when the judgement is not done within the call's wait, the judgement runs on to its own bound, and
// claude_code_judged collects its answer.
func TestALongJudgementIsCollectedByItsTicket(t *testing.T) {
var deadline time.Time
j := testJudge(true, true, func(ctx context.Context, _ []string, _ []byte) ([]byte, error) {
deadline, _ = ctx.Deadline()
select {
case <-time.After(400 * time.Millisecond):
return []byte(`{"structured_output":{"verdict":"none"}}`), nil
case <-ctx.Done():
return nil, ctx.Err()
}
})
j.Wait = 50 * time.Millisecond
start := time.Now()
out, err := j.Judge(JudgeAsk{Question: "a long record", Schema: verdictSchema, Within: 90 * time.Second})
if err != nil {
t.Fatal(err)
}
if !out.Pending || out.Ticket == "" || len(out.Answer) != 0 || out.Node != "control" {
t.Fatalf("not a ticket: %+v", out)
}
var got *Judged
for i := 0; i < 100; i++ {
got, err = j.Collect(out.Ticket)
if err != nil {
t.Fatal(err)
}
if !got.Pending {
break
}
}
if got.Pending || string(got.Answer) != `{"verdict":"none"}` || got.Ticket != out.Ticket {
t.Fatalf("%+v", got)
}
// Read once the judgement is collected: its end is what orders the agent's write before this read.
if left := deadline.Sub(start); left < 80*time.Second || left > 91*time.Second {
t.Fatalf("the judgement's bound is %s, not the 90s asked", left)
}
if _, err := j.Collect(out.Ticket); err == nil || !strings.Contains(err.Error(), "no judgement") {
t.Fatalf("a ticket collected twice: %v", err)
}
}
// A long judgement that fails says why when it is collected, and its bound is never past MostJudgeBound.
func TestALongJudgementIsBoundedAndSaysWhyItFailed(t *testing.T) {
j := testJudge(true, true, func(ctx context.Context, _ []string, _ []byte) ([]byte, error) {
<-ctx.Done()
return nil, ctx.Err()
})
if _, err := j.Judge(JudgeAsk{Question: "q", Schema: verdictSchema, Within: MostJudgeBound + time.Second}); err == nil ||
!strings.Contains(err.Error(), "at most") {
t.Fatalf("a bound past the most taken: %v", err)
}
j.Wait = 20 * time.Millisecond
j.Bound = 30 * time.Millisecond // the test's stand-in for JudgeBound: within is past it, so a ticket
out, err := j.Judge(JudgeAsk{Question: "q", Schema: verdictSchema, Within: 200 * time.Millisecond})
if err != nil || !out.Pending {
t.Fatalf("%+v %v", out, err)
}
for i := 0; i < 100; i++ {
got, err := j.Collect(out.Ticket)
if err != nil {
if !strings.Contains(err.Error(), "did not answer within") {
t.Fatalf("%v", err)
}
return
}
if !got.Pending {
t.Fatalf("answered: %+v", got)
}
}
t.Fatal("never ended")
}
// The tools take within and ticket, and nothing else.
func TestTheToolsTakeWithinAndTicket(t *testing.T) {
j := testJudge(true, true, func(context.Context, []string, []byte) ([]byte, error) {
return []byte(`{"structured_output":{"verdict":"none"}}`), nil
})
out, err := judgeTool(j).Run(map[string]any{"question": "q", "schema": verdictSchema, "within": float64(60)})
if err != nil {
t.Fatal(err)
}
if got := out.(*Judged); got.Pending || string(got.Answer) != `{"verdict":"none"}` {
t.Fatalf("a quick long judgement is answered in its call: %+v", got)
}
if _, err := judgeTool(j).Run(map[string]any{"question": "q", "schema": verdictSchema, "within": "long"}); err == nil {
t.Fatal("within as text was taken")
}
if _, err := judgedTool(j).Run(map[string]any{"ticket": "nope"}); err == nil || !strings.Contains(err.Error(), "no judgement") {
t.Fatalf("%v", err)
}
if _, err := judgedTool(j).Run(map[string]any{"ticket": "x", "by": "y"}); err == nil {
t.Fatal("an unknown argument was taken")
}
}
+2 -1
View File
@@ -355,7 +355,8 @@ func tools(p Paths, servers ServerState, view *ServerView, config ConfigState, c
Description: "Read-only: everything registered through this module that a Claude Code session runs or obeys — the tool servers, the hooks, the settings and the skills, subagents, commands, output styles and instructions — by key, with who registered it, when, and whether it is still rendered (novox/hq issue 340), beside what the operator set for this node at the controller's terminal (managed_settings, mcp_servers). `by` is the node whose tool runner served the call, never the caller.",
Run: func(map[string]any) (any, error) { return Registrations(p, servers, config) }})
out = append(out, proposalTools(p)...)
out = append(out, judgeTool(NewJudge(p)))
judge := NewJudge(p)
out = append(out, judgeTool(judge), judgedTool(judge))
return append(out, configTools(p, config, configView)...)
}
+2 -1
View File
@@ -67,7 +67,8 @@
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge"
"claude_code_judge",
"claude_code_judged"
],
"data": {
"own": [
+51 -1
View File
@@ -2,7 +2,7 @@
The forge, and the mesh's npm package registry: it claims the `git` and `npm-package-registry` seats.
Its tools, its events and the provisioner of registry accounts are the TypeScript bundle (`index.ts`,
`tools/`, `provisioner/`). This README covers the Go bundle beside it.
`tools/`, `provisioner/`). This README covers the two Go bundles beside it.
## Announcing pull requests to the merge check (novox/hq to-be 45 §9, issue 347)
@@ -28,6 +28,56 @@ Every new head of an open pull request is announced as `pull.updated`, once, and
- **A pull request or issue number that is not one is refused** before the forge is asked: a tool called with a
misnamed argument sent `pulls/NaN`.
## An agent's commit and the reads around it (novox/hq issue 375)
An agent that finished a change on a branch had no way to put it on the forge: the module had no commit or
push tool, `gitea_api` refused the forge's patch endpoint, and its only file write took one whole file
base64-encoded. The Go bundle `cmd/forge` serves the commit, and the reads an agent otherwise made through
`gitea_api` as raw JSON, summarised.
| tool | | what |
|---|---|---|
| `gitea_commit` | a | one commit on a branch of its own, created from `base` (default the default branch) when absent, carrying either `files` (each `{path, content \| content_base64 \| delete: true}`) or `patch` (a unified diff, as `git diff` writes it); answers the commit and its parent |
| `gitea_commits` | r | a ref's history, newest first, optionally only commits touching a path: sha, author, date, subject |
| `gitea_compare` | r | what head holds that base does not: its commits and the files they touch |
| `gitea_pull_reviews` | r | a pull request's reviews — state, the commit reviewed and whether it is the head now, stale, dismissed — with their line comments |
| `gitea_list_dir` | r | a directory's entries at a ref: name, type, a file's size |
| `gitea_search_code` | r | the lines of a ref's text files matching a literal string or a regular expression, optionally under a path |
**Where a commit may go** is the rule `gitea_api` already holds for a file written (novox/hq ADR 0266):
never the repository's default branch, never a branch a protection rule names. A rule covers a branch as the
forge itself decides it (Gitea 1.27, novox/hq issue 539): a plain name covers that branch in any case; a glob
(`*`, `**`, `?`, `[abc]`, `[!abc]`, `{a,b}`, `\` escaping) covers case-sensitively with `/` as separator; a glob
the forge cannot compile covers its own text. `cmd/forge/rules.go` and `guard.ts` hold this one rule. The rules are read before anything is sent; rules that cannot be read refuse the
commit. Nothing about merging changes: a merge still needs an independent review at the head and both
required checks green, through `gitea_merge_pull_request`.
**How the commit is made.** `files` become one call to the forge's multi-file contents endpoint: a path the
ref holds is updated, one it does not is created, and an update or delete carries the blob read just
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. The blobs are read at
the commit the branch (or, for a new branch, its base) was read at, never at its name. A `patch` goes to the
forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than the
file. Neither endpoint takes a commit to build on, only a branch (Gitea 1.27), so the parent answered is the
one the forge says it made the commit on; when the branch moved between the read and the commit, the answer
says so. One commit carries at most 200 files, 4 MiB per file and 16 MiB together, or a patch of 4 MiB. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
local commit's: the merge check judges the head on the forge. After the write the branch is read back, and a
branch not at the commit the forge named is a failure, never a success.
**Code search** has no endpoint on the forge, so the ref's archive (`/archive/{ref}.tar.gz`, at most 64 MiB)
is read once and searched in memory; a binary file is passed over, a file over 4 MiB is named under
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found. A
search that reaches the call's deadline stops there and answers what it found, with `stopped_at_deadline`.
It reaches the forge through the same HTTP client as the npm-registry bundle (`internal/forgehttp`), and one
tool call takes at most 50 seconds.
`go test ./cmd/forge` runs it against a fake forge: the files of one commit created, updated and deleted on
a new branch from its base; an existing branch committed on; the default branch, a protected branch and
unreadable rules refused before anything is sent; a patch sent to the forge; a branch read back at another
commit failing; a branch or base that moved under the commit said, with the parent the forge used; a file
changed under the commit refused; a commit over its size limits refused; protection globs matched as the
forge matches them; a code search stopped at its deadline; every malformed ask refused; and each read
summarised.
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
Every publish adds a version to the registry, and until this nothing removed one. The Go bundle
+13
View File
@@ -465,6 +465,19 @@ export class GiteaClient {
return out;
}
/** Every status of one context on a commit, newest first, as the forge keeps each one it was given: what tells
* when the check now pending on a head was opened, after a restart (novox/hq issue 456). */
async contextHistory(owner: string, repo: string, sha: string, context: string): Promise<HeadCheck[]> {
const raw = await this.request<any>(`${repoPath(owner, repo)}/statuses/${commitSegment(sha)}?sort=highestindex&limit=50`);
const out: HeadCheck[] = [];
for (const s of Array.isArray(raw) ? raw : []) {
if (s?.context !== context) continue;
out.push({ state: String(s.status ?? s.state ?? ""), description: String(s.description ?? ""),
updated_at: String(s.created_at ?? s.updated_at ?? "") });
}
return out;
}
/** A commit's statuses whole, as the forge combines them for a ref (a sha, or the start of one): the commit
* it resolved to and every status with its description, link, times and setter (novox/hq ADR 0239). */
async combinedStatus(owner: string, repo: string, ref: string): Promise<{ sha: string; state: string; statuses: unknown[] }> {
+406
View File
@@ -0,0 +1,406 @@
package main
import (
"context"
"encoding/base64"
"errors"
"fmt"
"net/http"
"net/url"
"strings"
"unicode/utf8"
)
// gitea_commit (novox/hq issue 375): one commit on a branch of its own, so an agent follows the branch / pull
// request / approval rule without anyone pushing by hand. By the rule gitea_api already enforces for a file
// written (novox/hq ADR 0266): never the repository's default branch, never a branch a protection rule names.
// Nothing about merging changes — that still needs a review at the head and both required checks green
// (gitea_merge_pull_request).
// FileChange is one file of a commit: its new content, as text or base64, or its deletion.
type FileChange struct {
Path string
Content *string
ContentBase64 *string
Delete bool
}
// CommitAsk is one gitea_commit call.
type CommitAsk struct {
Owner, Repo string
Branch string
Base string // the branch or commit a new branch starts from; the default branch when empty
Message string
Files []FileChange
Patch string // a unified diff, as git diff writes it, instead of Files
}
// Committed is what gitea_commit made.
type Committed struct {
Repository string `json:"repository"`
Branch string `json:"branch"`
Created bool `json:"created"`
Base string `json:"base,omitempty"`
Parent string `json:"parent"`
Commit string `json:"commit"`
Files []CommittedFile `json:"files,omitempty"`
Said []string `json:"said"`
}
// CommittedFile is one file the commit changed, and how.
type CommittedFile struct {
Path string `json:"path"`
Operation string `json:"operation"`
}
// What one commit carries at most (novox/hq issue 539): files, one file's content, one patch, and every
// file's content together, in bytes as committed (content_base64 decoded). Larger work is several commits.
const (
maxFiles = 200
maxFileBytes = 4 << 20
maxPatchBytes = 4 << 20
maxCommitBytes = 16 << 20
)
// readCommitAsk reads the tool's arguments.
func readCommitAsk(args map[string]any) (CommitAsk, error) {
a := CommitAsk{Owner: text(args, "owner"), Repo: text(args, "repo"), Branch: text(args, "branch"),
Base: text(args, "base"), Message: strings.TrimSpace(raw(args, "message"))}
a.Patch = raw(args, "patch")
if a.Branch == "" {
return a, refuse("branch is required: the branch the commit goes on, a branch of its own (created from base when absent). Nothing was sent")
}
if a.Message == "" {
return a, refuse("message is required: the commit's message. Nothing was sent")
}
files, given := args["files"]
if given && files != nil {
list, ok := files.([]any)
if !ok {
return a, refuse("files is a list of {path, content | content_base64 | delete: true}. Nothing was sent")
}
for i, item := range list {
m, ok := item.(map[string]any)
if !ok {
return a, refuse("files[%d] is not an object {path, content | content_base64 | delete: true}. Nothing was sent", i)
}
fc := FileChange{Path: text(m, "path")}
if v, ok := m["content"]; ok && v != nil {
s, ok := v.(string)
if !ok {
return a, refuse("files[%d].content is text. Nothing was sent", i)
}
fc.Content = &s
}
if v, ok := m["content_base64"]; ok && v != nil {
s, ok := v.(string)
if !ok {
return a, refuse("files[%d].content_base64 is text. Nothing was sent", i)
}
fc.ContentBase64 = &s
}
if v, ok := m["delete"]; ok && v != nil {
b, ok := v.(bool)
if !ok {
return a, refuse("files[%d].delete is true or false. Nothing was sent", i)
}
fc.Delete = b
}
a.Files = append(a.Files, fc)
}
}
switch {
case len(a.Files) == 0 && strings.TrimSpace(a.Patch) == "":
return a, refuse("give files (each {path, content | content_base64 | delete: true}) or patch (a unified diff, as git diff writes it). Nothing was sent")
case len(a.Files) > 0 && strings.TrimSpace(a.Patch) != "":
return a, refuse("give files or patch, not both: one commit carries one of them. Nothing was sent")
case len(a.Files) > maxFiles:
return a, refuse("%d files is more than the %d one commit carries. Nothing was sent", len(a.Files), maxFiles)
case len(a.Patch) > maxPatchBytes:
return a, refuse("the patch is %s, more than the %s one commit carries: split it into several commits. Nothing was sent", mib(len(a.Patch)), mib(maxPatchBytes))
}
seen := map[string]bool{}
total := 0
for i, fc := range a.Files {
if strings.HasPrefix(fc.Path, "/") {
return a, refuse("files[%d].path %q is relative to the repository's root, without a leading slash. Nothing was sent", i, fc.Path)
}
if _, err := segments("file path", fc.Path); err != nil {
return a, err
}
if seen[fc.Path] {
return a, refuse("%s is named twice in files. Nothing was sent", fc.Path)
}
seen[fc.Path] = true
given := 0
for _, g := range []bool{fc.Content != nil, fc.ContentBase64 != nil, fc.Delete} {
if g {
given++
}
}
if given != 1 {
return a, refuse("files[%d] (%s) needs exactly one of content, content_base64 or delete: true. Nothing was sent", i, fc.Path)
}
if fc.Content != nil && !utf8.ValidString(*fc.Content) {
return a, refuse("files[%d].content (%s) is not UTF-8 text: give it as content_base64. Nothing was sent", i, fc.Path)
}
size := 0
switch {
case fc.Content != nil:
size = len(*fc.Content)
case fc.ContentBase64 != nil:
if base64.StdEncoding.DecodedLen(len(*fc.ContentBase64)) > maxFileBytes+3 {
size = base64.StdEncoding.DecodedLen(len(*fc.ContentBase64))
break
}
raw, err := base64.StdEncoding.DecodeString(*fc.ContentBase64)
if err != nil {
return a, refuse("files[%d].content_base64 (%s) is not base64: %v. Nothing was sent", i, fc.Path, err)
}
size = len(raw)
}
if size > maxFileBytes {
return a, refuse("files[%d] (%s) is %s, more than the %s one file of a commit may be. Nothing was sent", i, fc.Path, mib(size), mib(maxFileBytes))
}
total += size
}
if total > maxCommitBytes {
return a, refuse("the files are %s together, more than the %s one commit carries: split them into several commits. Nothing was sent", mib(total), mib(maxCommitBytes))
}
return a, nil
}
// mib says a size in bytes in MiB, to a tenth.
func mib(n int) string { return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20)) }
// targetRefused says why a commit may not go on a branch: the default branch, or one a protection rule names.
func targetRefused(target, defaultBranch string, rules []Protection) error {
if target == defaultBranch {
return refuse("a commit on %s, the repository's default branch, is refused: a change reaches it by a pull request "+
"that passes its checks (novox/hq ADR 0266). Put it on a branch of its own. Nothing was sent", target)
}
var guarding []string
for _, r := range rules {
if n := r.name(); n != "" && ruleCovers(n, target) {
guarding = append(guarding, n)
}
}
if len(guarding) > 0 {
return refuse("a commit on %s is refused: the protection rule %s guards it (novox/hq ADR 0266). Put it on a "+
"branch of its own. Nothing was sent", target, strings.Join(guarding, ", "))
}
return nil
}
// commit puts one commit on the asked branch, after the judgement, and reads the branch back.
func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
repo, err := repoPath(a.Owner, a.Repo)
if err != nil {
return nil, err
}
if _, err := segments("branch", a.Branch); err != nil {
return nil, err
}
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.DefaultBranch == "" {
return nil, fmt.Errorf("the forge did not say %s/%s's default branch, so whether %s is it cannot be judged. Nothing was sent", a.Owner, a.Repo, a.Branch)
}
// Read the rules, or refuse: a branch the rules could not be read for is never written (fails closed).
var rules []Protection
if err := f.GetJSON(ctx, repo+"/branch_protections", &rules); err != nil {
return nil, fmt.Errorf("the protection rules of %s/%s could not be read, so the commit was not judged and nothing was sent: %w", a.Owner, a.Repo, err)
}
if err := targetRefused(a.Branch, r.DefaultBranch, rules); err != nil {
return nil, err
}
out := &Committed{Repository: a.Owner + "/" + a.Repo, Branch: a.Branch}
existing, err := f.branch(ctx, repo, a.Branch)
if err != nil {
return nil, err
}
// The files are judged at the commit just read, never at a branch name that could move between two
// reads (novox/hq issue 539): a file the branch changes meanwhile is refused by the forge, whose update
// and delete carry the blob read here.
var ref string
if existing != nil {
out.Parent = existing.Commit.ID
ref = existing.Commit.ID
if a.Base != "" {
out.Said = append(out.Said, fmt.Sprintf("%s exists, at %s: base %s was not used", a.Branch, short(existing.Commit.ID), a.Base))
}
} else {
out.Created = true
out.Base = a.Base
if out.Base == "" {
out.Base = r.DefaultBranch
}
b, err := f.branch(ctx, repo, out.Base)
if err != nil {
return nil, err
}
if b == nil {
return nil, refuse("%s does not exist, and its base %s is not a branch of %s/%s. Nothing was sent", a.Branch, out.Base, a.Owner, a.Repo)
}
out.Parent = b.Commit.ID
ref = b.Commit.ID
}
// Where the forge takes the commit: on the branch, or on a new one from the base's commit.
on := map[string]any{"message": a.Message}
if out.Created {
on["branch"] = out.Base
on["new_branch"] = a.Branch
} else {
on["branch"] = a.Branch
}
// The forge takes no commit to build on, only a branch (Gitea 1.27: neither /contents nor /diffpatch
// carries a last commit), so the parent is read from what it made, never assumed (novox/hq issue 539).
var made struct {
Commit madeCommit `json:"commit"`
}
if a.Patch != "" {
on["content"] = a.Patch
if err := f.SendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the patch: %w", err)
}
} else {
var ops []map[string]any
for _, fc := range a.Files {
sha, err := f.blobAt(ctx, repo, fc.Path, ref)
if err != nil {
return nil, err
}
op := map[string]any{"path": fc.Path}
switch {
case fc.Delete && sha == "":
return nil, refuse("%s is to be deleted, but %s has no such file at %s. Nothing was sent", fc.Path, a.Owner+"/"+a.Repo, short(ref))
case fc.Delete:
op["operation"], op["sha"] = "delete", sha
case sha == "":
op["operation"] = "create"
default:
op["operation"], op["sha"] = "update", sha
}
if !fc.Delete {
if fc.Content != nil {
op["content"] = base64.StdEncoding.EncodeToString([]byte(*fc.Content))
} else {
op["content"] = *fc.ContentBase64
}
}
ops = append(ops, op)
out.Files = append(out.Files, CommittedFile{Path: fc.Path, Operation: op["operation"].(string)})
}
on["files"] = ops
if err := f.SendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the commit: %w", err)
}
}
if made.Commit.SHA == "" {
return nil, fmt.Errorf("the forge answered the commit on %s without saying the commit it made: read the branch (gitea_list_branches) before trying again", a.Branch)
}
out.Commit = made.Commit.SHA
read := out.Parent
parent, err := f.parentOf(ctx, repo, made.Commit)
if err != nil {
return nil, fmt.Errorf("the forge made %s on %s, but which commit it went on could not be read: %w", out.Commit, a.Branch, err)
}
out.Parent = parent
if parent != read {
moved := "branch " + a.Branch
if out.Created {
moved = "base " + out.Base
}
how := "each file it changes was checked against " + short(read) + " by its blob, so none of them changed in between"
if a.Patch != "" {
how = "the forge applied the patch there, and it refuses a patch whose context changed"
}
out.Said = append(out.Said, fmt.Sprintf("the %s moved from %s to %s between the read and the commit, so the "+
"commit went on %s: %s. What else changed in between: gitea_compare %s...%s",
moved, short(read), short(parent), short(parent), how, short(read), short(parent)))
}
// Evidence over assertion: the branch is read back, and it must be at the commit the forge said.
after, err := f.branch(ctx, repo, a.Branch)
switch {
case err != nil:
return nil, fmt.Errorf("the forge made %s on %s, but the branch could not be read back: %w", out.Commit, a.Branch, err)
case after == nil:
return nil, fmt.Errorf("the forge said it made %s on %s, but the branch does not exist", out.Commit, a.Branch)
case after.Commit.ID != out.Commit:
return nil, fmt.Errorf("the forge said it made %s on %s, but the branch is at %s: something else moved it", out.Commit, a.Branch, after.Commit.ID)
}
what := "on"
if out.Created {
what = "on the new branch"
}
out.Said = append(out.Said, fmt.Sprintf("%s committed %s %s of %s, after %s; the branch is there now. A merge "+
"still needs a review at this head and both required checks green", short(out.Commit), what, a.Branch, out.Repository, short(out.Parent)))
return out, nil
}
// madeCommit is the commit the forge answers a write with.
type madeCommit struct {
SHA string `json:"sha"`
Parents []struct {
SHA string `json:"sha"`
} `json:"parents"`
}
// parentOf is the commit the forge made its commit on: from its answer, or read from the commit when the
// answer does not say it.
func (f *Forge) parentOf(ctx context.Context, repo string, made madeCommit) (string, error) {
if len(made.Parents) == 0 {
var read madeCommit
if err := f.GetJSON(ctx, repo+"/git/commits/"+url.PathEscape(made.SHA)+"?stat=false&verification=false&files=false", &read); err != nil {
return "", err
}
made = read
}
if len(made.Parents) != 1 || made.Parents[0].SHA == "" {
return "", fmt.Errorf("the forge says %s has %d parents, not one", made.SHA, len(made.Parents))
}
return made.Parents[0].SHA, nil
}
// blobAt is the blob sha of a file at a ref, "" when there is no file there.
func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, error) {
seg, err := segments("file path", path)
if err != nil {
return "", err
}
var entry struct {
Type string `json:"type"`
SHA string `json:"sha"`
}
var list []any
body, err := f.Do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return "", nil
}
return "", err
}
if jsonInto(body, &list) == nil {
return "", refuse("%s is a directory, not a file. Nothing was sent", path)
}
if err := jsonInto(body, &entry); err != nil {
return "", fmt.Errorf("the forge's answer about %s is not readable: %w", path, err)
}
if entry.Type != "file" && entry.Type != "symlink" {
return "", refuse("%s is a %s, not a file. Nothing was sent", path, entry.Type)
}
return entry.SHA, nil
}
func short(sha string) string {
if len(sha) > 12 {
return sha[:12]
}
return sha
}
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"context"
"errors"
"fmt"
"net/url"
"regexp"
"strings"
"gitea/internal/forgehttp"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the npm-registry bundle beside it).
// Forge is the forge's HTTP interface.
type Forge struct {
forgehttp.Client
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
c, missing := forgehttp.FromEnv()
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
}
c.Timeout = deadline
return &Forge{Client: c}, nil
}
// errNotFound is the forge answering 404.
var errNotFound = forgehttp.ErrNotFound
// maxBody is the largest JSON answer read from the forge; maxArchive the largest archive code search reads.
const (
maxBody = forgehttp.MaxBody
maxArchive = 64 << 20
)
// Refused is a refusal of policy or of an argument, in the words the caller sees.
type Refused struct{ msg string }
func (r *Refused) Error() string { return r.msg }
func refuse(format string, a ...any) error { return &Refused{fmt.Sprintf(format, a...)} }
var nameShape = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.-]*$`)
// repoPath is /api/v1/repos/{owner}/{repo} for names that cannot become another path.
func repoPath(owner, repo string) (string, error) {
for _, v := range [][2]string{{"owner", owner}, {"name", repo}} {
if !nameShape.MatchString(v[1]) || v[1] == "." || v[1] == ".." {
return "", refuse("%q is not a repository %s", v[1], v[0])
}
}
return "/api/v1/repos/" + owner + "/" + repo, nil
}
// segments encodes a value that is several path segments — a branch like fix/1-x, a file's path — refusing
// any segment that could make the request another path: empty, `.` or `..`, or holding `?`, `#`, `%`, `\`
// or a control character (the same rule as the TypeScript bundle's pathSegments, guard.ts).
func segments(what, value string) (string, error) {
parts := strings.Split(value, "/")
for i, p := range parts {
if p == "" || p == "." || p == ".." || strings.ContainsAny(p, "?#%\\") || strings.IndexFunc(p, func(r rune) bool { return r < 0x20 || r == 0x7f }) >= 0 {
return "", refuse("%q is not a %s this tool puts in a path. Nothing was sent", value, what)
}
parts[i] = url.PathEscape(p)
}
return strings.Join(parts, "/"), nil
}
// Repo is what the tools read of a repository.
type Repo struct {
FullName string `json:"full_name"`
DefaultBranch string `json:"default_branch"`
Empty bool `json:"empty"`
}
// Protection is one branch protection rule, as far as covering a branch goes.
type Protection struct {
RuleName string `json:"rule_name"`
BranchName string `json:"branch_name"`
}
func (p Protection) name() string {
if p.RuleName != "" {
return p.RuleName
}
return p.BranchName
}
// Branch is one branch with the commit it points at.
type Branch struct {
Name string `json:"name"`
Commit struct {
ID string `json:"id"`
} `json:"commit"`
}
// branch is the named branch, nil when the forge has none of that name.
func (f *Forge) branch(ctx context.Context, repo, name string) (*Branch, error) {
seg, err := segments("branch", name)
if err != nil {
return nil, err
}
var b Branch
if err := f.GetJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
if errors.Is(err, errNotFound) {
return nil, nil
}
return nil, err
}
return &b, nil
}
+744
View File
@@ -0,0 +1,744 @@
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"testing"
"gitea/internal/forgehttp"
)
// fakeForge is the forge's HTTP interface as far as these tools use it.
type fakeForge struct {
t *testing.T
mu sync.Mutex
password string
branches map[string]string // branch -> commit
trees map[string]map[string]string // commit -> path -> content
rules []Protection
rulesBroken bool
staleRead bool // the branch is not moved by a write: a forge that says what it did not do
writes []map[string]any
patches []map[string]any
next int
archive []byte
pullHead string
reviews []map[string]any
comments map[string][]map[string]any
commits []map[string]any
dir []map[string]any
// moves: a branch moved by someone else right after it is first read — to a commit, with its tree.
moves map[string]move
noParents bool // the forge's answer to a write does not say the parents
parentsBroken bool // and GET git/commits fails
parents map[string]string // commit -> parent, as GET git/commits says it
}
type move struct {
to string
tree map[string]string
}
func newFake(t *testing.T) *fakeForge {
return &fakeForge{
t: t,
password: "s3cret-never-shown",
branches: map[string]string{"main": "c0", "fix/1-old": "c1"},
trees: map[string]map[string]string{
"c0": {"README.md": "hello\n", "docs/a.md": "a\n"},
"c1": {"README.md": "hello\n", "docs/a.md": "a on fix\n"},
},
rules: []Protection{{RuleName: "main"}, {RuleName: "release/*"}},
}
}
func blob(content string) string { return fmt.Sprintf("%x", sha1.Sum([]byte(content))) }
func (f *fakeForge) serve() *Forge {
dir := f.t.TempDir()
pw := filepath.Join(dir, "admin.secret")
if err := os.WriteFile(pw, []byte("s3cret-never-shown\n"), 0o600); err != nil {
f.t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(f.handle))
f.t.Cleanup(srv.Close)
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}}
}
func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
if u, p, ok := r.BasicAuth(); !ok || u != "mesh-admin" || p != f.password {
http.Error(w, "unauthorized", 401)
return
}
reply := func(v any) { _ = json.NewEncoder(w).Encode(v) }
const repo = "/api/v1/repos/acme/app"
path := r.URL.EscapedPath()
switch {
case r.Method == "GET" && path == repo:
reply(map[string]any{"full_name": "acme/app", "default_branch": "main"})
case r.Method == "GET" && path == repo+"/branch_protections":
if f.rulesBroken {
http.Error(w, "boom", 500)
return
}
reply(f.rules)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/branches/"):
name := strings.TrimPrefix(r.URL.Path, repo+"/branches/")
c, ok := f.branches[name]
if !ok {
http.NotFound(w, r)
return
}
reply(map[string]any{"name": name, "commit": map[string]any{"id": c}})
if m, ok := f.moves[name]; ok {
delete(f.moves, name)
f.trees[m.to] = m.tree
f.branches[name] = m.to
}
case r.Method == "GET" && (path == repo+"/contents" || path == repo+"/contents/docs"):
reply(f.dir)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/contents/"):
p := strings.TrimPrefix(r.URL.Path, repo+"/contents/")
ref := r.URL.Query().Get("ref")
if c, isBranch := f.branches[ref]; isBranch {
ref = c
}
content, ok := f.trees[ref][p]
if !ok {
http.NotFound(w, r)
return
}
reply(map[string]any{"type": "file", "path": p, "sha": blob(content)})
case r.Method == "POST" && path == repo+"/contents":
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
f.writes = append(f.writes, body)
from := f.branches[body["branch"].(string)]
tree := map[string]string{}
for k, v := range f.trees[from] {
tree[k] = v
}
for _, raw := range body["files"].([]any) {
op := raw.(map[string]any)
p := op["path"].(string)
old, had := tree[p]
switch op["operation"] {
case "create":
if had {
http.Error(w, "exists", 422)
return
}
case "update", "delete":
if !had || op["sha"] != blob(old) {
http.Error(w, "sha does not match", 409)
return
}
}
if op["operation"] == "delete" {
delete(tree, p)
continue
}
c, _ := base64.StdEncoding.DecodeString(op["content"].(string))
tree[p] = string(c)
}
f.next++
made := fmt.Sprintf("n%d", f.next)
f.trees[made] = tree
target := body["branch"].(string)
if nb, ok := body["new_branch"].(string); ok && nb != "" {
target = nb
}
if !f.staleRead {
f.branches[target] = made
}
reply(map[string]any{"commit": f.made(made, from)})
case r.Method == "POST" && path == repo+"/diffpatch":
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
f.patches = append(f.patches, body)
f.next++
made := fmt.Sprintf("p%d", f.next)
from := f.branches[body["branch"].(string)]
target := body["branch"].(string)
if nb, ok := body["new_branch"].(string); ok && nb != "" {
target = nb
}
f.branches[target] = made
reply(map[string]any{"commit": f.made(made, from)})
case r.Method == "GET" && strings.HasPrefix(path, repo+"/git/commits/"):
sha := strings.TrimPrefix(path, repo+"/git/commits/")
p, ok := f.parents[sha]
if !ok || f.parentsBroken {
http.NotFound(w, r)
return
}
reply(map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": p}}})
case r.Method == "GET" && path == repo+"/commits":
f.writes = append(f.writes, map[string]any{"query": r.URL.RawQuery})
reply(f.commits)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/compare/"):
reply(map[string]any{"total_commits": len(f.commits), "commits": f.commits})
case r.Method == "GET" && path == repo+"/pulls/7":
reply(map[string]any{"head": map[string]any{"sha": f.pullHead}})
case r.Method == "GET" && path == repo+"/pulls/7/reviews":
reply(f.reviews)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/pulls/7/reviews/") && strings.HasSuffix(path, "/comments"):
id := strings.TrimSuffix(strings.TrimPrefix(path, repo+"/pulls/7/reviews/"), "/comments")
reply(f.comments[id])
case r.Method == "GET" && strings.HasPrefix(path, repo+"/archive/"):
_, _ = w.Write(f.archive)
default:
http.NotFound(w, r)
}
}
// made is the forge's answer about a commit it made on a parent, with the parents unless noParents.
func (f *fakeForge) made(sha, parent string) map[string]any {
if f.parents == nil {
f.parents = map[string]string{}
}
f.parents[sha] = parent
if f.noParents {
return map[string]any{"sha": sha}
}
return map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": parent}}}
}
func filesArg(files ...map[string]any) []any {
out := make([]any, len(files))
for i, f := range files {
out[i] = f
}
return out
}
func ask(t *testing.T, args map[string]any) CommitAsk {
t.Helper()
base := map[string]any{"owner": "acme", "repo": "app", "message": "Say why"}
for k, v := range args {
base[k] = v
}
a, err := readCommitAsk(base)
if err != nil {
t.Fatalf("ask refused: %v", err)
}
return a
}
func TestACommitOnANewBranchCreatesUpdatesAndDeletesInOneCommitFromTheBase(t *testing.T) {
f := newFake(t)
a := ask(t, map[string]any{"branch": "fix/375-x", "files": filesArg(
map[string]any{"path": "new.md", "content": "new\n"},
map[string]any{"path": "README.md", "content": "hello again\n"},
map[string]any{"path": "docs/a.md", "delete": true},
)})
got, err := commit(context.Background(), f.serve(), a)
if err != nil {
t.Fatal(err)
}
if !got.Created || got.Base != "main" || got.Parent != "c0" || got.Commit != "n1" || f.branches["fix/375-x"] != "n1" {
t.Fatalf("%+v; branches %v", got, f.branches)
}
if len(f.writes) != 1 {
t.Fatalf("%d writes", len(f.writes))
}
w := f.writes[0]
if w["branch"] != "main" || w["new_branch"] != "fix/375-x" || w["message"] != "Say why" {
t.Fatalf("sent %v", w)
}
ops := map[string]string{}
for _, raw := range w["files"].([]any) {
op := raw.(map[string]any)
ops[op["path"].(string)] = op["operation"].(string)
if op["operation"] == "create" && op["sha"] != nil {
t.Error("a created file carried a sha")
}
}
if ops["new.md"] != "create" || ops["README.md"] != "update" || ops["docs/a.md"] != "delete" {
t.Fatalf("ops %v", ops)
}
tree := f.trees["n1"]
if tree["README.md"] != "hello again\n" || tree["new.md"] != "new\n" || tree["docs/a.md"] != "" {
t.Fatalf("tree %v", tree)
}
if f.branches["main"] != "c0" {
t.Fatal("main moved")
}
}
func TestACommitOnAnExistingBranchGoesOnItAndReadsItsFilesThere(t *testing.T) {
f := newFake(t)
a := ask(t, map[string]any{"branch": "fix/1-old", "base": "main",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})})
got, err := commit(context.Background(), f.serve(), a)
if err != nil {
t.Fatal(err)
}
if got.Created || got.Parent != "c1" || f.branches["fix/1-old"] != got.Commit {
t.Fatalf("%+v", got)
}
if _, has := f.writes[0]["new_branch"]; has || f.writes[0]["branch"] != "fix/1-old" {
t.Fatalf("sent %v", f.writes[0])
}
if !strings.Contains(strings.Join(got.Said, " "), "base main was not used") {
t.Errorf("an unused base was not said: %v", got.Said)
}
}
func TestTheDefaultBranchAndProtectedBranchesAreRefusedBeforeAnythingIsSent(t *testing.T) {
for _, branch := range []string{"main", "release/1.0"} {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": branch,
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "refused") || !strings.Contains(err.Error(), "ADR 0266") {
t.Errorf("%s: %v", branch, err)
}
if len(f.writes)+len(f.patches) != 0 {
t.Errorf("%s: something was written", branch)
}
}
}
func TestRulesThatCannotBeReadRefuseTheCommit(t *testing.T) {
f := newFake(t)
f.rulesBroken = true
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/2",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "protection rules") || len(f.writes) != 0 {
t.Fatalf("%v, %d writes", err, len(f.writes))
}
}
func TestAPatchIsAppliedByTheForgeOnANewBranch(t *testing.T) {
f := newFake(t)
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/3", "patch": patch}))
if err != nil {
t.Fatal(err)
}
if len(f.patches) != 1 || f.patches[0]["content"] != patch || f.patches[0]["branch"] != "main" || f.patches[0]["new_branch"] != "fix/3" {
t.Fatalf("%v", f.patches)
}
if got.Commit != "p1" || len(f.writes) != 0 {
t.Fatalf("%+v", got)
}
}
func TestACommitTheBranchDoesNotShowIsAFailure(t *testing.T) {
f := newFake(t)
f.staleRead = true
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "branch is at c1") {
t.Fatalf("%v", err)
}
}
func TestDeletingAFileThatIsNotThereIsRefused(t *testing.T) {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/4",
"files": filesArg(map[string]any{"path": "gone.md", "delete": true})}))
if err == nil || !strings.Contains(err.Error(), "no such file") || len(f.writes) != 0 {
t.Fatalf("%v", err)
}
}
func TestAMissingBaseIsRefused(t *testing.T) {
f := newFake(t)
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/5", "base": "nope",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "base nope is not a branch") {
t.Fatalf("%v", err)
}
}
func TestACommitAskThatCannotBeReadIsRefused(t *testing.T) {
one := filesArg(map[string]any{"path": "x", "content": "x"})
for name, args := range map[string]map[string]any{
"no branch": {"files": one},
"no message": {"branch": "b", "message": " ", "files": one},
"neither": {"branch": "b"},
"both": {"branch": "b", "files": one, "patch": "diff"},
"dot path": {"branch": "b", "files": filesArg(map[string]any{"path": "a/../b", "content": "x"})},
"absolute path": {"branch": "b", "files": filesArg(map[string]any{"path": "/etc/x", "content": "x"})},
"twice": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content": "x"}, map[string]any{"path": "x", "delete": true})},
"two contents": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content": "x", "delete": true})},
"no content": {"branch": "b", "files": filesArg(map[string]any{"path": "x"})},
"bad base64": {"branch": "b", "files": filesArg(map[string]any{"path": "x", "content_base64": "!!"})},
"files not list": {"branch": "b", "files": "x"},
"path with query": {"branch": "b", "files": filesArg(map[string]any{"path": "a?b", "content": "x"})},
} {
full := map[string]any{"owner": "acme", "repo": "app", "message": "m"}
for k, v := range args {
full[k] = v
}
if _, err := readCommitAsk(full); err == nil {
t.Errorf("%s: taken", name)
}
}
f := newFake(t)
_, err := commit(context.Background(), f.serve(), CommitAsk{Owner: "acme", Repo: "app", Branch: "../x", Message: "m", Patch: "d"})
if err == nil {
t.Error("a branch with a dot segment was taken")
}
}
// The forge's own answers (Gitea 1.27's Match), one table for this bundle and guard.ts alike (novox/hq issue 539).
// `forge` is the forge's answer, which this port must give; `guard` is where guard.ts deliberately covers more.
func TestRuleCoversAnswersAsTheForgeDoes(t *testing.T) {
raw, err := os.ReadFile("../../test/rule-cases.json")
if err != nil {
t.Fatal(err)
}
var cases []struct {
Rule string `json:"rule"`
Branch string `json:"branch"`
Forge bool `json:"forge"`
}
if err := json.Unmarshal(raw, &cases); err != nil {
t.Fatal(err)
}
if len(cases) < 100 {
t.Fatalf("only %d cases", len(cases))
}
for _, c := range cases {
if got := ruleCovers(c.Rule, c.Branch); got != c.Forge {
t.Errorf("%q covers %q: %v, the forge says %v", c.Rule, c.Branch, got, c.Forge)
}
}
}
func TestHistoryAndCompareAreSummarised(t *testing.T) {
f := newFake(t)
mk := func(sha, msg string, files ...string) map[string]any {
var fs []map[string]any
for _, x := range files {
fs = append(fs, map[string]any{"filename": x, "status": "modified"})
}
return map[string]any{"sha": sha, "commit": map[string]any{"message": msg,
"author": map[string]any{"name": "jochen", "date": "2026-10-12T10:00:00Z"}}, "files": fs}
}
f.commits = []map[string]any{mk("a1", "First line\n\nbody", "x.md", "y.md"), mk("a2", "Second", "y.md")}
forge := f.serve()
h, err := history(context.Background(), forge, "acme", "app", "fix/1", "docs", 2, 1)
if err != nil {
t.Fatal(err)
}
if len(h.Commits) != 2 || h.Commits[0].Subject != "First line" || h.Commits[0].Author != "jochen" || !h.More {
t.Fatalf("%+v", h)
}
q := f.writes[0]["query"].(string)
for _, want := range []string{"sha=fix%2F1", "path=docs", "limit=2", "files=false"} {
if !strings.Contains(q, want) {
t.Errorf("query %s lacks %s", q, want)
}
}
c, err := compare(context.Background(), forge, "acme", "app", "main", "fix/1")
if err != nil {
t.Fatal(err)
}
if c.Total != 2 || len(c.Files) != 2 || c.Files[0].Path != "x.md" || c.Commits[1].Subject != "Second" {
t.Fatalf("%+v", c)
}
if _, err := compare(context.Background(), forge, "acme", "app", "a...b", "c"); err == nil {
t.Error("a ref holding ... was compared")
}
}
func TestReviewsSayWhichApproveTheHeadAndCarryTheirLineComments(t *testing.T) {
f := newFake(t)
f.pullHead = "h2"
f.reviews = []map[string]any{
{"id": 1, "user": map[string]any{"login": "reviewer"}, "state": "APPROVED", "commit_id": "h1", "comments_count": 0},
{"id": 2, "user": map[string]any{"login": "reviewer"}, "state": "REQUEST_CHANGES", "commit_id": "h2", "comments_count": 1, "body": "see line"},
{"id": 3, "user": map[string]any{"login": "op"}, "state": "APPROVED", "commit_id": "h2", "comments_count": 0},
}
f.comments = map[string][]map[string]any{"2": {{"path": "a.go", "position": 12, "user": map[string]any{"login": "reviewer"},
"body": "off by one", "resolver": map[string]any{"login": "op"}}}}
r, err := reviews(context.Background(), f.serve(), "acme", "app", 7)
if err != nil {
t.Fatal(err)
}
if len(r.Reviews) != 3 || r.Reviews[0].AtHead || !r.Reviews[1].AtHead || len(r.Reviews[1].Comments) != 1 {
t.Fatalf("%+v", r)
}
lc := r.Reviews[1].Comments[0]
if lc.Path != "a.go" || lc.Line != 12 || lc.Resolved != "op" {
t.Fatalf("%+v", lc)
}
if !strings.Contains(r.Said[0], "1 approve the head") {
t.Fatalf("%v", r.Said)
}
}
func TestADirectoryIsListed(t *testing.T) {
f := newFake(t)
f.dir = []map[string]any{{"name": "a.md", "type": "file", "size": 3}, {"name": "sub", "type": "dir", "size": 0}}
l, err := listDir(context.Background(), f.serve(), "acme", "app", "/docs/", "main")
if err != nil {
t.Fatal(err)
}
if l.Path != "/docs" || len(l.Entries) != 2 || l.Entries[0].Size != 3 || l.Entries[1].Type != "dir" {
t.Fatalf("%+v", l)
}
}
func archive(t *testing.T, files map[string]string) []byte {
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
_ = tw.WriteHeader(&tar.Header{Name: "app/", Typeflag: tar.TypeDir, Mode: 0o755})
for name, content := range files {
if err := tw.WriteHeader(&tar.Header{Name: "app/" + name, Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(content))}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(content))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
func TestCodeSearchReadsTheArchiveOnce(t *testing.T) {
f := newFake(t)
f.archive = archive(t, map[string]string{
"docs/a.md": "one\nthe Forge\nthree forge\n",
"src/b.go": "package b // forge\n",
"bin/x": "forge\x00binary",
"docs/c.txt": "nothing\n",
})
forge := f.serve()
found, err := searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "forge", Max: 100})
if err != nil {
t.Fatal(err)
}
if len(found.Matches) != 2 || found.Ref != "main" || found.Files != 3 {
t.Fatalf("%+v", found)
}
found, _ = searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "forge", IgnoreCase: true, Path: "docs", Max: 100})
if len(found.Matches) != 2 || found.Matches[0].Path != "docs/a.md" || found.Matches[0].Line != 2 {
t.Fatalf("%+v", found)
}
found, _ = searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "^(one|three)", Regex: true, Max: 1})
if len(found.Matches) != 1 || !found.Truncated {
t.Fatalf("%+v", found)
}
if _, err := searchCode(context.Background(), forge, SearchAsk{Owner: "acme", Repo: "app", Query: "(", Regex: true, Max: 1}); err == nil {
t.Error("a broken expression was searched")
}
}
func TestAWrongPasswordFailsWithoutSayingIt(t *testing.T) {
f := newFake(t)
forge := f.serve()
f.password = "rotated"
_, err := listDir(context.Background(), forge, "acme", "app", "", "")
if err == nil || !strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "s3cret") {
t.Fatalf("%v", err)
}
}
func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
tools := Tools(func() (*Forge, error) { return nil, fmt.Errorf("unset") })
var served []string
for _, tl := range tools {
served = append(served, tl.Name)
suffix := "(r)"
if tl.Name == "gitea_commit" {
suffix = "(a)"
}
if !strings.HasSuffix(tl.Description, suffix) {
t.Errorf("%s does not end with %s", tl.Name, suffix)
}
if _, err := tl.Run(map[string]any{}); err == nil {
t.Errorf("%s ran without a forge", tl.Name)
}
}
if strings.Join(served, ",") != strings.Join(ToolNames, ",") {
t.Fatalf("served %v, named %v", served, ToolNames)
}
readme, err := os.ReadFile("../../README.md")
if err != nil {
t.Fatal(err)
}
for _, n := range ToolNames {
if !strings.Contains(string(readme), "`"+n+"`") {
t.Errorf("the README does not name %s", n)
}
}
// No name the TypeScript bundle serves: two bundles serving one name is one tool answered twice.
ts, err := os.ReadFile("../../tools/index.ts")
if err != nil {
t.Fatal(err)
}
for _, n := range ToolNames {
if strings.Contains(string(ts), `name: "`+n+`"`) {
t.Errorf("the TypeScript bundle serves %s too", n)
}
}
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Tools []string `json:"tools"`
Build struct {
Artifacts []struct {
From string `json:"from"`
Binary string `json:"binary"`
Loads []string `json:"loads"`
Env map[string]string `json:"env"`
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if len(m.Tools) != 0 {
t.Error("the manifest lists tools: its claims would offer them as both seats' verbs")
}
found := false
for _, a := range m.Build.Artifacts {
if a.From == "cmd/forge" {
found = true
if len(a.Loads) != 1 || a.Loads[0] != a.Binary {
t.Errorf("the bundle loads %v, not its binary %s", a.Loads, a.Binary)
}
for _, k := range []string{"MESH_GITEA_URL", "MESH_GITEA_ADMIN_USER", "MESH_GITEA_ADMIN_PASSWORD_FILE"} {
if a.Env[k] == "" {
t.Errorf("the bundle is not given %s", k)
}
}
}
}
if !found {
t.Error("the manifest builds no cmd/forge bundle")
}
}
// novox/hq issue 539: the forge takes a branch, never a commit to build on, so the answer's parent is the one
// the forge made the commit on, and a branch that moves between the read and the commit is said.
func TestAnExistingBranchMovedUnderTheCommitIsSaidAndItsParentIsTheOneTheForgeUsed(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "a on fix\n", "other.md": "x\n"}}}
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c2" || f.branches["fix/1-old"] != got.Commit {
t.Fatalf("parent %s, want c2 (the commit the forge built on): %+v", got.Parent, got)
}
said := strings.Join(got.Said, " ")
if !strings.Contains(said, "moved from c1 to c2") || !strings.Contains(said, "after c2") {
t.Fatalf("the move was not said: %v", got.Said)
}
}
func TestAnExistingBranchsFilesAreJudgedAtTheCommitReadNotAtWhereTheBranchMovedTo(t *testing.T) {
f := newFake(t)
// Someone else changes docs/a.md on the branch between the read and the commit: that change is not
// overwritten by a blob read at the branch's new head.
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "theirs\n"}}}
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "mine\n"})}))
if err == nil || !strings.Contains(err.Error(), "sha does not match") {
t.Fatalf("a file changed under the commit was overwritten: %v", err)
}
if f.trees["c2"]["docs/a.md"] != "theirs\n" || f.branches["fix/1-old"] != "c2" {
t.Fatalf("the branch was changed: %v", f.branches)
}
}
func TestAPatchOnABaseThatMovedNamesTheCommitTheForgeAppliedItTo(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"main": {to: "c9", tree: map[string]string{"README.md": "hello\n"}}}
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/6", "patch": patch}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c9" || !strings.Contains(strings.Join(got.Said, " "), "base main moved from c0 to c9") {
t.Fatalf("%+v", got)
}
}
func TestAParentTheAnswerDoesNotSayIsReadFromTheCommit(t *testing.T) {
f := newFake(t)
f.noParents = true
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/7",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err != nil || got.Parent != "c0" {
t.Fatalf("%v %+v", err, got)
}
// A parent that cannot be read is a failure, said with the commit that was made.
f = newFake(t)
f.noParents, f.parentsBroken = true, true
_, err = commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/8",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "the forge made n1 on fix/8, but which commit it went on could not be read") {
t.Fatalf("%v", err)
}
}
func TestACommitLargerThanItsLimitsIsRefusedBeforeAnythingIsSent(t *testing.T) {
big := strings.Repeat("x", maxFileBytes+1)
quarter := strings.Repeat("y", maxFileBytes)
many := filesArg()
for i := 0; i < maxCommitBytes/maxFileBytes+1; i++ {
many = append(many, map[string]any{"path": fmt.Sprintf("f%d", i), "content": quarter})
}
for name, args := range map[string]map[string]any{
"one file": {"files": filesArg(map[string]any{"path": "x", "content": big})},
"one file b64": {"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(big))})},
"the patch": {"patch": "diff --git a/x b/x\n" + big},
"all files": {"files": many},
} {
full := map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b"}
for k, v := range args {
full[k] = v
}
_, err := readCommitAsk(full)
if err == nil || !strings.Contains(err.Error(), "MiB") || !strings.Contains(err.Error(), "Nothing was sent") {
t.Errorf("%s: %v", name, err)
}
}
// At the limit is taken.
if _, err := readCommitAsk(map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b",
"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(quarter))})}); err != nil {
t.Errorf("a file at the limit was refused: %v", err)
}
}
func TestCodeSearchStopsAtItsDeadlineAndSaysSo(t *testing.T) {
gz := archive(t, map[string]string{"a.md": "forge\n", "b.md": "forge\n"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
out := &Found{Matches: []Match{}}
if err := searchArchive(ctx, gz, "", regexp.MustCompile("forge"), 100, out); err != nil {
t.Fatal(err)
}
if !out.Stopped || out.Files != 0 {
t.Fatalf("a search past its deadline went on: %+v", out)
}
// Within its deadline, the same archive is searched whole.
out = &Found{Matches: []Match{}}
if err := searchArchive(context.Background(), gz, "", regexp.MustCompile("forge"), 100, out); err != nil || out.Stopped || len(out.Matches) != 2 {
t.Fatalf("%v %+v", err, out)
}
}
+238
View File
@@ -0,0 +1,238 @@
// forge: the forge module's Go bundle for an agent's commit and the reads around it (novox/hq issue 375).
// A process the node's runtime launches and speaks MCP over stdio to, beside the module's TypeScript bundle
// and its npm-registry bundle. gitea_commit puts one commit on a branch of its own — never the default branch,
// never one a protection rule names (novox/hq ADR 0266) — and the reads answer a commit history, a compare, a
// pull request's reviews with their line comments, a directory's entries and a code search, summarised.
// stdout is the protocol; it says what it says on stderr, and never the forge's credential.
package main
import (
"context"
"fmt"
"math"
"os"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// ToolNames are the tools this bundle serves. The module's manifest lists no `tools` (see the npm-registry
// bundle for why); the README names these, and a test holds the three together.
var ToolNames = []string{"gitea_commit", "gitea_commits", "gitea_compare", "gitea_pull_reviews", "gitea_list_dir", "gitea_search_code"}
func main() {
if err := stdio.Serve("", Tools(ForgeFromEnv)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// Tools are the bundle's tools over the forge the function gives; asked per call, so a manifest that does not
// set the environment is a refusal naming what is missing, not a bundle that will not start.
func Tools(forge func() (*Forge, error)) []stdio.Tool {
str := func(d string) map[string]any { return map[string]any{"type": "string", "description": d} }
owner := str("the repository's owner, e.g. novox")
repo := str("the repository's name, e.g. hq")
run := func(do func(ctx context.Context, f *Forge, args map[string]any) (any, error)) func(map[string]any) (any, error) {
return func(args map[string]any) (any, error) {
f, err := forge()
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), deadline)
defer cancel()
return do(ctx, f, args)
}
}
return []stdio.Tool{
{
Name: "gitea_commit",
Description: "Put one commit on a branch of its own in a forge repository, creating the branch from base " +
"when it does not exist. Refused on the repository's default branch and on any branch a protection " +
"rule names (novox/hq ADR 0266): a change reaches those only by a pull request whose head an " +
"independent review approved and whose required checks passed (gitea_merge_pull_request). Carries " +
"either files — each {path, content (text) | content_base64 | delete: true}, created or updated as " +
"the branch holds them — or patch, a unified diff as git diff writes it, which the forge applies. " +
"At most 200 files, 4 MiB each and 16 MiB together, or a 4 MiB patch. Answers the commit it made and " +
"the parent the forge made it on, read back from the branch, and says when the branch moved under " +
"it. Replaces git commit and git " +
"push of a feature branch. (a)",
Input: map[string]any{
"type": "object",
"properties": map[string]any{
"owner": owner,
"repo": repo,
"branch": str("the branch the commit goes on, e.g. fix/375-a-gitea-commit-tool; never the default branch or a protected one"),
"base": str("for a branch that does not exist yet: the branch it starts from (default the repository's default branch)"),
"message": str("the commit's message: concise, in the imperative, about why"),
"files": map[string]any{
"type": "array",
"description": "the files the commit changes; give this or patch",
"items": map[string]any{
"type": "object",
"properties": map[string]any{
"path": str("the file's path from the repository's root"),
"content": str("its whole new content, as UTF-8 text"),
"content_base64": str("its whole new content, base64-encoded, for a file that is not text"),
"delete": map[string]any{"type": "boolean", "description": "true to delete the file"},
},
"required": []string{"path"},
},
},
"patch": str("a unified diff of the change, as git diff or git format-patch writes it; give this or files"),
},
"required": []string{"owner", "repo", "branch", "message"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
a, err := readCommitAsk(args)
if err != nil {
return nil, err
}
return commit(ctx, f, a)
}),
},
{
Name: "gitea_commits",
Description: "A repository's commit history at a branch, tag or commit (default the default branch), " +
"newest first, optionally only the commits touching a path: each commit's sha, author, date and " +
"subject line. Replaces git log. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"ref": str("a branch, tag or commit (default the default branch)"),
"path": str("only commits touching this file or directory"),
"limit": map[string]any{"type": "integer", "description": "commits per page, 1 to 100 (default 20)"},
"page": map[string]any{"type": "integer", "description": "the page, from 1 (default 1)"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
limit, err := count(args, "limit", 20, 1, 100)
if err != nil {
return nil, err
}
page, err := count(args, "page", 1, 1, 10000)
if err != nil {
return nil, err
}
return history(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "ref"), strings.Trim(text(args, "path"), "/"), limit, page)
}),
},
{
Name: "gitea_compare",
Description: "What head holds that base does not, in a forge repository: its commits (sha, author, date, " +
"subject) and the files they touch with how. Replaces git log base..head and git diff --stat. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"base": str("the branch, tag or commit compared from, e.g. main"),
"head": str("the branch, tag or commit compared, e.g. fix/375-a-gitea-commit-tool"),
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
return compare(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "base"), text(args, "head"))
}),
},
{
Name: "gitea_pull_reviews",
Description: "A pull request's reviews, oldest first: who, its state (APPROVED, REQUEST_CHANGES, COMMENT, " +
"PENDING), the commit it reviewed and whether that is the pull request's head now, stale or dismissed, " +
"its body, and every line comment it left (file, line, body, who resolved it). (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"number": map[string]any{"type": "integer", "description": "the pull request's number"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
n, err := count(args, "number", 0, 1, math.MaxInt32)
if err != nil {
return nil, err
}
if n == 0 {
return nil, refuse("number is required: the pull request's number")
}
return reviews(ctx, f, text(args, "owner"), text(args, "repo"), n)
}),
},
{
Name: "gitea_list_dir",
Description: "A directory's entries in a forge repository at a branch, tag or commit (default the default " +
"branch): each entry's name, its type (file, dir, symlink, submodule) and a file's size. Replaces ls " +
"and git ls-tree. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"path": str("the directory from the repository's root (default the root)"),
"ref": str("a branch, tag or commit (default the default branch)"),
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
return listDir(ctx, f, text(args, "owner"), text(args, "repo"), text(args, "path"), text(args, "ref"))
}),
},
{
Name: "gitea_search_code",
Description: "Search a forge repository's text files at a branch, tag or commit (default the default " +
"branch) for a literal string, or a regular expression with regex: true, optionally under a path: " +
"each matching line's file, number and text. The forge has no code search of its own, so the " +
"repository's archive is read once and searched; files over 4 MiB are named, not searched. Replaces " +
"git grep. (r)",
Input: map[string]any{
"owner": owner, "repo": repo,
"query": str("what to find: a literal string, or a Go regular expression with regex: true"),
"regex": map[string]any{"type": "boolean", "description": "true to read query as a regular expression"},
"ignore_case": map[string]any{"type": "boolean", "description": "true to match regardless of case"},
"path": str("only files under this directory, or this one file"),
"ref": str("a branch, tag or commit (default the default branch)"),
"max": map[string]any{"type": "integer", "description": "the most matches answered, 1 to 1000 (default 100)"},
},
Run: run(func(ctx context.Context, f *Forge, args map[string]any) (any, error) {
max, err := count(args, "max", 100, 1, 1000)
if err != nil {
return nil, err
}
regex, err := flag(args, "regex")
if err != nil {
return nil, err
}
ignore, err := flag(args, "ignore_case")
if err != nil {
return nil, err
}
return searchCode(ctx, f, SearchAsk{Owner: text(args, "owner"), Repo: text(args, "repo"), Ref: text(args, "ref"),
Query: raw(args, "query"), Path: text(args, "path"), Regex: regex, IgnoreCase: ignore, Max: max})
}),
},
}
}
// text is an argument trimmed; raw is it as given.
func text(args map[string]any, key string) string { return strings.TrimSpace(raw(args, key)) }
func raw(args map[string]any, key string) string {
s, _ := args[key].(string)
return s
}
func count(args map[string]any, key string, fallback, least, most int) (int, error) {
v, given := args[key]
if !given || v == nil {
return fallback, nil
}
x, ok := v.(float64)
if !ok || x != math.Trunc(x) {
return 0, refuse("%s must be a whole number", key)
}
if x < float64(least) || x > float64(most) {
return 0, refuse("%s must be from %d to %d", key, least, most)
}
return int(x), nil
}
func flag(args map[string]any, key string) (bool, error) {
v, given := args[key]
if !given || v == nil {
return false, nil
}
switch b := v.(type) {
case bool:
return b, nil
case string:
if b == "true" || b == "false" {
return b == "true", nil
}
}
return false, refuse("%s is true or false, not %v", key, v)
}
+334
View File
@@ -0,0 +1,334 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"strings"
"time"
)
// The forge's reads an agent otherwise made through gitea_api, answered summarised rather than as the forge's
// JSON (novox/hq issue 375): a commit's history, a compare, a pull request's reviews with their line comments,
// a directory's entries.
// CommitLine is one commit, summarised.
type CommitLine struct {
SHA string `json:"sha"`
Author string `json:"author"`
Date string `json:"date"`
Subject string `json:"subject"`
Files int `json:"files,omitempty"`
}
type forgeCommit struct {
SHA string `json:"sha"`
Commit struct {
Message string `json:"message"`
Author struct {
Name string `json:"name"`
Email string `json:"email"`
Date string `json:"date"`
} `json:"author"`
} `json:"commit"`
Files []struct {
Filename string `json:"filename"`
Status string `json:"status"`
} `json:"files"`
}
func (c forgeCommit) line() CommitLine {
subject, _, _ := strings.Cut(strings.TrimSpace(c.Commit.Message), "\n")
return CommitLine{SHA: c.SHA, Author: c.Commit.Author.Name, Date: c.Commit.Author.Date, Subject: subject, Files: len(c.Files)}
}
// History is a ref's commits, newest first.
type History struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
Path string `json:"path,omitempty"`
Page int `json:"page"`
Commits []CommitLine `json:"commits"`
More bool `json:"more"`
}
func history(ctx context.Context, f *Forge, owner, name, ref, path string, limit, page int) (*History, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
q := url.Values{}
if ref != "" {
q.Set("sha", ref)
}
if path != "" {
q.Set("path", path)
}
q.Set("limit", fmt.Sprint(limit))
q.Set("page", fmt.Sprint(page))
q.Set("stat", "false")
q.Set("verification", "false")
q.Set("files", "false")
var got []forgeCommit
if err := f.GetJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
return nil, err
}
h := &History{Repository: owner + "/" + name, Ref: ref, Path: path, Page: page, Commits: []CommitLine{}, More: len(got) >= limit}
for _, c := range got {
l := c.line()
l.Files = 0
h.Commits = append(h.Commits, l)
}
return h, nil
}
// Comparison is what head holds that base does not.
type Comparison struct {
Repository string `json:"repository"`
Base string `json:"base"`
Head string `json:"head"`
Total int `json:"total_commits"`
Commits []CommitLine `json:"commits"`
Files []FileLine `json:"files"`
Said []string `json:"said"`
}
// FileLine is one file a compare touched, and how the last commit that touched it changed it.
type FileLine struct {
Path string `json:"path"`
Status string `json:"status"`
}
func compare(ctx context.Context, f *Forge, owner, name, base, head string) (*Comparison, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
b, err := segments("base", base)
if err != nil {
return nil, err
}
h, err := segments("head", head)
if err != nil {
return nil, err
}
if strings.Contains(base, "...") || strings.Contains(head, "...") {
return nil, refuse("a base or head holding ... cannot be compared: the forge splits on it. Nothing was sent")
}
var got struct {
Total int `json:"total_commits"`
Commits []forgeCommit `json:"commits"`
}
if err := f.GetJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
return nil, err
}
c := &Comparison{Repository: owner + "/" + name, Base: base, Head: head, Total: got.Total, Commits: []CommitLine{}, Files: []FileLine{}}
at := map[string]int{}
for _, fc := range got.Commits {
c.Commits = append(c.Commits, fc.line())
for _, file := range fc.Files {
if i, seen := at[file.Filename]; seen {
c.Files[i].Status = file.Status
continue
}
at[file.Filename] = len(c.Files)
c.Files = append(c.Files, FileLine{Path: file.Filename, Status: file.Status})
}
}
c.Said = append(c.Said, fmt.Sprintf("%s holds %d commits %s does not, touching %d files", head, got.Total, base, len(c.Files)))
if got.Total > len(got.Commits) {
c.Said = append(c.Said, fmt.Sprintf("the forge listed %d of the %d commits; the files are those of the commits listed", len(got.Commits), got.Total))
}
c.Said = append(c.Said, "a file's status is the last listed commit's change to it; for the lines, gitea_pull_request_diff or gitea_get_file")
return c, nil
}
// Review is one review of a pull request with its line comments.
type Review struct {
ID int64 `json:"id"`
By string `json:"by"`
State string `json:"state"`
Commit string `json:"commit"`
AtHead bool `json:"at_head"`
Submitted string `json:"submitted,omitempty"`
Stale bool `json:"stale,omitempty"`
Dismissed bool `json:"dismissed,omitempty"`
Official bool `json:"official,omitempty"`
Body string `json:"body,omitempty"`
Comments []LineComment `json:"comments"`
}
// LineComment is one comment a review left on a line.
type LineComment struct {
Path string `json:"path"`
Line int `json:"line,omitempty"`
OldLine int `json:"old_line,omitempty"`
By string `json:"by"`
Body string `json:"body"`
Commit string `json:"commit,omitempty"`
Resolved string `json:"resolved_by,omitempty"`
}
// Reviews is a pull request's reviews, oldest first.
type Reviews struct {
Repository string `json:"repository"`
Number int `json:"number"`
Head string `json:"head"`
Reviews []Review `json:"reviews"`
Said []string `json:"said"`
}
type forgeUser struct {
Login string `json:"login"`
}
func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Reviews, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
pr := fmt.Sprintf("%s/pulls/%d", repo, number)
var pull struct {
Head struct {
SHA string `json:"sha"`
} `json:"head"`
}
if err := f.GetJSON(ctx, pr, &pull); err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no pull request %d", owner, name, number)
}
return nil, err
}
out := &Reviews{Repository: owner + "/" + name, Number: number, Head: pull.Head.SHA, Reviews: []Review{}}
type forgeReview struct {
ID int64 `json:"id"`
User forgeUser `json:"user"`
State string `json:"state"`
Body string `json:"body"`
CommitID string `json:"commit_id"`
Submitted string `json:"submitted_at"`
Stale bool `json:"stale"`
Dismissed bool `json:"dismissed"`
Official bool `json:"official"`
Comments int `json:"comments_count"`
}
const page = 50
var all []forgeReview
for p := 1; ; p++ {
var batch []forgeReview
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
return nil, err
}
all = append(all, batch...)
if len(batch) < page || p >= 20 {
break
}
}
approvedAtHead := 0
for _, r := range all {
rv := Review{ID: r.ID, By: r.User.Login, State: r.State, Commit: r.CommitID, AtHead: r.CommitID != "" && r.CommitID == pull.Head.SHA,
Submitted: r.Submitted, Stale: r.Stale, Dismissed: r.Dismissed, Official: r.Official, Body: r.Body, Comments: []LineComment{}}
if r.Comments > 0 {
var cs []struct {
Path string `json:"path"`
Position int `json:"position"`
Original int `json:"original_position"`
User forgeUser `json:"user"`
Body string `json:"body"`
CommitID string `json:"commit_id"`
Resolver *forgeUser `json:"resolver"`
}
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
return nil, fmt.Errorf("review %d's line comments could not be read: %w", r.ID, err)
}
for _, c := range cs {
lc := LineComment{Path: c.Path, Line: c.Position, OldLine: c.Original, By: c.User.Login, Body: c.Body, Commit: c.CommitID}
if c.Resolver != nil {
lc.Resolved = c.Resolver.Login
}
rv.Comments = append(rv.Comments, lc)
}
}
if rv.AtHead && r.State == "APPROVED" && !r.Dismissed && !r.Stale {
approvedAtHead++
}
out.Reviews = append(out.Reviews, rv)
}
out.Said = append(out.Said, fmt.Sprintf("%d reviews; %d approve the head %s", len(out.Reviews), approvedAtHead, short(pull.Head.SHA)))
return out, nil
}
// Entry is one entry of a directory.
type Entry struct {
Name string `json:"name"`
Type string `json:"type"`
Size int64 `json:"size,omitempty"`
}
// Listing is a directory's entries at a ref.
type Listing struct {
Repository string `json:"repository"`
Path string `json:"path"`
Ref string `json:"ref,omitempty"`
Entries []Entry `json:"entries"`
}
func listDir(ctx context.Context, f *Forge, owner, name, path, ref string) (*Listing, error) {
repo, err := repoPath(owner, name)
if err != nil {
return nil, err
}
path = strings.Trim(path, "/")
at := repo + "/contents"
if path != "" {
seg, err := segments("directory path", path)
if err != nil {
return nil, err
}
at += "/" + seg
}
if ref != "" {
at += "?ref=" + queryEscape(ref)
}
body, err := f.Do(ctx, "GET", at, nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no %q at %s", owner, name, path, orDefault(ref))
}
return nil, err
}
var got []struct {
Name string `json:"name"`
Type string `json:"type"`
Size int64 `json:"size"`
}
if err := jsonInto(body, &got); err != nil {
return nil, refuse("%s is a file, not a directory: gitea_get_file reads it", path)
}
l := &Listing{Repository: owner + "/" + name, Path: "/" + path, Ref: ref, Entries: []Entry{}}
for _, e := range got {
en := Entry{Name: e.Name, Type: e.Type}
if e.Type == "file" {
en.Size = e.Size
}
l.Entries = append(l.Entries, en)
}
return l, nil
}
func orDefault(ref string) string {
if ref == "" {
return "the default branch"
}
return ref
}
func jsonInto(body []byte, into any) error { return json.Unmarshal(body, into) }
// queryEscape is a query value as the forge reads it.
func queryEscape(v string) string { return url.QueryEscape(v) }
// deadline is how long one tool call may take: inside a module's 60-second ask.
const deadline = 50 * time.Second
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"errors"
"regexp"
"strings"
)
// ruleCovers is whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
// models/git/protected_branch.go Match and modules/glob, novox/hq issue 539): a name without a glob character
// covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any
// run within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]`
// one character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge
// cannot compile covers its own text exactly. The TypeScript bundle's ruleCovers (guard.ts) is the same rule.
func ruleCovers(rule, branch string) bool {
if !strings.ContainsAny(rule, `*?\[]{}`) {
return strings.EqualFold(rule, branch)
}
re, err := globRegexp(rule)
if err != nil {
return rule == branch
}
return re.MatchString(branch)
}
// globRegexp is the forge's glob as a regular expression, compiled as modules/glob compiles it with `/` as
// the separator.
func globRegexp(glob string) (*regexp.Regexp, error) {
g := []rune(glob)
var b strings.Builder
pos := 0
var compile func(sub bool) error
chars := func() error {
b.WriteByte('[')
if pos < len(g) && g[pos] == '!' {
pos++
b.WriteByte('^')
}
for pos < len(g) {
c := g[pos]
pos++
switch {
case c == ']':
b.WriteByte(']')
return nil
case c == '\\':
if pos >= len(g) {
return errors.New("unterminated character class escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
default:
b.WriteRune(c)
}
}
return errors.New("unterminated character class")
}
compile = func(sub bool) error {
for pos < len(g) {
c := g[pos]
pos++
if sub && c == '}' {
b.WriteByte(')')
return nil
}
switch c {
case '*':
if pos < len(g) && g[pos] == '*' {
pos++
b.WriteString(".*")
} else {
b.WriteString("[^/]*")
}
case '?':
b.WriteString("[^/]")
case '+':
b.WriteString(`\+`)
case '[':
if err := chars(); err != nil {
return err
}
case '{':
b.WriteByte('(')
if err := compile(true); err != nil {
return err
}
case ',':
if sub {
b.WriteByte('|')
} else {
b.WriteByte(',')
}
case '\\':
if pos >= len(g) {
return errors.New("no character to escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
case '.', '^', '$', '(', ')', '|':
b.WriteByte('\\')
b.WriteRune(c)
default:
b.WriteRune(c)
}
}
return nil
}
b.WriteByte('^')
if err := compile(false); err != nil {
return nil, err
}
b.WriteByte('$')
return regexp.Compile(b.String())
}
+206
View File
@@ -0,0 +1,206 @@
package main
import (
"archive/tar"
"bufio"
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"regexp"
"strings"
)
// gitea_search_code: the forge's API has no code search, so the repository's archive at a ref is read once
// and searched here (novox/hq issue 375). One request, nothing kept.
// Match is one line that matched.
type Match struct {
Path string `json:"path"`
Line int `json:"line"`
Text string `json:"text"`
}
// Found is a search's answer.
type Found struct {
Repository string `json:"repository"`
Ref string `json:"ref"`
Query string `json:"query"`
Matches []Match `json:"matches"`
Files int `json:"files_searched"`
Skipped []string `json:"skipped,omitempty"`
Truncated bool `json:"truncated"`
Stopped bool `json:"stopped_at_deadline,omitempty"`
Said []string `json:"said"`
}
// SearchAsk is one gitea_search_code call.
type SearchAsk struct {
Owner, Repo, Ref, Query, Path string
Regex, IgnoreCase bool
Max int
}
const (
maxFileSearched = 4 << 20 // a larger file is skipped and named
maxLineShown = 300
)
func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
repo, err := repoPath(a.Owner, a.Repo)
if err != nil {
return nil, err
}
if a.Query == "" {
return nil, refuse("query is required. Nothing was sent")
}
pattern := a.Query
if !a.Regex {
pattern = regexp.QuoteMeta(pattern)
}
if a.IgnoreCase {
pattern = "(?i)" + pattern
}
re, err := regexp.Compile(pattern)
if err != nil {
return nil, refuse("query is not a regular expression Go reads: %v. Nothing was sent", err)
}
ref := a.Ref
if ref == "" {
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.Empty || r.DefaultBranch == "" {
return &Found{Repository: a.Owner + "/" + a.Repo, Query: a.Query, Matches: []Match{}, Said: []string{"the repository is empty"}}, nil
}
ref = r.DefaultBranch
}
seg, err := segments("ref", ref)
if err != nil {
return nil, err
}
prefix := strings.Trim(a.Path, "/")
if prefix != "" {
if _, err := segments("path", prefix); err != nil {
return nil, err
}
}
body, err := f.Do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no ref %s", a.Owner, a.Repo, ref)
}
return nil, err
}
out := &Found{Repository: a.Owner + "/" + a.Repo, Ref: ref, Query: a.Query, Matches: []Match{}}
if err := searchArchive(ctx, body, prefix, re, a.Max, out); err != nil {
return nil, fmt.Errorf("the archive of %s at %s could not be read: %w", out.Repository, ref, err)
}
said := fmt.Sprintf("%d matches in %d files searched", len(out.Matches), out.Files)
switch {
case out.Stopped:
said = fmt.Sprintf("the search stopped at its deadline of %s after %d files, with %d matches: the rest of the "+
"archive was not searched. Narrow with path", deadline, out.Files, len(out.Matches))
case out.Truncated:
said = fmt.Sprintf("the first %d matches, and more were found: narrow with path or the query", len(out.Matches))
}
out.Said = append(out.Said, said)
if len(out.Skipped) > 0 {
out.Said = append(out.Said, fmt.Sprintf("%d files larger than %d MiB were not searched: they are under skipped", len(out.Skipped), maxFileSearched>>20))
}
return out, nil
}
// searchArchive reads a gzipped tar the forge made — every path under one top directory — and collects the
// lines that match, from text files under the prefix. It stops at the call's deadline (novox/hq issue 539):
// what it found by then is kept, and out.Stopped says the rest was not searched.
func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
zr, err := gzip.NewReader(bytes.NewReader(gz))
if err != nil {
return err
}
tr := tar.NewReader(ctxReader{ctx, zr})
stopped := func(err error) bool {
if ctx.Err() != nil && (err == nil || errors.Is(err, ctx.Err())) {
out.Stopped = true
return true
}
return false
}
for {
if stopped(nil) {
return nil
}
h, err := tr.Next()
if err == io.EOF {
return nil
}
if stopped(err) {
return nil
}
if err != nil {
return err
}
if h.Typeflag != tar.TypeReg {
continue
}
_, path, found := strings.Cut(h.Name, "/")
if !found || path == "" {
continue
}
if prefix != "" && path != prefix && !strings.HasPrefix(path, prefix+"/") {
continue
}
if h.Size > maxFileSearched {
out.Skipped = append(out.Skipped, path)
continue
}
content, err := io.ReadAll(io.LimitReader(tr, maxFileSearched+1))
if stopped(err) {
return nil
}
if err != nil {
return err
}
head := content
if len(head) > 8000 {
head = head[:8000]
}
if bytes.IndexByte(head, 0) >= 0 {
continue // binary
}
out.Files++
sc := bufio.NewScanner(bytes.NewReader(content))
sc.Buffer(make([]byte, 64<<10), maxFileSearched+1)
for n := 1; sc.Scan(); n++ {
line := sc.Text()
if !re.MatchString(line) {
continue
}
if len(out.Matches) >= max {
out.Truncated = true
return nil
}
if len(line) > maxLineShown {
line = strings.ToValidUTF8(line[:maxLineShown], "") + "…"
}
out.Matches = append(out.Matches, Match{Path: path, Line: n, Text: line})
}
}
}
// ctxReader is a reader that ends when its context does: decompressing is the long part of a search.
type ctxReader struct {
ctx context.Context
r io.Reader
}
func (c ctxReader) Read(p []byte) (int, error) {
if err := c.ctx.Err(); err != nil {
return 0, err
}
return c.r.Read(p)
}
+20 -104
View File
@@ -2,45 +2,33 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"time"
"gitea/internal/forgehttp"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin
// account, which the vault delivered into a file this module owns (novox/hq ADR 0086). Basic
// authentication, so this bundle mints nothing and keeps nothing: the TypeScript bundle's token is its
// own. The password is read when a call is made and never logged, answered or put in an error.
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the forge bundle beside it). Basic authentication, so this bundle mints
// nothing and keeps nothing: the TypeScript bundle's token is its own.
// Forge is the forge's HTTP interface.
type Forge struct {
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
Owner string // the package registry's owner, as the seat serves it
HTTP *http.Client
forgehttp.Client
Owner string // the package registry's owner, as the seat serves it
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
f := &Forge{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
Owner: os.Getenv("MESH_NPM_OWNER"),
}
var missing []string
for name, v := range map[string]string{"MESH_GITEA_URL": f.URL, "MESH_GITEA_ADMIN_USER": f.User,
"MESH_GITEA_ADMIN_PASSWORD_FILE": f.PasswordFile, "MESH_NPM_OWNER": f.Owner} {
if v == "" {
missing = append(missing, name)
}
c, missing := forgehttp.FromEnv()
f := &Forge{Client: c, Owner: os.Getenv("MESH_NPM_OWNER")}
if f.Owner == "" {
missing = append(missing, "MESH_NPM_OWNER")
}
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
@@ -49,86 +37,14 @@ func ForgeFromEnv() (*Forge, error) {
}
// errNotFound is the forge answering 404.
var errNotFound = errors.New("not found")
// statusError is an answer the forge gave that is not a success, said without the request's credential.
type statusError struct {
method, path string
status int
said string
}
func (e *statusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
}
func (f *Forge) client() *http.Client {
if f.HTTP != nil {
return f.HTTP
}
return &http.Client{Timeout: 20 * time.Second}
}
func (f *Forge) password() (string, error) {
raw, err := os.ReadFile(f.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
var errNotFound = forgehttp.ErrNotFound
// maxBody is the largest answer read from the forge.
const maxBody = 32 << 20
const maxBody = forgehttp.MaxBody
// do sends one request and answers the body; 404 is errNotFound.
// do sends one request without a body and answers the body; 404 is errNotFound.
func (f *Forge) do(ctx context.Context, method, path string) ([]byte, error) {
password, err := f.password()
if err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, nil)
if err != nil {
return nil, err
}
req.SetBasicAuth(f.User, password)
req.Header.Set("Accept", "application/json")
res, err := f.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
// One byte past the cap is read, so a body the cap cut is told from one that fits it exactly: a
// cut body is refused, never parsed as if it were whole.
body, err := io.ReadAll(io.LimitReader(res.Body, maxBody+1))
if err != nil {
return nil, err
}
if len(body) > maxBody {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, maxBody>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(body))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &statusError{method, path, res.StatusCode, said}
}
return body, nil
}
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
body, err := f.do(ctx, http.MethodGet, path)
if err != nil {
return err
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s is not readable: %w", path, err)
}
return nil
return f.Do(ctx, method, path, nil, maxBody)
}
// PackageVersion is one version of one package, as the forge lists it.
@@ -146,7 +62,7 @@ func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
for page := 1; page <= 1000; page++ {
var batch []PackageVersion
path := fmt.Sprintf("/api/v1/packages/%s?type=npm&page=%d&limit=%d", url.PathEscape(f.Owner), page, pageSize)
if err := f.getJSON(ctx, path, &batch); err != nil {
if err := f.GetJSON(ctx, path, &batch); err != nil {
return nil, err
}
all = append(all, batch...)
@@ -166,7 +82,7 @@ func (f *Forge) Size(ctx context.Context, name, version string) (int64, error) {
var files []struct {
Size int64 `json:"size"`
}
if err := f.getJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
if err := f.GetJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
return 0, err
}
var total int64
@@ -182,7 +98,7 @@ func (f *Forge) DistTags(ctx context.Context, name string) (map[string]string, e
DistTags map[string]string `json:"dist-tags"`
}
path := fmt.Sprintf("/api/packages/%s/npm/%s", url.PathEscape(f.Owner), url.PathEscape(name))
if err := f.getJSON(ctx, path, &meta); err != nil {
if err := f.GetJSON(ctx, path, &meta); err != nil {
return nil, err
}
return meta.DistTags, nil
@@ -208,7 +124,7 @@ func (f *Forge) Repositories(ctx context.Context) ([]Repository, error) {
var found struct {
Data []Repository `json:"data"`
}
if err := f.getJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
if err := f.GetJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
return nil, err
}
all = append(all, found.Data...)
@@ -239,7 +155,7 @@ func (f *Forge) Files(ctx context.Context, repo Repository) ([]string, error) {
}
path := fmt.Sprintf("/api/v1/repos/%s/git/trees/%s?recursive=true&page=%d&per_page=1000",
repoPath(repo.FullName), url.PathEscape(repo.DefaultBranch), page)
if err := f.getJSON(ctx, path, &tree); err != nil {
if err := f.GetJSON(ctx, path, &tree); err != nil {
return nil, err
}
for _, e := range tree.Tree {
@@ -15,6 +15,8 @@ import (
"sync"
"testing"
"time"
"gitea/internal/forgehttp"
)
// fakeForge answers the forge's interface from maps, and records every delete.
@@ -38,7 +40,7 @@ func (f *fakeForge) serve() *Forge {
if err := os.WriteFile(file, []byte(f.password+"\n"), 0o600); err != nil {
f.t.Fatal(err)
}
return &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
}
var (
@@ -501,7 +503,7 @@ func TestAnAnswerTheCapCutIsRefused(t *testing.T) {
if err := os.WriteFile(file, []byte("x\n"), 0o600); err != nil {
t.Fatal(err)
}
f := &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
f := &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
if _, err := f.do(context.Background(), http.MethodGet, "/big"); err == nil || !strings.Contains(err.Error(), "not read whole") {
t.Fatalf("a cut answer: %v", err)
}
+47
View File
@@ -13,6 +13,9 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus } from "./client.js";
import { checkName } from "./guard.js";
/** How a command is run: docker on the forge's machine, or a test's. */
export type Runner = (file: string, args: string[]) => Promise<{ stdout: string; code: number }>;
@@ -106,3 +109,47 @@ export function deliveryStatus(context: string, state: string, description: stri
return { state: state as "pending" | "success" | "error" | "failure" | "warning", context, description: d,
...(target && /^https?:\/\//.test(target) ? { target_url: target } : {}) };
}
/** What the status tool writes with: the forge, and the gate writer the merge check's statuses go through. */
export interface StatusWriters {
setCommitStatus(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void>;
gates: { rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> };
}
/**
* The `gitea_commit_status` tool. **The merge check's pending goes through the module's one writer** (novox/hq issue
* 456): a recheck's pending on mesh/merge-gate or mesh/repo-check takes its turn on the head's chain (gate.ts), so an
* old run's verdict in flight lands before it and a redelivered one is dropped. Served from the watcher's bundle
* (index.ts), the process that holds that writer: each bundle the runtime serves is its own process. The mesh's other
* statuses (mesh/delivery, mesh/delivery-group) are written as they were.
*/
export function commitStatusTool(writers: StatusWriters): ToolDefinition {
return {
name: "gitea_commit_status",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. The merge check's statuses (mesh/merge-gate, mesh/repo-check) take pending only — a recheck putting them back until the fresh verdict sets them — on the head's whole sha, in their turn after the check's earlier writes.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit" },
context: { type: "string", description: "the status's name, mesh/…" },
state: { type: "string", description: "pending | success | error | failure | warning" },
description: { type: "string", description: "one line" },
target_url: { type: "string", description: "the page it links to (the pull request)" },
},
run: async (args) => {
const owner = checkName("owner", args.owner), repo = checkName("name", args.repo), sha = String(args.sha ?? "");
const status = deliveryStatus(String(args.context), String(args.state), String(args.description ?? ""),
args.target_url ? String(args.target_url) : undefined);
if (mergeCheckContexts.has(status.context)) {
// The writer keeps a head by its whole sha, as the forge announced it: a short one would be another head.
if (!/^([0-9a-f]{40}|[0-9a-f]{64})$/i.test(sha)) {
throw new Error(`${JSON.stringify(args.sha)} is not a head's whole sha: the merge check's status names the head whole. Nothing was sent`);
}
await writers.gates.rechecked(owner, repo, sha, status);
} else {
await writers.setCommitStatus(owner, repo, sha, status);
}
return { set: status };
},
};
}
+115 -5
View File
@@ -8,6 +8,14 @@
// between its read and its write. So this module is the one writer of mesh/merge-gate, and every write of one head's
// gate goes through one chain: a verdict is marked the moment it is heard, before it waits its turn, and a "building"
// still waiting behind or taking its turn after that mark is dropped, never written.
//
// **A recheck's pending is written here too** (novox/hq issue 456). mesh-delivery's recheck puts the gate back to
// pending through the gitea_commit_status tool, which this module serves from the watcher's own bundle so that it
// reaches this writer: the pending clears the verdict mark, so the new run's "building" is said, and queues behind an
// old verdict in flight, so that verdict lands first. A verdict is weighed by the ask it answers — the controller's
// checked.id, `build-<unix nanoseconds>` of when the check was asked — against when the check now pending was opened:
// an older run's verdict, redelivered after its write failed, is dropped, never written over the pending. After a
// restart, when the check was opened is read from the forge's own history of the gate.
import { CHECK_CONTEXT, pendingWords } from "./pulls.js";
import type { CommitStatus } from "./client.js";
@@ -27,15 +35,53 @@ export interface GateForge {
setCommitStatus(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void>;
/** The head's mesh/merge-gate, newest: its state and when it was set; undefined when it has none. */
gateOf(owner: string, repo: string, sha: string): Promise<{ state: string; updated_at?: string } | undefined>;
/** The head's mesh/merge-gate statuses, every one the forge keeps, newest first (novox/hq issue 456). */
gateHistory(owner: string, repo: string, sha: string): Promise<GateStatus[]>;
}
/** One status of a head's gate as the forge keeps it: its state, its words, and when it was set. */
export interface GateStatus {
state: string;
description?: string;
updated_at?: string;
}
/**
* When a check was asked, from its id as mesh-controller's link.NewBuildID writes it: `build-<unix nanoseconds>`.
* Undefined for an id of any other shape, whose ask time is not known (novox/hq issue 456).
*/
export function askedAt(id: string | undefined): number | undefined {
const m = /^build-(\d{1,20})$/.exec(String(id ?? ""));
if (!m) return undefined;
const ms = Number(BigInt(m[1]) / 1_000_000n);
return ms > 0 ? ms : undefined;
}
/**
* When the check now pending on a head was opened, from the gate's history, newest first: the newest status, past the
* "building" words this writer says only inside an open check, when it is pending; undefined when the gate holds a
* verdict, holds nothing, or its time cannot be read (novox/hq issue 456).
*/
export function openedAt(history: GateStatus[]): number | undefined {
for (const s of history) {
if (s.state !== "pending") return undefined;
if ((s.description ?? "").startsWith(BUILDING)) continue;
const t = Date.parse(s.updated_at ?? "");
return Number.isNaN(t) ? undefined : t;
}
return undefined;
}
/** How far a build machine's clock may stand behind this one's when a start is weighed against the pending words. */
const CLOCK_SLACK_MS = 60_000;
/** How the words of a check a machine is running begin. */
const BUILDING = "building on ";
/** The words of a check a machine is running, in the time the operator reads. */
export function buildingWords(on: string, at: Date): string {
const two = (n: number) => String(n).padStart(2, "0");
return `building on ${on} since ${two(at.getHours())}:${two(at.getMinutes())}`;
return `${BUILDING}${on} since ${two(at.getHours())}:${two(at.getMinutes())}`;
}
/** Whether a repository as the mesh clones it (`http://forge/novox/hq.git`) is owner/repo. */
@@ -50,6 +96,8 @@ interface Head {
pendingAt?: number;
/** A verdict was heard for the head: nothing pending is written after it. */
verdict: boolean;
/** Which verdict set the mark: one dropped in its turn takes the mark back only while it is still its own. */
mark?: object;
/** The chain the head's writes take their turns on. */
chain: Promise<void>;
}
@@ -94,12 +142,38 @@ export class GateWriter {
pending(owner: string, repo: string, sha: string): Promise<void> {
const h = this.head(owner, repo, sha);
const at = this.now();
h.pendingAt = at.getTime();
h.verdict = false;
this.opened(h, at);
return this.said(h, `${owner}/${repo}@${sha.slice(0, 8)}: could not say a check is pending`, () =>
this.forge.setCommitStatus(owner, repo, sha, { state: "pending", context: CHECK_CONTEXT, description: pendingWords(at) }));
}
/** A new check of the head opened now: no verdict heard for it yet, and nothing older than it is its own. */
private opened(h: Head, at: Date): void {
h.pendingAt = at.getTime();
h.verdict = false;
h.mark = undefined;
}
/**
* A recheck's pending, asked through the gitea_commit_status tool (novox/hq issue 456): on mesh/merge-gate it opens
* a new check of the head — the verdict mark cleared, so the new run's "building" is said — and takes its turn
* after every write already asked for, an old verdict in flight included, so it lands over that verdict. Any
* other status of the merge check (mesh/repo-check, which the verdict also writes) takes its turn on the same
* chain. A write that fails fails the caller, and the head is as it was before it.
*/
rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> {
const h = this.head(owner, repo, sha);
const write = () => this.forge.setCommitStatus(owner, repo, sha, status);
if (status.context !== CHECK_CONTEXT || status.state !== "pending") return this.turn(h, write);
const before = { pendingAt: h.pendingAt, verdict: h.verdict, mark: h.mark };
const at = this.now();
this.opened(h, at);
return this.turn(h, write).catch((err) => {
if (h.pendingAt === at.getTime() && !h.verdict) Object.assign(h, before);
throw err;
});
}
/**
* A build machine took an ask of this head: said while no verdict was heard, and only for the check now pending —
* a start from before the pending words were set (a run before a recheck, the stream's history) is not this check's.
@@ -130,10 +204,46 @@ export class GateWriter {
/**
* The verdict: marked at once, so no pending word is written after it, then written in its turn — last. A write
* that fails fails the caller, so the event is offered again.
*
* `id` is the ask it answers (checked.id): a verdict of a check asked before the check now pending was opened is
* an older run's — redelivered after its write failed, or heard late — and is dropped, never written over the
* pending (novox/hq issue 456). When this process opened no check of the head (it restarted since), the opening
* is read from the forge's history of the gate in the verdict's turn; a history that cannot be read writes the
* verdict, as one with no readable id is: a verdict is never lost to what cannot be weighed.
*/
verdict(owner: string, repo: string, sha: string, write: () => Promise<void>): Promise<void> {
verdict(owner: string, repo: string, sha: string, write: () => Promise<void>, id?: string): Promise<void> {
const h = this.head(owner, repo, sha);
const asked = askedAt(id);
const head = `${owner}/${repo}@${sha.slice(0, 8)}`;
if (asked !== undefined && h.pendingAt !== undefined && older(asked, h.pendingAt)) {
console.log(`[gitea] ${head}: the verdict of ${id} is an older run's than the check now pending; not written`);
return Promise.resolve();
}
const before = h.verdict;
const mark = {};
h.verdict = true;
return this.turn(h, write);
h.mark = mark;
return this.turn(h, async () => {
if (asked !== undefined && h.pendingAt === undefined) {
let opened: number | undefined;
try {
opened = openedAt(await this.forge.gateHistory(owner, repo, sha));
} catch (err) {
console.error(`[gitea] ${head}: the gate's history could not be read, so the verdict of ${id} is written ` +
`unweighed — ${err instanceof Error ? err.message : err}`);
}
if (opened !== undefined && older(asked, opened)) {
if (h.mark === mark) h.verdict = before;
console.log(`[gitea] ${head}: the verdict of ${id} is an older run's than the check the forge holds pending; not written`);
return;
}
}
await write();
});
}
}
/** Whether a check asked at `asked` was asked before a check opened at `opened`, past what the clocks may differ by. */
function older(asked: number, opened: number): boolean {
return asked < opened - CLOCK_SLACK_MS;
}
+98 -13
View File
@@ -164,21 +164,106 @@ export function judgeApiCall(method: string, rawPath: string, body: unknown): Ap
};
}
/** Whether a protection rule's name — a branch, or a glob of them (`*`, `**`) — covers a branch. */
/**
* Whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
* models/git/protected_branch.go Match and modules/glob; novox/hq issue 539): a name without a glob character
* covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any run
* within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]` one
* character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge cannot
* compile covers its own text exactly. The forge bundle's ruleCovers (cmd/forge/rules.go) is the same rule, and
* both are held to one table of answers (test/rule-cases.json).
*
* The forge hands the compiled glob to Go's regexp, which reads some things JavaScript's reads otherwise: a
* backslash before a letter or digit (`\d`, `\w`, `\pL`, `\b`, `\A` are classes or anchors there), a class
* that opens with `]` (`[]a]` holds `]` and `a` there, an empty class here) or with `^` (a negation there),
* and a POSIX class (`[[:digit:]]`).
* A rule using any of them is taken to cover every branch: refusing a write the forge would take is safe,
* taking one it guards is not.
*/
export function ruleCovers(rule: string, branch: string): boolean {
if (rule === branch) return true;
if (!/[*?[]/.test(rule)) return false;
let re = "";
for (let i = 0; i < rule.length; i++) {
const c = rule[i];
if (c === "*" && rule[i + 1] === "*") {
re += ".*";
i++;
} else if (c === "*") re += "[^/]*";
else if (c === "?") re += "[^/]";
else re += c.replace(/[.+^${}()|[\]\\]/g, "\\$&");
if (!/[*?\\[\]{}]/.test(rule)) return rule.toLowerCase() === branch.toLowerCase();
if (goReadsOtherwise(rule)) return true;
const re = globRegExp(rule);
return re ? re.test(branch) : rule === branch;
}
/** Whether a glob holds what Go's regexp reads otherwise than JavaScript's (see ruleCovers): fail closed. */
function goReadsOtherwise(rule: string): boolean {
return /\\[\p{L}\p{N}]/u.test(rule) || /\[!?[\]^]/.test(rule) || /\[[^\]]*\[:/.test(rule);
}
/** The forge's glob as a regular expression, compiled as its modules/glob compiles it with `/` as separator. */
function globRegExp(glob: string): RegExp | null {
const g = Array.from(glob);
let out = "";
let pos = 0;
// A character as itself: escaped when it is syntax to a regular expression, and only then — the `u` flag
// refuses any other escape, and `-` may be escaped only inside a class.
const lit = (c: string) => c.replace(/[.*+?^${}()|[\]\\/]/g, "\\$&");
const litInClass = (c: string) => c.replace(/[\]\\[^-]/g, "\\$&");
const chars = (): boolean => {
out += "[";
if (g[pos] === "!") {
pos++;
out += "^";
}
while (pos < g.length) {
const c = g[pos++];
if (c === "]") {
out += "]";
return true;
}
if (c === "\\") {
if (pos >= g.length) return false;
out += litInClass(g[pos++]);
} else out += c === "[" || c === "^" ? "\\" + c : c;
}
return false;
};
const compile = (sub: boolean): boolean => {
while (pos < g.length) {
const c = g[pos++];
if (sub && c === "}") {
out += ")";
return true;
}
switch (c) {
case "*":
if (g[pos] === "*") {
pos++;
out += ".*";
} else out += "[^/]*";
break;
case "?":
out += "[^/]";
break;
case "[":
if (!chars()) return false;
break;
case "{":
out += "(";
if (!compile(true)) return false;
break;
case ",":
out += sub ? "|" : ",";
break;
case "\\":
if (pos >= g.length) return false;
out += lit(g[pos++]);
break;
default:
out += lit(c);
}
}
// An alternative left open is a glob the forge cannot compile.
return !sub;
};
if (!compile(false)) return null;
try {
return new RegExp(`^${out}$`, "u");
} catch {
return null;
}
return new RegExp(`^${re}$`).test(branch);
}
/** Refuses a file written to the default branch or to a branch a protection rule names. */
+18 -1
View File
@@ -17,6 +17,9 @@
// Consumes node-build-agent.started — a build machine took an ask — and says, on the head a pending check is of,
// which machine builds it since when (novox/hq issue 446). This module is the one writer of mesh/merge-gate.
//
// Serves gitea_commit_status from this bundle, not the tools', so a recheck's pending reaches that one writer
// (novox/hq issue 456).
//
// Every pull request the forge holds is announced, whatever its repository: the controller holds the
// module graph and decides what is checked — a repository is never asked to opt in.
//
@@ -29,6 +32,8 @@
// the forge for an immediacy nobody asked for is not.
import { emit, on } from "@novox/mesh-sdk/events";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { commitStatusTool } from "./delivery.js";
import { GiteaClient, movedSince } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, commentFor, headsToAnnounce, statusesFor, type Announced, type Checked,
type RepoCheckFacts, reposToLook } from "./pulls.js";
@@ -344,8 +349,17 @@ async function repoCheckFacts(client: GiteaClient, c: Checked, base: string | un
const gates = new GateWriter({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gateOf: async (owner, repo, sha) => (await (gitea as GiteaClient).headChecks(owner, repo, sha))[CHECK_CONTEXT],
gateHistory: (owner, repo, sha) => (gitea as GiteaClient).contextHistory(owner, repo, sha, CHECK_CONTEXT),
});
// **The status tool is served here, beside the writer** (novox/hq issue 456): the runtime starts each bundle as its own
// process, so a recheck's pending written from the tools' bundle bypassed this writer's chain. Without a forge to
// reach, nothing is served, as the tools' bundle serves nothing then.
registerModuleTools("gitea", () => (gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
})] : []));
// **A build machine took an ask** (the build seat's `started`, novox/hq ADR 0157): said on the head it checks, when
// the head is an open pull request's this module announced. Any other build — a branch, a merge's — says nothing.
async function buildStarted(event: { body: unknown }): Promise<void> {
@@ -392,7 +406,10 @@ if (gitea) {
await on("mesh-controller.checked", (event) => {
const c = (event.body ?? {}) as Checked;
// A head's own verdict takes its turn after every write of that head's gate already asked for (gate.ts).
if (!c.group && c.owner && c.repo && c.commit) return gates.verdict(c.owner, c.repo, c.commit, () => setVerdict(client, event));
// A verdict of a check asked before the one now pending is an older run's, and dropped (novox/hq issue 456).
if (!c.group && c.owner && c.repo && c.commit) {
return gates.verdict(c.owner, c.repo, c.commit, () => setVerdict(client, event), c.id);
}
return setVerdict(client, event);
});
await on("node-build-agent.started", buildStarted);
@@ -0,0 +1,151 @@
// Package forgehttp is the forge's HTTP interface as this module's Go bundles reach it (cmd/forge,
// cmd/npm-registry): the forge on its own machine, as its admin account, whose password the vault delivered
// into a file this module owns (novox/hq ADR 0086), by basic authentication. The password is read when a call
// is made and never logged, answered or put in an error. One copy for both bundles (novox/hq issue 539).
package forgehttp
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
// MaxBody is the largest JSON answer read from the forge.
const MaxBody = 32 << 20
// Client is the forge's HTTP interface.
type Client struct {
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
HTTP *http.Client // when nil, a client with Timeout
Timeout time.Duration // one request's limit when HTTP is nil; 20 s when zero
}
// FromEnv is the forge as the manifest hands it to a bundle, and the names of the variables it left unset.
func FromEnv() (Client, []string) {
c := Client{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
}
var missing []string
for _, kv := range [][2]string{{"MESH_GITEA_URL", c.URL}, {"MESH_GITEA_ADMIN_USER", c.User},
{"MESH_GITEA_ADMIN_PASSWORD_FILE", c.PasswordFile}} {
if kv[1] == "" {
missing = append(missing, kv[0])
}
}
return c, missing
}
// ErrNotFound is the forge answering 404.
var ErrNotFound = errors.New("not found")
// StatusError is an answer the forge gave that is not a success, said without the request's credential.
type StatusError struct {
Method, Path string
Status int
Said string
}
func (e *StatusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.Status, e.Method, e.Path, e.Said)
}
func (c *Client) client() *http.Client {
if c.HTTP != nil {
return c.HTTP
}
timeout := c.Timeout
if timeout <= 0 {
timeout = 20 * time.Second
}
return &http.Client{Timeout: timeout}
}
func (c *Client) password() (string, error) {
raw, err := os.ReadFile(c.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
// Do sends one request, with payload as its JSON body when not nil, and answers the body, read up to limit;
// 404 is ErrNotFound. One byte past the limit is read, so a body the limit cut is told from one that fits it
// exactly: a cut body is refused, never read as if it were whole.
func (c *Client) Do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
password, err := c.password()
if err != nil {
return nil, err
}
var body io.Reader
if payload != nil {
raw, err := json.Marshal(payload)
if err != nil {
return nil, err
}
body = bytes.NewReader(raw)
}
req, err := http.NewRequestWithContext(ctx, method, c.URL+path, body)
if err != nil {
return nil, err
}
req.SetBasicAuth(c.User, password)
req.Header.Set("Accept", "application/json")
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
res, err := c.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
if err != nil {
return nil, err
}
if int64(len(got)) > limit {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, ErrNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(got))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &StatusError{method, path, res.StatusCode, said}
}
return got, nil
}
// GetJSON reads one answer into into.
func (c *Client) GetJSON(ctx context.Context, path string, into any) error {
return c.SendJSON(ctx, http.MethodGet, path, nil, into)
}
// SendJSON sends one request and reads its answer into into, when into is not nil.
func (c *Client) SendJSON(ctx context.Context, method, path string, payload, into any) error {
body, err := c.Do(ctx, method, path, payload, MaxBody)
if err != nil {
return err
}
if into == nil {
return nil
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
}
return nil
}
+16
View File
@@ -257,6 +257,22 @@
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_NPM_OWNER": "novox"
}
},
{
"name": "forge",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/forge",
"binary": "forge",
"loads": [
"forge"
],
"env": {
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_GITEA_ADMIN_USER": "mesh-admin",
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret"
}
}
]
},
+164 -4
View File
@@ -18,12 +18,19 @@ const startedOnShanks = { id: "build-1791680941048622697", repository: "http://1
function aForge() {
const kept: Record<string, { state: string; description: string; updated_at: string }> = {};
const written: string[] = [];
/** Every mesh/merge-gate status the forge was given, newest first, as gitea keeps each one. */
const history: { state: string; description: string; updated_at: string }[] = [];
let failNext = 0;
let hold: Promise<void> | null = null;
let release = () => {};
let readHold: Promise<void> | null = null;
let readRelease = () => {};
return {
kept, written,
kept, written, history,
/** The next writes fail: a request the forge did not take. */
failWrites(n = 1) {
failNext = n;
},
/** The next writes wait until released: a request in flight. */
holdWrites() {
hold = new Promise<void>((r) => (release = r));
@@ -34,7 +41,12 @@ function aForge() {
},
async setCommitStatus(_o: string, _r: string, _sha: string, s: { state: string; context: string; description: string }) {
if (hold) await hold;
if (failNext > 0) {
failNext--;
throw new Error("the forge answered 502");
}
kept[s.context] = { state: s.state, description: s.description, updated_at: new Date().toISOString() };
if (s.context === "mesh/merge-gate") history.unshift(kept[s.context]);
written.push(`${s.state} ${s.description}`);
},
/** Reads answer what the forge held when asked, as late as the reads are held: the answer of a request in flight. */
@@ -50,6 +62,9 @@ function aForge() {
if (readHold) await readHold;
return held;
},
async gateHistory(_o: string, _r: string, _sha: string) {
return history.map((h) => ({ ...h }));
},
};
}
@@ -138,14 +153,159 @@ test("a start from before the check now pending (a run before a recheck, the str
await gates.pending("novox", "hq", SHA);
await gates.verdict("novox", "hq", SHA, () =>
forge.setCommitStatus("novox", "hq", SHA, { state: "failure", context: "mesh/merge-gate", description: "error" }));
now = new Date("2026-10-11T01:30:00Z"); // checked again: pending once more
await gates.pending("novox", "hq", SHA);
now = new Date("2026-10-11T01:30:00Z"); // checked again: the recheck's pending, as the tool writes it (issue 456)
await gates.rechecked("novox", "hq", SHA, { state: "pending", context: "mesh/merge-gate", description: "checking again: flaky" });
assert.equal(await gates.started("novox", "hq", SHA, startedOnShanks), false, "the first run's start");
assert.match(forge.kept["mesh/merge-gate"].description, /^waiting in the build queue since /);
assert.equal(forge.kept["mesh/merge-gate"].description, "checking again: flaky");
assert.equal(await gates.started("novox", "hq", SHA, { ...startedOnShanks, at: "2026-10-11T01:31:00Z" }), true,
"the second run's start");
});
// **A recheck's pending goes through the one writer** (novox/hq issue 456). mesh-delivery's recheck puts the gate back
// to pending through the gitea_commit_status tool; before, that write bypassed the head's chain: the first run's
// verdict mark stayed set, so the new run never said "building", and an old run's verdict could land over the pending.
/** The ask of a check, as the controller's link.NewBuildID names it: when it was asked, in unix nanoseconds. */
const askOf = (at: string) => `build-${BigInt(Date.parse(at)) * 1_000_000n}`;
const firstRun = askOf("2026-10-11T01:09:01Z");
const recheckAt = new Date("2026-10-11T01:30:00Z");
const secondRun = askOf("2026-10-11T01:30:02Z");
const checkingAgain = { state: "pending" as const, context: "mesh/merge-gate", description: "checking again: main moved on" };
const writeOf = (forge: ReturnType<typeof aForge>, state: string, description: string) => () =>
forge.setCommitStatus("novox", "hq", SHA, { state, context: "mesh/merge-gate", description });
const gateOf = (forge: ReturnType<typeof aForge>) =>
`${forge.kept["mesh/merge-gate"].state} ${forge.kept["mesh/merge-gate"].description}`;
test("an old run's verdict redelivered after a recheck's pending is not written over it", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.failWrites(); // the first run's verdict is not taken: the event is offered again
await assert.rejects(gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun));
now = recheckAt;
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun); // redelivered
assert.equal(gateOf(forge), "pending checking again: main moved on", forge.written.join(" | "));
// The recheck's own verdict is written.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail: 0 of 4 compose"), secondRun);
assert.equal(gateOf(forge), "failure fail: 0 of 4 compose");
});
test("an old run's verdict in flight when the recheck comes lands before the recheck's pending", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.holdWrites(); // the first run's verdict is being written when the recheck comes
const verdict = gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
await new Promise((r) => setImmediate(r));
now = recheckAt;
const pending = gates.rechecked("novox", "hq", SHA, checkingAgain);
forge.release();
await Promise.all([verdict, pending]);
assert.equal(gateOf(forge), "pending checking again: main moved on", forge.written.join(" | "));
});
test("a rechecked head says where its new run is: building on which machine", async () => {
const { GateWriter, buildingWords } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
now = recheckAt;
await gates.rechecked("novox", "hq", SHA, checkingAgain);
const at = "2026-10-11T01:30:03Z";
assert.equal(await gates.started("novox", "hq", SHA, { ...startedOnShanks, id: secondRun, at }), true);
assert.equal(forge.kept["mesh/merge-gate"].description, buildingWords("shanks", new Date(at)));
});
test("the recheck's own verdict is written, even asked a little before the pending by another machine's clock", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
const gates = new GateWriter(forge, () => recheckAt);
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), askOf("2026-10-11T01:29:30Z"));
assert.equal(gateOf(forge), "success pass");
// A verdict whose ask cannot be read is written: a verdict is never lost to what cannot be weighed.
await gates.rechecked("novox", "hq", SHA, checkingAgain);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail"), "an id of another shape");
assert.equal(gateOf(forge), "failure fail");
});
test("after a restart, an old run's verdict is weighed against the check the forge's history holds open", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
// What the forge kept before this process started: the announcement, the first run's verdict never written,
// the recheck's pending, and the second run's "building" over it.
forge.history.unshift({ state: "pending", description: "waiting in the build queue since 03:08", updated_at: "2026-10-11T01:08:50Z" });
forge.history.unshift({ state: "pending", description: "checking again: main moved on", updated_at: "2026-10-11T01:30:00Z" });
forge.history.unshift({ state: "pending", description: "building on shanks since 03:30", updated_at: "2026-10-11T01:30:05Z" });
forge.kept["mesh/merge-gate"] = forge.history[0];
const gates = new GateWriter(forge, announcedAt);
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(forge.kept["mesh/merge-gate"].description, "building on shanks since 03:30", forge.written.join(" | "));
await gates.verdict("novox", "hq", SHA, writeOf(forge, "failure", "fail"), secondRun);
assert.equal(gateOf(forge), "failure fail");
});
test("a recheck the forge does not take fails its caller, and the head is as it was", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
let now = announcedAt();
const gates = new GateWriter(forge, () => now);
await gates.pending("novox", "hq", SHA);
forge.failWrites();
await assert.rejects(gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun));
now = recheckAt;
forge.failWrites();
await assert.rejects(gates.rechecked("novox", "hq", SHA, checkingAgain));
// The recheck was refused, so its check is not asked: the first run's verdict, offered again, is the head's.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(gateOf(forge), "success pass");
});
test("the status tool writes the merge check's pending through the writer, and the mesh's other statuses as they were", async () => {
const { GateWriter } = await import("../gate.ts");
const { commitStatusTool } = await import("../delivery.ts");
const forge = aForge();
const direct: string[] = [];
const gates = new GateWriter(forge, () => recheckAt);
const tool = commitStatusTool({
setCommitStatus: async (_o, _r, _sha, s) => void direct.push(`${s.context} ${s.state}`),
gates,
});
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/merge-gate", state: "pending", description: "checking again: x" });
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/repo-check", state: "pending", description: "checking again: x" });
await tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/delivery", state: "success", description: "delivered" });
assert.equal(gateOf(forge), "pending checking again: x");
assert.equal(forge.kept["mesh/repo-check"].state, "pending");
assert.deepEqual(direct, ["mesh/delivery success"]);
// The redelivered first run's verdict is the writer's to drop.
await gates.verdict("novox", "hq", SHA, writeOf(forge, "success", "pass"), firstRun);
assert.equal(gateOf(forge), "pending checking again: x");
// A head named short is another head to the writer: refused, nothing sent.
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: SHA.slice(0, 12), context: "mesh/merge-gate", state: "pending" }),
/whole sha/);
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: SHA, context: "mesh/merge-gate", state: "success" }),
/set by its verdict only/);
});
test("when a check was asked is read from its id; the opening of the check now pending from the gate's history", async () => {
const { askedAt, openedAt } = await import("../gate.ts");
assert.equal(askedAt("build-1791680941048622697"), Date.parse("2026-10-11T01:09:01.048Z"));
assert.equal(askedAt("build-mesh-catalog"), undefined);
assert.equal(askedAt(undefined), undefined);
assert.equal(openedAt([{ state: "pending", description: "building on ace since 03:31", updated_at: "2026-10-11T01:31:00Z" },
{ state: "pending", description: "checking again: x", updated_at: "2026-10-11T01:30:00Z" }]), Date.parse("2026-10-11T01:30:00Z"));
assert.equal(openedAt([{ state: "success", description: "pass", updated_at: "2026-10-11T01:30:00Z" }]), undefined);
assert.equal(openedAt([]), undefined);
});
test("a head this process did not announce (it restarted) is said only while the forge holds its gate pending", async () => {
const { GateWriter } = await import("../gate.ts");
const forge = aForge();
+29 -1
View File
@@ -7,10 +7,13 @@
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { readFileSync } from "node:fs";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { commitStatusTool } from "../dist/delivery.js";
import { GateWriter } from "../dist/gate.js";
import { judgeApiCall, judgeProtectionChange, ruleCovers, routedPath } from "../dist/guard.js";
interface Sent { method: string; path: string; body: any }
@@ -97,7 +100,13 @@ const server = createServer(async (req, res) => {
await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r()));
after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const tools = getGiteaTools(new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t")));
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
// gitea_commit_status is served from the watcher's bundle, beside the merge check's one writer (novox/hq issue 456).
const tools = [...getGiteaTools(client), commitStatusTool({
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gates: new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [] }),
})];
const tool = (name: string) => tools.find((t) => t.name === name)!;
const writes = () => state.sent.map((s) => `${s.method} ${s.path}`);
@@ -138,6 +147,25 @@ test("gitea_api refuses a file written to the default branch or one a rule guard
assert.deepEqual(writes(), ["PUT /repos/novox/mesh-catalog/contents/modules/sudo/module.json"]);
});
test("a protection rule covers a branch as the forge decides it: classes, alternatives, a plain name in any case (issue 539)", async () => {
// One table with the forge bundle (cmd/forge/rules.go): `forge` is the forge's answer; `guard` is where this
// guard deliberately covers more — a rule Go's regexp reads otherwise than JavaScript's — and never less.
const cases = JSON.parse(readFileSync(new URL("./rule-cases.json", import.meta.url), "utf8")) as
Array<{ rule: string; branch: string; forge: boolean; guard?: boolean }>;
assert.ok(cases.length >= 100);
for (const c of cases) {
assert.ok(c.guard === undefined || (c.guard && !c.forge), `${c.rule}: the guard may only cover more than the forge`);
assert.equal(ruleCovers(c.rule, c.branch), c.guard ?? c.forge, `${c.rule} covers ${c.branch}`);
}
const put = (branch: string) => tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/contents/README.md",
method: "PUT", body: { content: "e30=", sha: "x", branch } });
state.rules.push({ rule_name: "release-[ab]" }, { rule_name: "{stable,lts}/*" });
await assert.rejects(put("release-a"), /protection rule release-\[ab\]/);
await assert.rejects(put("lts/2"), /protection rule \{stable,lts\}\/\*/);
await assert.rejects(put("MAIN"), /protection rule main/);
assert.deepEqual(writes(), []);
});
test("gitea_api still writes what an author writes", async () => {
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/issues/3/comments", method: "POST", body: { body: "hi" } });
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/pulls/7/reviews", method: "POST", body: { event: "COMMENT" } });
+285
View File
@@ -0,0 +1,285 @@
[
{"rule":"main","branch":"main","forge":true,"why":"a plain name covers its branch"},
{"rule":"main","branch":"mainx","forge":false},
{"rule":"main","branch":"MAIN","forge":true,"why":"a plain name covers in any case (EqualFold)"},
{"rule":"Main","branch":"main","forge":true},
{"rule":"release/*","branch":"release/1.0","forge":true,"why":"* stays within a segment"},
{"rule":"release/*","branch":"release/a/b","forge":false},
{"rule":"release/**","branch":"release/a/b","forge":true,"why":"** crosses segments"},
{"rule":"v?","branch":"v1","forge":true},
{"rule":"v?","branch":"v/","forge":false,"why":"? is not the separator"},
{"rule":"a.b","branch":"axb","forge":false,"why":". is itself"},
{"rule":"Feat/*","branch":"feat/x","forge":false,"why":"a glob is matched with case"},
{"rule":"release-[ab]","branch":"release-a","forge":true,"why":"a class"},
{"rule":"release-[ab]","branch":"release-c","forge":false},
{"rule":"release-[!ab]","branch":"release-c","forge":true,"why":"a negated class"},
{"rule":"release-[!ab]","branch":"release-a","forge":false},
{"rule":"v[0-9].x","branch":"v7.x","forge":true,"why":"a range"},
{"rule":"v[0-9].x","branch":"vA.x","forge":false},
{"rule":"{main,master}","branch":"master","forge":true,"why":"alternatives"},
{"rule":"{main,master}","branch":"mainline","forge":false},
{"rule":"rel/{a,b}/*","branch":"rel/b/1","forge":true},
{"rule":"rel/{a,b}/*","branch":"rel/c/1","forge":false},
{"rule":"a\\*","branch":"a*","forge":true,"why":"an escape"},
{"rule":"a\\*","branch":"ab","forge":false},
{"rule":"a+b","branch":"a+b","forge":true,"why":"+ is itself"},
{"rule":"a+b","branch":"aab","forge":false},
{"rule":"{open","branch":"{open","forge":true,"why":"a glob the forge cannot compile covers its own text"},
{"rule":"[open","branch":"[open","forge":true},
{"rule":"[open","branch":"o","forge":false},
{"rule":"release-\\d","branch":"release-1","forge":true,"why":"Go reads \\d as a digit"},
{"rule":"release-\\d","branch":"release-d","forge":false,"guard":true},
{"rule":"v\\w*","branch":"v2","forge":true},
{"rule":"main\\s","branch":"main","forge":false,"guard":true},
{"rule":"[]a]","branch":"a","forge":true,"why":"a class opening with ] holds ]"},
{"rule":"[]a]","branch":"]","forge":true},
{"rule":"[]a]","branch":"b","forge":false,"guard":true},
{"rule":"rel-[\\d]","branch":"rel-1","forge":true},
{"rule":"v[\\pL]","branch":"va","forge":true,"why":"\\p is a Unicode class"},
{"rule":"v\\pL","branch":"va","forge":true},
{"rule":"[[:alpha:]]","branch":"a","forge":true,"why":"a POSIX class"},
{"rule":"rel-[[:digit:]]","branch":"rel-1","forge":true},
{"rule":"rel-[[:digit:]]","branch":"rel-x","forge":false,"guard":true},
{"rule":"[^+]*","branch":"-[1","forge":true,"why":"a class opening with ^ is negated in Go"},
{"rule":"d[^](","branch":"d[^](","forge":true},
{"rule":"]?","branch":":,1},","forge":false},
{"rule":"Ab,","branch":"Ab,","forge":true},
{"rule":"\\:[}-|","branch":",{","forge":false},
{"rule":"^,*.d::","branch":"/x","forge":false},
{"rule":"","branch":"A*:","forge":false},
{"rule":"!|A{","branch":":.{]","forge":false},
{"rule":"","branch":"]éb","forge":false},
{"rule":"/{)w","branch":"/{)w","forge":true},
{"rule":"*^w]","branch":"}!.","forge":false},
{"rule":"*),^?*d","branch":"*),^?*d","forge":true},
{"rule":"A)^}!","branch":"","forge":false},
{"rule":"}^d?","branch":"1\\.","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"$?[","branch":"$?[","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"+","forge":true},
{"rule":"","branch":"},[","forge":false},
{"rule":"xA[d:a)","branch":"]1.!bx","forge":false},
{"rule":"^/}(","branch":"","forge":false},
{"rule":"-|x.x1?","branch":"-|x.x1?","forge":true},
{"rule":"x+)A]-A","branch":"x+)A]-A","forge":true},
{"rule":"x*a","branch":"x*a","forge":true},
{"rule":"$xb","branch":"}é","forge":false},
{"rule":"wAx?/a,","branch":"","forge":false},
{"rule":"[,","branch":"[,","forge":true},
{"rule":"}!","branch":"}!","forge":true},
{"rule":"*:|\\w","branch":"","forge":false,"guard":true},
{"rule":"","branch":"","forge":true},
{"rule":"é","branch":"é","forge":true},
{"rule":"..(","branch":"{\\{+","forge":false},
{"rule":"*|{","branch":"1.}-","forge":false},
{"rule":"!+,","branch":"}++-*","forge":false},
{"rule":"x\\({","branch":"*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-{+-","branch":"?-{+-","forge":true},
{"rule":"/-{é|$","branch":"b!.[","forge":false},
{"rule":"a-+dxA","branch":"1AA,xA","forge":false},
{"rule":"]|(","branch":"]|(","forge":true},
{"rule":"}","branch":"{!x{d","forge":false},
{"rule":"/{*](","branch":"!]","forge":false},
{"rule":"($+{-}|","branch":"/","forge":false},
{"rule":"(-b.-1}","branch":"*A*+,1","forge":false},
{"rule":"1[d|)wa","branch":"+}[!}}","forge":false},
{"rule":"w","branch":":.","forge":false},
{"rule":"?}\\*","branch":"","forge":false},
{"rule":"?$*","branch":"?$*","forge":true},
{"rule":"","branch":"!/{béa","forge":false},
{"rule":"","branch":"A1","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"!a$1/b{","branch":"!a$1/b{","forge":true},
{"rule":":dww/|d","branch":":dww/|d","forge":true},
{"rule":"!|*}","branch":"xéxé{b","forge":false},
{"rule":"1é||","branch":"a+.","forge":false},
{"rule":"/+],","branch":"!-\\[{é","forge":false},
{"rule":"a}(","branch":"a}(","forge":true},
{"rule":"|A?.","branch":"}.","forge":false},
{"rule":"d$a^d","branch":"*{","forge":false},
{"rule":"-ba$+A","branch":"-ba$+A","forge":true},
{"rule":"-.|,?a!","branch":"-.|,?a!","forge":true},
{"rule":"A","branch":"A","forge":true},
{"rule":"w","branch":"w","forge":true},
{"rule":".!1!+a","branch":".!1!+a","forge":true},
{"rule":"+$w:}","branch":"+$w:}","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":")]","branch":")]","forge":true},
{"rule":"a})^,}1","branch":"a})^,}1","forge":true},
{"rule":":[:!:","branch":":[:!:","forge":true},
{"rule":"w?w:]1!","branch":"\\.]-","forge":false},
{"rule":"*$/","branch":"*$/","forge":true},
{"rule":"é](","branch":"é](","forge":true},
{"rule":"x$a/","branch":"x$a/","forge":true},
{"rule":"*)b+[^","branch":"*)b+[^","forge":true},
{"rule":"/]/?","branch":"/]/?","forge":true},
{"rule":"{x,A","branch":"!Ab","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"-,","branch":"xé}A[.","forge":false},
{"rule":"w)Ad","branch":"w)Ad","forge":true},
{"rule":":??","branch":":??","forge":true},
{"rule":"x/xé|","branch":"x/xé|","forge":true},
{"rule":"","branch":"1!","forge":false},
{"rule":"!![é[","branch":"x","forge":false},
{"rule":"wb$","branch":"wb$","forge":true},
{"rule":")","branch":")","forge":true},
{"rule":"Aw:)(1","branch":"{b\\,","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"1*}?d","branch":"1*}?d","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-b.","branch":"?-b.","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"d","branch":",aé1","forge":false},
{"rule":"+","branch":"+{","forge":false},
{"rule":"]d","branch":"]}:/bA","forge":false},
{"rule":"*?+:|","branch":"!é.","forge":false},
{"rule":"}$d","branch":"}$d","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"a$+b)b","branch":"d}","forge":false},
{"rule":"?$[/A","branch":"","forge":false},
{"rule":"é){","branch":"é){","forge":true},
{"rule":"$(d","branch":"{é}","forge":false},
{"rule":"b(","branch":"b(","forge":true},
{"rule":"^^+]","branch":"^^+]","forge":true},
{"rule":",)","branch":",)","forge":true},
{"rule":"d-.A-","branch":"d-.A-","forge":true},
{"rule":"$","branch":"\\/","forge":false},
{"rule":"}+[\\|\\$","branch":"x[!b","forge":false},
{"rule":"(d1aé!é","branch":"A[-","forge":false},
{"rule":"x","branch":"1","forge":false},
{"rule":"x","branch":"dx","forge":false},
{"rule":"]|)]","branch":"]|)]","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"[.*?","branch":"[.*?","forge":true},
{"rule":"[?","branch":"[?","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"),/","branch":"a1,,1","forge":false},
{"rule":"}+","branch":"}+","forge":true},
{"rule":"$d\\,d","branch":"$d\\,d","forge":false},
{"rule":"d1","branch":"d1","forge":true},
{"rule":"-!^.:","branch":"-!^.:","forge":true},
{"rule":"x","branch":"+].A11","forge":false},
{"rule":"","branch":"-]","forge":false},
{"rule":"}]}.","branch":"","forge":false},
{"rule":"^{wx","branch":"]1a!","forge":false},
{"rule":"^","branch":"*]x,","forge":false},
{"rule":"(:x*x","branch":"","forge":false},
{"rule":"|[/*a/","branch":"}","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"1\\^","branch":"!é!x{","forge":false},
{"rule":"(.b,AA+","branch":"(.b,AA+","forge":true},
{"rule":"xx","branch":"","forge":false},
{"rule":"d)^):","branch":"/}!","forge":false},
{"rule":"|","branch":",é*]*[","forge":false},
{"rule":"d*d:*1","branch":"\\b,","forge":false},
{"rule":":]","branch":"[","forge":false},
{"rule":"([|!\\^é","branch":"([|!\\^é","forge":true},
{"rule":"$[w)x-*","branch":"","forge":false},
{"rule":")","branch":")","forge":true},
{"rule":".d^)?","branch":"x\\é","forge":false},
{"rule":"1|{w","branch":"1|{w","forge":true},
{"rule":"é$,(]$$","branch":"é$,(]$$","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"","forge":false},
{"rule":"]","branch":"","forge":false},
{"rule":"+:a:","branch":"+:a:","forge":true},
{"rule":"/*","branch":"d.}","forge":false},
{"rule":"|/ba1-","branch":"|/ba1-","forge":true},
{"rule":"","branch":"+","forge":false},
{"rule":"A","branch":"A","forge":true},
{"rule":",-!]","branch":",-!]","forge":true},
{"rule":"$*d!)","branch":"","forge":false},
{"rule":")é+x.d","branch":")é+x.d","forge":true},
{"rule":"éa$\\[1","branch":"1/,","forge":false},
{"rule":"}?1|]d","branch":"}?1|]d","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"-1.*+","branch":"-1.*+","forge":true},
{"rule":"Aw|{","branch":"Aw|{","forge":true},
{"rule":"adAAAa","branch":"A","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"d/^{:","branch":"d/^{:","forge":true},
{"rule":"é:(,^","branch":"é:(,^","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"!ébé({+","branch":"!ébé({+","forge":true},
{"rule":"x","branch":"x","forge":true},
{"rule":"|","branch":",:*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"x|","branch":"x|","forge":true},
{"rule":"a","branch":"aaxéb,","forge":false},
{"rule":"/w-),1é","branch":"/w-),1é","forge":true},
{"rule":"|*)^,","branch":"|*)^,","forge":true},
{"rule":"){é.!b.","branch":"a-}:","forge":false},
{"rule":"(\\","branch":"(\\","forge":true},
{"rule":".w(\\","branch":"+","forge":false},
{"rule":"/d[","branch":"/d[","forge":true},
{"rule":"(}$x!}","branch":"a,bb","forge":false},
{"rule":"w{","branch":"w{","forge":true},
{"rule":"[+a\\A(+","branch":"[+a\\A(+","forge":true},
{"rule":"(1^{)bb","branch":"(1^{)bb","forge":true},
{"rule":"^1","branch":"é-}","forge":false},
{"rule":"*,+}1):","branch":"*,+}1):","forge":true},
{"rule":"[A[(.x,","branch":"]db!*","forge":false},
{"rule":"","branch":"Ad}","forge":false},
{"rule":".","branch":".","forge":true},
{"rule":"w!é{x","branch":"é.{","forge":false},
{"rule":"$\\^","branch":"bx]+","forge":false},
{"rule":"1)","branch":"+*d","forge":false},
{"rule":"x)}/])","branch":"]","forge":false},
{"rule":",a}.1}","branch":",a}.1}","forge":true},
{"rule":"-a{(","branch":"-a{(","forge":true},
{"rule":"1","branch":"1","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"}[é$","branch":"]d-!","forge":false},
{"rule":"xé{é+?","branch":"xé{é+?","forge":true},
{"rule":"/A1!","branch":"xAA\\::","forge":false},
{"rule":"1][b-.(","branch":"{/-b![","forge":false},
{"rule":"(d}(","branch":"(d}(","forge":true},
{"rule":"{*}","branch":"!ébd","forge":true},
{"rule":"()^^)","branch":"()^^)","forge":true},
{"rule":"1(1A{$","branch":"*,[","forge":false},
{"rule":"A:!^\\","branch":"A:!^\\","forge":true},
{"rule":"-]:b]","branch":"-]:b]","forge":true},
{"rule":",","branch":",","forge":true},
{"rule":"[-:]","branch":"[-:]","forge":false},
{"rule":"","branch":"é:b1:d","forge":false},
{"rule":"1\\!]?d","branch":"","forge":false},
{"rule":"","branch":"bb!]+","forge":false},
{"rule":"/da","branch":"b]d[","forge":false},
{"rule":",]","branch":",]","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"a]{A-","branch":"-+","forge":false},
{"rule":"A?A^1","branch":"A?A^1","forge":true},
{"rule":"\\-$","branch":"","forge":false},
{"rule":"+","branch":"*[}[","forge":false},
{"rule":",b?{dw","branch":",b?{dw","forge":true},
{"rule":"d[$b1","branch":"/","forge":false},
{"rule":"})|xAA","branch":"é,!","forge":false},
{"rule":"]--","branch":"]--","forge":true},
{"rule":"],/b","branch":"],/b","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"*aé","branch":"*aé","forge":true},
{"rule":"{","branch":"{","forge":true},
{"rule":":aé","branch":"","forge":false},
{"rule":".\\}","branch":".\\}","forge":false},
{"rule":"wA]*","branch":"\\]!d{+","forge":false},
{"rule":"a/1x$/","branch":"[1!:/A","forge":false},
{"rule":"[.wb}d.","branch":"-!*..a","forge":false},
{"rule":".[{()a","branch":"+}","forge":false},
{"rule":"]/!:,","branch":"{Aéa","forge":false},
{"rule":"é{ba[-","branch":"],,1d/","forge":false},
{"rule":"(w*","branch":"1}.!","forge":false},
{"rule":"{?","branch":"1","forge":false},
{"rule":")","branch":",+!","forge":false},
{"rule":":","branch":"{/+A][","forge":false},
{"rule":".1{)","branch":"a\\bA","forge":false},
{"rule":"w-1+x1-","branch":"*]1}bb","forge":false},
{"rule":"xb$$","branch":"éx:]-:","forge":false},
{"rule":"++$!?*$","branch":"bA","forge":false},
{"rule":"]{","branch":"[xd*x.","forge":false}
]
+1 -1
View File
@@ -426,7 +426,7 @@ test("the tools register once there is a way to a token, and the first call mint
"gitea_list_branches",
"gitea_delete_branch",
"gitea_branch_protection_get", "gitea_branch_protection_set",
"gitea_note_append", "gitea_delivery_view", "gitea_commit_statuses", "gitea_open_pulls", "gitea_contains", "gitea_commit_status",
"gitea_note_append", "gitea_delivery_view", "gitea_commit_statuses", "gitea_open_pulls", "gitea_contains",
"gitea_list_labels", "gitea_create_label",
"gitea_api",
],
+3 -20
View File
@@ -11,7 +11,7 @@
import { registerModuleTools, type ToolDefinition } from "@novox/mesh-sdk/tools";
import { emit } from "@novox/mesh-sdk/events";
import { GiteaClient, type BranchProtection } from "../client.js";
import { appendNote, deliveryStatus, run, viewBody, viewComment } from "../delivery.js";
import { appendNote, run, viewBody, viewComment } from "../delivery.js";
import { containedFrom, isCommit, mergeableUnder, newestByContext } from "../checks.js";
import {
MERGE_METHODS, Refused, checkName, judgeApiCall, judgeContentsWrite, judgeDelete, judgeProtectionChange, pathSegments, routedPath, sentQuery,
@@ -639,25 +639,8 @@ function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return { commit, in: within, contains, ahead };
},
},
{
name: "gitea_commit_status",
description: "Set one of the mesh's statuses on a commit (mesh/delivery, mesh/delivery-group): pending, success, error, failure or warning, a short description, and the page it links to. The merge check's statuses (mesh/merge-gate, mesh/repo-check) take pending only — a recheck putting them back until the fresh verdict sets them.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit" },
context: { type: "string", description: "the status's name, mesh/…" },
state: { type: "string", description: "pending | success | error | failure | warning" },
description: { type: "string", description: "one line" },
target_url: { type: "string", description: "the page it links to (the pull request)" },
},
run: async (args) => {
const status = deliveryStatus(String(args.context), String(args.state), String(args.description ?? ""),
args.target_url ? String(args.target_url) : undefined);
await gitea.setCommitStatus(String(args.owner), String(args.repo), String(args.sha), status);
return { set: status };
},
},
// gitea_commit_status is served from the watcher's bundle (index.ts), beside the merge check's one writer
// (novox/hq issue 456).
// ---- Labels ----
{
@@ -366,6 +366,7 @@ func (h *Holder) words() []func() {
}
sort.Strings(walks)
var asks []func()
h.waitsWhy = map[string]string{}
for _, walk := range walks {
want := h.turning[walk]
var waiting []string
@@ -377,11 +378,20 @@ func (h *Holder) words() []func() {
}
if len(waiting) > 0 {
sort.Strings(waiting)
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s carried by it not at its turn", walk,
strings.Join(waiting, ", "))
h.waitsWhy[walk] = strings.Join(waiting, ", ") + " carried by it not at its turn"
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s", walk, h.waitsWhy[walk])
continue
}
ids, why := append([]string(nil), want.ids...), strings.Join(want.whys, "; ")
// **Never the word for a walk that carries a stopped change** (ADR 0299, review of #221): it is asked to
// leave the merge out first (leaveStopped).
if stopped, _ := h.carriesStopped(walk); len(stopped) > 0 {
h.waitsWhy[walk] = "it carries " + strings.Join(stopped, ", ") + ", whose delivery was stopped: it is " +
"asked to leave that merge out first"
h.Logf("[mesh-delivery] the walk %s is not let go: it carries %s, whose delivery was stopped", walk,
strings.Join(stopped, ", "))
continue
}
if h.asking[walk] {
continue // its word is being asked
}
@@ -412,3 +422,94 @@ func (h *Holder) words() []func() {
h.turning = nil
return asks
}
// endedMovingNothing is whether a delivery ended delivered with nothing for a walk to move: merged, its plan
// moving no module the mesh holds, and no walk followed.
func endedMovingNothing(d *Delivery) bool {
return d.State == Delivered && d.MergedAs != "" && d.Walk == nil && d.MovesNothing()
}
// endedAsNothingToMove are the deliveries a walk answers that ended delivered as nothing for a walk to move (novox/hq
// issues 362, 529: the nzbgeek case); empty when it answers any other. Such a walk is held by nothing: said, never
// let go by this owner. Under the lock.
func (h *Holder) endedAsNothingToMove(w Walk) []string {
var out []string
for _, d := range h.deliveries {
_, answers := walkAnswers(w, d)
if !answers && (d.Walk == nil || d.Walk.ID != w.ID) {
continue
}
if !endedMovingNothing(d) {
return nil
}
out = append(out, d.ID)
}
sort.Strings(out)
return out
}
// WalksWaiting is `stalled` asked with `walks` (novox/hq issue 529): for each walk waiting for this owner's word,
// what holds it, in the shape of a stalled line naming the walk — each held delivery it carries (State held, its
// reason and the verb that releases it), else why its word was not given at the last tick (State published), else
// that no live delivery holds it (State unheld). A walk whose word is being given is not listed. The controller's
// `plans` says these instead of pointing at a hand start; the self-check never raises a condition for them.
func (h *Holder) WalksWaiting() []StalledLine {
h.mu.Lock()
defer h.mu.Unlock()
now := h.Now()
var out []StalledLine
for id, s := range h.walks {
w := s.walk
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.asking[id] {
continue
}
waited := now.Sub(w.Created).Round(time.Second).String()
var held []StalledLine
live := false
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != id {
continue
}
live = true
if d.State == Held {
held = append(held, StalledLine{ID: d.ID, Number: d.Number, State: Held, Walk: id, For: waited,
Bound: "0s", H2: "none: held for a person — the operator's", Says: HeldLine(d)})
}
}
switch {
case len(held) > 0:
out = append(out, held...)
case h.waitsWhy[id] != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: "not let go yet: " + h.waitsWhy[id]})
case !live:
says := "no delivery that has not ended follows it or is answered by it, so nothing gives it its word: " +
"`mesh-delivery.show` for the deliveries of " + w.Repository
if ended := h.endedAsNothingToMove(w); len(ended) > 0 {
// The nzbgeek case (novox/hq issues 362, 529): its merges' deliveries ended as nothing for a walk to
// move, yet the walk moves modules — nothing judged those, so nothing here lets it go.
says = "held by nothing: its merges' deliveries ended as nothing-to-move (" + strings.Join(ended, ", ") +
"), yet the walk moves " + strings.Join(walkModules(w), ", ") + " — nothing gives it its word"
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: id, For: waited,
Bound: "0s", H2: "none: no delivery holds the walk — the operator's", Says: says})
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Walk != out[j].Walk {
return out[i].Walk < out[j].Walk
}
return out[i].ID < out[j].ID
})
return out
}
// walkModules are the modules a walk's record names, sorted.
func walkModules(w Walk) []string {
out := make([]string, 0, len(w.Modules))
for m := range w.Modules {
out = append(out, m)
}
sort.Strings(out)
return out
}
@@ -65,6 +65,11 @@ type ChecksOf struct {
Since string `json:"since"`
Waits string `json:"waits,omitempty"`
Group string `json:"group,omitempty"`
// Class, Opened and MergedAt are what the issue tracker measures a fix's time by (novox/hq ADR 0313): the
// class of what it moves, when its pull request was first heard, and when it merged.
Class string `json:"class,omitempty"`
Opened string `json:"opened,omitempty"`
MergedAt string `json:"merged_at,omitempty"`
}
// Checks is the `checks` verb: a repository and a pull request's number, or a repository and a commit.
@@ -104,7 +109,13 @@ func (h *Holder) Checks(repository string, number int, commit string) (*ChecksAn
}
if d != nil {
a.Delivery = &ChecksOf{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), Waits: d.waits,
Group: d.Group}
Group: d.Group, Class: fixClass(&d.Delivery)}
if !d.opened.IsZero() {
a.Delivery.Opened = d.opened.UTC().Format(time.RFC3339)
}
if !d.MergedAt.IsZero() {
a.Delivery.MergedAt = d.MergedAt.UTC().Format(time.RFC3339)
}
if d.Check != nil && sameCommit(d.Commit, a.Commit) {
a.Verdict = d.Check
}
@@ -139,6 +150,7 @@ type checksDelivery struct {
Delivery
waits string
groupSaid string
opened time.Time
}
// deliveryOf is the delivery of a commit — its head, or the commit it merged as — else, for a pull request,
@@ -167,7 +179,7 @@ func (h *Holder) deliveryOf(repository string, number int, commit string) *check
if found == nil {
return nil
}
c := &checksDelivery{Delivery: *found, waits: h.waitsFor(found)}
c := &checksDelivery{Delivery: *found, waits: h.waitsFor(found), opened: h.openedOf(found)}
if found.Check != nil {
v := *found.Check
c.Check = &v
@@ -14,8 +14,10 @@ import (
// the forge's holder, asked through its tools). Owed, never lost: kept with the delivery before anything is
// tried, and tried again until done.
// transitionEvent is what `mesh-delivery.transition` says: no secret, no address.
func transitionEvent(d *Delivery, t Transition) map[string]any {
// transitionEvent is what `mesh-delivery.transition` says: no secret, no address. opened is when its pull
// request was first heard (zero when not known); with its class and its merge's moment it is what the issue
// tracker measures a fix's time by (novox/hq ADR 0313 §2, §4).
func transitionEvent(d *Delivery, t Transition, opened time.Time) map[string]any {
e := map[string]any{"id": d.ID, "repository": d.Repository, "commit": d.Commit, "from": t.From, "to": t.To,
"event": t.Event, "why": t.Why, "at": t.At}
if t.By != "" {
@@ -36,9 +38,57 @@ func transitionEvent(d *Delivery, t Transition) map[string]any {
if d.Plan != nil {
e["plan"] = d.Plan.Summary
}
if c := fixClass(d); c != "" {
e["class"] = c
}
if !opened.IsZero() {
e["opened"] = opened.UTC()
}
if !d.MergedAt.IsZero() {
e["merged_at"] = d.MergedAt.UTC()
}
return e
}
// fixClass is the class of what a delivery moves (novox/hq ADR 0313 §2): its plan's, as the planner said it;
// none for a plan that moves no module, from a planner that did not say a class; else its walk's class once its
// delivery time is known; empty when nothing says it.
func fixClass(d *Delivery) string {
switch {
case d.Plan != nil && d.Plan.Class != "":
return d.Plan.Class
case d.Plan != nil && len(d.Plan.Moved) == 0 && len(d.Plan.Dependents) == 0 && len(d.Plan.New) == 0:
return classNone
case d.Time != nil:
return d.Time.Class
}
return ""
}
// classNone is the class of a delivery that moves no module: nothing for a walk to move, and no fix budget.
const classNone = "none"
// openedOf is when a delivery's pull request was first heard: the earliest head of it this owner holds, never
// one made from the forge's word at its merge; zero when none is held or it has no pull request. Under the lock.
func (h *Holder) openedOf(d *Delivery) time.Time {
if d.Number <= 0 {
return time.Time{}
}
var first time.Time
for _, c := range h.deliveries {
if c.FromForge || c.Number != d.Number || !strings.EqualFold(c.Repository, d.Repository) || c.Created.IsZero() {
continue
}
if first.IsZero() || c.Created.Before(first) {
first = c.Created
}
}
if first.IsZero() && !d.FromForge {
first = d.Created
}
return first
}
// noteLine is one transition as the commit's note keeps it.
func noteLine(d *Delivery, t Transition) string {
by := ""
@@ -104,6 +154,9 @@ func statusOf(d *Delivery) (string, string) {
case Superseded:
return "warning", clip("superseded: "+why, 140)
case Stopped:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its change reached the machines (novox/hq issue 480)
}
return "error", clip("stopped: "+why, 140)
case Retired:
return "success", "history: merged before this owner delivered anything, never walked"
@@ -124,6 +177,9 @@ const viewMarker = "<!-- mesh-delivery:view -->"
func ViewBody(d *Delivery, g *Group) string {
var b strings.Builder
fmt.Fprintf(&b, "%s\n**Delivery** `%s` — **%s** since %s\n\n", viewMarker, d.ID, d.State, d.Since.UTC().Format(time.RFC3339))
if d.coveredStop() {
fmt.Fprintf(&b, "**Covered** after its stop: a later merge that contains it was delivered: %s\n\n", d.CoveredBy)
}
if d.HeldWhy != "" && d.State == Held {
fmt.Fprintf(&b, "Held for a person: %s — `mesh-delivery.release` with why lets it go on.\n\n", d.HeldWhy)
}
@@ -278,6 +334,16 @@ func (h *Holder) Flush() {
d.Owed = owed(d.Owed)
h.keep(d)
}
// Counted for delivery_status, so a window's writes can be judged against its tries (novox/hq issue 407).
h.tried.last = h.Now()
switch {
case err == nil:
h.tried.done++
case dropped != "":
h.tried.givenUp++
default:
h.tried.failed++
}
if dropped != "" {
h.refuse(fmt.Errorf("%s: its owed %s given up — %s: %v", j.id, j.e.Kind, dropped, err))
}
@@ -94,8 +94,18 @@ type fakeController struct {
asked int
delivers []string
stops []string
plans int
down bool
// withouts are the stops that left merges out (novox/hq ADR 0299): "<walk> without <merge>,…".
withouts []string
// refuseWithout refuses every stop with merges to leave out, as a controller before ADR 0299 does; mergedAt are
// the merges' times, by commit, for what a later merge contains.
refuseWithout bool
mergedAt map[string]time.Time
// refuseStop refuses every whole stop too; moves are the modules each merge moves, by commit, for what a left-out
// merge shares (ADR 0299).
refuseStop bool
moves map[string][]string
plans int
down bool
// onCheck runs as a check is asked, before it is answered: what the forge held at that moment.
onCheck func()
// planned is what Plan answers instead of "builds app", when set.
@@ -103,7 +113,8 @@ type fakeController struct {
}
func newFakeController() *fakeController {
return &fakeController{walks: map[string]Walk{}, held: true}
return &fakeController{walks: map[string]Walk{}, held: true, mergedAt: map[string]time.Time{},
moves: map[string][]string{}}
}
func (c *fakeController) Plan(repository, base, head string, paths, dirs, removed []string) (*DeliveryPlan, error) {
@@ -177,7 +188,13 @@ func (c *fakeController) Stop(walk, why, by string) error {
return errors.New("down")
}
c.stops = append(c.stops, walk)
if w, known := c.walks[walk]; known {
if c.refuseStop {
return errors.New("the controller refuses")
}
if w, known := c.walks[walk]; known && !w.Open() {
// As the controller: only a walk that is building or rolling is stopped (`delivery-stop`).
return fmt.Errorf("%s is already %s", walk, w.State)
} else if known {
w.State = walkFailed
if w.Delivery == nil {
w.Delivery = &WalkDelivery{}
@@ -188,6 +205,128 @@ func (c *fakeController) Stop(walk, why, by string) error {
}
return nil
}
// StopWithout ends a walk without the merges named, as the controller does (novox/hq ADR 0299): each merge named,
// and each later merge of its repository by merge time (mergedAt), is left out; the others are put in a batch,
// "<walk>-again", assembling until a test cuts it. Given a batch, the merges left out go to a stopped record,
// "<walk>-left", and the batch goes on with the others. refuseWithout refuses every such stop, as an older
// controller does.
func (c *fakeController) StopWithout(walk, why, by string, without []string) (StopAnswer, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.down {
return StopAnswer{}, errors.New("down")
}
c.withouts = append(c.withouts, walk+" without "+strings.Join(without, ","))
if c.refuseWithout {
return StopAnswer{}, errors.New(`delivery-stop takes no argument "without"`)
}
w, known := c.walks[walk]
if !known || !(w.Open() || w.IsBatch()) {
return StopAnswer{}, fmt.Errorf("%s is no open walk or batch", walk)
}
wd := WalkDelivery{}
if w.Delivery != nil {
wd = *w.Delivery
}
named := func(m WalkMerge) bool { return containsString(without, m.Repository+"@"+m.Commit) }
contains := func(m WalkMerge) string {
for _, s := range wd.Merges {
if named(s) && s.Repository == m.Repository && c.mergedAt[m.Commit].After(c.mergedAt[s.Commit]) {
return s.Commit
}
}
return ""
}
shares := func(m WalkMerge) (string, []string) {
for _, s := range wd.Merges {
if named(s) {
var both []string
for _, x := range c.moves[m.Commit] {
if containsString(c.moves[s.Commit], x) {
both = append(both, x)
}
}
if len(both) > 0 {
return s.Commit, both
}
}
}
return "", nil
}
a := StopAnswer{Stopped: walk}
var again, left []WalkMerge
for _, m := range wd.Merges {
s, both := shares(m)
switch {
case named(m):
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit})
left = append(left, m)
case len(both) > 0:
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: s, Shares: both})
left = append(left, m)
case contains(m) != "":
a.LeftOut = append(a.LeftOut, WalkLeftOut{Repository: m.Repository, Commit: m.Commit, Contains: contains(m)})
left = append(left, m)
default:
m.Carried = ""
again = append(again, m)
a.WalkedAgain = append(a.WalkedAgain, m.Repository+"@"+m.Commit)
}
}
if len(a.LeftOut) == 0 {
return StopAnswer{}, fmt.Errorf("%s answers no merge %s", walk, strings.Join(without, ", "))
}
if w.IsBatch() && len(again) == 0 {
// Every merge left out: the batch itself is stopped, as the controller does.
wd.Stopped, wd.StoppedWhy, wd.LeftOut, wd.Batch = by, why, a.LeftOut, nil
w.Delivery, w.State = &wd, walkFailed
w.Revision++
c.walks[walk] = w
return a, nil
}
if w.IsBatch() {
rid := walk + "-left"
c.walks[rid] = Walk{ID: rid, Repository: left[0].Repository, Branch: "main", Commit: left[0].Commit,
Created: w.Created, State: walkFailed, Revision: 1, Modules: map[string]*WalkModule{},
Delivery: &WalkDelivery{Stopped: by, StoppedWhy: why, Merges: left, LeftOut: a.LeftOut, WalkedAgainBy: walk}}
wd.Merges = again
w.Delivery = &wd
w.Repository, w.Commit = again[0].Repository, again[0].Commit
w.Revision++
c.walks[walk] = w
a.Stopped, a.WalkedAgainBy = rid, walk
return a, nil
}
if len(again) > 0 {
id := walk + "-again"
closes := w.Created.Add(time.Minute)
c.walks[id] = Walk{ID: id, Repository: again[0].Repository, Branch: "main", Commit: again[0].Commit,
Created: w.Created.Add(time.Minute), State: walkAssembling, Revision: 1,
Modules: map[string]*WalkModule{}, Delivery: &WalkDelivery{Merges: again, Batch: &WalkBatch{ClosesAt: &closes}}}
wd.WalkedAgainBy, a.WalkedAgainBy = id, id
}
wd.Stopped, wd.StoppedWhy, wd.LeftOut = by, why, a.LeftOut
w.Delivery = &wd
w.State = walkFailed
w.Note = "stopped by " + by + ": " + why
w.Revision++
c.walks[walk] = w
return a, nil
}
// cut makes a batch the walk under the same id, waiting for its word, as the controller cuts it.
func (c *fakeController) cut(id string) Walk {
c.mu.Lock()
w := c.walks[id]
c.mu.Unlock()
wd := *w.Delivery
wd.Batch, wd.Awaits = nil, "mesh-delivery"
w.Delivery = &wd
w.State, w.Tiers, w.Modules = walkBuilding, [][]string{{"app"}}, map[string]*WalkModule{"app": {}}
return c.put(w)
}
func (c *fakeController) Walks(walk string) (bool, []Walk, error) {
c.mu.Lock()
defer c.mu.Unlock()
@@ -55,8 +55,28 @@ type Holder struct {
walks map[string]seenWalk
orphans map[string]time.Time
turning map[string]*turnWant
// waitsWhy is, by walk, why the last tick did not give a waiting walk its word: a delivery it carries not at
// its turn, or a stopped change it carries (novox/hq issue 529). Said by WalksWaiting.
waitsWhy map[string]string
// asking are the walks whose word is being asked right now, outside the lock: not asked twice meanwhile.
asking map[string]bool
// leaving are, by walk, the merges a stop gathered to leave out of it, asked once (novox/hq ADR 0299).
leaving map[string][]string
// leaveAsked is when a walk was last asked to leave out the stopped merges it carries (leaveStopped).
leaveAsked map[string]leaveAsk
// leaveRefused is, by walk, what the controller answered when it refused both to leave a stopped merge out and
// to stop the walk whole.
leaveRefused map[string]string
// tried counts the owed effects Flush tried since this owner started, by how each ended (novox/hq issue 407).
tried owedTries
}
// owedTries is how many owed effects were tried since this owner started, and when the last was: three numbers
// and a time, however many were tried. Not kept: a restarted owner counts from its start, and says since when, so a
// window's count is the difference of two readings in the same run (novox/hq issue 407).
type owedTries struct {
done, failed, givenUp int
last time.Time
}
// unmatched is a walk no delivery is known for yet: its merge not heard, or merged before this owner held.
@@ -96,6 +116,7 @@ func (h *Holder) init() {
h.written = map[string][]byte{}
h.dirtyGroup = map[string]bool{}
h.unmatched = map[string]unmatched{}
h.waitsWhy = map[string]string{}
h.askTried = map[string]time.Time{}
h.walkAsked = map[string]time.Time{}
h.unchecked = map[string]unchecked{}
@@ -151,9 +172,10 @@ func (h *Holder) refuse(err error) {
// moved records what a delivery's transitions owe outside, and keeps it.
func (h *Holder) moved(d *Delivery, ts []Transition) {
now := h.Now()
opened := h.openedOf(d)
for _, t := range ts {
h.Logf("[mesh-delivery] %s: %s → %s (%s): %s", d.ID, orNothing(t.From), t.To, t.Event, t.Why)
d.Owed = append(d.Owed, Effect{Kind: EffectEmit, Line: string(mustJSON(transitionEvent(d, t))), Since: now})
d.Owed = append(d.Owed, Effect{Kind: EffectEmit, Line: string(mustJSON(transitionEvent(d, t, opened))), Since: now})
line := noteLine(d, t)
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: line, Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) && afterTheTrunk(t.To) {
@@ -642,6 +664,15 @@ func (h *Holder) linkWalk(d *Delivery, w Walk, carried string) {
d.LetGo, d.LetGoWhy = time.Time{}, "" // a new walk waits for a word of its own
}
d.Walk = w.Seen()
// Stopped without some of its merges (novox/hq ADR 0299): this delivery's merge was left out, maybe for
// containing a stopped one, or is walked again by the batch the record names.
if w.Delivery != nil && (w.Delivery.WalkedAgainBy != "" || len(w.Delivery.LeftOut) > 0) {
if o, out := w.leftOut(d.Repository, d.MergedAs); out {
d.Walk.Contains, d.Walk.Shares = o.Contains, o.Shares
} else {
d.Walk.WalkedAgainBy = w.Delivery.WalkedAgainBy
}
}
if carried != "" {
d.AnsweredBy = carried
}
@@ -821,6 +852,9 @@ func (h *Holder) Tick() {
for _, ask := range h.plan() {
ask()
}
for _, ask := range h.leaveStopped() {
ask()
}
}
// plan works out, under the lock, what to ask; each ask applies its own answer.
@@ -930,12 +964,16 @@ func (h *Holder) plan() []func() {
// grace, it is stopped saying so — never left waiting for a start that cannot come.
for _, d := range h.deliveries {
ended := d.endedUnstarted()
if d.coveredStop() && !d.CoverTold {
h.tellCovered(d, now) // covered before this holder told it, or before a restart (novox/hq issue 480)
continue
}
if strandedUncovered(d) {
// Stopped and final, but still read for a later merge that delivered its modules: until one did,
// `stalled` keeps listing it (review of novox/hq issue 474).
var ask []string
if d.CoveredBy, ask = h.coverOf(d); d.CoveredBy != "" {
h.keep(d)
h.tellCovered(d, now)
}
for _, in := range ask {
if a := h.askContains(d, in, now); a != nil {
@@ -1080,6 +1118,25 @@ func (h *Holder) laterOnItsWay(d *Delivery) string {
return ""
}
// tellCovered tells the forge that a stop is covered since, once: its status says covered and by which merge, its view
// says so, and its note keeps the line on its head and its merge commit (novox/hq issue 480). No transition: the stop
// stays final, and the bus's transition event is not said again.
func (h *Holder) tellCovered(d *Delivery, now time.Time) {
d.CoverTold = true
line := oneLine(fmt.Sprintf("%s delivery %s: stopped, %s", now.UTC().Format(time.RFC3339), d.ID, d.coveredStopSaid()))
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: line, Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) {
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.MergedAs, Line: line, Since: now})
}
if d.shownOnItsPullRequest() {
d.owe(Effect{Kind: EffectView, Since: now})
state, desc := statusOf(d)
d.owe(Effect{Kind: EffectStatus, Commit: d.Commit, Context: "mesh/delivery", State: state, Line: desc, Since: now})
}
h.Logf("[mesh-delivery] %s: stopped, %s", d.ID, d.coveredStopSaid())
h.keep(d)
}
// strandedUncovered is whether a delivery stopped because its ended walk delivered nothing is still uncovered: a
// merged change that never reached the machines, which `stalled` keeps listing until a later delivered merge
// carries its modules (review of novox/hq issue 474).
@@ -1807,7 +1864,13 @@ func mayCarry(c, d *Delivery, module string) bool {
c.Plan != nil && slices.Contains(c.Plan.Moved, module)
}
// stopInGroup stops a member after one that failed, and its walk if it has one open.
// stopInGroup stops a member after one that failed, and leaves its merge out of its walk if it has one open.
//
// **Only its own change** (novox/hq issue 459, ADR 0299): one walk carries every merge of its batch, so ending the
// member's walk ended every other delivery on it, which nobody stopped. Its merge is left out instead: the
// controller ends the walk and walks the other merges again without it, and without each later merge of its
// repository, which contains it. Every member a pass stops on one walk is gathered first, and the walk is asked
// once, by the first member's ask, after the pass: a member is never walked again because its stop came second.
func (h *Holder) stopInGroup(d *Delivery, why string) func() {
if d == nil || d.State.Final() {
return nil
@@ -1818,15 +1881,266 @@ func (h *Holder) stopInGroup(d *Delivery, why string) func() {
return nil
}
h.moved(d, []Transition{t})
if d.Walk == nil || d.Walk.State == walkDone || d.Walk.State == walkFailed || d.Walk.State == walkSuperseded {
// Its open walk; or, its walk stopped for another merge, the batch or walk that walks its merge again (review
// of #221): a merge walked again and then stopped by its group is left out there too.
var walk string
switch {
case d.Walk == nil:
return nil
case d.Walk.State == walkBuilding || d.Walk.State == walkRolling:
walk = d.Walk.ID
case d.walkedAgain():
walk = d.Walk.WalkedAgainBy
default:
return nil
}
walk := d.Walk.ID
if !h.leave(walk, d) {
return nil // the first member left out of this walk asks for every one
}
return func() {
if err := h.Controller.Stop(walk, why, byOwner); err != nil {
h.Logf("[mesh-delivery] the walk %s could not be stopped: %v", walk, err)
without, shared := h.leaving459(walk)
if !shared {
if err := h.Controller.Stop(walk, why, byOwner); err != nil {
h.Logf("[mesh-delivery] the walk %s could not be stopped: %v", walk, err)
}
return
}
h.Logf("[mesh-delivery] %s", h.stopLeaving(walk, why, byOwner, without))
}
}
// stopLeaving asks the controller to stop a walk or batch without merges (ADR 0299) and says what came of it. A
// refusal — an older controller takes no `without`, or the leave-out could not be made — falls back to stopping the
// whole walk: the stopped change is held back, and every delivery on it ends stopped with the reason (review of
// #221). Outside the lock.
func (h *Holder) stopLeaving(walk, why, by string, without []string) string {
h.mu.Lock()
h.askedToLeave(walk)
h.mu.Unlock()
a, err := h.Controller.StopWithout(walk, why, by, without)
if err == nil {
return fmt.Sprintf("the walk %s is stopped without %s; %s", walk, strings.Join(without, ", "), againWords(a))
}
refused := err
if err := h.Controller.Stop(walk, fmt.Sprintf("%s, and it could not be left out alone (%v)", why, refused), by); err != nil {
h.mu.Lock()
if h.leaveRefused == nil {
h.leaveRefused = map[string]string{}
}
h.leaveRefused[walk] = fmt.Sprintf("leaving it out (%v), and stopping the walk whole (%v)", refused, err)
h.mu.Unlock()
return fmt.Sprintf("the walk %s could not be stopped without %s (%v), nor whole (%v): `plans stop %s --why …` "+
"ends it by hand", walk, strings.Join(without, ", "), refused, err, walk)
}
return fmt.Sprintf("the walk %s could not be stopped without %s (%v), so it was stopped whole: every delivery on it "+
"ends stopped", walk, strings.Join(without, ", "), refused)
}
// carriesStopped are the merges a walk or batch carries whose deliveries were stopped, as `<repository>@<commit>`,
// and why the first was: a record that names a stopped delivery's merge and has not left it out would deliver a
// stopped change (review of #221). Under the lock.
func (h *Holder) carriesStopped(walk string) ([]string, string) {
s, known := h.walks[walk]
if !known || !(s.walk.Open() || s.walk.IsBatch()) || s.walk.Delivery == nil || len(s.walk.Delivery.Merges) == 0 {
return nil, ""
}
var out []string
why := ""
ids := make([]string, 0, len(h.deliveries))
for id := range h.deliveries {
ids = append(ids, id)
}
slices.Sort(ids)
for _, id := range ids {
d := h.deliveries[id]
// Stopped by a person or by its group, the acts that hold a change back: never one ended by what its walk
// did (the observed rows, which name nobody), such as a merge stranded after its walk (issue 474).
if d.State != Stopped || d.MergedAs == "" || len(d.Transitions) == 0 {
continue
}
if last := d.Transitions[len(d.Transitions)-1]; last.Event != EvStop || last.By == "" {
continue
}
// By the merges it names alone: its own commit may be another's merge it no longer answers.
if !slices.ContainsFunc(s.walk.Delivery.Merges, func(m WalkMerge) bool {
return strings.EqualFold(m.Repository, d.Repository) && sameCommit(m.Commit, d.MergedAs)
}) {
continue
}
if _, out := s.walk.leftOut(d.Repository, d.MergedAs); out {
continue
}
out = append(out, d.Repository+"@"+d.MergedAs)
if why == "" {
why = fmt.Sprintf("%s was stopped: %s", d.ID, lastWhyOf(d))
}
}
return out, why
}
// lastWhyOf is the reason of a delivery's last transition.
func lastWhyOf(d *Delivery) string {
if len(d.Transitions) == 0 {
return ""
}
return d.Transitions[len(d.Transitions)-1].Why
}
// leaveAsk is when a walk was last asked to leave merges out, and at which revision of its record.
type leaveAsk struct {
at time.Time
revision int64
}
// askedToLeave notes an ask to leave merges out of a walk, at its record's revision: under the lock.
func (h *Holder) askedToLeave(walk string) {
if h.leaveAsked == nil {
h.leaveAsked = map[string]leaveAsk{}
}
h.leaveAsked[walk] = leaveAsk{at: h.Now(), revision: h.walks[walk].walk.Revision}
}
// leaveStopped asks, for every open walk or batch that carries a stopped delivery's merge, that the merge be left
// out (ADR 0299, review of #221): a merge walked again and then stopped, or stopped while its batch assembled, never
// ships. Asked once per record of it while it is carried; such a walk is never given its word (words).
func (h *Holder) leaveStopped() []func() {
h.mu.Lock()
defer h.mu.Unlock()
now := h.Now()
var asks []func()
held := map[string]string{}
for id := range h.walks {
if without, _ := h.carriesStopped(id); len(without) > 0 {
said := fmt.Sprintf("its walk %s is not let go: it carries %s, whose delivery was stopped, and is asked to "+
"leave it out (novox/hq ADR 0299)", id, strings.Join(without, ", "))
if r := h.leaveRefused[id]; r != "" {
said += "; the controller refused: " + r
}
held[id] = said
}
}
// What the waiting deliveries say: `waits`, and `stalled` past their bound.
for _, d := range h.deliveries {
said := ""
if d.Walk != nil && !d.State.Final() {
said = held[d.Walk.ID]
}
if d.HeldBack != said {
d.HeldBack = said
h.keep(d)
}
}
for id, seen := range h.walks {
without, why := h.carriesStopped(id)
if len(without) == 0 {
delete(h.leaveRefused, id)
continue
}
// Once per record of it: the record that answers an ask is newer. Asked again after five minutes, should
// the ask have gone nowhere.
if a, asked := h.leaveAsked[id]; asked && seen.walk.Revision <= a.revision && now.Sub(a.at) < 5*time.Minute {
continue
}
walk := id
asks = append(asks, func() { h.Logf("[mesh-delivery] %s", h.stopLeaving(walk, why, byOwner, without)) })
}
return asks
}
// leave gathers a delivery's merge to leave out of a walk (ADR 0299), for a caller holding the lock; true for the
// first of that walk, whose ask asks for all.
func (h *Holder) leave(walk string, d *Delivery) bool {
if h.leaving == nil {
h.leaving = map[string][]string{}
}
first := len(h.leaving[walk]) == 0
if m := d.Repository + "@" + d.MergedAs; d.MergedAs != "" && !containsString(h.leaving[walk], m) {
h.leaving[walk] = append(h.leaving[walk], m)
}
return first
}
// leaving459 takes the merges gathered to leave out of a walk, and whether the walk carries anything else: another
// merge it names, or a delivery not ended that follows it. Taken under the lock, outside any.
func (h *Holder) leaving459(walk string) ([]string, bool) {
h.mu.Lock()
defer h.mu.Unlock()
without := h.leaving[walk]
delete(h.leaving, walk)
return without, len(h.othersOn(walk, without)) > 0
}
// othersOn are what a walk carries besides the merges named: each delivery not ended that follows it, by id, and
// each merge it names that no such delivery holds, as `<repository>@<commit>`.
func (h *Holder) othersOn(walk string, without []string) []string {
out := map[string]bool{}
held := map[string]bool{}
for _, d := range h.deliveries {
if d.Walk == nil || d.Walk.ID != walk || d.MergedAs == "" {
continue
}
m := d.Repository + "@" + d.MergedAs
held[strings.ToLower(m)] = true
if !d.State.Final() && !containsString(without, m) {
out[d.ID] = true
}
}
if s, known := h.walks[walk]; known && s.walk.Delivery != nil {
for _, m := range s.walk.Delivery.Merges {
key := m.Repository + "@" + m.Commit
if !held[strings.ToLower(key)] && !slices.ContainsFunc(without, func(w string) bool { return strings.EqualFold(w, key) }) {
out[key] = true
}
}
}
keys := make([]string, 0, len(out))
for k := range out {
keys = append(keys, k)
}
slices.Sort(keys)
return keys
}
// againWords is what a stop that left merges out did, in a person's words: the walk that walks the others again,
// what it left out (the controller's word on what contains a stopped merge), what stays on the machines, and what
// will still deliver a stopped change (ADR 0299).
func againWords(a StopAnswer) string {
var left []string
for _, o := range a.LeftOut {
s := o.Repository + "@" + shortOf(o.Commit, 12)
switch {
case len(o.Shares) > 0:
s += " (moves " + strings.Join(o.Shares, ", ") + ", as " + shortOf(o.Contains, 12) + " does)"
case o.Contains != "":
s += " (contains " + shortOf(o.Contains, 12) + ")"
}
left = append(left, s)
}
var out string
switch {
case a.WalkedAgainBy != "":
out = fmt.Sprintf("%s walks its other merges again: %s; left out: %s", a.WalkedAgainBy,
strings.Join(a.WalkedAgain, ", "), orNone(strings.Join(left, ", ")))
case len(left) > 0:
out = "left out: " + strings.Join(left, ", ") + "; nothing else to walk"
default:
out = "nothing else to walk"
}
if len(a.Withheld) > 0 {
out += "; its walk withholds " + strings.Join(a.Withheld, ", ")
}
if len(a.Kept) > 0 {
var kept []string
for _, k := range a.Kept {
kept = append(kept, k.Module+" on "+strings.Join(k.Machines, ", "))
}
out += "; what the stopped walk already sent stays on the machines: " + strings.Join(kept, "; ")
}
if len(a.StillCarriedBy) > 0 {
out += "; still to deliver the stopped change when walked, a revert's to hold back: " +
strings.Join(a.StillCarriedBy, ", ")
}
return out
}
// GroupState is a group's state, derived from its members and its composed check — never set apart (novox/hq
@@ -0,0 +1,95 @@
package main
import (
"testing"
"time"
)
// novox/hq issue 403, ADR 0313: the issue tracker measures a fix's time from the move to fixing to delivered and
// splits it at the pull request's opening and its merge. A transition says what it reads them by: when the pull
// request was first heard (its first head, not a later one), when it merged, and the class of what it moves —
// core, leaf, or none for a change that moves no module. `checks` says the same for a fix the tracker hears of late.
// transitionsTo are the transition events said for a delivery that moved to a state.
func transitionsTo(w *world, id string, to State) []map[string]any {
var out []map[string]any
for _, e := range w.said {
if e["event-kind"] == "transition" && e["id"] == id && e["to"] == string(to) {
out = append(out, e)
}
}
return out
}
func TestATransitionSaysWhenItsPullRequestOpenedMergedAndItsClass(t *testing.T) {
w := newWorld(t)
repo := "novox/mesh-catalog"
opened := w.now
first := pr(repo, 9, "head9first", "fix/403-x", "modules/app/a.go")
w.h.PullUpdated(first)
w.later(3 * time.Minute)
second := pr(repo, 9, "head9second", "fix/403-x", "modules/app/a.go")
w.h.PullUpdated(second)
c := verdictMoving(repo, 9, "head9second", "app")
c.Plan.Class = "core"
w.h.Checked(c)
w.later(2 * time.Minute)
mergedWhen := w.now
w.h.PullMerged(mergedAt(second, "merge9aaaaaa", mergedWhen))
walk := aWalk("plan-merge9", repo, "merge9aaaaaa", false)
walk.Delivery = &WalkDelivery{Merges: []WalkMerge{{Repository: repo, Commit: "merge9aaaaaa"}}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
walk = w.ctl.walk(walk.ID)
walk.State, walk.Revision = walkDone, walk.Revision+1
walk.Modules = map[string]*WalkModule{"app": {State: "built"}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
id := IDOf(repo, "head9second")
if s := w.state(id); s != Delivered {
t.Fatalf("the merge is %s, not delivered", s)
}
said := transitionsTo(w, id, Delivered)
if len(said) == 0 {
t.Fatalf("no transition to delivered was said: %v", w.said)
}
e := said[len(said)-1]
if e["opened"] != opened.UTC().Format(time.RFC3339) {
t.Errorf("opened %v, want the first head's %s (not the second head's)", e["opened"], opened.Format(time.RFC3339))
}
if e["merged_at"] != mergedWhen.UTC().Format(time.RFC3339) {
t.Errorf("merged_at %v, want %s", e["merged_at"], mergedWhen.Format(time.RFC3339))
}
if e["class"] != "core" {
t.Errorf("class %v, want the plan's core", e["class"])
}
a, err := w.h.Checks(repo, 9, "")
if err != nil {
t.Fatal(err)
}
if a.Delivery == nil || a.Delivery.Opened != opened.UTC().Format(time.RFC3339) || a.Delivery.Class != "core" ||
a.Delivery.MergedAt != mergedWhen.UTC().Format(time.RFC3339) {
t.Errorf("checks says %+v", a.Delivery)
}
}
// A plan that moves no module is class none, so the tracker gives its fix no budget; a plan from a controller that
// says no class falls back to its walk's class once known, and says nothing before.
func TestAFixClassIsNoneWhenItMovesNothing(t *testing.T) {
for _, c := range []struct {
why string
d Delivery
want string
}{
{"the planner's class", Delivery{Plan: &DeliveryPlan{Moved: []string{"app"}, Class: "leaf"}}, "leaf"},
{"a plan that moves nothing, no class said", Delivery{Plan: planMoving("novox/x")}, "none"},
{"no class said, its walk's known", Delivery{Plan: planMoving("novox/x", "app"), Time: &DeliveryTime{Class: "core"}}, "core"},
{"no class said, nothing known", Delivery{Plan: planMoving("novox/x", "app")}, ""},
} {
if got := fixClass(&c.d); got != c.want {
t.Errorf("%s: %q, want %q", c.why, got, c.want)
}
}
}
@@ -0,0 +1,105 @@
package main
import (
"testing"
"time"
)
// triedSaid is delivery_status's count of owed effects tried (novox/hq issue 407), read as a test reads it.
func triedSaid(t *testing.T, w *world) map[string]any {
t.Helper()
got, ok := w.h.Status("listening")["owed-tried"].(map[string]any)
if !ok {
t.Fatalf("delivery_status says no owed-tried: %v", w.h.Status("listening"))
}
return got
}
func countOf(t *testing.T, m map[string]any, key string) int {
t.Helper()
n, ok := m[key].(int)
if !ok {
t.Fatalf("owed-tried has no count %q: %v", key, m)
}
return n
}
// novox/hq issue 407: delivery_status says how many owed effects were tried since this owner started, how
// many were done, failed to be tried again, or were given up, and when the last was tried — so a window's
// writes can be checked against the tries made in it (issue 326's rule) by reading it at both ends.
func TestDeliveryStatusCountsTheOwedEffectsTried(t *testing.T) {
w := newWorld(t)
start := w.now
tried := triedSaid(t, w)
if countOf(t, tried, "tried") != 0 || countOf(t, tried, "done") != 0 || countOf(t, tried, "failed") != 0 ||
countOf(t, tried, "given-up") != 0 {
t.Fatalf("a holder that tried nothing says %v", tried)
}
if tried["since"] != start.Format(time.RFC3339) {
t.Fatalf("its count is since %v, the holder started %s", tried["since"], start.Format(time.RFC3339))
}
if _, there := tried["last"]; there {
t.Fatalf("nothing tried, yet a last try is said: %v", tried)
}
// A head seen: its event and its note are owed and done.
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
tried = triedSaid(t, w)
done := countOf(t, tried, "done")
if done < len(w.said)+len(w.forge.notesOn(head)) || done == 0 {
t.Fatalf("said %d and noted %d, yet done counts %d", len(w.said), len(w.forge.notesOn(head)), done)
}
if countOf(t, tried, "tried") != done || countOf(t, tried, "failed") != 0 {
t.Fatalf("every try succeeded, yet %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
// The forge away: its effects fail and are counted failed; quiet passes try nothing more.
w.forge.down = true
w.later(time.Minute)
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.Flush()
tried = triedSaid(t, w)
failed := countOf(t, tried, "failed")
if failed == 0 {
t.Fatalf("the forge away failed nothing: %v", tried)
}
if countOf(t, tried, "tried") != countOf(t, tried, "done")+failed+countOf(t, tried, "given-up") {
t.Fatalf("tries do not add up: %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
before := countOf(t, tried, "tried")
w.h.Flush() // before each failure's next try: nothing is tried
if n := countOf(t, triedSaid(t, w), "tried"); n != before {
t.Fatalf("a pass before any next try tried %d", n-before)
}
// The forge back, past every next try: the failed effects are done.
w.forge.down = false
w.later(time.Hour)
w.h.Flush()
tried = triedSaid(t, w)
if countOf(t, tried, "tried") != before+failed || countOf(t, tried, "done") <= done {
t.Fatalf("the forge back: tried %d (was %d, %d failed), %v", countOf(t, tried, "tried"), before, failed, tried)
}
}
// The count is this run's: a restarted holder says it counts since its own start, from nothing.
func TestOwedTriedCountsSinceTheStart(t *testing.T) {
w := newWorld(t)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
if countOf(t, triedSaid(t, w), "tried") == 0 {
t.Fatal("nothing counted before the restart")
}
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Minute))
tried := triedSaid(t, again)
if countOf(t, tried, "tried") != 0 || tried["since"] != again.now.Format(time.RFC3339) {
t.Fatalf("a restarted holder says %v", tried)
}
}
@@ -0,0 +1,48 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
)
// **An unanswered line says the state of the delivery it holds, as data** (novox/hq issue 451): for a delivery in
// ready or rejected, whose check answered once, the controller advises a recheck rather than the build queue, and
// it reads that state from the line, never from its words. A head with no delivery held says none.
func TestAnUnansweredLineNamesTheStateOfItsDelivery(t *testing.T) {
at := time.Date(2026, 10, 11, 0, 43, 0, 0, time.UTC)
for _, c := range []struct {
gate string
want State
}{{"pass", Ready}, {"fail", Rejected}} {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
id := heldIn(t, w, c.gate, c.want)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].ID != id || lines[0].State != Unanswered || lines[0].Held != c.want {
t.Fatalf("a %s delivery's unanswered line does not say its state as data: %+v", c.want, lines)
}
raw, _ := json.Marshal(lines[0])
if !strings.Contains(string(raw), `"held":"`+string(c.want)+`"`) {
t.Fatalf("the line the controller reads carries no held state: %s", raw)
}
}
// The same head with no delivery held here: unanswered, and no state to say.
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].State != Unanswered || lines[0].Held != "" {
t.Fatalf("a head with no delivery held says a state: %+v", lines)
}
if raw, _ := json.Marshal(lines[0]); strings.Contains(string(raw), `"held"`) {
t.Fatalf("a line with no delivery held carries a held field: %s", raw)
}
}
@@ -0,0 +1,475 @@
package main
import (
"fmt"
"strings"
"testing"
"time"
)
// novox/hq issue 459, ADR 0299: since the merge window (ADR 0276) one walk carries every merge of its batch. A group
// stop asked the controller to end the stopped member's walk, and with it every other delivery on it, which no one
// stopped. Now a group stop asks `delivery-stop` with the stopped members' merges to leave out: the walk ends, and
// the controller walks its other merges again in a fresh batch; a later merge of a stopped merge's repository
// contains it and ends with it, named. A person's stop of a delivery on a shared walk answers first with what
// ending the walk would take down, and acts only on a way given: `again` or `walk`.
// shape459 is a world of a delivery group feat/g (its members in the repositories group names, in that order) and
// lone deliveries (the repositories alone names), each checked and merged. The group's first member is on a walk of
// its own, plan-a; every other merge is carried by one walk, plan-2, waiting for its word.
type shape459 struct {
*world
ids, heads, merges map[string]string // by repository
}
func open459(t *testing.T, group, alone []string) *shape459 {
t.Helper()
w := newWorld(t)
s := &shape459{world: w, ids: map[string]string{}, heads: map[string]string{}, merges: map[string]string{}}
n := 0
add := func(repo, branch string) {
n++
head := fmt.Sprintf("%02d%s", n, strings.Repeat("h", 14))
s.ids[repo], s.heads[repo] = IDOf(repo, head), head
s.merges[repo] = fmt.Sprintf("%02d%s", n, strings.Repeat("m", 10))
w.h.PullUpdated(pr(repo, n, head, branch, "x"))
w.h.Checked(verdict(repo, n, head, "pass"))
}
for _, r := range group {
add(r, "feat/g")
}
for _, r := range alone {
add(r, "fix/"+r[strings.LastIndex(r, "/")+1:])
}
w.settleAll()
w.h.Checked(CheckedEvent{Group: "feat/g", ID: "check-1", Verdict: "pass", Summary: "compose"})
w.settleAll()
for i, r := range append(append([]string{}, group...), alone...) {
w.h.PullMerged(merged(r, i+1, s.heads[r], s.merges[r]))
}
first := aWalk("plan-a", group[0], s.merges[group[0]], true)
first.Delivery.Merges = []WalkMerge{{Repository: group[0], Commit: s.merges[group[0]]}}
w.h.WalkMoved(w.ctl.put(first))
two := aWalk("plan-2", alone[0], s.merges[alone[0]], true)
two.Created = two.Created.Add(time.Minute)
for _, r := range append(append([]string{}, group[1:]...), alone...) {
two.Delivery.Merges = append(two.Delivery.Merges, WalkMerge{Repository: r, Commit: s.merges[r]})
}
w.h.WalkMoved(w.ctl.put(two))
w.settleAll()
return s
}
// failFirst fails the group's first member's walk: the members after it are stopped by their group.
func (s *shape459) failFirst() {
a := s.ctl.walk("plan-a")
a.State, a.Note = walkFailed, "the first member failed its gate on anchor"
s.h.WalkMoved(s.ctl.put(a))
s.settleAll()
}
// heard is the controller's records of the stopped walk and the walk that walks its merges again, as plan-moved says
// them, and then a walk done.
func (s *shape459) heard(walks ...string) {
for _, id := range walks {
s.h.WalkMoved(s.ctl.walk(id))
}
s.settleAll()
}
// cutAgain is the controller cutting the batch that walks a stopped walk's merges again, as plan-moved says it.
func (s *shape459) cutAgain(id string) {
s.h.WalkMoved(s.ctl.cut(id))
s.settleAll()
}
func (s *shape459) done(walk string) {
r := s.ctl.walk(walk)
sent := s.now
for m := range r.Modules {
r.Modules[m] = &WalkModule{State: "built", SentAt: &sent}
}
r.State = walkDone
s.h.WalkMoved(s.ctl.put(r))
s.settleAll()
}
func lastWhy(d *Delivery) string {
if len(d.Transitions) == 0 {
return ""
}
return d.Transitions[len(d.Transitions)-1].Why
}
// A group member stopped on a walk it shares with a lone delivery: the walk is stopped without the member's merge,
// never whole, and the lone delivery follows the walk that walks its merge again and is delivered by it.
func TestAGroupStopWalksTheOtherDeliveriesAgainWithoutIt(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
if containsString(s.ctl.delivers, "plan-2") {
t.Fatalf("plan-2 was let go before b's turn: %v", s.ctl.delivers)
}
s.failFirst()
if b := s.delivery(s.ids["novox/b"]); b.State != Stopped || !strings.Contains(lastWhy(b), s.ids["novox/a"]) {
t.Fatalf("b is %s (%q); want it stopped by its group, naming a", b.State, lastWhy(b))
}
if containsString(s.ctl.stops, "plan-2") {
t.Fatalf("plan-2 was ended whole %v, and with it z, which is in no group and was not stopped", s.ctl.stops)
}
if want := "plan-2 without novox/b@" + s.merges["novox/b"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
s.heard("plan-2", "plan-2-again")
if containsString(s.ctl.delivers, "plan-2-again") {
t.Fatal("a batch not yet cut was given its word")
}
s.cutAgain("plan-2-again")
z := s.delivery(s.ids["novox/z"])
if z.State.Final() || z.Walk == nil || z.Walk.ID != "plan-2-again" || !containsString(s.ctl.delivers, "plan-2-again") {
t.Fatalf("z is %s on %+v, let go %v; want it following plan-2-again, let go", z.State, z.Walk, s.ctl.delivers)
}
s.done("plan-2-again")
if z, b := s.state(s.ids["novox/z"]), s.state(s.ids["novox/b"]); z != Delivered || b != Stopped {
t.Fatalf("once plan-2-again is done, z is %s and b %s; want z delivered and b stopped", z, b)
}
}
// Two members after the failed one, both on the shared walk: one ask leaves both out, so neither is walked again
// because its own stop came second.
func TestAGroupStopLeavesOutEveryMemberItStopsOnOneWalk(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b", "novox/c"}, []string{"novox/z"})
s.failFirst()
if len(s.ctl.withouts) != 1 || !strings.Contains(s.ctl.withouts[0], "novox/b@"+s.merges["novox/b"]) ||
!strings.Contains(s.ctl.withouts[0], "novox/c@"+s.merges["novox/c"]) {
t.Fatalf("the controller was asked %v; want one stop of plan-2 leaving out b and c", s.ctl.withouts)
}
s.heard("plan-2", "plan-2-again")
again := s.ctl.walk("plan-2-again")
if len(again.Delivery.Merges) != 1 || again.Delivery.Merges[0].Repository != "novox/z" {
t.Fatalf("plan-2-again walks %+v; want z's merge alone", again.Delivery.Merges)
}
for _, r := range []string{"novox/b", "novox/c"} {
if st := s.state(s.ids[r]); st != Stopped {
t.Fatalf("%s is %s", r, st)
}
}
}
// A lone delivery whose merge is a later merge of the stopped member's repository contains the member's merge: it
// is left out and ends stopped, naming the merge it contains; the other lone delivery is walked again.
func TestAGroupStopEndsAMergeThatContainsTheStoppedOne(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z", "novox/b2"})
// b2 is a later merge of b's repository: the walk carries it for b.
s.retarget("novox/b2", "novox/b")
s.failFirst()
s.heard("plan-2", "plan-2-again")
s.cutAgain("plan-2-again")
b2 := s.delivery(s.ids["novox/b2"])
if b2.State != Stopped || !strings.Contains(lastWhy(b2), shortOf(s.merges["novox/b"], 8)) ||
!strings.Contains(lastWhy(b2), "contains") {
t.Fatalf("b2 is %s (%q); want it stopped, naming b's merge it contains", b2.State, lastWhy(b2))
}
if z := s.delivery(s.ids["novox/z"]); z.Walk == nil || z.Walk.ID != "plan-2-again" || z.State.Final() {
t.Fatalf("z is %s on %+v; want it following plan-2-again", z.State, z.Walk)
}
}
// retarget makes the delivery of repo `later` a later merge of repository `of`: same repository, carried by plan-2
// for the merge of `of`.
func (s *shape459) retarget(later, of string) {
s.t.Helper()
s.h.mu.Lock()
d := s.h.deliveries[s.ids[later]]
d.Repository, d.MergedAt = of, d.MergedAt.Add(time.Minute)
s.h.mu.Unlock()
s.ctl.mu.Lock()
s.ctl.mergedAt[s.merges[of]], s.ctl.mergedAt[s.merges[later]] = d.MergedAt.Add(-time.Minute), d.MergedAt
s.ctl.mu.Unlock()
two := s.ctl.walk("plan-2")
merges := []WalkMerge{}
for _, m := range two.Delivery.Merges {
switch m.Repository {
case of:
m.Carried = s.merges[later]
case later:
m.Repository = of
}
merges = append(merges, m)
}
wd := *two.Delivery
wd.Merges = merges
two.Delivery = &wd
s.h.WalkMoved(s.ctl.put(two))
s.settleAll()
}
// shared is two lone deliveries, y and z, let go together on one walk, plan-2.
func shared459(t *testing.T) *shape459 {
t.Helper()
s := open459(t, []string{"novox/a"}, []string{"novox/y", "novox/z"})
for _, r := range []string{"novox/y", "novox/z"} {
if st := s.state(s.ids[r]); st != Delivering {
t.Fatalf("%s is %s, let go %v; want both delivering on plan-2", r, st, s.ctl.delivers)
}
}
return s
}
// A person's stop of y, which shares its walk with z, does nothing at first: it names z and the two ways.
func TestAPersonsStopOfASharedWalkAsksFirst(t *testing.T) {
s := shared459(t)
said, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "")
if err == nil {
t.Fatalf("the stop acted at once: %q", said)
}
for _, want := range []string{s.ids["novox/z"], "plan-2", "again", "walk", "not stopped"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the stop said %q; want it to name %q", err, want)
}
}
if st := s.state(s.ids["novox/y"]); st != Delivering || len(s.ctl.stops)+len(s.ctl.withouts) != 0 {
t.Fatalf("y is %s, the controller was asked %v %v; want nothing done", st, s.ctl.stops, s.ctl.withouts)
}
if _, err := s.h.StopWay(s.ids["novox/y"], "x", "jochen", "sideways"); err == nil {
t.Fatal("a way that is neither again nor walk was taken")
}
}
// With `again`, y is stopped, the walk is stopped without y's merge, and z goes back to published and follows the
// walk that walks it again.
func TestAPersonsStopAgainWalksTheOthersAgain(t *testing.T) {
s := shared459(t)
said, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "plan-2-again") || !strings.Contains(said, "novox/z@") {
t.Fatalf("the stop said %q; want it to name the walk that walks z again", said)
}
if want := "plan-2 without novox/y@" + s.merges["novox/y"]; !containsString(s.ctl.withouts, want) ||
containsString(s.ctl.stops, "plan-2") {
t.Fatalf("the controller was asked %v, ended whole %v; want %q", s.ctl.withouts, s.ctl.stops, want)
}
s.heard("plan-2")
if z := s.delivery(s.ids["novox/z"]); z.State != Published {
t.Fatalf("z is %s once plan-2 was stopped and walked again; want published, waiting for that walk", z.State)
}
s.heard("plan-2-again")
s.cutAgain("plan-2-again")
if z := s.delivery(s.ids["novox/z"]); z.Walk == nil || z.Walk.ID != "plan-2-again" || z.State != Delivering {
t.Fatalf("z is %s on %+v; want it delivering on plan-2-again", z.State, z.Walk)
}
if st := s.state(s.ids["novox/y"]); st != Stopped {
t.Fatalf("y is %s", st)
}
}
// With `walk`, the whole walk is stopped, and z ends stopped, naming y's stop.
func TestAPersonsStopOfTheWalkStopsEveryDelivery(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "walk"); err != nil {
t.Fatal(err)
}
if !containsString(s.ctl.stops, "plan-2") || len(s.ctl.withouts) != 0 {
t.Fatalf("the controller was asked %v %v; want plan-2 ended whole", s.ctl.stops, s.ctl.withouts)
}
s.heard("plan-2")
z := s.delivery(s.ids["novox/z"])
if z.State != Stopped || !strings.Contains(lastWhy(z), s.ids["novox/y"]) {
t.Fatalf("z is %s (%q); want it stopped, naming y's stop", z.State, lastWhy(z))
}
}
// The reviewer's case (review of #221): group [a, b]; y and z ride plan-2 with b. y is stopped `way: again`, and the
// batch that walks plan-2's others again carries b and z. Then a fails: b is stopped by its group, and its merge is
// left out of that batch — never shipped by it — and z alone is walked and delivered.
func TestAMergeWalkedAgainAndThenStoppedByItsGroupNeverShips(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/y", "novox/z"})
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.heard("plan-2", "plan-2-again")
s.failFirst()
if b := s.state(s.ids["novox/b"]); b != Stopped {
t.Fatalf("b is %s", b)
}
if want := "plan-2-again without novox/b@" + s.merges["novox/b"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
s.heard("plan-2-again", "plan-2-again-left")
s.cutAgain("plan-2-again")
again := s.ctl.walk("plan-2-again")
for _, m := range again.Delivery.Merges {
if m.Repository == "novox/b" {
t.Fatalf("the walk %s still carries b's stopped merge: %+v", again.ID, again.Delivery.Merges)
}
}
if !containsString(s.ctl.delivers, "plan-2-again") {
t.Fatalf("z's walk was not let go: %v", s.ctl.delivers)
}
s.done("plan-2-again")
if z := s.state(s.ids["novox/z"]); z != Delivered {
t.Fatalf("z is %s", z)
}
}
// Whatever brought it there, a walk that carries a stopped delivery's merge is asked to leave it out, and is never
// given its word while it carries it.
func TestAWalkCarryingAStoppedMergeIsAskedToLeaveItOutAndNeverLetGo(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
// A walk the controller made that carries y's merge anyway, waiting for its word, with z on it.
x := aWalk("plan-x", "novox/z", s.merges["novox/z"], true)
x.Created = x.Created.Add(10 * time.Minute)
x.Delivery.Merges = []WalkMerge{{Repository: "novox/y", Commit: s.merges["novox/y"]},
{Repository: "novox/z", Commit: s.merges["novox/z"]}}
s.ctl.refuseWithout = true // it cannot be left out: the walk is stopped whole
s.h.WalkMoved(s.ctl.put(x))
s.settleAll()
if want := "plan-x without novox/y@" + s.merges["novox/y"]; !containsString(s.ctl.withouts, want) {
t.Fatalf("the controller was asked %v; want %q", s.ctl.withouts, want)
}
if containsString(s.ctl.delivers, "plan-x") {
t.Fatalf("a walk carrying a stopped merge was let go: %v", s.ctl.delivers)
}
if !containsString(s.ctl.stops, "plan-x") {
t.Fatalf("refused the leave-out, the walk was not stopped whole: %v", s.ctl.stops)
}
}
// A group stop the controller refuses to make without the merge (an older controller) stops the whole walk: the
// stopped change is held back, and the delivery riding with it ends stopped, saying why.
func TestARefusedLeaveOutStopsTheWholeWalk(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
s.ctl.refuseWithout = true
s.failFirst()
if len(s.ctl.withouts) != 1 || !containsString(s.ctl.stops, "plan-2") {
t.Fatalf("asked %v, ended whole %v; want the leave-out tried, then plan-2 ended whole", s.ctl.withouts, s.ctl.stops)
}
s.heard("plan-2")
if z := s.delivery(s.ids["novox/z"]); z.State != Stopped || !strings.Contains(lastWhy(z), "could not be left out") {
t.Fatalf("z is %s (%q); want it stopped, saying the leave-out was refused", z.State, lastWhy(z))
}
}
// A person's stop the controller cannot make leaves the delivery as it was, and says how to go on.
func TestAPersonsStopThatCannotBeMadeStopsNothing(t *testing.T) {
s := shared459(t)
s.ctl.refuseWithout = true
_, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again")
if err == nil || !strings.Contains(err.Error(), "is not stopped") || !strings.Contains(err.Error(), "plans stop") {
t.Fatalf("the refused stop said %v", err)
}
if st := s.state(s.ids["novox/y"]); st != Delivering {
t.Fatalf("y is %s; want it delivering, its stop not made", st)
}
}
// What a stop that left merges out says: the walk that walks the others, what it left out and what each contains,
// as the controller says it, what stays on which machines, and what will still deliver the stopped change.
func TestAStopSaysWhatStaysAndWhatStillCarriesIt(t *testing.T) {
said := againWords(StopAnswer{Stopped: "plan-2", WalkedAgainBy: "plan-3", WalkedAgain: []string{"novox/z@zz"},
LeftOut: []WalkLeftOut{{Repository: "novox/b", Commit: "b1b1b1b1b1b1"},
{Repository: "novox/b", Commit: "b2b2b2b2b2b2", Contains: "b1b1b1b1b1b1"}},
Kept: []WalkKept{{Module: "lab", Machines: []string{"anchor"}}},
StillCarriedBy: []string{"novox/b@b3b3 (in plan-4)"}})
for _, want := range []string{"plan-3", "novox/z@zz", "b2b2b2b2b2b2 (contains b1b1b1b1b1b1)", "lab on anchor",
"stays on the machines", "novox/b@b3b3 (in plan-4)", "revert"} {
if !strings.Contains(said, want) {
t.Fatalf("the stop said %q; want %q in it", said, want)
}
}
}
// A merge left out for moving a module the stopped merge moves, from another repository, ends stopped saying so —
// never that it contains the stopped merge (review of #221).
func TestALeftOutMergeSharingAModuleSaysSoTruly(t *testing.T) {
s := open459(t, []string{"novox/a", "novox/b"}, []string{"novox/z"})
s.ctl.mu.Lock()
s.ctl.moves[s.merges["novox/b"]], s.ctl.moves[s.merges["novox/z"]] = []string{"lab"}, []string{"lab"}
s.ctl.mu.Unlock()
s.failFirst()
s.heard("plan-2")
z := s.delivery(s.ids["novox/z"])
why := lastWhy(z)
if z.State != Stopped || !strings.Contains(why, "moves lab, as the stopped merge "+shortOf(s.merges["novox/b"], 12)) ||
strings.Contains(why, "contains the stopped merge") {
t.Fatalf("z is %s (%q); want it stopped, saying it moves lab as b's merge does", z.State, why)
}
}
// A person's first answer names the deliveries that would be left out for moving a module this merge moves, and says
// the controller may leave out more.
func TestAPersonsFirstAnswerNamesWhatSharesItsModules(t *testing.T) {
s := shared459(t)
_, err := s.h.StopWay(s.ids["novox/y"], "x", "jochen", "")
if err == nil || !strings.Contains(err.Error(), "for moving a module this merge moves: "+s.ids["novox/z"]+" (moves app)") ||
!strings.Contains(err.Error(), "may leave out more") {
t.Fatalf("the first answer said %v", err)
}
}
// The safety net holds back only what a person or a group stopped: a delivery ended by what its walk did (the
// observed rows, which name nobody) is not asked out of a walk that carries its merge.
func TestTheSafetyNetLeavesAStrandedStopAlone(t *testing.T) {
s := shared459(t)
s.h.mu.Lock()
y := s.h.deliveries[s.ids["novox/y"]]
y.State = Stopped
y.Transitions = append(y.Transitions, Transition{At: s.now, From: Published, To: Stopped, Event: EvStop,
Why: "its walk ended done before its word"})
s.h.mu.Unlock()
s.settleAll()
if len(s.ctl.withouts) != 0 {
t.Fatalf("a stop nobody asked for was held back: %v", s.ctl.withouts)
}
}
// A walk the safety net holds — its leave-out and its whole stop both refused — says why on the deliveries waiting on
// it, in `waits`, and in `stalled` past their bound.
func TestAHeldWalkIsSaidOnItsDeliveriesAndStalls(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.ctl.refuseWithout, s.ctl.refuseStop = true, true
x := aWalk("plan-x", "novox/z", s.merges["novox/z"], true)
x.Created = x.Created.Add(10 * time.Minute)
x.Delivery.Merges = []WalkMerge{{Repository: "novox/y", Commit: s.merges["novox/y"]},
{Repository: "novox/z", Commit: s.merges["novox/z"]}}
s.h.WalkMoved(s.ctl.put(x))
s.settleAll()
z := s.delivery(s.ids["novox/z"])
if !strings.Contains(z.HeldBack, "plan-x") || !strings.Contains(z.HeldBack, "refused") {
t.Fatalf("z says it is held back by %q; want plan-x and the refusal", z.HeldBack)
}
if w := s.h.waitsFor(z); w != z.HeldBack {
t.Fatalf("z waits for %q", w)
}
s.later(Bounds[Published].For + time.Minute)
s.settleAll()
for _, l := range s.h.Stalled() {
if l.ID == z.ID && strings.Contains(l.Says, "plan-x") {
return
}
}
t.Fatalf("z is not listed stalled naming plan-x: %+v", s.h.Stalled())
}
// A person's stop of a delivery whose merge waits in a batch asks that batch now to leave it out, and says so.
func TestAPersonsStopOfAMergeInABatchAsksTheBatch(t *testing.T) {
s := shared459(t)
if _, err := s.h.StopWay(s.ids["novox/y"], "not wanted after all", "jochen", "again"); err != nil {
t.Fatal(err)
}
s.heard("plan-2", "plan-2-again")
said, err := s.h.StopWay(s.ids["novox/z"], "this one too", "jochen", "")
if err != nil {
t.Fatal(err)
}
if want := "plan-2-again without novox/z@" + s.merges["novox/z"]; !containsString(s.ctl.withouts, want) ||
!strings.Contains(said, "batch plan-2-again") {
t.Fatalf("said %q, asked %v; want the batch asked now, and said", said, s.ctl.withouts)
}
}
@@ -0,0 +1,134 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 480: a delivery stopped because its walk ended done before its word without building its modules
// (issue 474) is still read for a later merge that delivers them, and kept CoveredBy once one did. Its state stays
// stopped — a final state never moves, as for a covered superseded member (issue 442) — but the forge's status, its
// view, its note and `show` went on saying "stopped" for a change that reached every machine. Once covered they say
// covered, and by which merge. And while `stalled` lists such a stop, `deliveries` lists it too, past a day.
// strandedNight is theNight with #215 stopped, its walk past the grace with nothing later carrying build-agent.
func strandedNight(t *testing.T) (*world, string) {
w := theNight(t)
id := IDOf(ctl474, head215)
w.later(strandedAfter + 5*time.Minute)
w.settleAll()
if d := w.delivery(id); d.State != Stopped || d.Stranded == "" {
t.Fatalf("#215 is %s, stranded %q", d.State, d.Stranded)
}
return w, id
}
func TestAStopALaterMergeCoversReadsCoveredByThatMerge(t *testing.T) {
w, id := strandedNight(t)
if got := statusOn(w, ctl474, head215); !strings.HasPrefix(got, "error stopped: ") {
t.Fatalf("stopped, its status says %q", got)
}
w.forge.contains[merge215+" in "+merge219] = true
by := deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
d := w.delivery(id)
if d.State != Stopped || !strings.Contains(d.CoveredBy, by) {
t.Fatalf("after #219 it is %s, covered by %q", d.State, d.CoveredBy)
}
got := statusOn(w, ctl474, head215)
if !strings.HasPrefix(got, "success covered") || !strings.Contains(got, by) {
t.Fatalf("covered by %s, its status still says %q", by, got)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
if reads, _ := shown.(map[string]any)["reads"].(string); !strings.HasPrefix(reads, "covered") || !strings.Contains(reads, by) {
t.Fatalf("show reads %q", reads)
}
view := w.forge.views[ctl474+"#215"]
if !strings.Contains(view, "Covered") || !strings.Contains(view, by) {
t.Fatalf("its view does not say it was covered by %s:\n%s", by, view)
}
noted := false
for _, l := range w.forge.notesOn(merge215) {
noted = noted || strings.Contains(l, "covered") && strings.Contains(l, by)
}
if !noted {
t.Fatalf("its merge's note does not say it was covered: %v", w.forge.notesOn(merge215))
}
// Told once: further ticks owe nothing more.
w.later(time.Hour)
w.settleAll()
if d := w.delivery(id); len(d.Owed) != 0 || !d.CoverTold {
t.Fatalf("still owes %+v, told %v", d.Owed, d.CoverTold)
}
}
// A stop covered before this was built (CoveredBy kept, nothing told) is told after a restart, from what was kept.
func TestACoveredStopKeptBeforeIsToldAfterARestart(t *testing.T) {
w, id := strandedNight(t)
w.h.mu.Lock()
d := w.h.deliveries[id]
d.CoveredBy = IDOf(ctl474, head219) + " (merged as " + merge219[:8] + ")"
if err := w.store.PutDelivery(d); err != nil {
t.Fatal(err)
}
w.h.mu.Unlock()
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Hour))
again.settleAll()
if got := statusOn(again, ctl474, head215); !strings.HasPrefix(got, "success covered") ||
!strings.Contains(got, IDOf(ctl474, head219)) {
t.Fatalf("after a restart its status says %q", got)
}
if d := again.delivery(id); d.State != Stopped || len(d.Transitions) != len(w.delivery(id).Transitions) {
t.Fatalf("a restart moved it: %s", d.State)
}
}
// Only a stop for a walk that never built its modules reads covered: a person's stop, or one the cover test has
// not answered, keeps its error.
func TestOnlyACoveredStrandedStopReadsCovered(t *testing.T) {
for _, d := range []*Delivery{
{State: Stopped, CoveredBy: "x"}, // a person's stop, with a cover found while it was held
{State: Stopped, Stranded: "never reached"}, // stranded, not covered
} {
if state, _ := statusOf(d); state != "error" {
t.Fatalf("%+v reads %s", d, state)
}
if d.coveredStop() {
t.Fatalf("%+v reads as a covered stop", d)
}
}
}
func TestAStopStalledListsStaysInDeliveries(t *testing.T) {
w, id := strandedNight(t)
w.later(30 * time.Hour)
w.settleAll()
listed := func() *Line {
for _, l := range w.h.Deliveries("", "", "", false) {
if l.ID == id {
return &l
}
}
return nil
}
if len(w.h.Stalled()) != 1 {
t.Fatalf("stalled lists %+v", w.h.Stalled())
}
l := listed()
if l == nil || !l.Stalled || !strings.Contains(l.Waits, "build-agent") {
t.Fatalf("listed by stalled, deliveries shows %+v", l)
}
// Covered, it ends like any final delivery: a day after its stop.
w.forge.contains[merge215+" in "+merge219] = true
deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
if len(w.h.Stalled()) != 0 || listed() != nil {
t.Fatalf("covered, stalled lists %+v and deliveries %+v", w.h.Stalled(), listed())
}
}
@@ -0,0 +1,142 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 362's note of 2026-10-11 (the nzbgeek case) and issue 529: mesh-media-catalog #18 added a module
// whose delivery plan said "sent nowhere". This owner ended its delivery at once, "nothing for a walk to move", but
// the controller opened a walk for the merge that moved modules to a machine and waited for this owner's word.
// With its delivery ended, nothing gave the walk its word: it held 39 minutes and two walks queued behind it.
const (
media = "novox/mesh-media-catalog"
headNzb = "abababab00001818"
mergeNzb = "cdcdcdcd00001818"
nzbWalkID = "plan-1791748193836841649"
)
var atNzb = time.Date(2026, 10, 11, 19, 43, 0, 0, time.UTC)
// nzbgeekMerged is #18 checked on a plan that builds a new module and sends it nowhere, and merged: its delivery
// ends at once, nothing for a walk to move.
func nzbgeekMerged(t *testing.T) *world {
t.Helper()
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb.Add(-time.Minute))
p := pr(media, 18, headNzb, "feat/nzbgeek", "modules/nzbgeek/module.json")
w.h.PullUpdated(p)
v := verdict(media, 18, headNzb, "pass")
v.Plan = &DeliveryPlan{Repository: media, Base: "main", New: []string{"modules/nzbgeek"},
Summary: "builds new: modules/nzbgeek, sent nowhere; no bus step"}
w.h.Checked(v)
w.later(time.Minute)
w.h.PullMerged(mergedAt(p, mergeNzb, atNzb))
w.settleAll()
if d := w.delivery(IDOf(media, headNzb)); d.State != Delivered {
t.Fatalf("#18 is %s, not delivered as nothing for a walk to move", d.State)
}
return w
}
// nzbWalk is the walk the controller cut for the merge's batch: it names the merge and moves nzbgeek and nzbget.
func nzbWalk(created time.Time, merges ...WalkMerge) Walk {
w := Walk{ID: nzbWalkID, Repository: media, Branch: "main", Commit: mergeNzb, Created: created, State: walkBuilding,
Tiers: [][]string{{"nzbgeek", "nzbget"}}, Modules: map[string]*WalkModule{"nzbgeek": {}, "nzbget": {}},
Delivery: &WalkDelivery{Awaits: "mesh-delivery", Merges: merges}}
return w
}
// **A walk answering only merges that ended as nothing for a walk to move is named as held by nothing** (novox/hq
// issues 362 and 529): never let go by this owner — nothing judged what it moves — and said, by `stalled` with
// `walks`, with why, so the controller's `plans` says so instead of pointing nowhere.
func TestAWalkOfNothingToMoveMergesIsNamedAsHeldByNothing(t *testing.T) {
w := nzbgeekMerged(t)
w.later(90 * time.Second)
w.h.WalkMoved(w.ctl.put(nzbWalk(w.now, WalkMerge{Repository: media, Commit: mergeNzb})))
w.settleAll()
if len(w.ctl.delivers) != 0 {
t.Fatalf("a walk no live delivery holds was let go: %v", w.ctl.delivers)
}
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != nzbWalkID || lines[0].State != Unheld ||
!strings.Contains(lines[0].Says, "ended as nothing-to-move") || !strings.Contains(lines[0].Says, IDOf(media, headNzb)) ||
!strings.Contains(lines[0].Says, "nzbget") {
t.Fatalf("the walk is said as %+v", lines)
}
}
// **The status names each held delivery, why, and the verb that releases it** (novox/hq issue 529): "held: 1"
// alone pointed a reader nowhere.
func TestTheStatusNamesAHeldDeliveryAndWhy(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb)
p := pr(media, 22, headNzb, "adr-0307-state-history", "modules/nzbgeek/module.json")
w.h.PullUpdated(p)
w.h.PullMerged(mergedAt(p, mergeNzb, atNzb)) // merged before any verdict: held for a person
w.settleAll()
id := IDOf(media, headNzb)
d := w.delivery(id)
if d.State != Held {
t.Fatalf("#22 is %s", d.State)
}
held, _ := w.h.Status("listening")["held"].([]string)
if len(held) != 1 || !strings.Contains(held[0], id) || !strings.Contains(held[0], d.HeldWhy) ||
!strings.Contains(held[0], "mesh-delivery.release") {
t.Fatalf("the status says held %q", held)
}
}
// **A walk waiting for this owner's word is told why** (novox/hq issue 529): asked with `walks`, `stalled` says of
// each walk waiting for its word what holds it — a held delivery, naming it, its reason and `mesh-delivery.release`,
// or the deliveries not at their turn — so the controller's `plans` points at the person's decision, not at a hand
// start.
func TestAWaitingWalkIsToldWhatHoldsIt(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atDu.Add(5*time.Second))
cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json")
du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json")
w.h.PullUpdated(cc)
w.h.PullUpdated(du)
w.h.Checked(verdictMoving(cat, 101, headCC, "claude-code"))
w.h.PullMerged(mergedAt(cc, mergeCC, atCC))
w.h.PullMerged(mergedAt(du, mergeDu, atDu)) // unchecked: held for a person
walk := catWalk("plan-9", mergeDu, atDu.Add(5*time.Second), "claude-code", "dunst")
walk.Delivery.Merges = []WalkMerge{{Repository: cat, Commit: mergeCC, Carried: mergeDu}, {Repository: cat, Commit: mergeDu}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
idDu := IDOf(cat, headDu)
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != "plan-9" || lines[0].State != Held || lines[0].ID != idDu ||
!strings.Contains(lines[0].Says, w.delivery(idDu).HeldWhy) || !strings.Contains(lines[0].Says, "mesh-delivery.release") {
t.Fatalf("the waiting walk is said as %+v", lines)
}
// Not by default: `stalled` stays what the self-check raises conditions for.
for _, l := range w.h.Stalled() {
if l.Walk == "plan-9" {
t.Fatalf("stalled lists the waiting walk without being asked: %+v", l)
}
}
// Released, its word is asked: nothing holds it any more.
if _, err := w.h.Release(idDu, "checked by hand", "jochen"); err != nil {
t.Fatal(err)
}
w.h.WalkMoved(w.ctl.walk("plan-9"))
w.settleAll()
if lines := w.h.WalksWaiting(); len(lines) != 0 {
t.Fatalf("a walk let go is still said waiting: %+v", lines)
}
}
// A walk waiting while a delivery it carries is not at its turn says so, naming the delivery — what was said only
// in this owner's journal (novox/hq issue 529, note 2).
func TestAWaitingWalkSaysWhichDeliveryIsNotAtItsTurn(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atNzb)
w.h.mu.Lock()
w.h.waitsWhy = map[string]string{"plan-7": "novox/x@1 (published) carried by it not at its turn"}
w.h.walks["plan-7"] = seenWalk{walk: aWalk("plan-7", "novox/x", "1111", true), at: atNzb}
w.h.mu.Unlock()
lines := w.h.WalksWaiting()
if len(lines) != 1 || lines[0].Walk != "plan-7" || !strings.Contains(lines[0].Says, "not at its turn") {
t.Fatalf("the waiting walk is said as %+v", lines)
}
}
@@ -192,7 +192,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
seat := "mesh-delivery."
return []stdio.Tool{
{Name: seat + "deliveries",
Description: "Every delivery not final, and those that ended in the last day, one line each: its id " +
Description: "Every delivery not final, those that ended in the last day, and any stop `stalled` still lists, one line each: its id " +
"(owner/repository@commit), its state, its group, what it waits for and since when. Narrowed by state, " +
"repository or group.",
Input: map[string]any{"type": "object", "properties": map[string]any{"state": str("one state"),
@@ -205,7 +205,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: seat + "times",
Description: "Delivery time (novox/hq ADR 0282): from a merge to every machine of its walk running the build. " +
"Each class's budget (a leaf module five minutes, a core module ten), the last days' median and worst, " +
"Each class's budget (a leaf module three minutes, a core module ten), the last days' median and worst, " +
"how many within and over, the newest delivery of each class, every delivery over its budget with its " +
"longest phase, and the delivered ones whose time is not known and why. Given a delivery's id, its time " +
"phase by phase: window, queued, word, build, send-first, judgement, send-rest, between-tiers, apply — " +
@@ -268,8 +268,17 @@ func tools(h *Holder, l *listening) []stdio.Tool {
"and what healer H2 may do once it has passed; and the machine steps' table.",
Run: func(map[string]any) (any, error) { return TableText(), nil }},
{Name: seat + "stalled",
Description: "Every delivery held past its state's bound, with the transition the table lets healer H2 take.",
Run: func(map[string]any) (any, error) { return h.Stalled(), nil }},
Description: "Every delivery held past its state's bound, with the transition the table lets healer H2 take. " +
"With walks: true, instead each walk waiting for this owner's word and what holds it — a held delivery, " +
"its reason and the verb that releases it, or the deliveries not at their turn (novox/hq issue 529).",
Input: map[string]any{"type": "object", "properties": map[string]any{
"walks": map[string]any{"type": "boolean", "description": "the waiting walks and what holds each, instead"}}},
Run: func(a map[string]any) (any, error) {
if walks, _ := a["walks"].(bool); walks {
return h.WalksWaiting(), nil
}
return h.Stalled(), nil
}},
{Name: seat + "recheck",
Description: "Check a rejected or ready delivery again: its merge check statuses put back to pending on the forge, " +
"so nothing merges on the old verdict; proposed again, and its check asked. With why.",
@@ -293,14 +302,18 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: seat + "stop",
Description: "Stop a delivery that is not final, with why: its walk is ended through the controller, and " +
"in a delivering group every member after it is stopped too, naming it.",
"in a delivering group every member after it is stopped too, naming it. When its walk carries other " +
"deliveries, it first answers what ending the walk would take down, and acts only on a way: `again` " +
"ends the walk and walks the others again without this merge, `walk` stops the whole walk (novox/hq ADR 0299).",
Input: map[string]any{"type": "object", "properties": map[string]any{"id": str("the delivery's id"),
"why": str("why, kept with the transition")}, "required": []string{"id", "why"}},
"why": str("why, kept with the transition"),
"way": str("on a walk shared with other deliveries: again (the others walked again without it) or walk " +
"(the whole walk stopped)")}, "required": []string{"id", "why"}},
Run: func(a map[string]any) (any, error) {
if err := need(a, "id", "why"); err != nil {
return nil, err
}
return h.Stop(strArg(a, "id"), strArg(a, "why"), aPerson)
return h.StopWay(strArg(a, "id"), strArg(a, "why"), aPerson, strArg(a, "way"))
}},
{Name: seat + "close",
Description: "Healer H2's verb: for a delivery held past its bound, read its walk again and take the " +
@@ -334,7 +347,8 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: "delivery_status",
Description: "Whether this owner works: how many deliveries in each state, the groups, what is owed to " +
"the forge and the bus and not yet done, walks with no delivery yet, whether its events arrive, when " +
"the forge and the bus and not yet done, how many owed effects were tried since it started (done, failed " +
"and tried again, given up) and when the last was, walks with no delivery yet, whether its events arrive, when " +
"the controller's walks were last read or why they could not be, and the last refusals.",
Run: func(map[string]any) (any, error) { return h.Status(l.get()), nil }},
}
+102 -14
View File
@@ -68,10 +68,16 @@ type Delivery struct {
// Stranded says why a merge whose walk ended done before its word, without moving every module it moves, is
// stopped: nothing later delivered them either (novox/hq issue 474). Set once the cover test says no.
Stranded string `json:"stranded,omitempty"`
// CoverTold is whether the forge was told that such a stop was covered since: its status, its view and its note
// (novox/hq issue 480). Kept, so a restart tells it once, and a stop covered before this was built is told too.
CoverTold bool `json:"cover_told,omitempty"`
// TurnAwaits says, for a group member whose turn waits for a later merge to carry a superseded member before it,
// which merges and until when; TurnAwaitsUntil is that moment, which bounds the wait (ADR 0276 decision 9,
// novox/hq issue 445). `show` shows both, and `stalled` says them past that moment.
TurnAwaits string `json:"turn_awaits,omitempty"`
TurnAwaits string `json:"turn_awaits,omitempty"`
// HeldBack is why the walk it waits on is not let go: it carries a merge whose delivery was stopped, and is
// asked to leave it out (novox/hq ADR 0299). Said by `waits`, and by `stalled` past its bound.
HeldBack string `json:"held_back,omitempty"`
TurnAwaitsUntil *time.Time `json:"turn_awaits_until,omitempty"`
// ContainedIn is the forge's answer, by a later merge commit, to whether that commit contains this one's
// merge (gitea_contains): asked once per pair, and kept (ADR 0270).
@@ -159,6 +165,9 @@ type DeliveryPlan struct {
Machines []MachinePlan `json:"machines,omitempty"`
Steps []string `json:"steps,omitempty"`
Summary string `json:"summary"`
// Class is the class of the slowest-budgeted module it moves, as the planner says it: core, leaf, or none when
// it moves no module (novox/hq ADR 0313 §2). Empty in a plan from a controller that did not say it.
Class string `json:"class,omitempty"`
}
// receiving are the machines the plan sends a module to, in the plan's order; none when the plan names no
@@ -245,6 +254,20 @@ func (d *Delivery) FromForgesWord() bool {
// shownOnItsPullRequest is whether its view and status are kept on its pull request: not for history, whose
// pull request the forge may no longer hold — or holds under its number another one.
// coveredStop is whether a delivery stopped because its walk ended done before its word without building its modules
// (novox/hq issue 474) was covered since: a later merge into its trunk that contains it was delivered, carrying every
// module it moves to every machine running it (ADR 0270's cover test, kept as CoveredBy). Its state stays stopped —
// final, as a covered superseded member stays superseded (issue 442) — and it reads covered, by that merge (novox/hq
// issue 480). A stop for any other reason never reads covered.
func (d *Delivery) coveredStop() bool {
return d.State == Stopped && d.Stranded != "" && d.CoveredBy != ""
}
// coveredStopSaid is how a covered stop reads: on the forge's status, in its view and note, and in `show`.
func (d *Delivery) coveredStopSaid() string {
return "covered after its stop: a later merge that contains it was delivered: " + d.CoveredBy
}
func (d *Delivery) shownOnItsPullRequest() bool {
return d.Number > 0 && d.History == "" && d.State != Retired
}
@@ -383,6 +406,11 @@ type WalkDelivery struct {
// TakenOverBy is, on a walk folded before it started, the walk that took it over and answers every merge
// it answered.
TakenOverBy string `json:"taken_over_by,omitempty"`
// WalkedAgainBy is, on a walk stopped without some of its merges, the batch that walks its other merges again,
// and LeftOut the merges it left out (novox/hq ADR 0299): a delivery of a merge left out ends stopped with it,
// one of a merge walked again follows that batch's walk.
WalkedAgainBy string `json:"walked_again_by,omitempty"`
LeftOut []WalkLeftOut `json:"left_out,omitempty"`
// Batch is set while the record is a batch, not yet a walk: its state `assembling` or `queued`. A batch, once
// cut, is the walk under the same id: its next record is `building`. A merge this owner reads in a batch is
// not held as unwalked while the batch is heard of.
@@ -398,6 +426,30 @@ type WalkMerge struct {
Carried string `json:"carried,omitempty"`
}
// WalkLeftOut is a merge a stop left out (novox/hq ADR 0299): a merge stopped, or a later merge of its
// repository's branch that contains it, Contains naming the stopped merge.
type WalkLeftOut struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Contains string `json:"contains,omitempty"`
// Shares are the modules it moves that a stopped merge moves too, when that is why it was left out: Contains
// then names that stopped merge, which it does not contain.
Shares []string `json:"shares,omitempty"`
}
// leftOut is the merge of a repository a stop left out, when it did.
func (w Walk) leftOut(repository, merge string) (WalkLeftOut, bool) {
if w.Delivery == nil || merge == "" {
return WalkLeftOut{}, false
}
for _, o := range w.Delivery.LeftOut {
if strings.EqualFold(o.Repository, repository) && sameCommit(o.Commit, merge) {
return o, true
}
}
return WalkLeftOut{}, false
}
// WalkBatch is a batch's window: when it closes unless another merge arrives, when it closes at the latest,
// and, once closed, the walk it waits behind.
type WalkBatch struct {
@@ -454,18 +506,27 @@ func (w Walk) carriedFor(repository, merge string) (string, bool) {
// WalkSeen is what a delivery keeps of its walk.
type WalkSeen struct {
ID string `json:"id"`
Commit string `json:"commit"`
State string `json:"state"`
Note string `json:"note,omitempty"`
Revision int64 `json:"revision"`
Waits bool `json:"waits,omitempty"`
WaitedFor string `json:"waited_for,omitempty"`
GoBy string `json:"go_by,omitempty"`
StoppedBy string `json:"stopped_by,omitempty"`
Tier int `json:"tier"`
Tiers int `json:"tiers"`
Updated time.Time `json:"updated"`
ID string `json:"id"`
Commit string `json:"commit"`
State string `json:"state"`
Note string `json:"note,omitempty"`
Revision int64 `json:"revision"`
Waits bool `json:"waits,omitempty"`
WaitedFor string `json:"waited_for,omitempty"`
GoBy string `json:"go_by,omitempty"`
StoppedBy string `json:"stopped_by,omitempty"`
// StoppedWhy is why it was stopped; WalkedAgainBy, on a walk stopped without some merges, the batch that walks
// this delivery's merge again; Contains, when this delivery's merge was left out for containing a stopped
// merge, that merge (novox/hq ADR 0299).
StoppedWhy string `json:"stopped_why,omitempty"`
WalkedAgainBy string `json:"walked_again_by,omitempty"`
Contains string `json:"contains,omitempty"`
// Shares are, when this delivery's merge was left out for moving modules a stopped merge moves, those modules:
// built at the branch, which holds the stopped merge, they would deliver it (ADR 0299).
Shares []string `json:"shares,omitempty"`
Tier int `json:"tier"`
Tiers int `json:"tiers"`
Updated time.Time `json:"updated"`
// Branch is the branch the walk's commit was merged into; empty in a record kept before it was read.
Branch string `json:"branch,omitempty"`
// Created is when the controller made the walk: a delivery follows the newest walk that answers it.
@@ -487,7 +548,7 @@ func (w Walk) Seen() *WalkSeen {
s.WaitedFor = d.Awaits
s.Waits = d.Awaits != "" && d.Go == nil
s.GoBy = d.By
s.StoppedBy = d.Stopped
s.StoppedBy, s.StoppedWhy = d.Stopped, d.StoppedWhy
}
s.Branch = w.Branch
s.Created = w.Created
@@ -511,6 +572,33 @@ func (d *Delivery) endedUnstarted() bool {
return d.State == Published && d.Walk != nil && d.Walk.State == walkDone && !d.Walk.Started()
}
// walkStopped is whether the walk it follows was stopped through this owner, and walkedAgain whether its merge is
// walked again by another walk since (novox/hq ADR 0299).
func (d *Delivery) walkStopped() bool {
return d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy != ""
}
func (d *Delivery) walkedAgain() bool { return d.walkStopped() && d.Walk.WalkedAgainBy != "" }
// stoppedWords is why a delivery ends with the walk it follows stopped: the stop, and the stopped merge its own
// contains when it was left out for that.
func (d *Delivery) stoppedWords() string {
s := fmt.Sprintf("its walk %s was stopped by %s", d.Walk.ID, d.Walk.StoppedBy)
if d.Walk.StoppedWhy != "" {
s += ": " + d.Walk.StoppedWhy
}
switch {
case len(d.Walk.Shares) > 0:
s += fmt.Sprintf("; its merge moves %s, as the stopped merge %s does: built at the branch, which holds that "+
"merge, it would deliver it, so it was left out with it (novox/hq ADR 0299)", strings.Join(d.Walk.Shares, ", "),
shortOf(d.Walk.Contains, 12))
case d.Walk.Contains != "":
s += fmt.Sprintf("; its merge contains the stopped merge %s, so it cannot be walked without delivering it, "+
"and was left out with it (novox/hq ADR 0299)", shortOf(d.Walk.Contains, 12))
}
return s
}
// walkMovedItsModules is whether the walk it follows built every module its plan moves, and whether that is known:
// from the modules the walk's record names, or, for a record kept before they were, from a walk of no tier, which
// moved nothing. Unknown when its plan is not known.
@@ -61,6 +61,10 @@ type Controller interface {
Deliver(walk, why string) error
// Stop ends a walk, said as stopped by whom.
Stop(walk, why, by string) error
// StopWithout ends a walk and walks its other merges again without the merges named, each
// `<repository>@<commit>`, and without each later merge of their repository, which contains one (novox/hq
// ADR 0299).
StopWithout(walk, why, by string, without []string) (StopAnswer, error)
// Walks are the walks the controller keeps — one, given its id — and whether this seat has a holder on record.
Walks(walk string) (bool, []Walk, error)
}
@@ -269,6 +273,41 @@ func (c seatController) Stop(walk, why, by string) error {
return err
}
// StopAnswer is what the controller's `delivery-stop` with merges to leave out answers (novox/hq ADR 0299): the walk
// stopped, the batch that walks its other merges again with those merges, and every merge left out.
type StopAnswer struct {
Stopped string `json:"stopped"`
WalkedAgainBy string `json:"walked_again_by,omitempty"`
WalkedAgain []string `json:"walked_again,omitempty"`
LeftOut []WalkLeftOut `json:"left_out,omitempty"`
// Kept are the modules the stopped walk sent that nothing walks again, on the machines named; Withheld the
// modules the walk that walks the others withholds; StillCarriedBy the merges elsewhere that will deliver a
// stopped change when walked — a later merge of its repository, or one moving its modules (ADR 0299).
Kept []WalkKept `json:"kept,omitempty"`
Withheld []string `json:"withheld,omitempty"`
StillCarriedBy []string `json:"still_carried_by,omitempty"`
}
// WalkKept is a module a stopped walk left running its build, and where.
type WalkKept struct {
Module string `json:"module"`
Machines []string `json:"machines"`
}
func (c seatController) StopWithout(walk, why, by string, without []string) (StopAnswer, error) {
raw, err := c.verb("delivery-stop", map[string]any{"plan": walk, "why": why, "by": by,
"without": strings.Join(without, ",")})
if err != nil {
return StopAnswer{}, err
}
var a StopAnswer
if json.Unmarshal(raw, &a) != nil || a.Stopped == "" {
return StopAnswer{}, fmt.Errorf("the controller's delivery-stop did not say what it left out: %s",
lastLine(string(raw)))
}
return a, nil
}
func (c seatController) Walks(walk string) (bool, []Walk, error) {
args := map[string]any{}
if walk != "" {
@@ -84,7 +84,7 @@ const (
EvRetire Event = "retire" // a person's word that a held delivery is history, never to be walked
EvCovered Event = "covered" // a later merge into the same trunk delivered every module it moves
EvLifted Event = "lifted" // what held it for a person is no longer so: its group's composed check passed, or a walk now answers its merge
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed: it waits for a walk of its own commit
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed, or was stopped for another merge (ADR 0299): it waits for the walk that walks it again
)
// Facts are what a guard reads beyond the delivery itself: the moment, and who asks, for an act.
@@ -259,6 +259,9 @@ var Table = []Row{
if d.Walk != nil && d.Walk.State == walkFailed && d.carriedLater() {
return errors.New("its walk, which carried a later commit for it, failed: it waits for a walk of its own commit")
}
if d.walkStopped() {
return errors.New("its walk was stopped: it waits for the walk that walks its merge again, or ends stopped")
}
return unless(d.Walk != nil && d.Walk.Started(), "its walk waits")
}},
// **A walk that ended before its word settles its delivery** (novox/hq issue 474): a batch's walk the
@@ -284,11 +287,21 @@ var Table = []Row{
}},
{From: []State{Published}, Event: EvStop, To: Stopped,
Guard: "its walk ended done before its word without building every module it moves, and no later merge that " +
"contains it delivered them: no start can come, and nothing delivered it (novox/hq issue 474)",
"contains it delivered them: no start can come, and nothing delivered it (novox/hq issue 474); or the walk " +
"it waits on was stopped through this owner and walks its merge again nowhere — left out, or the walk " +
"ended whole (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
if d.walkStopped() && !d.walkedAgain() {
return nil
}
return unless(d.endedUnstarted() && d.Stranded != "", "its walk has not ended unstarted, or something may cover it")
},
Says: func(d *Delivery) string { return d.Stranded }},
Says: func(d *Delivery) string {
if d.walkStopped() {
return d.stoppedWords()
}
return d.Stranded
}},
{From: []State{Held}, Event: EvGo, To: Delivering,
Guard: "its walk started on a word that was not this owner's: the controller's own path, which waits for " +
"nobody, or a person's `plans go`",
@@ -332,10 +345,15 @@ var Table = []Row{
return "covered: a later merge that contains it was delivered: " + d.Carrier
}},
{From: []State{Delivering}, Event: EvStop, To: Stopped,
Guard: "its walk was stopped through this owner",
Guard: "its walk was stopped through this owner, and walks its merge again nowhere: left out, or the walk ended " +
"whole (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
return unless(d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy != "", "its walk was not stopped")
}},
if d.walkedAgain() {
return fmt.Errorf("its walk was stopped without it, and %s walks its merge again", d.Walk.WalkedAgainBy)
}
return unless(d.walkStopped(), "its walk was not stopped")
},
Says: func(d *Delivery) string { return d.stoppedWords() }},
{From: []State{Delivering}, Event: EvFailed, To: Failed,
Guard: "its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine",
Holds: func(d *Delivery, f Facts) error {
@@ -344,12 +362,20 @@ var Table = []Row{
}},
{From: []State{Delivering}, Event: EvRewalk, To: Published,
Guard: "its walk failed carrying a later merge of its branch for it: a failed walk marks no merge delivered, and this " +
"merge waits, published, for a walk of its own commit (novox/hq ADR 0276 decision 3)",
"merge waits, published, for a walk of its own commit (novox/hq ADR 0276 decision 3); or its walk was stopped " +
"for another merge, and its merge is walked again without that one (novox/hq ADR 0299)",
Holds: func(d *Delivery, f Facts) error {
if d.walkedAgain() {
return nil
}
return unless(d.Walk != nil && d.Walk.State == walkFailed && d.Walk.StoppedBy == "" && d.carriedLater(),
"its walk did not fail, or carried its own commit")
},
Says: func(d *Delivery) string {
if d.walkedAgain() {
return fmt.Sprintf("its walk %s was stopped for another merge (%s): %s walks its merge again without that one",
d.Walk.ID, d.Walk.StoppedWhy, d.Walk.WalkedAgainBy)
}
return fmt.Sprintf("its walk %s, which carried %s for it, failed: it waits for a walk of its own commit",
d.Walk.ID, shortOf(d.AnsweredBy, 12))
}},
@@ -542,6 +568,12 @@ func boundOf(d *Delivery) (Bound, bool) {
if d.waitsForItsOwnWalk() {
return rewalkBound, true
}
if (d.State == Published || d.State == Delivering) && d.HeldBack != "" {
// Its walk carries a stopped merge and is not let go (ADR 0299): said with why, past the bound of its state.
b := Bounds[d.State]
b.For, b.Says = Bounds[Published].For, d.HeldBack // a walk not let go waits for a word, whatever its state
return b, true
}
if d.State == Published && d.TurnAwaits != "" && d.TurnAwaitsUntil != nil {
// Its turn waits for a later merge to carry a member before it: bounded by that carrier's own bound, and
// said with it, never the generic "waits for its turn or its word" (novox/hq issue 445).
@@ -13,8 +13,8 @@ import (
// delivery time from its own merge, its class and its phases, and `times` answers the week's median and worst
// per class against its budget. Measurement only: nothing here holds, releases or changes a delivery.
// The budgets by class (ADR 0282 decision 1).
var budgets = map[string]time.Duration{"leaf": 5 * time.Minute, "core": 10 * time.Minute}
// The budgets by class (ADR 0282 decision 1, the leaf's as ADR 0304 decision 4 changed it).
var budgets = map[string]time.Duration{"leaf": 3 * time.Minute, "core": 10 * time.Minute}
// classes in the order `times` says them.
var classes = []string{"leaf", "core"}
@@ -68,8 +68,8 @@ func TestADeliveryKeepsItsTimeFromItsOwnMerge(t *testing.T) {
if d.Time == nil {
t.Fatalf("no delivery time kept: %+v", d)
}
if d.Time.TookMS != (280*time.Second).Milliseconds() || d.Time.Class != "leaf" || d.Time.Over {
t.Fatalf("time %dms (want 280000, from its own merge), class %s, over %v", d.Time.TookMS, d.Time.Class, d.Time.Over)
if d.Time.TookMS != (280*time.Second).Milliseconds() || d.Time.Class != "leaf" || !d.Time.Over {
t.Fatalf("time %dms (want 280000, from its own merge, over a leaf's three minutes), class %s, over %v", d.Time.TookMS, d.Time.Class, d.Time.Over)
}
var sum int64
for _, p := range d.Time.Phases {
@@ -133,9 +133,13 @@ func TestADeliveryKeepsItsTimeFromItsOwnMerge(t *testing.T) {
t.Fatalf("merged after its walk's end: %+v", dt)
}
// Exactly its budget is within it.
p = phasesOf("leaf", from, ph{"build", 5*time.Minute + 10*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); dt.Over || dt.TookMS != 300000 {
t.Fatalf("five minutes is within a leaf's budget: %+v", dt)
p = phasesOf("leaf", from, ph{"build", 3*time.Minute + 10*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); dt.Over || dt.TookMS != 180000 {
t.Fatalf("three minutes is within a leaf's budget: %+v", dt)
}
p = phasesOf("leaf", from, ph{"build", 3*time.Minute + 11*time.Second})
if dt := timeOf(&Delivery{MergedAt: from.Add(10 * time.Second)}, Walk{ID: "w", Phases: p}); !dt.Over {
t.Fatalf("a second past three minutes is over a leaf's budget: %+v", dt)
}
// Seen again, unchanged: not written again (issue 326).
before := w.store.puts
@@ -174,8 +178,8 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
return walkedOnce(w, n, "head"+string(rune('a'+n)), "merge"+string(rune('a'+n))+"zzzz",
phasesOf("leaf", t0.Add(at), ph{"window", 10 * time.Second}, ph{"judgement", d}))
}
leaf(1, 2*time.Minute, 0)
leaf(2, 4*time.Minute, time.Hour)
leaf(1, 1*time.Minute, 0)
leaf(2, 2*time.Minute, time.Hour)
over := leaf(3, 6*time.Minute, 2*time.Hour)
walkedOnce(w, 4, "headcore", "mergekore1", phasesOf("core", t0.Add(3*time.Hour), ph{"judgement", 9 * time.Minute}))
w.later(4 * time.Hour)
@@ -184,8 +188,8 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
t.Fatalf("classes: %+v", a.Classes)
}
l := a.Classes[0]
// Each delivery's time is from its own merge, ten seconds after the walk's: 2m, 4m, 6m.
if l.Count != 3 || l.Within != 2 || l.Over != 1 || l.MedianMS != (4*time.Minute).Milliseconds() ||
// Each delivery's time is from its own merge, ten seconds after the walk's: 1m, 2m, 6m.
if l.Count != 3 || l.Within != 2 || l.Over != 1 || l.MedianMS != (2*time.Minute).Milliseconds() ||
l.WorstMS != (6*time.Minute).Milliseconds() || l.WorstID != over {
t.Fatalf("leaf: %+v", l)
}
@@ -198,7 +202,7 @@ func TestTimesSaysEachClassAgainstItsBudget(t *testing.T) {
if len(a.Over) != 1 || a.Over[0].ID != over {
t.Fatalf("over: %+v", a.Over)
}
if !strings.Contains(a.Line, "leaf: median 4m0s, worst 6m0s, 2 of 3 within 5m0s") {
if !strings.Contains(a.Line, "leaf: median 2m0s, worst 6m0s, 2 of 3 within 3m0s") {
t.Fatalf("the line: %s", a.Line)
}
// Past the window, nothing.
@@ -250,8 +250,8 @@ func (h *Holder) uncheckedLines(now time.Time) []StalledLine {
h2 = "none: its delivery is " + string(u.held) + ", a state the operator moves — the operator's"
}
out = append(out, StalledLine{ID: id, Number: u.pull.Number, State: state, For: now.Sub(u.first).Round(time.Second).String(),
Bound: u.bound().String(),
H2: h2, Says: says, Waiting: waiting})
Bound: u.bound().String(), Held: u.held,
H2: h2, Says: says, Waiting: waiting})
}
return out
}
+210 -18
View File
@@ -29,6 +29,9 @@ type Line struct {
// waitsFor is what a delivery waits for, in words.
func (h *Holder) waitsFor(d *Delivery) string {
if d.HeldBack != "" && !d.State.Final() {
return d.HeldBack
}
switch d.State {
case Proposed:
return "its check"
@@ -68,6 +71,11 @@ func (h *Holder) waitsFor(d *Delivery) string {
if d.Walk != nil {
return "its walk: " + clip(d.Walk.Note, 120)
}
case Stopped:
if strandedUncovered(d) {
return "a later delivered merge to carry " + strings.Join(d.notMovedBy(), ", ") + ": it is stopped, and " +
"listed by stalled until one does (novox/hq issue 480)"
}
}
return ""
}
@@ -83,7 +91,8 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
group != "" && d.Group != group {
continue
}
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour {
// A final delivery `stalled` still lists stays, however old: the two never disagree (novox/hq issue 480).
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour && !strandedUncovered(d) {
continue
}
l := Line{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), For: now.Sub(d.Since).Round(time.Second).String(),
@@ -94,7 +103,7 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
if d.Walk != nil {
l.Walk = d.Walk.ID
}
l.Stalled = pastBound(d, now)
l.Stalled = pastBound(d, now) || strandedUncovered(d)
out = append(out, l)
}
sort.Slice(out, func(i, j int) bool {
@@ -112,6 +121,9 @@ func (h *Holder) Show(id string) (any, error) {
defer h.mu.Unlock()
if d := h.deliveries[id]; d != nil {
out := map[string]any{"delivery": d, "waits": h.waitsFor(d)}
if d.coveredStop() {
out["reads"] = d.coveredStopSaid() // stopped, and its change reached the machines (novox/hq issue 480)
}
if g := h.groups[d.Group]; g != nil {
state, why := GroupState(g, h.membersOf(g))
out["group"] = map[string]any{"group": g, "state": state, "why": why}
@@ -187,10 +199,14 @@ type StalledLine struct {
// the operator's own zone, which it cannot do inside a finished sentence (novox/hq issue 443). Waiting stays
// beside them for a controller that does not read them yet.
Checks []WaitingCheck `json:"checks,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
// Held is the state of the delivery a line of the state Unanswered is about, ready or rejected, when one is held
// here (novox/hq issues 441, 451): its check answered once, so the controller advises a recheck of it rather than
// the build queue, reading the state here and never from Says. Empty for a head with no delivery.
Held State `json:"held,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// Stalled is the `stalled` verb: what the controller's self-check raises as `delivery.<id>.stalled`.
@@ -326,10 +342,19 @@ func (h *Holder) Release(id, why, by string) (string, error) {
if d.Walk != nil && d.Walk.Waits {
walk = d.Walk.ID
}
var stopped []string
if walk != "" {
stopped, _ = h.carriesStopped(walk)
}
h.mu.Unlock()
if walk == "" {
return fmt.Sprintf("%s is delivering, on %s's word", id, by), nil
}
if len(stopped) > 0 {
// Never the word for a walk that carries a stopped change (ADR 0299): asked to leave it out first.
return "", fmt.Errorf("%s is delivering on %s's word, and its walk %s is not let go yet: it carries %s, whose "+
"delivery was stopped, and is asked to leave it out first", id, by, walk, strings.Join(stopped, ", "))
}
if err := h.Controller.Deliver(walk, "released by "+by+": "+why); err != nil {
return "", fmt.Errorf("%s is delivering on %s's word, and its walk %s could not be let go yet (%v): the next "+
"tick asks again", id, by, walk, err)
@@ -346,25 +371,98 @@ func (h *Holder) Release(id, why, by string) (string, error) {
return fmt.Sprintf("%s is delivering on %s's word; its walk %s is let go", id, by, walk), nil
}
// Stop is the `stop` verb: a delivery stopped, its walk ended through the controller, and in a group every
// Stop is the `stop` verb with no way given: see StopWay.
func (h *Holder) Stop(id, why, by string) (string, error) { return h.StopWay(id, why, by, "") }
// The ways a person's stop of a delivery on a shared walk takes (novox/hq ADR 0299).
const (
wayAgain = "again" // the walk ended, its other merges walked again without this one
wayWalk = "walk" // the whole walk stopped, every delivery on it with it
)
// StopWay is the `stop` verb: a delivery stopped, its walk ended through the controller, and in a group every
// member after it stopped too.
func (h *Holder) Stop(id, why, by string) (string, error) {
//
// **A stop never takes other deliveries down quietly** (novox/hq issue 459, ADR 0299): one walk carries every merge
// of its batch. When the delivery's open walk carries anything else, the stop does nothing at first: it says what
// ending the walk would take down — each other delivery, and which of them contain this merge and end either way —
// and the two ways, given with `way`: `again` ends the walk and walks the others again without this merge and the
// merges that contain it; `walk` stops the whole walk, and every delivery on it ends stopped, naming this stop.
func (h *Holder) StopWay(id, why, by, way string) (string, error) {
if way != "" && way != wayAgain && way != wayWalk {
return "", fmt.Errorf("way %q: %q or %q", way, wayAgain, wayWalk)
}
h.mu.Lock()
d := h.deliveries[id]
if d == nil {
h.mu.Unlock()
return "", fmt.Errorf("no delivery %q", id)
}
// Its guard is read on a copy: the delivery is stopped only once its walk is (review of #221).
trial := *d
if _, err := Apply(&trial, EvStop, true, Facts{Now: h.Now(), Why: why, By: by}); err != nil {
h.mu.Unlock()
return "", err
}
var walk string
if d.Walk != nil && (d.Walk.State == walkBuilding || d.Walk.State == walkRolling) {
walk = d.Walk.ID
}
merge := d.Repository + "@" + d.MergedAs
// A merge that waits in a batch not yet cut: the batch is asked to leave it out once it is stopped (ADR 0299).
var batch string
if b, batched := h.batchOf(d); batched && walk == "" {
batch = b.ID
}
var others []string
if walk != "" {
others = h.othersOn(walk, []string{merge})
}
if len(others) > 0 && way == "" {
said := h.takenDown(d, walk, others)
h.mu.Unlock()
return "", errors.New(said)
}
h.mu.Unlock()
var did string
switch {
case walk == "":
case len(others) == 0 || way == wayWalk:
whole := why
if len(others) > 0 {
whole = fmt.Sprintf("%s was stopped, and its whole walk with it: %s", id, why)
}
if err := h.Controller.Stop(walk, whole, by); err != nil {
return "", fmt.Errorf("%s is not stopped: its walk %s could not be ended yet: %v — `plans stop %s --why …` "+
"ends it by hand", id, walk, err, walk)
}
did = fmt.Sprintf("; its walk %s is ended", walk)
if len(others) > 0 {
did = fmt.Sprintf("; its walk %s is ended whole, and every delivery on it with it: %s", walk,
strings.Join(others, ", "))
}
default:
a, err := h.Controller.StopWithout(walk, why, by, []string{merge})
if err != nil {
return "", fmt.Errorf("%s is not stopped: its walk %s could not be ended without it: %v — `way: %s` stops "+
"the whole walk, and `plans stop %s --why …` ends it by hand", id, walk, err, wayWalk, walk)
}
did = fmt.Sprintf("; its walk %s is ended without it, and %s", walk, againWords(a))
}
h.mu.Lock()
d = h.deliveries[id]
if d == nil || d.State.Final() {
h.mu.Unlock()
return fmt.Sprintf("%s ended meanwhile%s", id, did), nil
}
t, err := Apply(d, EvStop, true, Facts{Now: h.Now(), Why: why, By: by})
if err != nil {
h.mu.Unlock()
return "", err
}
h.moved(d, []Transition{t})
var walk string
if d.Walk != nil && (d.Walk.State == walkBuilding || d.Walk.State == walkRolling) {
walk = d.Walk.ID
}
var asks []func()
if g := h.groups[d.Group]; g != nil && g.Closed {
asks = h.groupTurns(g, h.membersOf(g))
@@ -375,14 +473,73 @@ func (h *Holder) Stop(id, why, by string) (string, error) {
for _, a := range asks {
a()
}
if walk == "" {
return fmt.Sprintf("%s is stopped, on %s's word", id, by), nil
if batch != "" {
// Asked now, not at the next tick; and the batch's walk is never let go while it carries the merge.
did = fmt.Sprintf("; its merge waits in the batch %s, which was asked now to leave it out and is never let go "+
"while it carries it: %s", batch, h.stopLeaving(batch, why, by, []string{merge}))
}
if err := h.Controller.Stop(walk, why, by); err != nil {
return "", fmt.Errorf("%s is stopped, and its walk %s could not be ended yet: %v — `plans stop %s --why …` "+
"ends it by hand", id, walk, err, walk)
return fmt.Sprintf("%s is stopped, on %s's word%s", id, by, did), nil
}
// takenDown is what ending a delivery's shared walk would take down, and the two ways: for a caller holding the
// lock (ADR 0299).
func (h *Holder) takenDown(d *Delivery, walk string, others []string) string {
var ride, contain []string
for _, o := range others {
od := h.deliveries[o]
if od != nil && containsMerge(od, d) {
contain = append(contain, o)
continue
}
ride = append(ride, o)
}
return fmt.Sprintf("%s is stopped, on %s's word; its walk %s is ended", id, by, walk), nil
s := fmt.Sprintf("%s is not stopped: its walk %s carries other changes, and ending the walk would take them down: %s.",
d.ID, walk, orNone(strings.Join(ride, ", ")))
if len(contain) > 0 {
s += fmt.Sprintf(" These contain its merge and end with it either way: %s.", strings.Join(contain, ", "))
}
// A module is built at its branch, which holds this merge: a delivery moving a module this one moves would
// deliver it, and the controller leaves it out too (ADR 0299). Its plan is what is known here; the controller
// decides, and its answer names each.
var share []string
for _, o := range ride {
if od := h.deliveries[o]; od != nil && od.Plan != nil && d.Plan != nil {
if both := sharedModules(od.Plan.Moved, d.Plan.Moved); len(both) > 0 {
share = append(share, fmt.Sprintf("%s (moves %s)", o, strings.Join(both, ", ")))
}
}
}
if len(share) > 0 {
s += fmt.Sprintf(" With `way: %s` these are left out too, for moving a module this merge moves: %s.", wayAgain,
strings.Join(share, ", "))
}
s += fmt.Sprintf(" With `way: %s` the controller may leave out more for sharing a module, and its answer names each.",
wayAgain)
return s + fmt.Sprintf(" What the walk already sent stays where it was sent; the stop's answer names it."+
" Choose a way: `way: %s` ends the walk and walks the others again without this merge; "+
"`way: %s` stops the whole walk and every delivery on it.", wayAgain, wayWalk)
}
// sharedModules are the modules in both lists, sorted.
func sharedModules(a, b []string) []string {
var out []string
for _, m := range a {
if containsString(b, m) && !containsString(out, m) {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// containsMerge is whether o is a later merge of d's repository and branch, which contains d's merge on a linear
// trunk: merged after it, or the merge the walk carries for it.
func containsMerge(o, d *Delivery) bool {
if o.ID == d.ID || !strings.EqualFold(o.Repository, d.Repository) || !strings.EqualFold(o.Base, d.Base) || o.MergedAs == "" {
return false
}
return o.MergedAt.After(d.MergedAt) || d.AnsweredBy != "" && sameCommit(d.AnsweredBy, o.MergedAs) &&
!sameCommit(d.AnsweredBy, d.MergedAs)
}
// Close is healer H2's verb: for a delivery past its bound, the walk's record is read again and the
@@ -491,6 +648,27 @@ func (h *Holder) Status(listening string) map[string]any {
out := map[string]any{"deliveries": byState, "groups": len(h.groups), "owed": owed, "unkept": len(h.dirty),
"walks-unmatched": len(h.unmatched), "events": listening, "started": h.started.Format(time.RFC3339),
"seat-on-record": h.held}
// The owed effects tried since this owner started (novox/hq issue 407): done, failed and to be tried again,
// or given up, and when the last was tried. A count since the start, said with that start: two readings in
// one run give a window's tries.
tried := map[string]any{"since": h.started.Format(time.RFC3339), "done": h.tried.done, "failed": h.tried.failed,
"given-up": h.tried.givenUp, "tried": h.tried.done + h.tried.failed + h.tried.givenUp}
if !h.tried.last.IsZero() {
tried["last"] = h.tried.last.Format(time.RFC3339)
}
out["owed-tried"] = tried
// Each held delivery by name, with why and the verb that releases it (novox/hq issue 529): a count alone
// pointed a reader at nothing.
var held []string
for _, d := range h.deliveries {
if d.State == Held {
held = append(held, HeldLine(d))
}
}
if len(held) > 0 {
sort.Strings(held)
out["held"] = held
}
if !h.reconciled.IsZero() {
out["walks-read"] = h.reconciled.Format(time.RFC3339)
}
@@ -599,3 +777,17 @@ func withoutKinds(owed []Effect, kinds ...string) []Effect {
}
return out
}
// HeldLine says a held delivery in one line: which, since when, why, its walk when one waits, and the verb that
// releases it (novox/hq issue 529).
func HeldLine(d *Delivery) string {
why := d.HeldWhy
if why == "" {
why = lastWhyOf(d)
}
s := fmt.Sprintf("%s held since %s: %s", d.ID, d.Since.UTC().Format(time.RFC3339), why)
if d.Walk != nil && d.Walk.Waits {
s += "; its walk " + d.Walk.ID + " waits for it"
}
return s + " — a person decides: `mesh-delivery.release` (id " + d.ID + ", why) lets it go, `mesh-delivery.stop` ends it"
}
+123 -21
View File
@@ -19,7 +19,7 @@ what it refuses.
|---|---|---|
| `open` | `title`, `report`, optionally `located-in`, `record` | checks the issue, then takes the next number and keeps it under that number; answers the number |
| `get` | `number` | the issue whole: fields, notes, links |
| `list` | optionally `status`, `owner`, `phrase`, `limit`, `severity`, `needs`, `ready`, `flag` (`delivered-not-resolved` or `abandoned`) | short rows, newest first: number, status, owner, title, and each flag that holds |
| `list` | optionally `status`, `owner`, `phrase`, `limit`, `severity`, `needs`, `ready`, `flag` (`delivered-not-resolved` or `abandoned`), `search` (words, ranked), `before` (a number, the cursor) | short rows, newest first, or best first for a `search`: number, status, owner, title, and each flag that holds; `next`, the `before` that reads on, when more match |
| `move` | `number`, `status`, and what that status needs (`by` for `diagnosing` and `fixing`, optionally `take-over`) | changes the status under the development cycle's rules |
| `note` | `number`, `text` | appends a note; a note is never edited |
| `link` | `number`, `kind`, `target`, optionally `remove` | adds or removes a typed link |
@@ -28,6 +28,7 @@ what it refuses.
| `groom` | optionally `pass`, `confirm` | the triage over the backlog: without arguments a dry run in the background, answering its pass id; with `pass` its progress and findings; with `confirm` and `pass` exactly that pass's findings applied, each checked again; see below |
| `tracking` | optionally `conditions` (condition keys) | for every condition key an issue links (link kind `condition`), the issues that link it, those not closed first: status, severity, needs, worker with since, active and stale, and where the fix is (`fix`); the operator's channel says it in a condition's message (hq ADR 0285) |
| `reserve` | `kind` (`decision` or `research`), optionally `for` (one line: what the record is about) | the next number of that kind, spent forever; answers `kind` and `number` (and `prefix`, the number as the record's name starts with it); see below |
| `fix-times` | optionally `days` (1 to 90, default 7) | each class's fix budget, the median and worst fix time of the issues whose last fix was delivered in the window, within and over, and every issue over its budget with its longest phase; see *A fix's time* |
**The cycle's rules, at the write** (hq playbook 03, which `00-META/checks/cycle.py` enforced on the
folders until the import, and hq ADR 0284): a status is one of `open`,
@@ -134,11 +135,46 @@ carries `abandoned` on `get` and its `list` row, beside its `worker`. A move by
`take-over`'s reason, as any stale worker; the answer, the note and the `moved` event say it was abandoned
(`abandoned`).
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`): it is
said once — a note on the issue and a warning on the operator's channel (`seat:operator-channel.notify`, key
`issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue is delivered. A module that runs only
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`) of an
issue not closed: it is said once — a note on the issue and a warning on the operator's channel
(`seat:operator-channel.notify`, key `issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue
is delivered or the issue is closed. Resolved is the end of the warning as wontfix is: its worker proved the
acceptance live, so a fix whose own walk failed and that a later merge carried is not waited for (hq issue 547). An
issue resolved or wontfix is never said stuck. A module that runs only
its bundle cannot raise a controller condition (hq ADR 0240 rule 8, ADR 0256), so the warning is the channel's, not the controller's.
## A fix's time (hq ADR 0313)
An issue's **fix time** runs from its first move to `fixing` (kept as `fixing-from`; a take-over or a later move
back to `fixing` does not restart it) to the moment the last of its fixes is delivered. An issue moved before the
moment was kept has it read from its log: a move's own moment, or, for a value carried into the log after one at
another status, its worker's `active` (the move), never its worker's `since`, which a diagnosis first keeps. Such a
fix time says `from-log`: it is kept and shown, and never warned on, so the first fixes heard of after the tracker
starts measuring raise no warning for work done before it. Once every fix it names is delivered, `get` carries it as `fix-time`:
`from`, `end`, `took`, the issue's `class`, its **fix budget** (`budget`: five minutes for a leaf module, ten for a
core one, counted from the move to `fixing`), `over`, the fix delivered last (`fix`), the `worker`, and four
`phases` read from that fix, which add up to the fix time:
| phase | ends at | read from |
|---|---|---|
| implementing | its pull request opened | mesh-delivery's `opened` (the pull request's first head heard) |
| review | the first approval at its head | the forge's reviews (`gitea_pull_reviews`): the forge's own approval, or a comment review whose first line is `Verdict: approve at <sha>`, the sha the head it was posted at or a prefix of at least seven characters (the forge refuses an approval of one's own pull request, so the mesh's independent reviews are comments) |
| checks | its merge, required checks and any wait for the merge included | mesh-delivery's `merged_at` |
| merge to running | delivered | the delivery's `delivered` (its delivery time) |
A phase whose end came before the end of the phase before it counts zero; a phase whose end is not known (no
approval at the head on the forge, a delivery that did not say when the pull request opened) is `unknown`, and its
time counts in the next phase, which says so. The class is that of the slowest-budgeted module the fixes move, as
mesh-delivery says it from the delivery plan (`class`: `core`, `leaf`, or `none` for a fix that moves no module);
an issue whose fixes move no module, or whose class is not said, is measured and shown with no fix budget.
**Over its budget is loud.** When the last fix is delivered and the fix time exceeds the class's fix budget, the
operator's channel is told (`seat:operator-channel.notify`, key `issue-tracker.fix.<class>.over-budget`), naming the
issue, its worker and its longest phase. Nothing is said while a fix is undelivered: a merged fix not delivered is
loud as a stuck fix. The warning clears when the next issue of that class is delivered within its budget; the
newest fix time of each class decides, and every sweep says or clears what a failed ask left. The delivery budget
(mesh-delivery's `times`, from the merge) stays a measure inside the fix budget.
## The import of hq's issues (slice 3)
`issue-tracker.import` reads every `04-ISSUES/NNN-slug/00-report.md` of hq **through the records module
@@ -167,7 +203,7 @@ It is a verb on the seat, callable by any agent, because it can do only this:
`imported` event. What it writes comes from hq's `main` as the records module holds it, which is reviewed
in pull requests.
- **What it keeps is bounded** as every verb's text is: the title 200 bytes, the report 64 KiB, one issue
256 KiB; a report that trips a credential's shape is refused. An entry of `located-in` or `fixed-by` hq
252 KiB; a report that trips a credential's shape is refused. An entry of `located-in` or `fixed-by` hq
wrote is prose at times, so it is taken within a reason's bound (2000 bytes), not a line's.
- **An owner hq named that the registered list does not know is kept as written and marked**, in
`located-in-unverified` on the issue and `unverified-owners` in the answer, never refused: the list is
@@ -200,14 +236,15 @@ in one instance or two — never take one number. A write another opener won is
bounded number of times with a short random wait; past the bound the open is refused and takes no number.
- **A refused open takes no number**: an issue is checked before its number is taken.
- **A number is never handed out twice, and may be skipped.** A number taken and then not kept — the bus
failing between the counter and the issue — stays unused, and the answer says which. A write whose answer
was lost is recognised as its own when tried again, so it leaves no gap; when it cannot be confirmed either
way, the answer says the issue may have been written and to `get` that number before opening it again,
never that nothing was opened.
- **An issue is never overwritten.** It is kept by a write that is refused when the key exists; if the
counter ever hands out a number already kept, the open is refused loudly, the number stays unused, and
nothing is overwritten.
- **A number is never handed out twice, and may be skipped.** A number taken and then not taken by the log —
the bus failing between the counter and the issue's first operation, or a full log — is spent, and the open
answers the failure (hq ADR 0300 §2). An append whose answer was lost is recognised as its own by the open's
id when tried again or read back, so it leaves no gap; when it cannot be confirmed either way, the answer says
the issue may have been written and to `get` that number before opening it again, never that nothing was
opened.
- **An issue is never overwritten.** Its first operation is appended at `expect` 0, refused when the log
already holds one for that number; if the counter ever hands out a number already kept, the open is refused
loudly, the number stays unused, and nothing is overwritten.
- **The counter starts once**, the first time a number is taken: at the setting `first-number`, or after
the highest issue already kept, whichever is higher, made only if it does not exist. The setting is never
read again, so a number never moves.
@@ -242,16 +279,63 @@ sessions writing a record at once never take one number.
`decision-floor` or `research-floor` (the controller's settings verb) to the highest number on any open
branch of hq; once the counter has started, the floor is not read again.
## State and events
## The log, the bucket and the events (hq ADR 0300)
The states, declared in the manifest and made by the controller; the module never makes a bucket
(ADR 0201): `issues`, one key per issue by number, keeping 8 values per key; `counters`, one value per key
(`issue`, `decision`, `research`); `grooming`, the backlog's pass; `reservations`, one key per reserved number.
Neither expires. One issue is kept in at most 256 KiB; a note that would pass it is refused.
**The record is the log `changes`** (declared in the manifest's `logs`, made by the controller, hq ADR 0297),
one key per issue by number. Every write — open, move, note, link, set, the triage, a delivery's word on a
fix, a warning said or cleared, the import — is one operation appended under the issue's number with what
changed, who acted and when, at the sequence of the last operation the issue was folded at
(`AppendAt`). Another operation in between refuses it: nothing is written, the log is folded again and the
change applied to that, a bounded number of times. A write the log refused writes nothing; only an open's
number, once taken, stays spent. An operation that begins an issue (`opened`, `imported`, `carried`) holds it
whole; every other one names the fields given a new value (`set`), the lists added to at their end (`add`:
a note is kept as itself, not as every note again) and the fields taken away (`unset`). The fold applies them
as written, so a new kind of operation needs nothing new in the fold.
Every change is an event: `mesh-issues.opened`, `.moved`, `.noted`, `.linked`, `.imported`, `.reserved`, each with the
number, what changed, who changed it and when. The events are the audit trail. A change whose event the bus did not take
is kept, and its answer says the event was not said (`announced: false`).
**The bucket `issues` is the fold of the log**, each issue carrying `folded`, the sequence
of its last operation. It is written after the append, at the revision read; one whose write fails after the
log took the change is behind, and caught up at its next change (the stale append is refused and the log
folded) or at the next start. Agents and views watch it as before. It still keeps 8 values per key, so the bus
cannot drop the older values before the carry has read them; it goes to one value per key (ADR 0300 §3) in a
follow-up, once the carry has run with a read of those values.
**On start**, the tracker watches its own bucket, carries what the bucket holds and the log does not (below),
reads the whole log page by page, folds every issue whose last operation is newer than its `folded` — an
emptied bucket is rebuilt whole — and raises the issue counter to the highest number in the log where it is
below. Until then a `list` reads the bucket and a write carries the issue it changes first.
**The switch** (ADR 0300 §6) is the carry, run on every start and before any write of an issue not carried:
for every issue the bucket holds with no `folded`, each value the bucket still keeps of it, read through the
runtime's `mesh/state.history` (hq ADR 0307), is appended oldest first as a `carried` operation holding the
whole issue and the bucket's revision of that value, and the fold written. It resumes rather than restarts: a
kept value at or below the highest revision a carried operation holds, or one the tracker wrote itself (it has
`folded`), is known to the log, so an interrupted carry, or two instances carrying one issue at once, carry each
value once. Matched by revision, not content, because the same content may be kept twice. A value written beside
the log after the carry, by an instance from before the switch, is carried as one more operation and is never
written over by a fold. A deletion among the kept values is passed over and counted, since the tracker never
deletes an issue and every value after one holds the issue whole.
An issue that cannot be carried — the log refusing it, or a value this module cannot read — is a fault, named
with its reason in the start line; the others are carried and the start finishes. That issue stays in the
bucket as it was, read and listed as before, and every write to it is refused with the same reason until it
can be carried, so no operation reaches the log before its carry. One issue is kept in at most 252 KiB, so that
the operation holding it whole fits one entry of the log (256 KiB). An entry of the log that cannot be folded
faults its issue on start: nothing folded past it is written, and its bucket value is left as it is.
**In memory** the tracker holds every folded issue, kept current by the watch and its own writes, and a
full-text index of title, report and notes, ranked by BM25 (title words weigh three), rebuilt on every start.
`list` answers from memory, with `search` ranked best first, and `before` paging the newest-first list past
the 500 rows one answer holds. The index is the module's own: a few thousand issues need a hundred lines, not a
library of dozens of modules.
The other states: `counters`, one value per key (`issue`, `decision`, `research`); `grooming`, the backlog's
pass; `reservations`, one key per reserved number. None expires. One issue is kept in at most 252 KiB; a note
that would pass it is refused.
**The events announce; the log is the audit trail.** Every change is still an event: `mesh-issues.opened`,
`.moved`, `.noted`, `.linked`, `.imported`, `.reserved`, each with the number, what changed, who changed it and
when. A change whose event the bus did not take is kept, and its answer says the event was not said
(`announced: false`).
## Who acted
@@ -313,9 +397,27 @@ reserve spending nothing, the first reserve refused while the knowledge base doe
set, a kind held none of being no answer, hq's folders never read, and a
reservation that cannot be kept leaving its number spent.
A fix's time is tested in `fixtime_test.go`: the start at the first move to `fixing` through a diagnosis, a
take-over and a move away and back; phases that add up to the fix time, one ended early counting zero and one not
known counted in the next; the class of the slowest fix, none and not known with no budget; no warning while
undelivered, the warning once delivered over budget, cleared by the next of its class within budget and not by
another class, and said at the next sweep when the channel did not take it; `fix-times` over a window, an issue
before it not counted; the start read from the log; and the first approval at the head.
The import is tested over `testdata/hq`, copies of seven real reports of hq chosen for their shapes (a
bracketed list with commas inside parentheses, quoted block items, an apostrophe in prose, a `resolved:`
date, a replay, an empty list, a wontfix): the frontmatter read as written, each issue created at its own
number with the counter untouched, a second run changing nothing, an issue the tracker holds differently
left alone, a folder-backed issue filled once, a folder past the counter or past the first number refused,
a dry run writing nothing, and a report that cannot be read refused alone with why.
The log (hq ADR 0300) is tested in `log_test.go` over a log kept in the test with the runtime's semantics
(`expect`, `ErrChanged`): a write whose append is refused changing nothing, an operation in between not lost, an
open refused at validation taking no number, an append failing after the number spending it and answering the
failure, an emptied bucket restored exactly by the fold on start, a bucket behind its log catching up, the
counter raised to the log, each write saying its event beside its one operation, `before` paging 530 issues
without a gap or a repeat, a ranked search with the index rebuilt on start and kept current by the watch, and
the carry carrying each value kept, oldest first, once, passing over a deletion among them. Over the bus, a
write to an issue the bucket kept before the switch carries its three kept values first, read through the
runtime's history. Over the bus, the stream the controller makes holds one
operation per change of `TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus`.
+169 -1
View File
@@ -12,6 +12,7 @@ import (
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
@@ -48,6 +49,8 @@ type bus struct {
nc *nats.Conn
js jetstream.JetStream
kvs map[string]jetstream.KeyValue
// logs is each declared log's stream, made as the controller makes it (novox/hq ADR 0297).
logs map[string]jetstream.Stream
}
func newBus(t *testing.T) *bus {
@@ -73,7 +76,7 @@ func newBus(t *testing.T) *bus {
if err != nil {
t.Fatal(err)
}
b := &bus{nc: nc, js: js, kvs: map[string]jetstream.KeyValue{}}
b := &bus{nc: nc, js: js, kvs: map[string]jetstream.KeyValue{}, logs: map[string]jetstream.Stream{}}
// The controller's job: a bucket per declared state, with the options the manifest gives it.
m, _ := readManifest(t)
for _, d := range m.stateDeclarations(t) {
@@ -88,9 +91,98 @@ func newBus(t *testing.T) *bus {
}
b.kvs[d.Name] = kv
}
for _, name := range m.Logs {
st, err := js.CreateStream(context.Background(), jetstream.StreamConfig{
Name: "LOG_" + m.Module + "_" + name, Subjects: []string{"mesh.log." + m.Module + "." + name + ".>"},
Storage: jetstream.FileStorage, MaxMsgsPerSubject: -1, Discard: jetstream.DiscardNew, AllowDirect: true,
DenyDelete: true, DenyPurge: true})
if err != nil {
t.Fatal(err)
}
b.logs[name] = st
}
return b
}
// logVerb is the runtime's mesh/log verbs over a stream, as the tool runner serves them (ADR 0297).
func (b *bus) logVerb(ctx context.Context, verb string, params json.RawMessage) (any, error) {
var p struct {
Log string `json:"log"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Expect *uint64 `json:"expect"`
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
}
if err := json.Unmarshal(params, &p); err != nil {
return nil, err
}
st := b.logs[p.Log]
if st == nil {
return nil, fmt.Errorf("the log %q is not declared", p.Log)
}
subject := func(key string) string { return "mesh.log.mesh-issues." + p.Log + "." + key }
entry := func(m *jetstream.RawStreamMsg) map[string]any {
return map[string]any{"sequence": m.Sequence, "key": m.Subject[len(subject("")):], "value": json.RawMessage(m.Data),
"at": m.Time}
}
switch verb {
case "append":
msg := nats.NewMsg(subject(p.Key))
msg.Data = p.Value
if p.Expect != nil {
msg.Header.Set(jetstream.ExpectedLastSubjSeqHeader, strconv.FormatUint(*p.Expect, 10))
}
ack, err := b.js.PublishMsg(ctx, msg)
if err != nil {
return nil, err
}
return map[string]any{"sequence": ack.Sequence}, nil
case "last":
m, err := st.GetLastMsgForSubject(ctx, subject(p.Key))
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil, nil
}
if err != nil {
return nil, err
}
return entry(m), nil
case "read":
filter := subject(">")
if p.Keys != "" {
filter = subject(p.Keys)
}
limit := p.Limit
if limit <= 0 {
limit = 100
}
info, err := st.Info(ctx)
if err != nil {
return nil, err
}
entries := []map[string]any{}
var next uint64
for seq := max(p.From, 1); seq <= info.State.LastSeq; {
m, err := st.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(filter))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return nil, err
}
if len(entries) == limit {
next = m.Sequence
break
}
entries = append(entries, entry(m))
seq = m.Sequence + 1
}
return map[string]any{"entries": entries, "next": next, "last": info.State.LastSeq}, nil
}
return nil, fmt.Errorf("the runtime answers no mesh/log.%s", verb)
}
// instance is one running bundle, and the runtime's half of its conversation.
type instance struct {
t *testing.T
@@ -213,6 +305,9 @@ func (i *instance) answer(m rpc) {
}
return map[string]any{"decision": 291, "research": 45, "records": 336}, nil
}
if verb, ok := strings.CutPrefix(m.Method, "mesh/log."); ok {
return i.b.logVerb(ctx, verb, m.Params)
}
kv := i.b.kvs[p.State]
if kv == nil {
return nil, fmt.Errorf("the state %q is not declared", p.State)
@@ -239,6 +334,26 @@ func (i *instance) answer(m rpc) {
case "mesh/state.put":
rev, err := kv.Put(ctx, p.Key, p.Value)
return map[string]any{"revision": rev}, err
case "mesh/state.history":
// As the runtime serves it (novox/hq ADR 0307): every value kept, oldest first, puts and deletes.
kept, err := kv.History(ctx, p.Key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return []any{}, nil
}
if err != nil {
return nil, err
}
out := []map[string]any{}
for _, e := range kept {
r := map[string]any{"key": e.Key(), "revision": e.Revision(), "op": "put", "at": e.Created()}
if e.Operation() == jetstream.KeyValuePut {
r["value"] = json.RawMessage(e.Value())
} else {
r["op"] = "delete"
}
out = append(out, r)
}
return out, nil
case "mesh/state.keys":
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
@@ -471,6 +586,42 @@ func TestFortyReservationsAtOnceFromTwoInstancesGetFortyDistinctConsecutiveNumbe
}
}
// The carry over the bus (novox/hq ADR 0300 §6, ADR 0307): values the bucket kept before the switch are read
// through the runtime's history, oldest first, and carried before the write that touches the issue.
func TestAWriteOverTheBusCarriesEveryValueTheBucketKeptFirst(t *testing.T) {
b := newBus(t)
var events []event
var evMu sync.Mutex
kv := b.kvs[IssuesState]
for _, title := range []string{"first", "second", "third"} {
raw, _ := json.Marshal(Issue{Number: 300, Title: title, Status: StatusOpen, Report: "r", OpenedBy: "x", Opening: "o300"})
if _, err := kv.Put(context.Background(), "300", raw); err != nil {
t.Fatal(err)
}
}
i := b.start(t, writeSettings(t, 365), &events, &evMu)
if _, err := i.call("note", map[string]any{"number": 300, "text": "after the switch"}); err != nil {
t.Fatal(err)
}
var titles []string
var ops []string
for seq := uint64(1); ; seq++ {
m, err := b.logs[ChangesLog].GetMsg(context.Background(), seq)
if err != nil {
break
}
var op Operation
_ = json.Unmarshal(m.Data, &op)
ops = append(ops, op.Op)
if op.Issue != nil {
titles = append(titles, op.Issue.Title)
}
}
if fmt.Sprint(ops) != "[carried carried carried noted]" || fmt.Sprint(titles) != "[first second third]" {
t.Fatalf("the log holds %v of %v", ops, titles)
}
}
// One instance, every verb over the bus: what an agent does with an issue from opening to resolving.
func TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus(t *testing.T) {
b := newBus(t)
@@ -531,4 +682,21 @@ func TestAnIssueIsOpenedNotedLinkedMovedAndListedOverTheBus(t *testing.T) {
if fmt.Sprint(kinds) != "[opened noted linked moved moved moved]" {
t.Fatalf("events %v", kinds)
}
// The record is the log (novox/hq ADR 0300 §1): one operation per change, in the stream the controller makes,
// and the issue in the bucket folded at the last of them.
var ops []string
for seq := uint64(1); ; seq++ {
m, err := b.logs[ChangesLog].GetMsg(context.Background(), seq)
if err != nil {
break
}
var op Operation
if err := json.Unmarshal(m.Data, &op); err != nil || m.Subject != "mesh.log.mesh-issues.changes.365" {
t.Fatalf("entry %d on %s: %s", seq, m.Subject, m.Data)
}
ops = append(ops, op.Op)
}
if fmt.Sprint(ops) != fmt.Sprint(kinds) || is.Folded != uint64(len(ops)) {
t.Fatalf("the log holds %v, the issue is folded at %d", ops, is.Folded)
}
}
@@ -0,0 +1,831 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// The tracker's record is its log `changes`, not its bucket (novox/hq ADR 0300 §1). Every write is one
// operation appended under the issue's number, refused when another operation on that issue came in between;
// only then is the folded issue written to the bucket `issues`, which is what the operations add up to (§3).
const ChangesLog = "changes"
// Log is the module's log as the runtime offers it (novox/hq ADR 0297): an append at the key's last sequence,
// refused with stdio.ErrChanged when another entry came in between, and a read in pages.
type Log interface {
AppendAt(key string, value any, expect uint64) (uint64, error)
Read(from uint64, keys string, limit int) (*stdio.LogPage, error)
}
// runtimeLog is Log through the runtime that launched this bundle.
type runtimeLog struct{}
func (runtimeLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
return stdio.Log(ChangesLog).AppendAt(key, value, expect)
}
func (runtimeLog) Read(from uint64, keys string, limit int) (*stdio.LogPage, error) {
return stdio.Log(ChangesLog).Read(from, keys, limit)
}
// errNoLog is a tracker started without its log: the terminal's dry run, which writes nothing.
var errNoLog = errors.New("this tracker runs without its log, so it writes nothing")
// logPage is how many entries one read of the log asks for: the most the runtime gives.
const logPage = 1000
// The kinds of operation. An operation that begins an issue holds it whole: opened, imported, carried.
const (
OpOpened = "opened"
OpImported = "imported"
// OpCarried is a value the bucket kept when the tracker switched to its log (ADR 0300 §6), whole.
OpCarried = "carried"
)
// Operation is one change to one issue, as the log keeps it: what changed, who acted and when (ADR 0300 §1).
// One that begins an issue, or carries a value the bucket kept, holds the issue whole; every other one names
// the fields it gave a new value, the lists it added to at their end, and the fields it took away. The fold
// applies them as they are written, so it needs no knowledge of the kind: a new kind is folded as it comes.
type Operation struct {
Op string `json:"op"`
By string `json:"by"`
At time.Time `json:"at"`
Issue *Issue `json:"issue,omitempty"`
Set map[string]json.RawMessage `json:"set,omitempty"`
Add map[string]json.RawMessage `json:"add,omitempty"`
Unset []string `json:"unset,omitempty"`
// Revision is the bucket's revision of the value a carried operation holds: how a carry knows which kept
// values the log holds already.
Revision uint64 `json:"revision,omitempty"`
}
// Act is who changes an issue and as what kind of operation.
type Act struct {
Op string
By string
}
// fields is an issue as its fields in JSON, without the sequence it was folded at: that is the log's, not
// the issue's.
func fields(is *Issue) (map[string]json.RawMessage, error) {
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
var m map[string]json.RawMessage
if err := json.Unmarshal(raw, &m); err != nil {
return nil, err
}
delete(m, "folded")
return m, nil
}
// operation is what changed from before to after as one operation of act, and false when nothing did.
func operation(act Act, at time.Time, before, after *Issue) (*Operation, bool, error) {
b, err := fields(before)
if err != nil {
return nil, false, err
}
a, err := fields(after)
if err != nil {
return nil, false, err
}
op := &Operation{Op: act.Op, By: act.By, At: at}
for k, v := range a {
old, had := b[k]
if had && bytes.Equal(old, v) {
continue
}
if !had {
// A list left out while empty: its first element is added like any other.
old = json.RawMessage("[]")
}
if tail, ok := appended(old, v); ok {
if op.Add == nil {
op.Add = map[string]json.RawMessage{}
}
op.Add[k] = tail
continue
}
if op.Set == nil {
op.Set = map[string]json.RawMessage{}
}
op.Set[k] = v
}
for k := range b {
if _, still := a[k]; !still {
op.Unset = append(op.Unset, k)
}
}
sort.Strings(op.Unset)
return op, len(op.Set)+len(op.Add)+len(op.Unset) > 0, nil
}
// appended is what a list gained at its end, when after is before with elements added and nothing else
// changed: a note or a link is kept in the log as itself, not as every note again.
func appended(before, after json.RawMessage) (json.RawMessage, bool) {
var b, a []json.RawMessage
if len(after) == 0 || after[0] != '[' || json.Unmarshal(before, &b) != nil || json.Unmarshal(after, &a) != nil || len(a) <= len(b) {
return nil, false
}
for i := range b {
if !bytes.Equal(compact(b[i]), compact(a[i])) {
return nil, false
}
}
tail, err := json.Marshal(a[len(b):])
if err != nil {
return nil, false
}
return tail, true
}
func compact(raw json.RawMessage) []byte {
var buf bytes.Buffer
if json.Compact(&buf, raw) != nil {
return raw
}
return buf.Bytes()
}
// ErrNoBeginning is an operation on an issue the log holds no beginning for.
var ErrNoBeginning = errors.New("the log holds no operation that begins it")
// fold applies one operation to an issue (nil before its first) and answers the issue it adds up to, at seq.
func fold(base *Issue, op *Operation, seq uint64) (*Issue, error) {
var m map[string]json.RawMessage
switch {
case op.Issue != nil:
f, err := fields(op.Issue)
if err != nil {
return nil, err
}
m = f
case base == nil:
return nil, fmt.Errorf("a %s operation at %d: %w", op.Op, seq, ErrNoBeginning)
default:
f, err := fields(base)
if err != nil {
return nil, err
}
m = f
}
for k, v := range op.Set {
m[k] = v
}
for k, tail := range op.Add {
var have, more []json.RawMessage
if old, ok := m[k]; ok {
if err := json.Unmarshal(old, &have); err != nil {
return nil, fmt.Errorf("a %s operation at %d adds to %s, which is not a list", op.Op, seq, k)
}
}
if err := json.Unmarshal(tail, &more); err != nil {
return nil, fmt.Errorf("a %s operation at %d adds to %s what is not a list", op.Op, seq, k)
}
joined, err := json.Marshal(append(have, more...))
if err != nil {
return nil, err
}
m[k] = joined
}
for _, k := range op.Unset {
delete(m, k)
}
raw, err := json.Marshal(m)
if err != nil {
return nil, err
}
var is Issue
if err := json.Unmarshal(raw, &is); err != nil {
return nil, fmt.Errorf("a %s operation at %d folds to no issue: %v", op.Op, seq, err)
}
is.Folded = seq
return &is, nil
}
// foldEntry is fold of one entry of the log.
func foldEntry(base *Issue, e stdio.LogEntry) (*Issue, error) {
var op Operation
if err := json.Unmarshal(e.Value, &op); err != nil {
return nil, fmt.Errorf("the entry at %d is no operation this module reads: %v", e.Sequence, err)
}
return fold(base, &op, e.Sequence)
}
// readKey is every entry of one issue's key from the sequence from on, oldest first, read page by page.
func (t *Tracker) readKey(n uint64, from uint64) ([]stdio.LogEntry, error) {
if t.Log == nil {
return nil, errNoLog
}
var out []stdio.LogEntry
for from != 0 || out == nil {
p, err := t.Log.Read(max(from, 1), keyOf(n), logPage)
if err != nil {
return nil, fmt.Errorf("the log of issue %d could not be read: %v", n, err)
}
if out == nil {
out = []stdio.LogEntry{}
}
out = append(out, p.Entries...)
from = p.Next
}
return out, nil
}
// append adds one operation to an issue at the sequence its fold was read at. A refusal is stdio.ErrChanged:
// another operation on the issue came in between, and nothing was written.
func (t *Tracker) appendOp(n uint64, op *Operation, expect uint64) (uint64, error) {
if t.Log == nil {
return 0, errNoLog
}
raw, err := json.Marshal(op)
if err != nil {
return 0, err
}
return t.Log.AppendAt(keyOf(n), json.RawMessage(raw), expect)
}
// errBegun is a first operation refused because the log already begins that issue, with another one.
var errBegun = errors.New("the log already holds another issue under that number")
// begin appends the first operation of a new issue — expect 0 — and answers its sequence. A first operation
// whose answer was lost is known as its own by the issue's opening id. Any other refusal is errBegun; a
// failure is answered as it is (ADR 0300 §2: the number it was given stays spent).
func (t *Tracker) begin(is *Issue, act Act) (uint64, error) {
op := &Operation{Op: act.Op, By: act.By, At: t.now(), Issue: is}
var err error
for i := 0; i < createAttempts; i++ {
if i > 0 {
t.Sleep(jitter(time.Duration(i) * 50 * time.Millisecond))
}
var seq uint64
seq, err = t.appendOp(is.Number, op, 0)
if err == nil {
return seq, nil
}
if errors.Is(err, stdio.ErrChanged) {
first, lerr := t.firstOf(is.Number)
if lerr != nil {
return 0, fmt.Errorf("the log refused issue %d's first operation and could not be read back (%v): %w",
is.Number, lerr, errBegun)
}
if mine(first, is.Opening) {
return first.Sequence, nil
}
return 0, errBegun
}
}
// An append that timed out may still have landed: read back before saying it did not.
first, lerr := t.firstOf(is.Number)
switch {
case lerr != nil:
return 0, fmt.Errorf("%v, and the log could not be read back (%v): %w", err, lerr, errUnsure)
case mine(first, is.Opening):
return first.Sequence, nil
case first != nil:
return 0, errBegun
}
return 0, err
}
// firstOf is an issue's first entry in the log, the one that begins it, or nil when it has none. The first, not
// the last: an open whose answer was lost may have been followed by another operation on it already, such as the
// triage's.
func (t *Tracker) firstOf(n uint64) (*stdio.LogEntry, error) {
p, err := t.Log.Read(1, keyOf(n), 1)
if err != nil {
return nil, err
}
if len(p.Entries) == 0 {
return nil, nil
}
return &p.Entries[0], nil
}
// errUnsure is a first operation the log may have taken: its answer was lost, and the log could not be read back.
var errUnsure = errors.New("it may have been taken")
// mine says whether a log entry is the first operation of the issue opened by opening.
func mine(e *stdio.LogEntry, opening string) bool {
if e == nil || opening == "" {
return false
}
var op Operation
return json.Unmarshal(e.Value, &op) == nil && op.Issue != nil && op.Issue.Opening == opening
}
// change reads an issue, applies f, appends what changed to the log at the sequence the issue was folded at,
// and then writes the fold to the bucket (ADR 0300 §1). An append another operation won is read and applied
// again, so no note, link or move is lost and none is applied to a stale issue; a refused append writes
// nothing. A change that changes nothing appends nothing.
func (t *Tracker) change(n uint64, act Act, f func(*Issue) error) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := t.current(n)
if err != nil {
return nil, err
}
if is.Folded == 0 {
// Kept before the switch, and not carried yet: carried first, so the log begins it (ADR 0300 §6).
if _, _, err := t.carryOne(n); err != nil {
return nil, fmt.Errorf("issue %d is kept from before the tracker's log and is not carried into it yet, "+
"so nothing is changed: %v", n, err)
}
continue
}
before, err := clone(is)
if err != nil {
return nil, err
}
if err := f(is); err != nil {
return nil, err
}
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
if len(raw) > MostIssueBytes {
return nil, fmt.Errorf("issue %d would be %d bytes; one issue is kept in at most %d (252 KiB). Nothing "+
"was changed: say it shorter, or in a document the issue links", n, len(raw), MostIssueBytes)
}
op, changed, err := operation(act, t.now(), before, is)
if err != nil {
return nil, err
}
if !changed {
return is, nil
}
seq, err := t.appendOp(n, op, is.Folded)
if errors.Is(err, stdio.ErrChanged) {
// Another operation came in between, or the bucket is behind the log: fold what the log holds and
// apply this change again to that.
if _, err := t.catchUp(n); err != nil {
return nil, err
}
t.Sleep(jitter(time.Duration(i+1) * 5 * time.Millisecond))
continue
}
if err != nil {
return nil, fmt.Errorf("issue %d: the log did not take the change (%v), so the bucket was not written. If "+
"the log took it before the failure it is kept and the bucket folds it at the next change or start: "+
"get %d before changing it again", n, err, n)
}
is.Folded = seq
t.store(n, is, rev)
return is, nil
}
return nil, fmt.Errorf("issue %d was changed by another caller %d times while this change was made; nothing was "+
"changed. Try again", n, issueAttempts)
}
// current is an issue as the bucket holds it, and the revision it was read at; an issue the log holds and the
// bucket does not is folded into the bucket first.
func (t *Tracker) current(n uint64) (*Issue, uint64, error) {
is, rev, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) && t.Log != nil {
if _, ferr := t.catchUp(n); ferr != nil {
return nil, 0, ferr
}
is, rev, err = read(t.Store, n)
}
return is, rev, err
}
// store writes the fold of an issue at the revision read, or creates it at 0. The log already holds the
// change, so a write another writer won is folded again from the log, and one that fails is said and left for
// the next change or start, which fold the log again; it is never answered as a change not made.
func (t *Tracker) store(n uint64, is *Issue, rev uint64) {
var err error
if rev == 0 {
_, err = t.Store.Create(IssuesState, keyOf(n), is)
} else {
_, err = t.Store.Update(IssuesState, keyOf(n), is, rev)
}
if err == nil {
t.remember(is)
return
}
if errors.Is(err, stdio.ErrChanged) {
if folded, cerr := t.catchUp(n); cerr == nil {
if folded != nil {
t.remember(folded)
}
return
} else {
err = cerr
}
}
t.remember(is)
t.Logf("[mesh-issues] issue %d is kept in the log at %d and its bucket could not be written (%v): the bucket is "+
"behind until the next change of it or the next start folds it", n, is.Folded, err)
}
// catchUp folds into the bucket every operation of an issue the bucket has not folded, and answers the issue
// as folded (nil when neither holds it). The bucket is written only at the revision read; a write another
// writer won is read again.
func (t *Tracker) catchUp(n uint64) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil && !errors.Is(err, ErrNoIssue) {
return nil, err
}
var from uint64 = 1
if is != nil {
if is.Folded == 0 {
// Not carried yet: what the log holds is no fold of it.
return is, nil
}
from = is.Folded + 1
}
entries, err := t.readKey(n, from)
if err != nil {
return nil, err
}
if len(entries) == 0 {
return is, nil
}
folded := is
for _, e := range entries {
if folded, err = foldEntry(folded, e); err != nil {
return nil, fmt.Errorf("issue %d cannot be folded from its log: %v", n, err)
}
}
if is == nil {
_, err = t.Store.Create(IssuesState, keyOf(n), folded)
} else {
_, err = t.Store.Update(IssuesState, keyOf(n), folded, rev)
}
if err == nil {
t.remember(folded)
return folded, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return nil, fmt.Errorf("issue %d is behind its log in the bucket and could not be written (%v)", n, err)
}
}
return nil, fmt.Errorf("issue %d was written by another writer %d times while it was folded from its log", n, issueAttempts)
}
func clone(is *Issue) (*Issue, error) {
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
var c Issue
return &c, json.Unmarshal(raw, &c)
}
// Carried is what the carry did (ADR 0300 §6), as it happened: the values it appended, not the ones it meant to.
type Carried struct {
// Issues is how many issues it appended values of, Values how many values; Already how many issues the log
// already began whole, which it left as they are.
Issues int `json:"issues"`
Values int `json:"values"`
Already int `json:"already"`
// Deletes is how many deletions among the kept values were passed over: an issue is never deleted by the
// tracker, and each value after one holds the issue whole, so the fold needs none.
Deletes int `json:"deletes,omitempty"`
// Faults is each issue that could not be carried, and why. It stays in the bucket as it was, readable and
// listed; every write to it is refused with the same reason until it can be carried, so nothing reaches the
// log before its carry.
Faults []string `json:"faults,omitempty"`
Said string `json:"said"`
}
// carryOne carries one issue the bucket has not folded from the log: each value the bucket still keeps of it
// (novox/hq ADR 0307), oldest first, as `carried` operations holding the whole issue and the bucket's revision
// of that value, then the fold written to the bucket. Answers how many values it appended and how many
// deletions it passed over.
//
// It resumes rather than restarts. A value is known to the log when its revision is at or below the highest
// revision a carried operation of the issue holds, or when it is a value the tracker wrote itself (it carries
// `folded`): matched by revision, not by content, because the same content may be kept twice (a write that
// changed nothing) and a revision names one value exactly. So an interrupted carry, or another instance carrying
// the same issue at the same moment, carries each value once; a value written beside the log after the carry,
// by an instance from before the switch, is carried as one more operation and so is never overwritten.
func (t *Tracker) carryOne(n uint64) (int, int, error) {
appended := 0
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil {
return appended, 0, err
}
if is.Folded != 0 {
return appended, 0, nil
}
entries, err := t.readKey(n, 1)
if err != nil {
return appended, 0, err
}
var folded *Issue
var expect, known uint64
for _, e := range entries {
if folded, err = foldEntry(folded, e); err != nil {
return appended, 0, fmt.Errorf("issue %d cannot be folded from its log: %v", n, err)
}
expect = e.Sequence
var op Operation
if json.Unmarshal(e.Value, &op) == nil && op.Op == OpCarried {
known = max(known, op.Revision)
}
}
kept, err := t.Store.History(IssuesState, keyOf(n))
if err != nil {
return appended, 0, fmt.Errorf("the values the bucket keeps of issue %d could not be read: %v", n, err)
}
type value struct {
issue *Issue
revision uint64
}
var values []value
deletes := 0
for _, e := range kept {
if e.Op == "delete" {
deletes++
continue
}
var v Issue
if err := json.Unmarshal(e.Value, &v); err != nil {
return appended, deletes, fmt.Errorf("the value of issue %d at revision %d is in a form this module "+
"cannot read (%v): not carried, so that nothing kept is left out", n, e.Revision, err)
}
switch {
case v.Folded != 0:
// The tracker's own fold: the log holds it already, and whatever came before it.
values = nil
known = max(known, e.Revision)
case e.Revision > known:
values = append(values, value{&v, e.Revision})
}
}
if len(kept) == 0 || kept[len(kept)-1].Revision < rev {
// The value read is newer than the history read before it: carried too.
if rev > known {
values = append(values, value{is, rev})
}
}
raced := false
for _, v := range values {
v.issue.Folded = 0
seq, err := t.appendOp(n, &Operation{Op: OpCarried, By: Seat, At: t.now(), Issue: v.issue, Revision: v.revision}, expect)
if errors.Is(err, stdio.ErrChanged) {
raced = true
break
}
if err != nil {
return appended, deletes, fmt.Errorf("issue %d could not be carried into the log (%v); %d of its %d "+
"values not carried yet were carried. Nothing is lost: the bucket still holds it, and the carry "+
"resumes after the last value carried, at the next start or before its next change", n, err, appended, len(values))
}
appended++
expect = seq
if folded, err = fold(nil, &Operation{Op: OpCarried, Issue: v.issue}, seq); err != nil {
return appended, deletes, err
}
}
if raced {
// Another carrier was first: read again; what it appended is known by its revisions.
continue
}
if folded == nil {
return appended, deletes, fmt.Errorf("issue %d keeps no value to carry", n)
}
if _, err := t.Store.Update(IssuesState, keyOf(n), folded, rev); err != nil {
if errors.Is(err, stdio.ErrChanged) {
// Written meanwhile, by a carrier or an instance from before the switch: read again.
continue
}
return appended, deletes, fmt.Errorf("issue %d was carried into the log and its bucket could not be written (%v): "+
"it is folded at the next change or start", n, err)
}
t.remember(folded)
return appended, deletes, nil
}
return appended, 0, fmt.Errorf("issue %d was written by another writer %d times while it was carried", n, issueAttempts)
}
// same says whether two issues hold the same, whatever sequence each was folded at.
func same(a, b *Issue) bool {
fa, err1 := fields(a)
fb, err2 := fields(b)
if err1 != nil || err2 != nil || len(fa) != len(fb) {
return false
}
for k, v := range fa {
if !bytes.Equal(compact(v), compact(fb[k])) {
return false
}
}
return true
}
// Carry carries every issue the bucket holds and has not folded from the log (ADR 0300 §6). An issue that
// cannot be carried is a fault, named with its reason, and the others are carried. Run again, it carries
// nothing more, and resumes an issue whose carry was interrupted after the last value carried.
func (t *Tracker) Carry() (*Carried, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues kept cannot be listed: %v", err)
}
out := &Carried{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
out.Faults = append(out.Faults, err.Error())
continue
}
if is.Folded != 0 {
out.Already++
continue
}
v, d, err := t.carryOne(n)
out.Values += v
out.Deletes += d
if v > 0 {
out.Issues++
} else if err == nil {
out.Already++
}
if err != nil {
t.Logf("[mesh-issues] %v", err)
out.Faults = append(out.Faults, err.Error())
}
}
if out.Values == 0 {
out.Said = fmt.Sprintf("Carried nothing: the log begins every issue of the bucket it could (%d).", out.Already)
} else {
out.Said = fmt.Sprintf("Carried %d values of %d issues into the log, each value the bucket kept, oldest first; "+
"%d issues were there already.", out.Values, out.Issues, out.Already)
}
if out.Deletes > 0 {
out.Said += fmt.Sprintf(" %d deletions among the kept values were passed over: each value after one holds "+
"the issue whole.", out.Deletes)
}
if len(out.Faults) > 0 {
out.Said += fmt.Sprintf(" %d issues could not be carried, and stay as the bucket holds them, every write to "+
"them refused until they can be: %s.", len(out.Faults), strings.Join(out.Faults, "; "))
}
return out, nil
}
// Started is what the tracker did on start (ADR 0300 §3, §6).
type Started struct {
// Entries is how many operations the log holds; Issues how many issues they add up to.
Entries int `json:"entries"`
Issues int `json:"issues"`
// Refolded is how many issues the bucket held behind the log, Rebuilt how many it did not hold at all.
Refolded int `json:"refolded"`
Rebuilt int `json:"rebuilt"`
Carried *Carried `json:"carried"`
// Counter is the issue counter after start, raised to the highest number in the log where it was below.
Counter uint64 `json:"counter"`
Faults []string `json:"faults,omitempty"`
Said string `json:"said"`
}
// Start carries what the bucket holds and the log does not, reads the whole log, folds every issue whose
// last operation is newer than the bucket's fold of it — an emptied bucket is rebuilt whole — keeps the
// counter at or above the highest number in the log, and holds every issue in memory with its index
// (ADR 0300 §3, §5, §6).
func (t *Tracker) Start() (*Started, error) {
out := &Started{}
if t.Log == nil {
return out, errNoLog
}
carried, err := t.Carry()
if err != nil {
return out, fmt.Errorf("the carry could not start (%v); writes carry each issue before they change it", err)
}
out.Carried = carried
for _, f := range carried.Faults {
out.Faults = append(out.Faults, "not carried: "+f)
}
folds := map[uint64]*Issue{}
// broken is each issue an entry of which could not be folded: none of its later entries is folded on top,
// and its bucket value is left as it is, so no fold that skipped an operation is ever written.
broken := map[uint64]bool{}
var highest uint64
for from := uint64(1); from != 0; {
p, err := t.Log.Read(from, "", logPage)
if err != nil {
return out, fmt.Errorf("the log could not be read from %d: %v", from, err)
}
for _, e := range p.Entries {
out.Entries++
n, ok := numberOfKey(e.Key)
if !ok {
out.Faults = append(out.Faults, fmt.Sprintf("the entry at %d has the key %q, which is no issue's number", e.Sequence, e.Key))
continue
}
highest = max(highest, n)
if broken[n] {
continue
}
f, err := foldEntry(folds[n], e)
if err != nil {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d cannot be folded from its log, and its bucket "+
"value is left as it is: %v", n, err))
broken[n] = true
delete(folds, n)
continue
}
folds[n] = f
}
from = p.Next
}
out.Issues = len(folds) + len(broken)
numbers := make([]uint64, 0, len(folds))
for n := range folds {
numbers = append(numbers, n)
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] < numbers[j] })
for _, n := range numbers {
f := folds[n]
is, rev, err := read(t.Store, n)
switch {
case errors.Is(err, ErrNoIssue):
if _, err := t.Store.Create(IssuesState, keyOf(n), f); err != nil && !errors.Is(err, stdio.ErrChanged) {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d could not be rebuilt in the bucket: %v", n, err))
continue
} else if err == nil {
out.Rebuilt++
}
case err != nil:
out.Faults = append(out.Faults, err.Error())
continue
case is.Folded == 0:
// Written beside the log, by an instance from before the switch, after the carry, and not carried again
// (its carry is in the faults): the fold would overwrite that write. Left for its carry, which appends it
// as one more operation before the issue's next change.
out.Faults = append(out.Faults, fmt.Sprintf("issue %d holds a value written beside the log and not carried "+
"yet; it is not refolded over", n))
continue
case is.Folded < f.Folded:
if _, err := t.Store.Update(IssuesState, keyOf(n), f, rev); err != nil && !errors.Is(err, stdio.ErrChanged) {
out.Faults = append(out.Faults, fmt.Sprintf("issue %d could not be refolded in the bucket: %v", n, err))
continue
} else if err == nil {
out.Refolded++
}
}
}
if out.Counter, err = t.raiseCounter(highest); err != nil {
out.Faults = append(out.Faults, err.Error())
}
all, err := t.readAll()
if err != nil {
return out, err
}
t.view().load(all)
out.Said = fmt.Sprintf("The log holds %d operations of %d issues; the bucket was behind on %d and missed %d, now "+
"folded. %s The issue counter is at %d. %d issues are held in memory and indexed.", out.Entries, out.Issues,
out.Refolded, out.Rebuilt, carried.Said, out.Counter, len(all))
if len(out.Faults) > 0 {
out.Said += fmt.Sprintf(" %d faults: %v", len(out.Faults), out.Faults)
}
return out, nil
}
// raiseCounter keeps the issue counter at or above the highest number in the log, so no number the log holds
// is handed out again; a counter not started yet is left to the first open, which starts it above every issue
// the bucket holds.
func (t *Tracker) raiseCounter(highest uint64) (uint64, error) {
for i := 0; i < issueAttempts; i++ {
e, err := t.Store.Get(CountersState, IssueCounter)
if err != nil {
return 0, fmt.Errorf("the issue counter cannot be read: %v", err)
}
if e == nil {
return 0, nil
}
var n uint64
if err := json.Unmarshal(e.Value, &n); err != nil {
return 0, fmt.Errorf("the issue counter holds %s, no number", e.Value)
}
if n >= highest {
return n, nil
}
_, err = t.Store.Update(CountersState, IssueCounter, highest, e.Revision)
if err == nil {
t.Logf("[mesh-issues] the issue counter was at %d, below issue %d in the log: raised to %d", n, highest, highest)
return highest, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return n, fmt.Errorf("the issue counter is at %d, below issue %d in the log, and could not be raised: %v", n, highest, err)
}
}
return 0, errors.New("the issue counter was advanced by others while it was raised to the log's highest number")
}
@@ -91,7 +91,7 @@ func (t *Tracker) sayNotResolved(is *Issue, now time.Time) (bool, error) {
prs, ago, at.Format(time.RFC3339), worker), "", false); err != nil {
return false, err
}
_, err := t.change(is.Number, func(i *Issue) error { i.NotResolvedOpen = true; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.NotResolvedOpen = true; return nil })
return err == nil, err
}
@@ -105,7 +105,7 @@ func (t *Tracker) clearNotResolved(is *Issue) error {
"fix not delivered yet", is.Number, is.Status), true); err != nil {
return err
}
_, err := t.change(is.Number, func(i *Issue) error { i.NotResolvedOpen = false; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.NotResolvedOpen = false; return nil })
return err
}
@@ -145,7 +145,7 @@ func (t *Tracker) askUnheard() error {
if since.IsZero() {
since = t.now()
}
if _, err := t.recordFix(pr, d.State, d.Why, d.Merged, since, n); err != nil {
if _, err := t.recordFix(pr, d.heard(since), n); err != nil {
return err
}
heard = true
+121 -30
View File
@@ -37,6 +37,34 @@ type Fix struct {
Since time.Time `json:"since"`
// StuckSaid says it was said stuck, on the issue and to the operator's channel.
StuckSaid bool `json:"stuck-said,omitempty"`
// Class is the class of what its delivery moves, as mesh-delivery says it: core, leaf, or none when it moves
// no module; Opened when its pull request was first heard (novox/hq ADR 0313 §2, §4).
Class string `json:"class,omitempty"`
Opened *time.Time `json:"opened,omitempty"`
}
// heardFix is what the delivery said of a pull request at one moment: its state, why, whether it is merged, and
// what a fix's time is read by — the class of what it moves, when its pull request opened and when it merged,
// each when said.
type heardFix struct {
State string
Why string
Merged bool
At time.Time
Class string
Opened *time.Time
MergedAt *time.Time
}
// news says whether what was heard adds anything to what a fix keeps.
func (h heardFix) news(f *Fix, kept string) bool {
if f.State != h.State || f.Why != kept || (h.Merged && !f.Merged) || (h.Class != "" && h.Class != f.Class) {
return true
}
if h.Opened != nil && (f.Opened == nil || !f.Opened.Equal(*h.Opened)) {
return true
}
return h.MergedAt != nil && (f.MergedAt == nil || !f.MergedAt.Equal(*h.MergedAt))
}
// reached says whether a delivery's state says the change reached the machines (mesh-delivery's Reached).
@@ -147,7 +175,8 @@ func DeliveryLine(is *Issue) string {
// recordFix keeps what the delivery said of a pull request on every issue whose fixed-by names it (only, when
// given, on that one issue), and answers the issues it changed.
func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, only uint64) ([]uint64, error) {
func (t *Tracker) recordFix(pr string, h heardFix, only uint64) ([]uint64, error) {
state, why, merged, at := h.State, h.Why, h.Merged, h.At
all, err := t.allIssues()
if err != nil {
return nil, err
@@ -158,7 +187,7 @@ func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, on
continue
}
kept := cut(why, MostReasonKept)
if f := fixOf(is, pr); f != nil && (at.Before(f.Since) || (f.State == state && f.Why == kept && f.Merged == (merged || f.Merged))) {
if f := fixOf(is, pr); f != nil && (at.Before(f.Since) || !h.news(f, kept)) {
// Older than what is kept, or nothing new: transitions can arrive out of order.
continue
}
@@ -167,7 +196,7 @@ func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, on
// the same pull request at most, and an older one is history.
continue
}
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: "delivery", By: Seat}, func(i *Issue) error {
f := fixOf(i, pr)
if f == nil {
i.Fixes = append(i.Fixes, Fix{PullRequest: pr})
@@ -179,6 +208,17 @@ func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, on
m := at
f.MergedAt = &m
}
if h.MergedAt != nil && (merged || f.Merged) {
m := h.MergedAt.UTC()
f.MergedAt = &m // the delivery's own moment of the merge, not when it was heard
}
if h.Class != "" {
f.Class = h.Class
}
if h.Opened != nil {
o := h.Opened.UTC()
f.Opened = &o
}
if reached(state) && !contains(i.Delivered, pr) {
i.Delivered = append(i.Delivered, pr)
}
@@ -207,6 +247,10 @@ func (t *Tracker) afterFix(n uint64) error {
t.Logf("[mesh-issues] the warning that %d is delivered and not resolved could not be cleared on the operator's channel (tried again at the next sweep): %v", n, err)
}
if allDelivered(is) {
// Its fix time, final once its last fix is delivered (novox/hq ADR 0313).
if err := t.timeFix(n); err != nil {
return err
}
if unresolved(is) {
prs := liveFixes(is)
sort.Strings(prs)
@@ -242,7 +286,7 @@ func (t *Tracker) afterFix(n uint64) error {
}
}
claimed := false
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: EventReadyToVerify, By: Seat}, func(i *Issue) error {
claimed = false
if strings.Join(i.VerifyFor, ",") == strings.Join(prs, ",") {
return nil
@@ -266,9 +310,11 @@ func (t *Tracker) afterFix(n uint64) error {
}
// stillStuck says whether an issue has a fix said stuck that is still named, not delivered, and the issue not
// closed as wontfix.
// closed. A closed issue is the end of its warning: resolved means its worker proved the acceptance live, so a fix
// the tracker never heard delivered is not waited for any more (a failed walk that a later merge carried stays
// failed, novox/hq issue 547), and wontfix waits for nothing.
func stillStuck(is *Issue) bool {
if is.Status == StatusWontfix {
if closedStatus(is.Status) {
return false
}
named := liveFixes(is)
@@ -281,19 +327,30 @@ func stillStuck(is *Issue) bool {
}
// clearStuck clears the warning said for an issue on the operator's channel once nothing it was said for is
// stuck — delivered, no longer named, or the issue closed as wontfix; one that fails is tried again at the next
// sweep.
// stuck — delivered, no longer named, or the issue closed (resolved or wontfix); one that fails is tried again at
// the next sweep.
func (t *Tracker) clearStuck(is *Issue) error {
if !is.StuckOpen || stillStuck(is) || t.Notify == nil {
return nil
}
if err := t.Notify(stuckKey(is.Number), "", "", "", "", "the fix of issue "+keyOf(is.Number)+" is delivered or no longer waited for", true); err != nil {
if err := t.Notify(stuckKey(is.Number), "", "", "", "", stuckEnded(is), true); err != nil {
return err
}
_, err := t.change(is.Number, func(i *Issue) error { i.StuckOpen = false; return nil })
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.StuckOpen = false; return nil })
return err
}
// stuckEnded is what the operator's channel says when a stuck warning ends.
func stuckEnded(is *Issue) string {
switch is.Status {
case StatusResolved:
return "resolved: issue " + keyOf(is.Number) + " was proven live by its worker"
case StatusWontfix:
return "issue " + keyOf(is.Number) + " was closed as wontfix: its fix is no longer waited for"
}
return "the fix of issue " + keyOf(is.Number) + " is delivered or no longer waited for"
}
func stuckKey(n uint64) string { return "issue-tracker.fix-stuck." + keyOf(n) }
func prKey(repository string, number int) string {
@@ -308,12 +365,15 @@ func prKey(repository string, number int) string {
// to is kept on each issue whose fixed-by names it.
func (t *Tracker) OnTransition(e stdio.Envelope) error {
var body struct {
Repository string `json:"repository"`
Number int `json:"number"`
To string `json:"to"`
Why string `json:"why"`
MergedAs string `json:"merged_as"`
At time.Time `json:"at"`
Repository string `json:"repository"`
Number int `json:"number"`
To string `json:"to"`
Why string `json:"why"`
MergedAs string `json:"merged_as"`
At time.Time `json:"at"`
Class string `json:"class"`
Opened *time.Time `json:"opened"`
MergedAt *time.Time `json:"merged_at"`
}
if err := json.Unmarshal(e.Body, &body); err != nil || body.To == "" || body.Number <= 0 {
return nil
@@ -323,7 +383,8 @@ func (t *Tracker) OnTransition(e stdio.Envelope) error {
at = t.now()
}
merged := body.MergedAs != ""
changed, err := t.recordFix(prKey(body.Repository, body.Number), body.To, body.Why, merged, at, 0)
changed, err := t.recordFix(prKey(body.Repository, body.Number), heardFix{State: body.To, Why: body.Why, Merged: merged,
At: at, Class: body.Class, Opened: nonZero(body.Opened), MergedAt: nonZero(body.MergedAt)}, 0)
if err != nil {
return err
}
@@ -343,6 +404,24 @@ type Delivery struct {
Since time.Time
// Merged is the pull request's own merged flag, as the forge says it.
Merged bool
// Class, Opened and MergedAt are what a fix's time is read by (novox/hq ADR 0313), when the delivery says them.
Class string
Opened *time.Time
MergedAt *time.Time
}
// heard is what a delivery answered, as a fix keeps it.
func (d Delivery) heard(since time.Time) heardFix {
return heardFix{State: d.State, Why: d.Why, Merged: d.Merged, At: since, Class: d.Class, Opened: d.Opened, MergedAt: d.MergedAt}
}
// nonZero is a moment said, or nil for one not said.
func nonZero(t *time.Time) *time.Time {
if t == nil || t.IsZero() {
return nil
}
u := t.UTC()
return &u
}
// DeliveryState asks mesh-delivery for one pull request's delivery.
@@ -357,9 +436,12 @@ func busDeliveryState(owner string) DeliveryState {
}
var out struct {
Delivery struct {
State string `json:"state"`
Waits string `json:"waits"`
Since time.Time `json:"since"`
State string `json:"state"`
Waits string `json:"waits"`
Since time.Time `json:"since"`
Class string `json:"class"`
Opened *time.Time `json:"opened"`
MergedAt *time.Time `json:"merged_at"`
} `json:"delivery"`
Pull struct {
Merged bool `json:"merged"`
@@ -368,7 +450,8 @@ func busDeliveryState(owner string) DeliveryState {
if err := json.Unmarshal(raw, &out); err != nil {
return Delivery{}, fmt.Errorf("mesh-delivery's checks answered no delivery: %.200s", raw)
}
return Delivery{State: out.Delivery.State, Why: out.Delivery.Waits, Since: out.Delivery.Since.UTC(), Merged: out.Pull.Merged}, nil
return Delivery{State: out.Delivery.State, Why: out.Delivery.Waits, Since: out.Delivery.Since.UTC(), Merged: out.Pull.Merged,
Class: out.Delivery.Class, Opened: nonZero(out.Delivery.Opened), MergedAt: nonZero(out.Delivery.MergedAt)}, nil
}
}
@@ -417,7 +500,7 @@ func (t *Tracker) OnMoved(e stdio.Envelope) error {
if since.IsZero() {
since = t.now()
}
if _, err := t.recordFix(pr, state, d.Why, d.Merged, since, body.Number); err != nil {
if _, err := t.recordFix(pr, d.heard(since), body.Number); err != nil {
return err
}
}
@@ -453,7 +536,7 @@ func (t *Tracker) stuckAfter() time.Duration {
return DefaultStuckAfter
}
// Sweep says every fix merged and not delivered past the bound: a note on its issue and a warning on the
// Sweep says every fix of an issue not closed merged and not delivered past the bound: a note on its issue and a warning on the
// operator's channel, once; and every issue in fixing whose fixes were all delivered past its own bound and is not
// resolved (novox/hq issue 483). First it asks the delivery of a few fixes never heard of. It answers how many
// warnings it said. Run every few minutes.
@@ -468,21 +551,26 @@ func (t *Tracker) Sweep() (int, error) {
bound, now := t.stuckAfter(), t.now()
said := 0
for n, is := range all {
// A fix time missed at its delivery (a failure, or a restart between) is kept now; one of an issue moved to
// fixing before the tracker kept the moment is kept only when its fix is heard of again.
if is.FixingFrom != nil && is.FixTime == nil && allDelivered(is) {
if err := t.timeFix(n); err != nil {
t.Logf("[mesh-issues] the fix time of %d could not be kept (tried again at the next sweep): %v", n, err)
}
}
ok, err := t.sayNotResolved(is, now)
if err != nil {
t.Logf("[mesh-issues] that %d is delivered and not resolved could not be said on the operator's channel (said at the next sweep): %v", n, err)
} else if ok {
said++
}
if is.Status == StatusWontfix {
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
continue
}
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
if closedStatus(is.Status) {
// A resolved or wontfix issue waits for no fix: never said stuck (novox/hq issue 547).
continue
}
named := liveFixes(is)
for _, f := range is.Fixes {
if !contains(named, f.PullRequest) || contains(is.Delivered, f.PullRequest) || !f.Merged || f.MergedAt == nil ||
@@ -514,7 +602,7 @@ func (t *Tracker) Sweep() (int, error) {
}
}
pr := f.PullRequest
if _, err := t.change(n, func(i *Issue) error {
if _, err := t.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if g := fixOf(i, pr); g != nil {
g.StuckSaid = true
}
@@ -526,5 +614,8 @@ func (t *Tracker) Sweep() (int, error) {
said++
}
}
if err := t.judgeFixBudgets(); err != nil {
t.Logf("[mesh-issues] a fix over its budget could not be said or cleared on the operator's channel (tried again at the next sweep): %v", err)
}
return said, nil
}
@@ -11,10 +11,22 @@ import (
)
type notified struct {
mu sync.Mutex
said []string
clear []string
err error
mu sync.Mutex
said []string
clear []string
resolved []string
err error
}
// fixedAtFixing names a fix while the issue stays at fixing, as an agent does when the fix merges (ADR 0281 §3,
// ADR 0284): a fix stuck is said only for an issue not closed (issue 547).
func fixedAtFixing(t *testing.T, h *harness, n uint64, fixedBy ...string) {
t.Helper()
h.fixing(t, n)
if _, err := h.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", LocatedIn: []string{"mesh-catalog modules/records"},
FixedBy: fixedBy}); err != nil {
t.Fatal(err)
}
}
func (n *notified) notify(key, severity, summary, headline, explanation, resolved string, clear bool) error {
@@ -25,6 +37,7 @@ func (n *notified) notify(key, severity, summary, headline, explanation, resolve
}
if clear {
n.clear = append(n.clear, key)
n.resolved = append(n.resolved, resolved)
} else {
n.said = append(n.said, key+" "+severity+" "+summary)
}
@@ -121,7 +134,7 @@ func TestAnIssueKnowsWhetherItsFixIsDelivered(t *testing.T) {
func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
resolve(t, h, n, "mesh-catalog#9")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
@@ -152,7 +165,7 @@ func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
// Under the bound nothing is said.
h2 := newHarness(t)
m := h2.open(t, "y")
resolve(t, h2, m, "mesh-catalog#9")
fixedAtFixing(t, h2, m, "mesh-catalog#9")
n2 := &notified{}
h2.Notify = n2.notify
transition(t, h2, 9, "delivering", "", "abc", time.Date(2026, 10, 10, 11, 30, 0, 0, time.UTC))
@@ -194,7 +207,8 @@ func TestAFixIsNamedWhileTheIssueStaysLocated(t *testing.T) {
func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
h := newHarness(t)
n := h.open(t, "x")
resolve(t, h, n, "mesh-catalog#9")
fixedAtFixing(t, h, n, "mesh-catalog#9")
h.settings.DeliveredNotResolvedAfter = "24h" // only the stuck warning is under test here
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
@@ -244,3 +258,78 @@ func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
t.Fatalf("%d asks, %q", asks, v.Delivery)
}
}
// A resolved issue is the end of its stuck warning, as wontfix is: resolving clears it on the operator's channel,
// saying so, and the sweep never says a fix stuck for an issue resolved or wontfix, even one whose delivery failed
// and never reads delivered (issue 547).
func TestAResolvedIssueEndsItsStuckWarningAndIsNeverSaidStuck(t *testing.T) {
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
moved := func(t *testing.T, h *harness, n uint64) {
t.Helper()
raw, _ := json.Marshal(map[string]any{"number": n})
if err := h.OnMoved(stdio.Envelope{Body: raw}); err != nil {
t.Fatal(err)
}
}
for _, to := range []string{StatusResolved, StatusWontfix} {
t.Run(to, func(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed on one machine", "abc", merged)
if said, err := h.Sweep(); err != nil || said != 1 || len(note.said) != 1 {
t.Fatalf("not said stuck: %d %v %q", said, err, note.said)
}
ask := MoveAsk{Number: n, Status: to, FixedBy: []string{"mesh-catalog#9"}, ReplayNone: "proven live"}
if to == StatusWontfix {
ask = MoveAsk{Number: n, Status: to, Reason: "not worth it"}
}
if _, err := h.Move(ask); err != nil {
t.Fatal(err)
}
moved(t, h, n)
if len(note.clear) != 1 || note.clear[0] != stuckKey(n) || get(t, h, n).StuckOpen {
t.Fatalf("not cleared at the move: %q", note.clear)
}
if to == StatusResolved && !strings.HasPrefix(note.resolved[0], "resolved: ") {
t.Fatalf("%q", note.resolved)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 1 || len(note.clear) != 1 {
t.Fatalf("said again after %s: %d %q, cleared %q", to, said, note.said, note.clear)
}
})
}
// A warning left open by a tracker that kept it for a resolved issue is cleared at the first sweep, and an
// issue resolved before its fix was ever said stuck is never said stuck.
h := newHarness(t)
n := h.open(t, "x")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{err: errStuck}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed", "abc", merged)
if _, err := h.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
i.Fixes[0].StuckSaid, i.StuckOpen = true, true
return nil
}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: n, Status: StatusResolved, FixedBy: []string{"mesh-catalog#9"}}); err != nil {
t.Fatal(err)
}
moved(t, h, n) // the channel does not answer: kept open, tried at the sweep
note.err = nil
if said, _ := h.Sweep(); said != 0 || len(note.clear) != 1 || get(t, h, n).StuckOpen {
t.Fatalf("said %d, cleared %q", said, note.clear)
}
m := h.open(t, "y")
fixedAtFixing(t, h, m, "mesh-catalog#10")
transition(t, h, 10, "failed", "the gate failed", "abc", merged)
if _, err := h.Move(MoveAsk{Number: m, Status: StatusResolved, FixedBy: []string{"mesh-catalog#10"}}); err != nil {
t.Fatal(err)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 0 {
t.Fatalf("a resolved issue said stuck: %q", note.said)
}
}
@@ -0,0 +1,647 @@
package main
// A fix's time (novox/hq ADR 0313, issue 403). An issue's fix time runs from its first move to fixing to the moment
// the last of its fixes is delivered to every machine it should reach. It is held to the fix budget of its class —
// five minutes for a leaf module, ten for a core one — and split into four phases read from the fix delivered
// last: implementing (to its pull request opened), review (to an approval at its head), checks (to its merge) and
// merge to running (to delivered, its delivery time). A fix over its budget is said on the operator's channel once
// it is final, naming the issue, its worker and its longest phase; the warning clears when the next issue of that
// class is delivered within its budget. `fix-times` answers the last days' median and worst per class.
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// The classes of a fix (ADR 0282 §1, ADR 0313 §2) and the class of a fix that moves no module.
const (
ClassCore = "core"
ClassLeaf = "leaf"
ClassNone = "none"
)
// FixBudgets are the fix budgets by class, counted from the move to fixing (ADR 0313 §2).
var FixBudgets = map[string]time.Duration{ClassLeaf: 5 * time.Minute, ClassCore: 10 * time.Minute}
// fixClasses in the order `fix-times` says them.
var fixClasses = []string{ClassLeaf, ClassCore}
// The phases of a fix time, in order (ADR 0313 §4).
const (
PhaseImplementing = "implementing"
PhaseReview = "review"
PhaseChecks = "checks"
PhaseRunning = "merge to running"
)
// FixTime is an issue's fix time, kept on it once the last of its fixes is delivered.
type FixTime struct {
// From is the first move to fixing; End when the fix delivered last was delivered.
From time.Time `json:"from"`
End time.Time `json:"end"`
TookMS int64 `json:"took_ms"`
Took string `json:"took"`
// Class is core, leaf or none; empty when a fix's class is not known. A fix of class none, or one not known,
// has no budget: its time is shown and never warned on.
Class string `json:"class,omitempty"`
Budget string `json:"budget,omitempty"`
BudgetMS int64 `json:"budget_ms,omitempty"`
Over bool `json:"over,omitempty"`
// Fix is the pull request delivered last, which the phases are read from; Worker who worked on the issue.
Fix string `json:"fix"`
Worker string `json:"worker,omitempty"`
Phases []FixPhase `json:"phases"`
Longest string `json:"longest,omitempty"`
Said string `json:"said"`
// Warned says the over-budget warning was said for it on the operator's channel and not cleared since.
Warned bool `json:"warned,omitempty"`
// FromLog says its start was read back from the log, not kept by the move: an issue moved to fixing before the
// tracker measured fix times. Its fix time is kept and shown, and never warned on, so the first fixes heard of
// after the tracker starts measuring raise no warning for work done before it (novox/hq issue 403).
FromLog bool `json:"from-log,omitempty"`
}
// FixPhase is one phase of a fix time: measured, or unknown when the moment that ends it is not known — its time is
// then counted in the phase after it, which says so.
type FixPhase struct {
Name string `json:"name"`
State string `json:"state"`
TookMS int64 `json:"took_ms"`
Took string `json:"took"`
Said string `json:"said,omitempty"`
}
// Approval is when a pull request was first approved at its head, as the forge says it; a test stubs the asking.
// found is false when no approval at its head is on the forge.
type Approval func(repository string, number int) (at time.Time, found bool, err error)
// classOfFixes is the class of an issue's fixes: that of the slowest-budgeted module they move — core when any
// moves a core module, leaf when any moves a leaf one, none when none moves a module — and empty, with why, when
// a fix's class is not known and none is core.
func classOfFixes(is *Issue, prs []string) (string, string) {
var unknown []string
leaf := false
for _, pr := range prs {
f := fixOf(is, pr)
switch {
case f == nil || f.Class == "":
unknown = append(unknown, pr)
case f.Class == ClassCore:
return ClassCore, ""
case f.Class == ClassLeaf:
leaf = true
}
}
if len(unknown) > 0 {
return "", "the delivery did not say what " + strings.Join(unknown, ", ") + " moves"
}
if leaf {
return ClassLeaf, ""
}
return ClassNone, ""
}
// fixPhases splits the span from start to the delivered moment at the moments that end each phase, in order. A
// phase whose end is not known is unknown, and its time counts in the next phase known; a phase whose end came
// before the end of the phase before it counts zero (ADR 0313 §4). The phases add up to the fix time.
func fixPhases(start time.Time, ends []struct {
name string
at *time.Time
why string
}) ([]FixPhase, string) {
var out []FixPhase
prev := start
var carried []string
for _, e := range ends {
if e.at == nil {
out = append(out, FixPhase{Name: e.name, State: "unknown", Took: "0s", Said: e.why})
carried = append(carried, e.name)
continue
}
took := time.Duration(0)
if e.at.After(prev) {
took = e.at.Sub(prev)
prev = *e.at
}
p := FixPhase{Name: e.name, State: "measured", TookMS: took.Milliseconds(), Took: took.Round(time.Second).String()}
if len(carried) > 0 {
p.Said = "counts " + strings.Join(carried, " and ") + " too: its end is not known"
carried = nil
}
out = append(out, p)
}
best := -1
for i, p := range out {
if p.State == "measured" && (best < 0 || p.TookMS > out[best].TookMS) {
best = i
}
}
longest := ""
if best >= 0 {
longest = out[best].Name + " " + out[best].Took
}
return out, longest
}
// startOfFixing is when an issue was first moved to fixing (ADR 0313 §1), and whether that moment came from its
// log rather than from the move that kept it: the earliest of what it keeps (fixing-from, kept by the move since the
// tracker measures fix times) and what its log says. An issue moved before the moment was kept has it read from its
// log: a move's operation says its own moment. A value carried into the log, or imported, holds the issue whole and
// no moment of its own (it is stamped when it was carried); when the value before it was not at fixing it is the
// move's own write, and its worker's active is the move's moment (a move to fixing marks its worker active), not
// its worker's since, which a move from diagnosing by the same worker keeps (ADR 0284 §6). Only a first value
// already at fixing falls back to its worker's since, the best moment the log holds for it. Zero when it never was.
func (t *Tracker) startOfFixing(is *Issue) (time.Time, bool) {
var kept time.Time
if is.FixingFrom != nil {
kept = is.FixingFrom.UTC()
}
if t.Log == nil {
return kept, false
}
entries, err := t.readKey(is.Number, 0)
if err != nil {
t.Logf("[mesh-issues] the log of %d could not be read for its first move to fixing (its kept moment is used): %v", is.Number, err)
return kept, false
}
var logged time.Time
status, seen := "", false
for _, e := range entries {
var op Operation
if json.Unmarshal(e.Value, &op) != nil {
continue
}
at, became := op.At, ""
switch {
case op.Issue != nil:
became = op.Issue.Status
if w := op.Issue.Worker; w != nil {
switch {
case seen && status != StatusFixing && !w.Active.IsZero():
at = w.Active
case !w.Since.IsZero():
at = w.Since
}
}
case op.Set["status"] != nil:
_ = json.Unmarshal(op.Set["status"], &became)
default:
continue
}
if became == StatusFixing && status != StatusFixing && !at.IsZero() {
logged = at.UTC()
break
}
status, seen = became, true
}
switch {
case kept.IsZero():
return logged, !logged.IsZero()
case !logged.IsZero() && logged.Before(kept):
return logged, true
}
return kept, false
}
// fixTimeOf is an issue's fix time once every fix it names is delivered; nil, with why, when it is not measured.
func (t *Tracker) fixTimeOf(is *Issue) (*FixTime, string) {
if !allDelivered(is) {
return nil, "not every fix is delivered"
}
start, fromLog := t.startOfFixing(is)
if start.IsZero() {
return nil, "it was never moved to fixing"
}
prs := liveFixes(is)
var last *Fix
for _, pr := range prs {
if f := fixOf(is, pr); f != nil && (last == nil || f.Since.After(last.Since)) {
last = f
}
}
if last == nil {
return nil, "no fix is heard of"
}
end := last.Since.UTC()
took := time.Duration(0)
if end.After(start) {
took = end.Sub(start)
}
ft := &FixTime{From: start, End: end, TookMS: took.Milliseconds(), Took: took.Round(time.Second).String(), Fix: last.PullRequest,
FromLog: fromLog}
if is.Worker != nil {
ft.Worker = is.Worker.By
} else if is.FixTime != nil {
ft.Worker = is.FixTime.Worker
}
class, why := classOfFixes(is, prs)
ft.Class = class
if b, ok := FixBudgets[class]; ok {
ft.Budget, ft.BudgetMS, ft.Over = b.String(), b.Milliseconds(), took > b
}
// The phases, from the fix delivered last.
type end_ = struct {
name string
at *time.Time
why string
}
var approved *time.Time
approvalWhy := "no approval at its head is on the forge: review and checks are counted together"
if t.Approval != nil {
repo, number, _ := strings.Cut(last.PullRequest, "#")
var n int
if _, err := fmt.Sscanf(number, "%d", &n); err == nil {
at, found, err := t.Approval(repo, n)
switch {
case err != nil:
approvalWhy = "the forge could not be read for its approval (" + cut(err.Error(), 200) + "): review and checks are counted together"
case found:
a := at.UTC()
approved = &a
}
}
} else {
approvalWhy = "the forge is not asked here: review and checks are counted together"
}
openedWhy := "the delivery did not say when its pull request opened: implementing is counted in review"
mergedWhy := "the delivery did not say when it merged: checks are counted in merge to running"
ft.Phases, ft.Longest = fixPhases(start, []end_{
{PhaseImplementing, last.Opened, openedWhy},
{PhaseReview, approved, approvalWhy},
{PhaseChecks, last.MergedAt, mergedWhy},
{PhaseRunning, &end, ""},
})
var said []string
said = append(said, fmt.Sprintf("fixed in %s, from the first move to fixing (%s) to %s delivered (%s)", ft.Took,
start.Format(time.RFC3339), last.PullRequest, end.Format(time.RFC3339)))
switch {
case class == ClassNone:
said = append(said, "its fixes move no module, so it has no fix budget")
case class == "":
said = append(said, "no fix budget: "+why)
case ft.Over:
said = append(said, fmt.Sprintf("over its %s fix budget of %s", class, ft.Budget))
default:
said = append(said, fmt.Sprintf("within its %s fix budget of %s", class, ft.Budget))
}
if ft.Longest != "" {
said = append(said, "longest: "+ft.Longest)
}
if fromLog {
said = append(said, "its start was read from its log (moved to fixing before fix times were measured), so it is never warned on")
}
ft.Said = strings.Join(said, "; ")
return ft, ""
}
// sameFixTime says whether two fix times say the same, the warning's mark aside.
func sameFixTime(a, b *FixTime) bool {
if a == nil || b == nil {
return a == b
}
x, y := *a, *b
x.Warned, y.Warned = false, false
ra, _ := json.Marshal(x)
rb, _ := json.Marshal(y)
return string(ra) == string(rb)
}
// timeFix keeps an issue's fix time once every fix it names is delivered, and again when a later fix moves its
// end; then the warnings of its class are judged.
func (t *Tracker) timeFix(n uint64) error {
is, _, err := read(t.Store, n)
if err != nil {
return err
}
ft, _ := t.fixTimeOf(is)
if ft == nil || sameFixTime(ft, is.FixTime) {
return nil
}
if _, err := t.change(n, Act{Op: "fix-time", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
ft.Warned = i.FixTime.Warned
}
i.FixTime = ft
return nil
}); err != nil {
return err
}
if err := t.judgeFixBudgets(); err != nil {
t.Logf("[mesh-issues] a fix over its budget could not be said or cleared on the operator's channel (tried again at the next sweep): %v", err)
}
return nil
}
// fixOverKey is the warning a fix over its class's budget raises (ADR 0313 §6).
func fixOverKey(class string) string { return "issue-tracker.fix." + class + ".over-budget" }
// judgeFixBudgets says or clears each class's warning by the newest fix time of the class (ADR 0313 §6): the newest
// over its budget and not yet said is said; the newest within its budget clears what was said for the class. Run
// after every fix time kept and at every sweep, so a warning that failed is said, or cleared, at the next.
func (t *Tracker) judgeFixBudgets() error {
if t.Notify == nil {
return nil
}
all, err := t.allIssues()
if err != nil {
return err
}
var firstErr error
for _, class := range fixClasses {
var newest *Issue
var warned []uint64
for _, is := range all {
ft := is.FixTime
if ft == nil || ft.Class != class {
continue
}
if ft.Warned {
warned = append(warned, is.Number)
}
if newest == nil || ft.End.After(newest.FixTime.End) || (ft.End.Equal(newest.FixTime.End) && is.Number > newest.Number) {
newest = is
}
}
if newest == nil {
continue
}
ft := newest.FixTime
if ft.Over {
if ft.Warned || ft.FromLog {
continue
}
worker := ft.Worker
if worker == "" {
worker = "nobody named"
}
longest := ft.Longest
if longest == "" {
longest = "not known"
}
if err := t.Notify(fixOverKey(class), "warning",
fmt.Sprintf("issue %d: fixed in %s, over its %s fix budget of %s (longest: %s)", newest.Number, ft.Took, class, ft.Budget, longest),
fmt.Sprintf("A fix took longer than its budget (issue %d)", newest.Number),
fmt.Sprintf("Issue %d was fixed in %s, from %s's move to fixing to its last fix delivered, over the %s fix "+
"budget of %s. Its longest phase was %s. Nothing is needed from you: it says where fixes lose time. "+
"It clears when the next %s fix is delivered within its budget.", newest.Number, ft.Took, worker, class,
ft.Budget, longest, class), "", false); err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
if _, err := t.change(newest.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
i.FixTime.Warned = true
}
return nil
}); err != nil && firstErr == nil {
firstErr = err
}
continue
}
if len(warned) == 0 || ft.BudgetMS == 0 {
continue
}
if err := t.Notify(fixOverKey(class), "", "", "", "", fmt.Sprintf("issue %d's fix was delivered within its %s fix budget (%s of %s)",
newest.Number, class, ft.Took, ft.Budget), true); err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
sort.Slice(warned, func(i, j int) bool { return warned[i] < warned[j] })
for _, n := range warned {
if _, err := t.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
i.FixTime.Warned = false
}
return nil
}); err != nil && firstErr == nil {
firstErr = err
}
}
}
return firstErr
}
// FixTimesClass is one class's line of `fix-times`.
type FixTimesClass struct {
Class string `json:"class"`
Budget string `json:"budget"`
BudgetMS int64 `json:"budget_ms"`
Count int `json:"count"`
Within int `json:"within"`
Over int `json:"over"`
Median string `json:"median,omitempty"`
MedianMS int64 `json:"median_ms,omitempty"`
Worst string `json:"worst,omitempty"`
WorstMS int64 `json:"worst_ms,omitempty"`
// WorstIssue is the issue whose fix took the worst time.
WorstIssue uint64 `json:"worst_issue,omitempty"`
}
// FixTimesRow is an issue as `fix-times` names it.
type FixTimesRow struct {
Number uint64 `json:"number"`
Title string `json:"title"`
Class string `json:"class,omitempty"`
Took string `json:"took"`
TookMS int64 `json:"took_ms"`
Budget string `json:"budget,omitempty"`
Longest string `json:"longest,omitempty"`
Worker string `json:"worker,omitempty"`
Delivered string `json:"delivered"`
}
// FixTimes is the answer of `fix-times`.
type FixTimes struct {
Days int `json:"days"`
Since string `json:"since"`
Line string `json:"line"`
Classes []FixTimesClass `json:"classes"`
// Over is every issue over its budget in the window, newest first, with its longest phase.
Over []FixTimesRow `json:"over,omitempty"`
// Unbudgeted are the issues fixed in the window with no fix budget: their fixes move no module, or what they
// move is not known.
Unbudgeted []FixTimesRow `json:"unbudgeted,omitempty"`
}
// MostFixTimesDays is the longest window `fix-times` reads.
const MostFixTimesDays = 90
// FixTimes is the `fix-times` verb: each class's fix budget, the last days' median and worst fix time, and every
// issue over its budget with its longest phase (ADR 0313 §5, §7). An issue counts in the window its last fix was
// delivered in.
func (t *Tracker) FixTimes(days int) (*FixTimes, error) {
if days <= 0 {
days = 7
}
if days > MostFixTimesDays {
return nil, fmt.Errorf("days %d is more than the %d fix-times reads", days, MostFixTimesDays)
}
all, err := t.allIssues()
if err != nil {
return nil, err
}
since := t.now().Add(-time.Duration(days) * 24 * time.Hour)
a := &FixTimes{Days: days, Since: since.Format(time.RFC3339)}
byClass := map[string][]*Issue{}
row := func(is *Issue) FixTimesRow {
ft := is.FixTime
return FixTimesRow{Number: is.Number, Title: is.Title, Class: ft.Class, Took: ft.Took, TookMS: ft.TookMS, Budget: ft.Budget,
Longest: ft.Longest, Worker: ft.Worker, Delivered: ft.End.Format(time.RFC3339)}
}
for _, is := range all {
ft := is.FixTime
if ft == nil || ft.End.Before(since) {
continue
}
if _, ok := FixBudgets[ft.Class]; !ok {
a.Unbudgeted = append(a.Unbudgeted, row(is))
continue
}
byClass[ft.Class] = append(byClass[ft.Class], is)
if ft.Over {
a.Over = append(a.Over, row(is))
}
}
newestFirst := func(rows []FixTimesRow) {
sort.Slice(rows, func(i, j int) bool {
if rows[i].Delivered != rows[j].Delivered {
return rows[i].Delivered > rows[j].Delivered
}
return rows[i].Number > rows[j].Number
})
}
newestFirst(a.Over)
newestFirst(a.Unbudgeted)
var line []string
for _, class := range fixClasses {
b := FixBudgets[class]
c := FixTimesClass{Class: class, Budget: b.String(), BudgetMS: b.Milliseconds()}
is := byClass[class]
sort.Slice(is, func(i, j int) bool {
if is[i].FixTime.TookMS != is[j].FixTime.TookMS {
return is[i].FixTime.TookMS < is[j].FixTime.TookMS
}
return is[i].Number < is[j].Number
})
c.Count = len(is)
for _, i := range is {
if i.FixTime.Over {
c.Over++
} else {
c.Within++
}
}
if n := len(is); n > 0 {
ms := is[n/2].FixTime.TookMS
if n%2 == 0 {
ms = (is[n/2-1].FixTime.TookMS + is[n/2].FixTime.TookMS) / 2
}
med := time.Duration(ms) * time.Millisecond
c.MedianMS, c.Median = ms, med.Round(time.Second).String()
worst := is[n-1]
c.WorstMS, c.Worst, c.WorstIssue = worst.FixTime.TookMS, worst.FixTime.Took, worst.Number
line = append(line, fmt.Sprintf("%s: median %s, worst %s (issue %d), %d of %d within %s", class, c.Median,
c.Worst, c.WorstIssue, c.Within, c.Count, c.Budget))
} else {
line = append(line, fmt.Sprintf("%s: no issue fixed", class))
}
a.Classes = append(a.Classes, c)
}
a.Line = fmt.Sprintf("fix time over the last %d day(s), from the move to fixing to delivered — %s", days, strings.Join(line, "; "))
if len(a.Unbudgeted) > 0 {
a.Line += fmt.Sprintf("; %d fixed with no fix budget", len(a.Unbudgeted))
}
return a, nil
}
// firstApproval is when a pull request was first approved at its head among the forge's reviews: the forge's own
// approval, or — the forge refusing an approval of one's own pull request, so the mesh's independent reviews are
// posted as comments by the account that opened it — a comment review whose first line is "Verdict: approve at
// <sha>", the sha the head it was posted at or a prefix of it of at least seven characters. An approval dismissed,
// stale, or of an older head does not count.
func firstApproval(raw json.RawMessage) (time.Time, bool, error) {
var a struct {
Reviews []struct {
State string `json:"state"`
AtHead bool `json:"at_head"`
Commit string `json:"commit"`
Body string `json:"body"`
Submitted string `json:"submitted"`
Stale bool `json:"stale"`
Dismissed bool `json:"dismissed"`
} `json:"reviews"`
}
var s string
if json.Unmarshal(raw, &s) == nil {
raw = json.RawMessage(s)
}
var wrapped struct {
Content []struct {
Text string `json:"text"`
} `json:"content"`
}
if json.Unmarshal(raw, &wrapped) == nil && len(wrapped.Content) > 0 {
raw = json.RawMessage(wrapped.Content[0].Text)
}
if err := json.Unmarshal(raw, &a); err != nil || a.Reviews == nil {
return time.Time{}, false, fmt.Errorf("the forge's gitea_pull_reviews answered no reviews: %.200s", raw)
}
var first time.Time
for _, r := range a.Reviews {
if !r.AtHead || r.Stale || r.Dismissed {
continue
}
switch r.State {
case "APPROVED":
case "COMMENT":
if !verdictApproves(r.Body, r.Commit) {
continue
}
default:
continue
}
at, err := time.Parse(time.RFC3339, r.Submitted)
if err != nil {
continue
}
if first.IsZero() || at.Before(first) {
first = at
}
}
return first, !first.IsZero(), nil
}
// verdictApproves says whether a review's body opens with the mesh's approving verdict on the commit it was posted
// at: "Verdict: approve at <sha>", the sha that commit or a prefix of it of at least seven characters.
func verdictApproves(body, commit string) bool {
line, _, _ := strings.Cut(strings.TrimLeft(body, " \t\r\n"), "\n")
rest, ok := strings.CutPrefix(strings.TrimSpace(line), "Verdict: approve at ")
if !ok {
return false
}
sha := strings.ToLower(strings.TrimRight(strings.TrimSpace(rest), "."))
commit = strings.ToLower(strings.TrimSpace(commit))
return len(sha) >= 7 && commit != "" && strings.Trim(sha, "0123456789abcdef") == "" && strings.HasPrefix(commit, sha)
}
// busApproval asks the forge's reviews of a pull request over the bus (gitea_pull_reviews).
func busApproval(owner func() string) Approval {
return func(repository string, number int) (time.Time, bool, error) {
raw, err := stdio.Ask("gitea.gitea_pull_reviews", map[string]any{"owner": owner(), "repo": repository, "number": number})
if err != nil {
return time.Time{}, false, err
}
return firstApproval(raw)
}
}
@@ -0,0 +1,480 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// novox/hq ADR 0313 (issue 403): an issue's fix time runs from its first move to fixing to its last fix
// delivered; it is held to its class's fix budget, split into four phases that add up to it, shown in get and
// fix-times, and a fix over its budget is said on the operator's channel once final, cleared at the next of its
// class delivered within budget.
// fixClock is a harness whose clock a test moves.
type fixClock struct {
*harness
at time.Time
// approvals is when each pull request was approved at its head, as the forge would say it.
approvals map[int]time.Time
}
func newFixClock(t *testing.T) *fixClock {
c := &fixClock{harness: newHarness(t), at: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC), approvals: map[int]time.Time{}}
c.Tracker.Now = func() time.Time { return c.at }
return c
}
func (c *fixClock) approval(repository string, number int) (time.Time, bool, error) {
a, ok := c.approvals[number]
return a, ok, nil
}
func (c *fixClock) later(d time.Duration) time.Time { c.at = c.at.Add(d); return c.at }
// delivery says one transition of a pull request as mesh-delivery says it, with what a fix's time is read by.
func (c *fixClock) delivery(t *testing.T, number int, to, class string, opened, mergedAt *time.Time) {
t.Helper()
body := map[string]any{"repository": "novox/mesh-catalog", "number": number, "to": to, "at": c.at}
if mergedAt != nil {
body["merged_as"] = "m" + strings.Repeat("a", 11)
body["merged_at"] = *mergedAt
}
if class != "" {
body["class"] = class
}
if opened != nil {
body["opened"] = *opened
}
raw, _ := json.Marshal(body)
if err := c.OnTransition(stdio.Envelope{Key: "mesh-delivery.transition", Body: raw}); err != nil {
t.Fatal(err)
}
}
func at(t time.Time) *time.Time { return &t }
func phaseTook(t *testing.T, ft *FixTime) map[string]time.Duration {
t.Helper()
out := map[string]time.Duration{}
var sum int64
for _, p := range ft.Phases {
out[p.Name] = time.Duration(p.TookMS) * time.Millisecond
sum += p.TookMS
}
if sum != ft.TookMS {
t.Fatalf("the phases add up to %dms, the fix time is %dms: %+v", sum, ft.TookMS, ft.Phases)
}
return out
}
// fixOne takes an issue through a fix: opened, moved to fixing, a pull request opened, approved, merged and
// delivered after the given spans; its fix time as kept.
func (c *fixClock) fixOne(t *testing.T, title string, pr int, class string, implementing, review, checks, running time.Duration) uint64 {
t.Helper()
n := c.open(t, title)
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#" + itoa(pr)}}); err != nil {
t.Fatal(err)
}
opened := c.later(implementing)
c.delivery(t, pr, "proposed", "", &opened, nil)
approved := c.later(review)
c.approvals[pr] = approved
merged := c.later(checks)
c.delivery(t, pr, "published", class, &opened, &merged)
c.later(running)
c.delivery(t, pr, "delivered", class, &opened, &merged)
return n
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
func TestAFixTimeRunsFromTheFirstMoveToFixingAndAddsUpItsPhases(t *testing.T) {
c := newFixClock(t)
c.Tracker.Approval = c.approval
note := &notified{}
c.Notify = note.notify
n := c.open(t, "a fix measured")
c.later(time.Minute)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusDiagnosing, By: "fix@g14"}); err != nil {
t.Fatal(err)
}
c.later(10 * time.Minute) // finding the cause is not the fix's time
start := c.at
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#31"}}); err != nil {
t.Fatal(err)
}
c.later(30 * time.Second)
// A take-over, and a move away and back, restart nothing.
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "other@shanks", TakeOver: "the first worker's session ended"}); err != nil {
t.Fatal(err)
}
if _, err := c.Move(MoveAsk{Number: n, Status: StatusOpen}); err != nil {
t.Fatal(err)
}
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "other@shanks"}); err != nil {
t.Fatal(err)
}
opened := c.later(90 * time.Second) // implementing: 2m
c.delivery(t, 31, "proposed", "", &opened, nil)
c.approvals[31] = c.later(time.Minute) // review: 1m
merged := c.later(3 * time.Minute) // checks: 3m
c.delivery(t, 31, "published", "leaf", &opened, &merged)
if is := get(t, c.harness, n); is.FixTime != nil {
t.Fatalf("a fix merged and not delivered has a fix time: %+v", is.FixTime)
}
c.later(2 * time.Minute) // merge to running: 2m
c.delivery(t, 31, "delivered", "leaf", &opened, &merged)
v, err := c.View(n)
if err != nil {
t.Fatal(err)
}
ft := v.FixTime
if ft == nil {
t.Fatalf("no fix time once the fix is delivered: %+v", v.Issue)
}
if !ft.From.Equal(start) || ft.TookMS != (8*time.Minute).Milliseconds() {
t.Fatalf("fix time from %s took %s, want from %s (the first move to fixing) and 8m0s", ft.From, ft.Took, start)
}
got := phaseTook(t, ft)
if got[PhaseImplementing] != 2*time.Minute || got[PhaseReview] != time.Minute || got[PhaseChecks] != 3*time.Minute ||
got[PhaseRunning] != 2*time.Minute {
t.Fatalf("phases %v", got)
}
if ft.Class != ClassLeaf || !ft.Over || ft.Budget != "5m0s" || ft.Longest != "checks 3m0s" || ft.Worker != "other@shanks" {
t.Fatalf("class %q over %v budget %q longest %q worker %q", ft.Class, ft.Over, ft.Budget, ft.Longest, ft.Worker)
}
// Over its budget: said once, naming the issue, its worker and its longest phase.
if len(note.said) != 1 || !strings.HasPrefix(note.said[0], "issue-tracker.fix.leaf.over-budget warning issue "+itoa(int(n))) ||
!strings.Contains(note.said[0], "checks 3m0s") {
t.Fatalf("said %q", note.said)
}
// Heard again: nothing new is kept or said.
c.delivery(t, 31, "delivered", "leaf", &opened, &merged)
if len(note.said) != 1 {
t.Fatalf("said again: %q", note.said)
}
}
// A phase whose end is not known is unknown and counted in the next; one that ended before the phase before it
// counts zero (a pull request opened before the move to fixing).
func TestAPhaseNotKnownIsCountedInTheNextAndOneEndedEarlyCountsZero(t *testing.T) {
c := newFixClock(t)
n := c.open(t, "a fix whose pull request came first")
opened := c.at
c.later(time.Minute)
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#32"}}); err != nil {
t.Fatal(err)
}
merged := c.later(4 * time.Minute)
c.later(time.Minute)
c.delivery(t, 32, "delivered", "leaf", &opened, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil {
t.Fatal("no fix time")
}
got := phaseTook(t, ft)
if got[PhaseImplementing] != 0 || got[PhaseChecks] != 4*time.Minute || got[PhaseRunning] != time.Minute {
t.Fatalf("phases %v", got)
}
for _, p := range ft.Phases {
if p.Name == PhaseReview && (p.State != "unknown" || p.Said == "") {
t.Fatalf("review with no approval asked: %+v", p)
}
if p.Name == PhaseChecks && !strings.Contains(p.Said, "review") {
t.Fatalf("checks does not say it counts review: %+v", p)
}
}
if ft.Over {
t.Fatalf("5m of a leaf's 5m is within it: %+v", ft)
}
}
// The class is the slowest-budgeted of the fixes'; a fix that moves no module has no budget and is never warned
// on; one whose class is not said has none either.
func TestAnIssuesClassIsItsSlowestFixAndOneMovingNothingHasNoBudget(t *testing.T) {
c := newFixClock(t)
note := &notified{}
c.Notify = note.notify
n := c.open(t, "two fixes, one core")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#41", "mesh-catalog#42"}}); err != nil {
t.Fatal(err)
}
merged := c.later(time.Minute)
c.delivery(t, 41, "delivered", "leaf", nil, &merged)
if get(t, c.harness, n).FixTime != nil {
t.Fatal("a fix time before the last fix is delivered")
}
c.later(7 * time.Minute)
c.delivery(t, 42, "delivered", "core", nil, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil || ft.Class != ClassCore || ft.Over || ft.Fix != "mesh-catalog#42" {
t.Fatalf("8m of a core fix: %+v", ft)
}
none := c.fixOne(t, "a fix in hq alone", 43, ClassNone, time.Minute, time.Minute, 10*time.Minute, time.Second)
ft = get(t, c.harness, none).FixTime
if ft == nil || ft.Class != ClassNone || ft.Over || ft.BudgetMS != 0 || !strings.Contains(ft.Said, "no fix budget") {
t.Fatalf("a fix moving nothing: %+v", ft)
}
unknown := c.fixOne(t, "a fix whose class is not said", 44, "", time.Minute, time.Minute, 10*time.Minute, time.Second)
ft = get(t, c.harness, unknown).FixTime
if ft == nil || ft.Class != "" || ft.Over || !strings.Contains(ft.Said, "no fix budget") {
t.Fatalf("a fix of no known class: %+v", ft)
}
if len(note.said) != 0 {
t.Fatalf("warned on a fix with no budget: %q", note.said)
}
}
// Nothing is said while a fix is undelivered, however long; once delivered over budget it is said; the next leaf
// fix delivered within budget clears it.
func TestAFixOverItsBudgetIsSaidOnceFinalAndClearedByTheNextWithin(t *testing.T) {
c := newFixClock(t)
c.Tracker.Approval = c.approval
note := &notified{}
c.Notify = note.notify
n := c.open(t, "a slow fix")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#51"}}); err != nil {
t.Fatal(err)
}
merged := c.later(20 * time.Minute)
c.delivery(t, 51, "published", "leaf", nil, &merged)
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 0 {
t.Fatalf("said while undelivered: %q", note.said)
}
c.later(time.Minute)
c.delivery(t, 51, "delivered", "leaf", nil, &merged)
if len(note.said) != 1 || !strings.HasPrefix(note.said[0], fixOverKey(ClassLeaf)+" warning") {
t.Fatalf("said %q", note.said)
}
if !get(t, c.harness, n).FixTime.Warned {
t.Fatal("not marked said")
}
// A core fix within its budget clears nothing of the leaf's.
c.fixOne(t, "a quick core fix", 52, ClassCore, time.Minute, time.Minute, time.Minute, time.Minute)
if len(note.clear) != 0 {
t.Fatalf("cleared by another class: %q", note.clear)
}
c.fixOne(t, "a quick leaf fix", 53, ClassLeaf, time.Minute, 30*time.Second, time.Minute, time.Minute)
if len(note.clear) != 1 || note.clear[0] != fixOverKey(ClassLeaf) {
t.Fatalf("cleared %q", note.clear)
}
if get(t, c.harness, n).FixTime.Warned {
t.Fatal("still marked said after the clear")
}
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 1 || len(note.clear) != 1 {
t.Fatalf("the sweep said again: %q, cleared %q", note.said, note.clear)
}
}
// A warning the channel did not take is said at the next sweep.
func TestAWarningNotTakenIsSaidAtTheNextSweep(t *testing.T) {
c := newFixClock(t)
note := &notified{err: errFake("the channel is away")}
c.Notify = note.notify
n := c.fixOne(t, "slow", 61, ClassLeaf, 5*time.Minute, time.Minute, time.Minute, time.Minute)
if get(t, c.harness, n).FixTime == nil {
t.Fatal("a fix time is kept although the warning was not taken")
}
note.err = nil
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 1 || !get(t, c.harness, n).FixTime.Warned {
t.Fatalf("said %q", note.said)
}
}
// fix-times answers each class's budget, the window's median and worst, and every issue over its budget with its
// longest phase; an issue delivered before the window does not count, so a lever is judged by the medians before
// and after it.
func TestFixTimesAnswersTheWindowsMedianAndWorst(t *testing.T) {
c := newFixClock(t)
old := c.fixOne(t, "long ago", 70, ClassLeaf, 30*time.Minute, time.Minute, time.Minute, time.Minute)
c.later(8 * 24 * time.Hour)
a := c.fixOne(t, "four minutes", 71, ClassLeaf, time.Minute, time.Minute, time.Minute, time.Minute)
b := c.fixOne(t, "six minutes", 72, ClassLeaf, 3*time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "eight minutes", 73, ClassLeaf, 5*time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "a core fix", 74, ClassCore, time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "hq alone", 75, ClassNone, time.Minute, time.Minute, time.Minute, time.Minute)
out, err := c.call(t, "fix-times", map[string]any{})
if err != nil {
t.Fatal(err)
}
ft := out.(*FixTimes)
leaf, core := ft.Classes[0], ft.Classes[1]
if ft.Days != 7 || leaf.Class != ClassLeaf || leaf.Count != 3 || leaf.Median != "6m0s" || leaf.Worst != "8m0s" ||
leaf.Within != 1 || leaf.Over != 2 || leaf.Budget != "5m0s" {
t.Fatalf("leaf %+v (issue %d, delivered before the window, must not count)", leaf, old)
}
if core.Count != 1 || core.Median != "4m0s" || core.Within != 1 {
t.Fatalf("core %+v", core)
}
if len(ft.Over) != 2 || ft.Over[0].Title != "eight minutes" || ft.Over[0].Longest != "implementing 5m0s" || ft.Over[1].Number != b {
t.Fatalf("over %+v", ft.Over)
}
if len(ft.Unbudgeted) != 1 || ft.Unbudgeted[0].Title != "hq alone" {
t.Fatalf("unbudgeted %+v", ft.Unbudgeted)
}
if !strings.Contains(ft.Line, "leaf: median 6m0s, worst 8m0s") {
t.Fatalf("line %q", ft.Line)
}
// Wider, the old one counts and moves the median.
out, _ = c.call(t, "fix-times", map[string]any{"days": 30})
if l := out.(*FixTimes).Classes[0]; l.Count != 4 || l.Median != "7m0s" {
t.Fatalf("over 30 days: %+v", l)
}
if _, err := c.call(t, "fix-times", map[string]any{"days": 91}); err == nil {
t.Fatal("days 91 not refused")
}
_ = a
}
// An issue moved to fixing before the tracker kept the moment has its start read from its log.
func TestTheStartOfAnIssueMovedBeforeItWasKeptIsReadFromItsLog(t *testing.T) {
c := newFixClock(t)
n := c.open(t, "moved to fixing long ago")
c.later(time.Minute)
start := c.at
c.fixing(t, n)
c.later(time.Hour)
is := get(t, c.harness, n)
if is.FixingFrom == nil || !is.FixingFrom.Equal(start) {
t.Fatalf("fixing-from %v, want %s", is.FixingFrom, start)
}
if got, fromLog := c.startOfFixing(is); !got.Equal(start) || fromLog {
t.Fatalf("kept: %s from the log %v, want %s kept by the move", got, fromLog, start)
}
is.FixingFrom = nil
if got, fromLog := c.startOfFixing(is); !got.Equal(start) || !fromLog {
t.Fatalf("from the log: %s (from the log %v), want %s", got, fromLog, start)
}
}
// A value carried into the log holds no moment of its own: after a value at diagnosing, the first value at fixing
// is the move's own write, and its start is its worker's active (the move), not its worker's since (the start of
// the diagnosis, ADR 0313 §1). A first value already at fixing falls back to its worker's since.
func TestACarriedIssueStartsAtTheMoveNotTheDiagnosis(t *testing.T) {
c := newFixClock(t)
diagnosed := time.Date(2026, 10, 1, 8, 0, 0, 0, time.UTC)
moved := diagnosed.Add(40 * time.Minute)
carry := func(n uint64, is Issue) {
t.Helper()
if _, err := c.log.AppendAt(keyOf(n), Operation{Op: OpCarried, By: Seat, At: c.at, Issue: &is}, c.log.lastOf(keyOf(n))); err != nil {
t.Fatal(err)
}
}
carry(900, Issue{Number: 900, Title: "x", Status: StatusDiagnosing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: diagnosed}})
carry(900, Issue{Number: 900, Title: "x", Status: StatusFixing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: moved}})
if got, fromLog := c.startOfFixing(&Issue{Number: 900}); !got.Equal(moved) || !fromLog {
t.Fatalf("start %s (from the log %v), want the move at %s, not the diagnosis at %s", got, fromLog, moved, diagnosed)
}
carry(901, Issue{Number: 901, Title: "y", Status: StatusFixing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: moved}})
if got, _ := c.startOfFixing(&Issue{Number: 901}); !got.Equal(diagnosed) {
t.Fatalf("a first value at fixing: %s, want its worker's since %s", got, diagnosed)
}
}
// An issue moved to fixing before fix times were measured gets its fix time when its fix is next heard of, with its
// start from its log; it is kept and shown, never warned on, and leaves the class's warning to a fix measured whole.
func TestAFixWhoseStartCameFromTheLogIsNeverWarnedOn(t *testing.T) {
c := newFixClock(t)
note := &notified{}
c.Notify = note.notify
n := c.open(t, "fixed before the tracker measured")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#81"}}); err != nil {
t.Fatal(err)
}
// As kept before this change: no fixing-from.
if _, err := c.change(n, Act{Op: "test", By: Seat}, func(i *Issue) error { i.FixingFrom = nil; return nil }); err != nil {
t.Fatal(err)
}
merged := c.later(3 * time.Hour)
c.delivery(t, 81, "delivered", ClassLeaf, nil, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil || !ft.Over || !ft.FromLog || ft.Warned || ft.TookMS != (3*time.Hour).Milliseconds() {
t.Fatalf("fix time %+v", ft)
}
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 0 {
t.Fatalf("warned on a fix whose start came from the log: %q", note.said)
}
// A fix measured whole after it is warned on as before.
c.fixOne(t, "measured whole", 82, ClassLeaf, 6*time.Minute, time.Minute, time.Minute, time.Minute)
if len(note.said) != 1 {
t.Fatalf("said %q", note.said)
}
}
// The forge's reviews: the first approval at the head, never one dismissed, stale or of an older head.
func TestTheReviewEndsAtTheFirstApprovalAtItsHead(t *testing.T) {
raw := json.RawMessage(`{"reviews":[
{"state":"APPROVED","at_head":false,"submitted":"2026-10-12T09:01:00Z"},
{"state":"REQUEST_CHANGES","at_head":true,"submitted":"2026-10-12T09:02:00Z"},
{"state":"APPROVED","at_head":true,"dismissed":true,"submitted":"2026-10-12T09:03:00Z"},
{"state":"APPROVED","at_head":true,"submitted":"2026-10-12T09:05:00Z"},
{"state":"APPROVED","at_head":true,"submitted":"2026-10-12T09:04:00Z"}]}`)
at, found, err := firstApproval(raw)
if err != nil || !found || at.Format(time.RFC3339) != "2026-10-12T09:04:00Z" {
t.Fatalf("%s %v %v", at, found, err)
}
// The mesh's own reviews: comments by the account that opened the pull request, whose first line is the verdict.
head := "43df7ebc5aa21c0bceacd1580f01f67d83d3495d"
comments := json.RawMessage(`{"reviews":[
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: request changes at ` + head + `\n\nBlocking: …","submitted":"2026-10-12T09:01:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Looks fine. Verdict: approve at ` + head + `","submitted":"2026-10-12T09:02:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: approve at 43df7eb\n\nAll clauses hold.","submitted":"2026-10-12T09:07:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: approve at ` + head + `\n\nAgain.","submitted":"2026-10-12T09:08:00Z"}]}`)
at, found, err = firstApproval(comments)
if err != nil || !found || at.Format(time.RFC3339) != "2026-10-12T09:07:00Z" {
t.Fatalf("a comment's verdict: %s %v %v", at, found, err)
}
for _, c := range []struct {
body, commit string
approves bool
}{
{"Verdict: approve at " + head, head, true},
{"Verdict: approve at 43df7eb", head, true},
{"Verdict: approve at 43df7e", head, false}, // shorter than seven characters
{"Verdict: approve at 0123456789", head, false}, // another commit
{"Verdict: request changes at " + head, head, false}, // not an approval
{"Not yet.\nVerdict: approve at " + head, head, false}, // not the first line
{"Verdict: approve at 43df7ebc", "", false}, // no commit to hold it to
} {
if got := verdictApproves(c.body, c.commit); got != c.approves {
t.Errorf("%q at %q approves %v, want %v", c.body, c.commit, got, c.approves)
}
}
if _, found, err := firstApproval(json.RawMessage(`{"reviews":[]}`)); err != nil || found {
t.Fatalf("no reviews: %v %v", found, err)
}
if _, _, err := firstApproval(json.RawMessage(`"nothing"`)); err == nil {
t.Fatal("an answer with no reviews read as none")
}
}
type errFake string
func (e errFake) Error() string { return string(e) }
+1 -25
View File
@@ -188,30 +188,6 @@ func hasLink(is *Issue, kind string, target uint64) bool {
return false
}
// allIssues is every issue kept, by number.
func (t *Tracker) allIssues() (map[uint64]*Issue, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues cannot be listed: %v", err)
}
out := map[uint64]*Issue{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
return nil, err
}
out[n] = is
}
return out, nil
}
// unresolvedIssues is every issue at open, diagnosing or located, oldest first.
func (t *Tracker) unresolvedIssues() ([]*Issue, error) {
all, err := t.allIssues()
@@ -538,7 +514,7 @@ func (t *Tracker) apply(newer uint64, v *Verdict) ([]string, error) {
}
}
kept := *v
if _, err := t.change(newer, func(i *Issue) error {
if _, err := t.change(newer, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error {
if i.Triage == nil {
i.Triage = &kept
}
+25 -12
View File
@@ -600,29 +600,42 @@ func differences(kept, folder *Issue) []string {
return d
}
// importCreate writes a new issue under its number, never over one.
// importCreate writes a new issue under its number, never over one: its first operation appended to the log at
// expect 0 (novox/hq ADR 0300 §1), then its fold to the bucket.
func (t *Tracker) importCreate(is *Issue, out *Imported) error {
_, err := t.Store.Create(IssuesState, keyOf(is.Number), is)
if errors.Is(err, stdio.ErrChanged) {
if there, _, err := read(t.Store, is.Number); err == nil && there.Opening != is.Opening {
return fmt.Errorf("an issue %d appeared while the import ran; nothing was overwritten. Run the import again", is.Number)
}
if err != nil {
// A write that timed out may still have landed: read back before saying it did not.
if there, _, rerr := read(t.Store, is.Number); rerr == nil && there.Opening == is.Opening {
t.announceImport(is, out)
return nil
}
return fmt.Errorf("issue %d could not be written: %v", is.Number, err)
seq, err := t.begin(is, Act{Op: OpImported, By: importer(is)})
if errors.Is(err, errBegun) {
return fmt.Errorf("an issue %d appeared while the import ran; nothing was overwritten. Run the import again", is.Number)
}
if errors.Is(err, errUnsure) {
return fmt.Errorf("issue %d may have been written: the log did not confirm it (%v); run the import again, "+
"which finds it the same or creates it", is.Number, err)
}
if err != nil {
return fmt.Errorf("issue %d could not be written: the log did not take it (%v)", is.Number, err)
}
is.Folded = seq
t.store(is.Number, is, 0)
t.announceImport(is, out)
return nil
}
// importer is who ran the import that took an issue from hq.
func importer(is *Issue) string {
if is.Imported != nil && is.Imported.By != "" {
return is.Imported.By
}
return ImportedBy
}
// importFill takes the status and the fields of an issue the tracker holds without a status: opened with a
// folder before slice 3, whose status lived in that folder. Its title, report, opening, notes and links
// stay; links the report names are added.
func (t *Tracker) importFill(from *Issue, out *Imported) error {
is, err := t.change(from.Number, func(is *Issue) error {
is, err := t.change(from.Number, Act{Op: OpImported, By: importer(from)}, func(is *Issue) error {
if is.Status != "" {
return fmt.Errorf("issue %d took a status while the import ran; nothing was changed", is.Number)
}
@@ -663,7 +676,7 @@ func (t *Tracker) announceImport(is *Issue, out *Imported) {
// fillOpen gives an issue the tracker holds without a status — opened with a folder before slice 3, whose
// folder hq does not hold at the import's commit — the status open, so no issue is left without one.
func (t *Tracker) fillOpen(n uint64, commit string, at time.Time, by string) error {
is, err := t.change(n, func(is *Issue) error {
is, err := t.change(n, Act{Op: OpImported, By: by}, func(is *Issue) error {
if is.Status != "" {
return fmt.Errorf("issue %d took a status while the import ran; nothing was changed", is.Number)
}
@@ -1,7 +1,6 @@
package main
import (
"errors"
"os"
"path/filepath"
"reflect"
@@ -304,7 +303,7 @@ func TestTheImportRefusesWhatItMustNotWrite(t *testing.T) {
// An issue that appears under the number while the import runs is never overwritten.
h = newHarness(t)
h.mem.fail[IssuesState] = errors.New("the bus is away")
h.log.failNext = createAttempts
out, err = h.Import(ImportAsk{}, newSample(t))
if err != nil {
t.Fatal(err)
+21 -3
View File
@@ -112,12 +112,21 @@ type Issue struct {
Triage *Verdict `json:"triage,omitempty"`
// Worker is who works on it, while it is diagnosing or fixing (ADR 0284); nil at every other status.
Worker *Worker `json:"worker,omitempty"`
// FixingFrom is when it was first moved to fixing: where its fix time starts (novox/hq ADR 0313 §1). A
// take-over or a later move back to fixing keeps it.
FixingFrom *time.Time `json:"fixing-from,omitempty"`
// FixTime is its fix time, phase by phase, once the last of its fixes is delivered (ADR 0313 §4): measured,
// never set by a verb.
FixTime *FixTime `json:"fix-time,omitempty"`
// Opening is the id of the call that opened it, so a write whose answer was lost is known as its own.
Opening string `json:"opening"`
// Imported says that the issue, or its status, was taken from its folder in hq by the import of
// slice 3: when, by whom, at which commit of hq, from which document. An issue the import wrote has
// OpenedBy ImportedBy, and its Opened is the date its report gave.
Imported *Import `json:"imported,omitempty"`
// Folded is the sequence in the log `changes` of the last operation this issue folds (novox/hq ADR 0300
// §3); 0 for an issue kept before the switch and not carried into the log yet.
Folded uint64 `json:"folded,omitempty"`
}
// ImportedBy is who opened an issue the import wrote: not a caller of `open`, and not a name anybody typed.
@@ -148,9 +157,10 @@ type Link struct {
At time.Time `json:"at"`
}
// MostIssueBytes is the most one issue may be as kept: a value the bus holds (to-be 53 §4). The largest
// issue folder in hq was 25 KiB on 2026-10-10.
const MostIssueBytes = 256 << 10
// MostIssueBytes is the most one issue may be as kept: 252 KiB, so the operation holding it whole (an open, an
// import, a carry) fits in one entry of the log, which takes at most 256 KiB (novox/hq ADR 0297, ADR 0300). The
// largest issue folder in hq was 25 KiB on 2026-10-10.
const MostIssueBytes = 252 << 10
// Entry is one key's value and the revision it was read at.
type Entry struct {
@@ -166,6 +176,9 @@ type Store interface {
Update(state, key string, value any, revision uint64) (uint64, error)
Keys(state string) ([]string, error)
Increment(state, key string, attempts int) (uint64, error)
// History is every value a key still keeps, oldest first, puts and deletes (novox/hq ADR 0307): what the
// carry into the log reads (ADR 0300 §6).
History(state, key string) ([]stdio.StateRevision, error)
}
// runtimeStore is Store through the runtime that launched this bundle.
@@ -185,6 +198,11 @@ func (runtimeStore) Update(state, key string, value any, revision uint64) (uint6
return stdio.State(state).Update(key, value, revision)
}
func (runtimeStore) Keys(state string) ([]string, error) { return stdio.State(state).Keys() }
// History is every value the bucket still keeps of a key, oldest first (novox/hq ADR 0307).
func (runtimeStore) History(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
func (runtimeStore) Increment(state, key string, attempts int) (uint64, error) {
return stdio.State(state).Increment(key, attempts)
}
@@ -0,0 +1,819 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// fakeLog is the module's log kept in this process with the runtime's semantics (novox/hq ADR 0297, mesh-sdk
// go/log.go): an append at expect is refused with stdio.ErrChanged when the key's last sequence is another,
// 0 standing for a key with no entry; a read pages from a sequence, narrowed to one key or none.
type fakeLog struct {
mu sync.Mutex
entries []stdio.LogEntry
// fail, when set, is what every append answers, writing nothing: a full log, or a bus that is down.
fail error
// failNext is how many of the next appends fail, writing nothing, as a bus that does not answer.
failNext int
// lost, when set, is what the next append answers after it was written: an answer lost on the way.
lost error
// between, when set, runs once before the next append is judged: another writer coming in between.
between func()
clock time.Time
}
func newFakeLog() *fakeLog {
return &fakeLog{clock: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)}
}
func (l *fakeLog) lastOf(key string) uint64 {
for i := len(l.entries) - 1; i >= 0; i-- {
if l.entries[i].Key == key {
return l.entries[i].Sequence
}
}
return 0
}
func (l *fakeLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
l.mu.Lock()
if b := l.between; b != nil {
l.between = nil
l.mu.Unlock()
b()
l.mu.Lock()
}
defer l.mu.Unlock()
if l.fail != nil {
return 0, l.fail
}
if l.failNext > 0 {
l.failNext--
return 0, errors.New("the bus did not answer")
}
raw, err := json.Marshal(value)
if err != nil {
return 0, err
}
if len(raw) > 256<<10 {
return 0, fmt.Errorf("the value for %s is %d bytes; a log entry holds at most 256 KiB", key, len(raw))
}
if last := l.lastOf(key); last != expect {
return 0, fmt.Errorf("the last entry of %s is %d, not %d: %w", key, last, expect, stdio.ErrChanged)
}
seq := uint64(len(l.entries) + 1)
l.entries = append(l.entries, stdio.LogEntry{Sequence: seq, Key: key, Value: raw, At: l.clock})
if lost := l.lost; lost != nil {
l.lost = nil
return 0, lost
}
return seq, nil
}
func (l *fakeLog) Read(from uint64, keys string, limit int) (*stdio.LogPage, error) {
l.mu.Lock()
defer l.mu.Unlock()
if limit <= 0 {
limit = 100
}
limit = min(limit, 1000)
p := &stdio.LogPage{Entries: []stdio.LogEntry{}, Last: uint64(len(l.entries))}
for _, e := range l.entries {
if e.Sequence < from || (keys != "" && e.Key != keys) {
continue
}
if len(p.Entries) == limit {
p.Next = e.Sequence
break
}
p.Entries = append(p.Entries, e)
}
return p, nil
}
// ops is every operation the log holds of one issue, oldest first.
func (l *fakeLog) ops(t *testing.T, n uint64) []Operation {
t.Helper()
l.mu.Lock()
defer l.mu.Unlock()
var out []Operation
for _, e := range l.entries {
if e.Key != keyOf(n) {
continue
}
var op Operation
if err := json.Unmarshal(e.Value, &op); err != nil {
t.Fatal(err)
}
out = append(out, op)
}
return out
}
// foldCheck holds every issue the bucket folded to what its whole log adds up to, byte for byte: run after every
// test of a harness on this log (the review of PR 228).
func (l *fakeLog) foldCheck(t *testing.T, m *memory) {
l.mu.Lock()
entries := append([]stdio.LogEntry(nil), l.entries...)
l.mu.Unlock()
for k, v := range m.snapshot(IssuesState) {
var is Issue
if json.Unmarshal([]byte(v), &is) != nil || is.Folded == 0 {
continue
}
var f *Issue
var err error
for _, e := range entries {
if e.Key != k || e.Sequence > is.Folded {
continue
}
if f, err = foldEntry(f, e); err != nil {
t.Errorf("issue %s does not fold from its log: %v", k, err)
break
}
}
if raw, _ := json.Marshal(f); err == nil && string(raw) != v {
t.Errorf("issue %s in the bucket is not the fold of its log:\nbucket %s\nfold %s", k, v, raw)
}
}
}
func (l *fakeLog) size() int {
l.mu.Lock()
defer l.mu.Unlock()
return len(l.entries)
}
// another is a second tracker on the same bucket and log: another instance, or the same after a restart.
func (h *harness) another(t *testing.T) *harness {
t.Helper()
o := newHarness(t)
o.mem, o.log = h.mem, h.log
o.Tracker.Store, o.Tracker.Log = h.mem, h.log
o.settings = h.settings
return o
}
// §1: a write whose append is refused changes nothing: not the bucket, not the log, and says no event.
func TestAWriteWhoseAppendIsRefusedChangesNothing(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the bucket fills", "mesh-catalog modules/mesh-issues")
bucket, entries, events := h.mem.snapshot(IssuesState), h.log.size(), len(h.said())
h.log.fail = fmt.Errorf("the log is full: %w", errors.New("maximum bytes exceeded"))
if _, err := h.Note(n, "a note the log refuses"); err == nil || !strings.Contains(err.Error(), "log did not take") {
t.Fatalf("a note whose append is refused answers %v", err)
}
if _, err := h.Move(MoveAsk{Number: n, Status: StatusDiagnosing, By: "fix@g14"}); err == nil {
t.Fatal("a move whose append is refused was answered as made")
}
if got := h.mem.snapshot(IssuesState); fmt.Sprint(got) != fmt.Sprint(bucket) {
t.Fatalf("the bucket changed:\n%v\n%v", bucket, got)
}
if h.log.size() != entries || len(h.said()) != events {
t.Fatalf("the log holds %d entries (was %d), %d events said (was %d)", h.log.size(), entries, len(h.said()), events)
}
}
// §1: an operation that came in between refuses the stale append; the change is read again and applied to
// what the log holds, so neither is lost.
func TestAnOperationInBetweenIsNotLost(t *testing.T) {
h := newHarness(t)
n := h.open(t, "two notes at once", "hq")
other := h.another(t)
h.log.between = func() {
if _, err := other.Note(n, "the other's note"); err != nil {
t.Errorf("the other note: %v", err)
}
}
if _, err := h.Note(n, "this note"); err != nil {
t.Fatal(err)
}
is, err := h.Get(n)
if err != nil {
t.Fatal(err)
}
if len(is.Notes) != 2 || is.Notes[0].Text != "the other's note" || is.Notes[1].Text != "this note" {
t.Fatalf("notes %+v", is.Notes)
}
ops := h.log.ops(t, n)
if len(ops) != 3 || ops[0].Op != OpOpened || ops[1].Op != EventNoted || ops[2].Op != EventNoted {
t.Fatalf("the log of %d: %+v", n, ops)
}
if is.Folded != uint64(h.log.size()) {
t.Fatalf("folded at %d, the log's last is %d", is.Folded, h.log.size())
}
}
// §2: an open refused at validation takes no number.
func TestAnOpenRefusedAtValidationTakesNoNumber(t *testing.T) {
h := newHarness(t)
if _, err := h.Open(OpenAsk{Title: "", Report: "no title"}); err == nil {
t.Fatal("an open without a title was taken")
}
if _, err := h.Open(OpenAsk{Title: "t", Report: "r", LocatedIn: []string{"nowhere"}}); err == nil {
t.Fatal("an open naming an unregistered owner was taken")
}
if c := h.mem.counter(t); c != 0 || h.log.size() != 0 {
t.Fatalf("counter %d, log %d entries", c, h.log.size())
}
if n := h.open(t, "the first"); n != 365 {
t.Fatalf("the first open got %d", n)
}
}
// §2: an append that fails after the number was taken answers the failure and spends that number.
func TestAnAppendFailingAfterTheNumberSpendsIt(t *testing.T) {
h := newHarness(t)
h.log.fail = errors.New("the log is full")
_, err := h.Open(OpenAsk{Title: "refused by a full log", Report: "r"})
if err == nil || !strings.Contains(err.Error(), "365 is spent") || !strings.Contains(err.Error(), "the log is full") {
t.Fatalf("the open answered %v", err)
}
if c := h.mem.counter(t); c != 365 {
t.Fatalf("the counter is at %d: the number was not taken", c)
}
if len(h.mem.snapshot(IssuesState)) != 0 || h.log.size() != 0 || len(h.said()) != 0 {
t.Fatal("an open the log refused left an issue, an entry or an event")
}
h.log.fail = nil
if n := h.open(t, "the next"); n != 366 {
t.Fatalf("the next open got %d, not 366", n)
}
}
// §2: an append whose answer was lost is known as the open's own by its opening id: one issue, one number.
func TestAnAppendWhoseAnswerWasLostIsTheOpensOwn(t *testing.T) {
h := newHarness(t)
h.log.lost = errors.New("timeout")
o, err := h.Open(OpenAsk{Title: "lost answer", Report: "r"})
if err != nil {
t.Fatal(err)
}
if o.Number != 365 || h.log.size() != 1 {
t.Fatalf("number %d, %d entries", o.Number, h.log.size())
}
if is, err := h.Get(365); err != nil || is.Folded != 1 {
t.Fatalf("issue %+v, %v", is, err)
}
}
// writeEverything makes issues with every kind of write: open, move, note, link, set, delivery, triage.
func writeEverything(t *testing.T, h *harness) []uint64 {
t.Helper()
a := h.open(t, "the issues bucket fills toward its cap", "mesh-catalog modules/mesh-issues")
b := h.open(t, "status changes are kept for seven days", "mesh-controller")
c := h.open(t, "an unrelated one")
if _, err := h.Note(a, "measured 10.9 MB of 64 MiB"); err != nil {
t.Fatal(err)
}
if _, err := h.Link(LinkAsk{Number: b, Kind: LinkRelatesTo, Target: keyOf(a)}); err != nil {
t.Fatal(err)
}
if _, err := h.Set(SetAsk{Number: a, Severity: "major", Needs: "decision"}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: a, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"novox/mesh-catalog#1"}}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: c, Status: StatusWontfix, Reason: "not ours"}); err != nil {
t.Fatal(err)
}
if _, err := h.Link(LinkAsk{Number: b, Kind: LinkRelatesTo, Target: keyOf(a), Remove: true}); err != nil {
t.Fatal(err)
}
return []uint64{a, b, c}
}
// §3: the bucket emptied, the fold on start restores every issue exactly.
func TestTheFoldOnStartRestoresAnEmptiedBucketExactly(t *testing.T) {
h := newHarness(t)
writeEverything(t, h)
before := h.mem.snapshot(IssuesState)
if len(before) != 3 {
t.Fatalf("%d issues", len(before))
}
h.mem.empty(IssuesState)
r := h.another(t)
s, err := r.Start()
if err != nil {
t.Fatal(err)
}
if s.Rebuilt != 3 || s.Issues != 3 || len(s.Faults) != 0 {
t.Fatalf("started %+v", s)
}
after := h.mem.snapshot(IssuesState)
for k, v := range before {
if after[k] != v {
t.Errorf("issue %s was\n%s\nand is folded as\n%s", k, v, after[k])
}
}
if len(after) != len(before) {
t.Fatalf("%d issues after, %d before", len(after), len(before))
}
}
// §3: a bucket behind its log (the write after the append failed) catches up on start, and at the next change.
func TestABucketBehindItsLogCatchesUp(t *testing.T) {
h := newHarness(t)
n := h.open(t, "crash between the two", "hq")
h.mem.fail[IssuesState] = errors.New("the bus went away")
if _, err := h.Note(n, "kept in the log only"); err != nil {
t.Fatalf("a note the log took is a note made: %v", err)
}
is, _ := h.Get(n)
if len(is.Notes) != 0 {
t.Fatal("the bucket write was meant to fail")
}
r := h.another(t)
s, err := r.Start()
if err != nil {
t.Fatal(err)
}
if s.Refolded != 1 {
t.Fatalf("started %+v", s)
}
if is, _ = h.Get(n); len(is.Notes) != 1 || is.Folded != 2 {
t.Fatalf("after start: %+v", is)
}
// And at the next change without a start: the stale append is refused, the log folded, the change applied.
h.mem.fail[IssuesState] = errors.New("the bus went away")
if _, err := h.Note(n, "second"); err != nil {
t.Fatal(err)
}
if _, err := h.Note(n, "third"); err != nil {
t.Fatal(err)
}
if is, _ = h.Get(n); len(is.Notes) != 3 || is.Folded != 4 {
t.Fatalf("after the next change: notes %d, folded %d", len(is.Notes), is.Folded)
}
}
// §3: the counter is kept at or above the highest number in the log.
func TestStartRaisesTheCounterToTheLog(t *testing.T) {
h := newHarness(t)
h.open(t, "one")
h.open(t, "two")
h.mem.put(CountersState, IssueCounter, uint64(365))
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if s.Counter != 366 || h.mem.counter(t) != 366 {
t.Fatalf("counter %d, kept %d", s.Counter, h.mem.counter(t))
}
if n := h.open(t, "three"); n != 367 {
t.Fatalf("the next open got %d", n)
}
}
// §4: each write still says its event, alongside its one operation in the log.
func TestEachWriteSaysItsEventAlongsideItsOperation(t *testing.T) {
h := newHarness(t)
type step struct {
event string
do func() error
}
var n uint64
steps := []step{
{EventOpened, func() error {
o, err := h.Open(OpenAsk{Title: "t", Report: "r", LocatedIn: []string{"hq"}})
n = o.Number
return err
}},
{EventNoted, func() error { _, err := h.Note(n, "a note"); return err }},
{EventLinked, func() error { _, err := h.Link(LinkAsk{Number: n, Kind: LinkDecision, Target: "0300"}); return err }},
{EventSet, func() error { _, err := h.Set(SetAsk{Number: n, Severity: "minor"}); return err }},
{EventMoved, func() error { _, err := h.Move(MoveAsk{Number: n, Status: StatusLocated}); return err }},
}
for i, s := range steps {
events, entries := len(h.said()), h.log.size()
if err := s.do(); err != nil {
t.Fatalf("%s: %v", s.event, err)
}
said := h.said()
if len(said) != events+1 || said[len(said)-1].Event != s.event {
t.Fatalf("step %d said %v", i, said[events:])
}
ops := h.log.ops(t, n)
if h.log.size() != entries+1 || ops[len(ops)-1].Op != s.event || ops[len(ops)-1].By != Unproven {
t.Fatalf("step %d: %d entries (was %d), last %+v", i, h.log.size(), entries, ops[len(ops)-1])
}
}
// A note is kept as itself, not as every note again.
if op := h.log.ops(t, n)[1]; op.Set != nil || len(op.Add) != 1 || op.Add["notes"] == nil {
t.Fatalf("the note's operation is %+v", op)
}
}
// §5: before pages through more than 500 issues, without a gap or a repeat.
func TestBeforePagesThroughMoreThanFiveHundredIssues(t *testing.T) {
h := newHarness(t)
const issues = 530
for i := 0; i < issues; i++ {
h.open(t, fmt.Sprintf("issue %d", i))
}
if _, err := h.Start(); err != nil {
t.Fatal(err)
}
seen := map[uint64]bool{}
var last uint64 = 0
var before uint64
pages := 0
for {
l, err := h.List(ListAsk{Limit: 100, Before: before})
if err != nil {
t.Fatal(err)
}
pages++
for _, r := range l.Rows {
if seen[r.Number] {
t.Fatalf("%d repeated", r.Number)
}
if last != 0 && r.Number != last-1 {
t.Fatalf("a gap from %d to %d", last, r.Number)
}
seen[r.Number], last = true, r.Number
}
if l.Next == 0 {
break
}
before = l.Next
}
if len(seen) != issues || pages != 6 || last != 365 {
t.Fatalf("%d issues seen in %d pages, ending at %d", len(seen), pages, last)
}
// Over the tool too, as JSON gives the cursor.
out, err := h.call(t, "list", map[string]any{"before": 400, "limit": 2})
if err != nil {
t.Fatal(err)
}
if l := out.(*Listed); len(l.Rows) != 2 || l.Rows[0].Number != 399 || l.Next != 398 {
t.Fatalf("listed %+v", l)
}
}
// §5: a search is ranked by relevance; the index is rebuilt on start and kept current by the bucket's watch.
func TestASearchIsRankedAndTheIndexRebuiltOnStart(t *testing.T) {
h := newHarness(t)
full := h.open(t, "the issues bucket fills toward its cap")
h.open(t, "a bucket of the controller")
week := h.open(t, "status changes are kept for seven days")
h.open(t, "nothing alike")
if _, err := h.Note(week, "the events stream drops them after a week; the bucket is not involved"); err != nil {
t.Fatal(err)
}
r := h.another(t)
if _, err := r.Start(); err != nil {
t.Fatal(err)
}
if !r.view().ready() {
t.Fatal("the index was not built on start")
}
l, err := r.List(ListAsk{Search: "issues bucket cap"})
if err != nil {
t.Fatal(err)
}
if len(l.Rows) != 3 || l.Rows[0].Number != full {
t.Fatalf("ranked %+v", l.Rows)
}
// A note's words are found: notes are indexed.
if l, _ = r.List(ListAsk{Search: "week"}); len(l.Rows) != 1 || l.Rows[0].Number != week {
t.Fatalf("a note's word: %+v", l.Rows)
}
// Another instance's write reaches this one's index through the watch of the bucket.
if _, err := h.Note(full, "seen again by the operator on shanks"); err != nil {
t.Fatal(err)
}
e, _ := h.mem.Get(IssuesState, keyOf(full))
if err := r.Watched(stdio.StateChange{State: IssuesState, Key: keyOf(full), Op: "put", Value: e.Value}); err != nil {
t.Fatal(err)
}
if l, _ = r.List(ListAsk{Search: "operator"}); len(l.Rows) != 1 || l.Rows[0].Number != full {
t.Fatalf("after the watch: %+v", l.Rows)
}
if _, err := r.List(ListAsk{Search: "bucket", Before: 400}); err == nil {
t.Fatal("a search with before was taken")
}
}
// keptBefore puts an issue in the bucket as the tracker kept it before the switch: values, no log.
func keptBefore(t *testing.T, h *harness, n uint64, titles ...string) {
t.Helper()
for _, title := range titles {
h.mem.put(IssuesState, keyOf(n), Issue{Number: n, Title: title, Status: StatusOpen, Report: "r",
Opened: time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC), OpenedBy: "x", Opening: fmt.Sprint("o", n)})
}
}
// §6: the carry appends each value the bucket keeps, oldest first, as carried operations holding the whole
// issue; run again it carries nothing; a write after it is one entry in the log.
func TestTheCarryIsIdempotentAndCarriesEveryValueKept(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "first title", "second title", "third title")
keptBefore(t, h, 301, "only")
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Issues != 2 || c.Values != 4 || len(c.Faults) != 0 {
t.Fatalf("carried %+v", c)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 {
t.Fatalf("%d operations of 300", len(ops))
}
for i, title := range []string{"first title", "second title", "third title"} {
if ops[i].Op != OpCarried || ops[i].Issue == nil || ops[i].Issue.Title != title {
t.Fatalf("operation %d: %+v", i, ops[i])
}
}
is, _ := h.Get(300)
if is.Title != "third title" || is.Folded != 3 {
t.Fatalf("300 is %+v", is)
}
again, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if again.Issues != 0 || again.Values != 0 || again.Already != 2 || h.log.size() != 4 {
t.Fatalf("carried again %+v, %d entries", again, h.log.size())
}
if _, err := h.Move(MoveAsk{Number: 300, Status: StatusDiagnosing, By: "fix@g14"}); err != nil {
t.Fatal(err)
}
if h.log.size() != 5 || h.log.ops(t, 300)[3].Op != EventMoved {
t.Fatalf("a move after the carry: %d entries", h.log.size())
}
// The start carries too, and finds nothing more.
s, err := h.another(t).Start()
if err != nil || s.Carried.Issues != 0 || h.log.size() != 5 {
t.Fatalf("start %+v, %v", s, err)
}
}
// §6: a deletion among the values the bucket kept is passed over, said, and the values after it carried.
func TestTheCarryPassesOverADeletionAmongTheKeptValues(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "before the deletion")
h.mem.deleteKey(IssuesState, keyOf(300))
keptBefore(t, h, 300, "after the deletion", "now")
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Issues != 1 || c.Values != 3 || c.Deletes != 1 || !strings.Contains(c.Said, "1 deletions") {
t.Fatalf("carried %+v", c)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 || ops[0].Issue.Title != "before the deletion" || ops[2].Issue.Title != "now" {
t.Fatalf("the log of 300: %+v", ops)
}
if is, _ := h.Get(300); is.Title != "now" || is.Folded != 3 {
t.Fatalf("300 is %+v", is)
}
}
// §6: a write to an issue not carried yet carries it first, so its first operation in the log is a carried one.
func TestAWriteBeforeTheCarryCarriesItsIssueFirst(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "before", "now")
if _, err := h.Note(300, "after the switch"); err != nil {
t.Fatal(err)
}
ops := h.log.ops(t, 300)
if len(ops) != 3 || ops[0].Op != OpCarried || ops[1].Op != OpCarried || ops[2].Op != EventNoted {
t.Fatalf("the log of 300: %+v", ops)
}
h.mem.empty(IssuesState)
if _, err := h.another(t).Start(); err != nil {
t.Fatal(err)
}
if is, _ := h.Get(300); is.Title != "now" || len(is.Notes) != 1 {
t.Fatalf("300 folds to %+v", is)
}
}
// The counter behind an issue the bucket kept before the switch: the open refuses, overwrites nothing.
func TestAnOpenOverAnIssueKeptBeforeTheSwitchOverwritesNothing(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 365, "kept")
h.mem.put(CountersState, IssueCounter, uint64(364))
if _, err := h.Open(OpenAsk{Title: "new", Report: "r"}); err == nil || !strings.Contains(err.Error(), "fault") {
t.Fatalf("the open answered %v", err)
}
if is, _ := h.Get(365); is.Title != "kept" {
t.Fatalf("365 is %+v", is)
}
}
// titlesOf is each operation of an issue's log as its kind and, for one holding the issue whole, its title.
func titlesOf(t *testing.T, h *harness, n uint64) string {
t.Helper()
var out []string
for _, op := range h.log.ops(t, n) {
if op.Issue != nil {
out = append(out, op.Op+":"+op.Issue.Title)
} else {
out = append(out, op.Op)
}
}
return strings.Join(out, " ")
}
// §6, the review of PR 228: a carry that fails partway, then a second carry, leaves every kept value carried
// once, oldest first, and each carry says what it really carried.
func TestAnInterruptedCarryResumesAfterTheLastValueCarried(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "v1", "v2", "v3", "v4")
// The bus goes away after the first value is appended.
h.log.between = func() { h.log.between = func() { h.log.fail = errors.New("the bus went away") } }
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Values != 1 || len(c.Faults) != 1 || !strings.Contains(c.Said, "could not be carried") ||
!strings.Contains(c.Faults[0], "the bus went away") {
t.Fatalf("the first carry said %+v", c)
}
if is, _ := h.Get(300); is.Folded != 0 || is.Title != "v4" {
t.Fatalf("after the failure the bucket holds %+v", is)
}
// A write meanwhile is refused, and changes nothing.
if _, err := h.Note(300, "while the bus is away"); err == nil || !strings.Contains(err.Error(), "not carried") {
t.Fatalf("a note on an issue half carried: %v", err)
}
h.log.fail = nil
c, err = h.Carry()
if err != nil {
t.Fatal(err)
}
if c.Values != 3 || c.Issues != 1 || len(c.Faults) != 0 {
t.Fatalf("the second carry said %+v", c)
}
if got := titlesOf(t, h, 300); got != "carried:v1 carried:v2 carried:v3 carried:v4" {
t.Fatalf("the log of 300: %s", got)
}
if is, _ := h.Get(300); is.Title != "v4" || is.Folded != 4 {
t.Fatalf("300 is %+v", is)
}
}
// §6, the review of PR 228: one issue that cannot be carried is a fault, named, and the start carries the others,
// folds, raises the counter and loads memory. Writes to that issue are refused with the reason.
func TestOneIssueThatCannotBeCarriedDoesNotStopTheStart(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "small")
// Kept before the switch at the bucket's limit: whole, in a carried operation, past what one log entry holds.
h.mem.put(IssuesState, "301", Issue{Number: 301, Title: "huge", Status: StatusOpen, Report: strings.Repeat("x", 256<<10-200),
OpenedBy: "x", Opening: "o301"})
keptBefore(t, h, 302, "also small")
s, err := h.Start()
if err != nil {
t.Fatal(err)
}
if s.Carried.Issues != 2 || len(s.Faults) != 1 || !strings.Contains(s.Faults[0], "issue 301") ||
!strings.Contains(s.Faults[0], "256 KiB") || !strings.Contains(s.Said, "issue 301") {
t.Fatalf("started %+v", s)
}
if !h.view().ready() {
t.Fatal("memory was not loaded")
}
l, err := h.List(ListAsk{})
if err != nil || l.Matched != 3 {
t.Fatalf("listed %+v, %v", l, err)
}
if _, err := h.Note(301, "a note"); err == nil || !strings.Contains(err.Error(), "256 KiB") {
t.Fatalf("a note on the issue not carried: %v", err)
}
if is, _ := h.Get(301); is.Title != "huge" || is.Folded != 0 || len(is.Notes) != 0 {
t.Fatal("the issue not carried was changed")
}
if is, _ := h.Get(302); is.Folded == 0 {
t.Fatal("302 was not carried")
}
}
// §6, the review of PR 228: two instances carrying the same issue at the same moment carry each value once.
func TestTwoInstancesCarryingOneIssueAtOnceCarryEachValueOnce(t *testing.T) {
h := newHarness(t)
keptBefore(t, h, 300, "v1", "v2", "v3")
other := h.another(t)
var otherSaid *Carried
h.log.between = func() {
c, err := other.Carry()
if err != nil {
t.Errorf("the other carry: %v", err)
}
otherSaid = c
}
c, err := h.Carry()
if err != nil {
t.Fatal(err)
}
if got := titlesOf(t, h, 300); got != "carried:v1 carried:v2 carried:v3" {
t.Fatalf("the log of 300: %s", got)
}
if otherSaid.Values+c.Values != 3 || len(c.Faults)+len(otherSaid.Faults) != 0 {
t.Fatalf("this carry %+v, the other %+v", c, otherSaid)
}
}
// §6, the review of PR 228: a value written beside the log after the carry, by an instance from before the switch,
// is never overwritten by the fold on start; it is carried as one more operation.
func TestAValueWrittenBesideTheLogIsCarriedNotOverwritten(t *testing.T) {
h := newHarness(t)
n := h.open(t, "carried", "hq")
if _, err := h.Note(n, "after the switch"); err != nil {
t.Fatal(err)
}
// An instance from before the switch writes the bucket: no folded.
is, _ := h.Get(n)
is.Folded, is.Title = 0, "written beside the log"
h.mem.put(IssuesState, keyOf(n), is)
// Its carry fails at this start: the fold is not written over it.
h.log.failNext = 1
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if got, _ := h.Get(n); got.Title != "written beside the log" || got.Folded != 0 || len(s.Faults) != 2 {
t.Fatalf("issue %d is %+v after a start whose carry failed (faults %v)", n, got, s.Faults)
}
// The next start carries it.
if _, err := h.another(t).Start(); err != nil {
t.Fatal(err)
}
got, _ := h.Get(n)
if got.Title != "written beside the log" || got.Folded != 3 || len(got.Notes) != 1 {
t.Fatalf("issue %d is %+v", n, got)
}
if ops := titlesOf(t, h, n); ops != "opened:carried noted carried:written beside the log" {
t.Fatalf("the log: %s", ops)
}
}
// §3, the review of PR 228: an entry that cannot be folded faults its issue on start, and nothing folded past it
// is written.
func TestAnEntryThatCannotBeFoldedLeavesItsIssueAsItIs(t *testing.T) {
h := newHarness(t)
n := h.open(t, "fine")
m := h.open(t, "broken")
if _, err := h.Note(m, "kept"); err != nil {
t.Fatal(err)
}
h.log.mu.Lock()
for i, e := range h.log.entries {
if e.Key == keyOf(m) && e.Sequence == 2 {
h.log.entries[i].Value = json.RawMessage(`{"op":"noted","add":{"notes":"not a list"}}`)
}
}
h.log.mu.Unlock()
before := h.mem.snapshot(IssuesState)[keyOf(m)]
h.mem.empty(IssuesState)
h.mem.put(IssuesState, keyOf(m), json.RawMessage(before))
s, err := h.another(t).Start()
if err != nil {
t.Fatal(err)
}
if s.Rebuilt != 1 || len(s.Faults) != 1 || !strings.Contains(s.Faults[0], fmt.Sprintf("issue %d cannot be folded", m)) {
t.Fatalf("started %+v", s)
}
if h.mem.snapshot(IssuesState)[keyOf(m)] != before {
t.Fatal("the broken issue's bucket value was written over")
}
if is, _ := h.Get(n); is.Title != "fine" {
t.Fatalf("%d is %+v", n, is)
}
// The broken entry is the test's doing: the fold check would find it, so take the issue out of the bucket.
h.mem.empty(IssuesState)
}
// §2, the review of PR 228: an open whose answer was lost, its issue already changed by another operation, is
// still known as its own by its first entry.
func TestALostOpenIsKnownByItsFirstEntryNotItsLast(t *testing.T) {
h := newHarness(t)
h.log.lost = errors.New("no answer in 30 s")
// After the first operation lands and its answer is lost, another operation follows before the retry.
h.log.between = func() {
h.log.between = func() {
h.log.mu.Lock()
h.log.entries = append(h.log.entries, stdio.LogEntry{Sequence: uint64(len(h.log.entries) + 1), Key: "365",
Value: json.RawMessage(`{"op":"triaged","by":"triage","set":{"severity":"\"minor\""}}`)})
h.log.mu.Unlock()
}
}
o, err := h.Open(OpenAsk{Title: "lost and followed", Report: "r"})
if err != nil || o.Number != 365 {
t.Fatalf("%+v %v", o, err)
}
// The entry added by hand is not one the tracker folded: take the issue out before the fold check.
h.mem.empty(IssuesState)
}
+76 -11
View File
@@ -22,7 +22,7 @@ import (
const Seat = "issue-tracker"
// Verbs are the seat's verbs, as the manifest declares and claims them.
var Verbs = []string{"open", "get", "list", "move", "note", "link", "import", "groom", "set", "tracking", "reserve"}
var Verbs = []string{"open", "get", "list", "move", "note", "link", "import", "groom", "set", "tracking", "reserve", "fix-times"}
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
@@ -95,19 +95,24 @@ func main() {
CallerOf: stdio.CallerOf,
}
t.Notify = busNotifier()
t.Deliveries = func(repository, number string) (Delivery, error) {
owner := "novox"
forgeOwner := func() string {
if s, err := t.Settings(); err == nil && strings.TrimSpace(s.ForgeOwner) != "" {
owner = strings.TrimSpace(s.ForgeOwner)
return strings.TrimSpace(s.ForgeOwner)
}
return busDeliveryState(owner)(repository, number)
return "novox"
}
t.Deliveries = func(repository, number string) (Delivery, error) {
return busDeliveryState(forgeOwner())(repository, number)
}
t.Approval = busApproval(forgeOwner)
if len(os.Args) > 1 && os.Args[1] == "dry-run" {
os.Exit(dryRunAtTheTerminal(t, os.Args[2:]))
}
if _, err := t.Settings(); err != nil {
logf("[mesh-issues] the settings cannot be read (%v): every open and move will say so", err)
}
t.Log = runtimeLog{}
go start(t)
go hearOpened(t)
go sweep(t)
if err := stdio.Serve("", Tools(t)); err != nil {
@@ -116,6 +121,30 @@ func main() {
}
}
// start watches the module's own bucket into memory, then carries, folds and indexes (novox/hq ADR 0300 §3, §5,
// §6), each asked again until it is done: the runtime answers only once Serve runs. Until it is done, a list
// reads the bucket, and a write carries the issue it changes first, so no write reaches the log before its
// issue's carry.
func start(t *Tracker) {
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
time.Sleep(wait)
if err := stdio.State(IssuesState).Watch("", t.Watched); err != nil {
logf("[mesh-issues] the bucket is not watched yet (%v); asking again in %s", err, wait)
continue
}
break
}
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
s, err := t.Start()
if err == nil {
logf("[mesh-issues] started: %s", s.Said)
return
}
logf("[mesh-issues] the start did not finish (%v); trying again in %s", err, wait)
time.Sleep(wait)
}
}
// hearOpened subscribes the triage to the tracker's own `opened` and `moved` events and to mesh-delivery's
// `transition` (ADR 0278 §1, ADR 0279 §3), each asked again until the runtime takes it: the runtime answers only
// once Serve runs.
@@ -181,6 +210,9 @@ func (emptyStore) Update(string, string, any, uint64) (uint64, error) {
return 0, errors.New("a dry run at the terminal writes nothing")
}
func (emptyStore) Keys(string) ([]string, error) { return nil, nil }
func (emptyStore) History(string, string) ([]stdio.StateRevision, error) {
return []stdio.StateRevision{}, nil
}
func (emptyStore) Increment(string, string, int) (uint64, error) {
return 0, errors.New("a dry run at the terminal writes nothing")
}
@@ -352,8 +384,10 @@ func Tools(t *Tracker) []stdio.Tool {
Description: "One issue whole, by its number: title, status, severity, what it needs, report, who opened it " +
"and when, the owners it is located in, who works on it and since when (worker, and worker-stale when " +
"nobody did anything on it for a day), what fixed it and whether that is delivered, its notes and its " +
"links; the issues it waits for, those that depend on it (needed-by), and whether it is ready; and " +
"whether its fix is delivered, as the delivery last said it (merged, held and why, delivered).",
"links; the issues it waits for, those that depend on it (needed-by), and whether it is ready; " +
"whether its fix is delivered, as the delivery last said it (merged, held and why, delivered); and, once " +
"the last of its fixes is delivered, its fix time (fix-time) from its first move to fixing, its class and " +
"fix budget, and its phases: implementing, review, checks, merge to running.",
Input: schema([]string{"number"}, map[string]any{"number": number("the issue's number")}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("get", m, "number")
@@ -376,7 +410,8 @@ func Tools(t *Tracker) []stdio.Tool {
"taken over. flag delivered-not-resolved keeps the issues in fixing whose every fix (fixed-by and the " +
"pull requests linked) is delivered; flag abandoned the issues whose worker did nothing on them for a " +
"day with no pull request open, which a move takes over without a reason. Each row says both. Answers " +
"what is open in one call.",
"what is open in one call. search ranks the issues whose title, report or notes hold its words, best first; " +
"before pages the newest-first list: give the answer's next to read on below it.",
Input: schema(nil, map[string]any{
"status": map[string]any{"type": "string", "enum": Statuses, "description": "only issues at this status"},
"owner": text("only issues located in this owner, such as mesh-controller"),
@@ -386,9 +421,11 @@ func Tools(t *Tracker) []stdio.Tool {
"needs": map[string]any{"type": "string", "enum": NeedsKinds, "description": "only issues that need this"},
"ready": map[string]any{"type": "boolean", "description": "only issues ready (true) or waiting (false)"},
"flag": map[string]any{"type": "string", "enum": Flags, "description": "only issues delivered and not resolved, or the abandoned ones"},
"search": text("rank the issues whose title, report or notes hold these words, best first"),
"before": number("only issues numbered below this: the next of the page before, to read on"),
}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("list", m, "status", "owner", "phrase", "limit", "severity", "needs", "ready", "flag")
a, err := readArgs("list", m, "status", "owner", "phrase", "limit", "severity", "needs", "ready", "flag", "search", "before")
if err != nil {
return nil, err
}
@@ -411,6 +448,12 @@ func Tools(t *Tracker) []stdio.Tool {
if ask.Flag, err = a.str("flag"); err != nil {
return nil, err
}
if ask.Search, err = a.str("search"); err != nil {
return nil, err
}
if ask.Before, err = a.number("before", false); err != nil {
return nil, err
}
if v, given := m["ready"]; given && v != nil {
r, err := a.flag("ready")
if err != nil {
@@ -646,13 +689,14 @@ func Tools(t *Tracker) []stdio.Tool {
}},
{Name: Seat + ".reserve",
Description: "Reserve the next decision or research number, before the record is written, and get it: never " +
"one another caller got, and always above every number hq already holds. The answer is the kind and the " +
"one another caller got, and always above the highest number the knowledge base holds (knowledge_highest; " +
"hq issue 493). The answer is the kind and the " +
"number, and the number as a record's name starts with it (0292 for a decision, 046 for research). " +
"Optionally for: what the record is about, in one line. A reserved number is spent forever, whether or " +
"not its record lands, and is never handed out again; who reserved it, when and for what are kept. " +
"Refused, taking no number: a kind that is not decision or research (an issue's number comes from " +
"open), a for that is not one line of plain text or carries a secret, an unknown argument, and the " +
"first number of a kind while what hq holds cannot be read.",
"first number of a kind while the knowledge base does not answer its highest number.",
Input: schema([]string{"kind"}, map[string]any{
"kind": map[string]any{"type": "string", "enum": ReserveKinds, "description": "which number: decision or research"},
"for": text("what the record is about, in one line, such as: the knowledge base holds decisions"),
@@ -671,5 +715,26 @@ func Tools(t *Tracker) []stdio.Tool {
}
return t.reserve(t.callerOf(m), ask)
}},
{Name: Seat + ".fix-times",
Description: "How long fixes take, from the move to fixing to the last fix delivered (the fix time): each class's fix budget (leaf 5m, core 10m), the median and worst fix time of the issues whose last " +
"fix was delivered in the last days (7 unless days says otherwise), how many were within and over, and every " +
"issue over its budget with its longest phase (implementing, review, checks, merge to running), newest first; " +
"issues whose fixes move no module, or whose class is not known, are listed apart with no budget. One issue's " +
"fix time, phase by phase, is in its get. A lever meant to shorten fixes is judged by comparing this median " +
"over the seven days before it was delivered with the seven days after. Refused: days above 90.",
Input: schema(nil, map[string]any{
"days": map[string]any{"type": "integer", "minimum": 1, "maximum": MostFixTimesDays, "description": "the window, in days back from now (default 7)"},
}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("fix-times", m, "days")
if err != nil {
return nil, err
}
days, err := a.number("days", false)
if err != nil {
return nil, err
}
return t.FixTimes(int(days))
}},
}
}
@@ -22,6 +22,7 @@ type manifest struct {
Consumes []string `json:"consumes"`
Invokes []string `json:"invokes"`
State []json.RawMessage `json:"state"`
Logs []string `json:"logs"`
Tools []string `json:"tools"`
Resources []struct {
ID string `json:"id"`
@@ -87,8 +88,13 @@ func TestItDeclaresAndHoldsTheIssueTrackerSeatOnceForTheMesh(t *testing.T) {
EventTriaged, EventReady, EventReadyToVerify, EventReserved}) {
t.Fatalf("emits %v", m.Emits)
}
// Four states, none expiring: issues keep a short history per key, the counters one value each, the
// backlog's pass of the duplicate check one value (ADR 0278 §6), each reserved number one value (issue 424).
// The record is the log `changes` (ADR 0300 §1). Four states, none expiring: issues the fold of the log (§3),
// still keeping its older values until the carry has read them (they go to one value in a follow-up), the
// counters one value each, the backlog's pass of the duplicate check one value (ADR 0278 §6), each reserved
// number one value (issue 424).
if !reflect.DeepEqual(m.Logs, []string{ChangesLog}) {
t.Fatalf("logs %v", m.Logs)
}
d := m.stateDeclarations(t)
if len(d) != 4 || d[0].Name != IssuesState || d[0].History < 2 || d[0].TTLSeconds != 0 ||
d[1].Name != CountersState || d[1].History > 1 || d[1].TTLSeconds != 0 ||
@@ -100,7 +106,7 @@ func TestItDeclaresAndHoldsTheIssueTrackerSeatOnceForTheMesh(t *testing.T) {
// the import and a reservation read the record through the records module (issue 424).
if !reflect.DeepEqual(m.Consumes, []string{"mesh-issues." + EventOpened, "mesh-issues." + EventMoved, "mesh-delivery.transition"}) ||
!reflect.DeepEqual(m.Invokes, []string{"claude-code.claude_code_judge", "seat:mesh-delivery.checks", "seat:operator-channel.notify",
"records.records_status", "records.records_list", "records.records_read", "mesh-knowledge.knowledge_highest"}) {
"gitea.gitea_pull_reviews", "records.records_status", "records.records_list", "records.records_read", "mesh-knowledge.knowledge_highest"}) {
t.Fatalf("consumes %v, invokes %v", m.Consumes, m.Invokes)
}
if len(m.Build.Artifacts) != 1 || m.Build.Artifacts[0].From != "cmd/mesh-issues" ||
@@ -22,10 +22,14 @@ type memory struct {
data map[string]map[string]Entry
// fail, when set, is the error the next write of that state answers.
fail map[string]error
// hist is every value each key was given, the last eight kept, as the bucket kept them before the tracker
// switched to its log (novox/hq ADR 0300 §6).
hist map[string]map[string][]Entry
}
func newMemory() *memory {
return &memory{data: map[string]map[string]Entry{IssuesState: {}, CountersState: {}, GroomingState: {}, ReservationsState: {}}, fail: map[string]error{}}
return &memory{data: map[string]map[string]Entry{IssuesState: {}, CountersState: {}, GroomingState: {}, ReservationsState: {}}, fail: map[string]error{},
hist: map[string]map[string][]Entry{}}
}
func (m *memory) bucket(state string) (map[string]Entry, error) {
@@ -71,9 +75,50 @@ func (m *memory) write(state, key string, value any, check func(Entry, bool) boo
}
m.rev++
b[key] = Entry{Value: raw, Revision: m.rev}
if m.hist[state] == nil {
m.hist[state] = map[string][]Entry{}
}
kept := append(m.hist[state][key], b[key])
if len(kept) > 8 {
kept = kept[len(kept)-8:]
}
m.hist[state][key] = kept
return m.rev, nil
}
// History is every value the key still keeps, oldest first, as the SDK's State(...).History answers it: puts and
// deletes, empty for a key that keeps nothing.
func (m *memory) History(state, key string) ([]stdio.StateRevision, error) {
m.mu.Lock()
defer m.mu.Unlock()
out := []stdio.StateRevision{}
for _, e := range m.hist[state][key] {
op, value := "put", e.Value
if e.Value == nil {
op = "delete"
}
out = append(out, stdio.StateRevision{Key: key, Value: value, Revision: e.Revision, Op: op})
}
return out, nil
}
// deleteKey deletes a key, keeping the deletion among its values as the bucket does.
func (m *memory) deleteKey(state, key string) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.data[state], key)
m.rev++
m.hist[state][key] = append(m.hist[state][key], Entry{Revision: m.rev})
}
// empty takes every key out of a state, as an emptied bucket.
func (m *memory) empty(state string) {
m.mu.Lock()
defer m.mu.Unlock()
m.data[state] = map[string]Entry{}
delete(m.hist, state)
}
func (m *memory) Create(state, key string, value any) (uint64, error) {
return m.write(state, key, value, func(_ Entry, there bool) bool { return !there })
}
@@ -154,6 +199,7 @@ type said struct {
type harness struct {
*Tracker
mem *memory
log *fakeLog
mu sync.Mutex
events []said
settings Settings
@@ -162,12 +208,13 @@ type harness struct {
func newHarness(t *testing.T) *harness {
t.Helper()
h := &harness{mem: newMemory(), settings: Settings{FirstNumber: 365, Owners: []string{
h := &harness{mem: newMemory(), log: newFakeLog(), settings: Settings{FirstNumber: 365, Owners: []string{
"hq", "mesh-catalog", "mesh-controller", "mesh-host", "mesh-sdk", "mesh-tools",
"mesh-catalog modules/nats", "mesh-catalog modules/records", "mesh-catalog modules/mesh-issues"}}}
clock := time.Date(2026, 10, 10, 12, 0, 0, 0, time.UTC)
h.Tracker = &Tracker{
Store: h.mem,
Log: h.log,
Emit: func(event string, body any) error {
h.mu.Lock()
defer h.mu.Unlock()
@@ -185,6 +232,11 @@ func newHarness(t *testing.T) *harness {
Logf: t.Logf,
Sleep: func(time.Duration) {},
}
t.Cleanup(func() {
if h.Tracker.Log == Log(h.log) {
h.log.foldCheck(t, h.mem)
}
})
return h
}
+10 -2
View File
@@ -45,6 +45,14 @@ type SetDone struct {
// Set changes an issue's severity, needs or suggested owner, as a person decides.
func (t *Tracker) Set(a SetAsk) (*SetDone, error) { return t.set(t.by(), a, false) }
// setOp is the kind of operation a set is in the log: a person's set, or the triage filling what is empty.
func setOp(onlyEmpty bool) string {
if onlyEmpty {
return EventTriaged
}
return EventSet
}
// set is Set as the one who acts. onlyEmpty fills only a value that is empty, as the triage does: a value a
// person gave meanwhile stands.
func (t *Tracker) set(by string, a SetAsk, onlyEmpty bool) (*SetDone, error) {
@@ -72,7 +80,7 @@ func (t *Tracker) set(by string, a SetAsk, onlyEmpty bool) (*SetDone, error) {
return nil, errors.New("set changes severity, needs or suggested-owner: give at least one")
}
var changed []string
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: setOp(onlyEmpty), By: by}, func(is *Issue) error {
changed = nil
for _, f := range []struct {
name string
@@ -344,7 +352,7 @@ func (t *Tracker) letGo(dep uint64) error {
deps := dependsOn(is)
sort.Slice(deps, func(i, j int) bool { return deps[i] < deps[j] })
claimed := false
if _, err := t.change(w, func(i *Issue) error {
if _, err := t.change(w, Act{Op: EventReady, By: Seat}, func(i *Issue) error {
claimed = false
if equalNumbers(i.ReadyFor, deps) {
return nil
+109 -91
View File
@@ -9,6 +9,7 @@ import (
"math/big"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -19,7 +20,7 @@ import (
// change as an event.
type Tracker struct {
Store Store
// Emit says an event as this module (ADR 0041); the events are the audit trail.
// Emit says an event as this module (ADR 0041); the events announce, the log is the audit trail (ADR 0300 §4).
Emit func(event string, body any) error
Settings func() (Settings, error)
Now func() time.Time
@@ -35,11 +36,19 @@ type Tracker struct {
Judge Judgement
// Deliveries answers a pull request's delivery (ADR 0279 §3, ADR 0281); nil asks nothing.
Deliveries DeliveryState
// Notify tells the operator's channel of a stuck fix (ADR 0281); nil says nothing there.
// Notify tells the operator's channel of a stuck fix (ADR 0281) and of a fix over its budget (ADR 0313); nil
// says nothing there.
Notify Notifier
// Approval answers when a pull request was first approved at its head (ADR 0313 §4); nil asks nothing, and a
// fix's review is counted with its checks.
Approval Approval
// CallerOf is who called, for the arguments one call was handed (stdio.CallerOf); nil proves nobody.
CallerOf func(map[string]any) string
// Log is the tracker's record, the log `changes` (novox/hq ADR 0300); the bucket `issues` is its fold.
Log Log
groom grooming
held *memoryView
heldOnce sync.Once
}
// The events this module says, one per kind of change.
@@ -122,9 +131,10 @@ type Opened struct {
Warning string `json:"warning,omitempty"`
}
// Open checks an issue, then takes the next number and writes the issue under it (to-be 53 §4). A refused
// issue takes no number. A number taken and then not written — the bus failing between the two — stays
// unused, and the answer says which: numbers may skip, never repeat.
// Open checks an issue, then takes the next number, appends the issue's first operation to the log at expect
// 0, and writes its fold under its number (to-be 53 §4, novox/hq ADR 0300 §2). An issue refused at validation
// takes no number. A number taken and then not taken by the log — the bus failing between the two, or a full
// log — is spent, and the open answers the failure: numbers may skip, never repeat.
func (t *Tracker) Open(a OpenAsk) (*Opened, error) { return t.open(t.by(), a) }
// open is Open as the one who acts.
@@ -174,9 +184,33 @@ func (t *Tracker) open(by string, a OpenAsk) (*Opened, error) {
if record != "" {
is.Record = RecordOf(n, record)
}
if err := t.create(&is); err != nil {
return nil, err
// The first operation, at expect 0 (ADR 0300 §2): one that fails after the number was taken spends the
// number, and the open answers the failure.
var seq uint64
if there, _, rerr := read(t.Store, n); rerr == nil && there.Opening != is.Opening {
// Kept from before the switch and not carried yet: the log does not know it, the bucket does.
err = errBegun
} else {
seq, err = t.begin(&is, Act{Op: OpOpened, By: by})
}
if errors.Is(err, errBegun) {
t.Logf("[mesh-issues] the counter handed out %d and an issue %d is already kept: not overwritten", n, n)
return nil, fmt.Errorf("a fault, not your mistake: the counter handed out %d, and an issue %d is already kept. "+
"Nothing was overwritten and nothing was opened; %d stays unused. Open it again for the next number, "+
"and tell the operator the counter is behind the issues kept", n, n, n)
}
if errors.Is(err, errUnsure) {
t.Logf("[mesh-issues] number %d was taken and the log's answer to the issue was lost: %v", n, err)
return nil, fmt.Errorf("number %d was taken and the log did not confirm the issue (%v): it may have been written. "+
"Call get %d before opening it again; if there is no issue %d, it is spent", n, err, n, n)
}
if err != nil {
t.Logf("[mesh-issues] number %d was taken and the log did not take the issue: %v", n, err)
return nil, fmt.Errorf("number %d was taken and the log did not take the issue (%v): nothing was opened and %d "+
"is spent. Open it again for the next number", n, err, n)
}
is.Folded = seq
t.store(n, &is, 0)
out := &Opened{Number: n, Title: title, Status: is.Status, LocatedIn: located, Record: is.Record, By: is.OpenedBy}
out.Said = fmt.Sprintf("Opened issue %d: %s.", n, title)
if is.Record != "" {
@@ -188,45 +222,6 @@ func (t *Tracker) open(by string, a OpenAsk) (*Opened, error) {
return out, nil
}
// create writes a new issue under its number, never over one: a key that is there already is a fault said
// loudly, and the number stays unused.
func (t *Tracker) create(is *Issue) error {
key := keyOf(is.Number)
var err error
for i := 0; i < createAttempts; i++ {
if i > 0 {
t.Sleep(jitter(time.Duration(i) * 50 * time.Millisecond))
}
_, err = t.Store.Create(IssuesState, key, is)
if err == nil {
return nil
}
if errors.Is(err, stdio.ErrChanged) {
// Ours, written by an attempt whose answer was lost?
there, _, rerr := read(t.Store, is.Number)
if rerr != nil {
t.Logf("[mesh-issues] issue %d is kept and could not be read back: %v", is.Number, rerr)
return fmt.Errorf("issue %d may have been written by this open: the key is taken and could not be read "+
"back (%v). Call get %d before opening it again, so it is not opened twice", is.Number, rerr, is.Number)
}
if there.Opening == is.Opening {
return nil
}
t.Logf("[mesh-issues] the counter handed out %d and an issue %d is already kept: not overwritten", is.Number, is.Number)
return fmt.Errorf("a fault, not your mistake: the counter handed out %d, and an issue %d is already kept. "+
"Nothing was overwritten and nothing was opened; %d stays unused. Open it again for the next number, "+
"and tell the operator the counter is behind the issues kept", is.Number, is.Number, is.Number)
}
}
// A write that timed out may still have landed: say so, never "nothing was opened".
t.Logf("[mesh-issues] number %d was taken and the issue's write was not confirmed: %v", is.Number, err)
if there, _, rerr := read(t.Store, is.Number); rerr == nil && there.Opening == is.Opening {
return nil
}
return fmt.Errorf("number %d was taken and the issue's write was not confirmed (%v): it may have been written. "+
"Call get %d before opening it again; if there is no issue %d, it stays unused", is.Number, err, is.Number, is.Number)
}
// next takes the next issue number: the counter advanced only at the revision read, so two openers — in
// one instance or two — never take one number (ADR 0275 §3).
func (t *Tracker) next() (uint64, error) {
@@ -285,38 +280,6 @@ func (t *Tracker) seed() error {
return nil
}
// change reads an issue, applies f, and writes it only at the revision read; a write another writer won
// is read and applied again, so no note, link or move is lost and none is applied to a stale issue.
func (t *Tracker) change(n uint64, f func(*Issue) error) (*Issue, error) {
for i := 0; i < issueAttempts; i++ {
is, rev, err := read(t.Store, n)
if err != nil {
return nil, err
}
if err := f(is); err != nil {
return nil, err
}
raw, err := json.Marshal(is)
if err != nil {
return nil, err
}
if len(raw) > MostIssueBytes {
return nil, fmt.Errorf("issue %d would be %d bytes; one issue is kept in at most %d (256 KiB). Nothing "+
"was changed: say it shorter, or in a document the issue links", n, len(raw), MostIssueBytes)
}
_, err = t.Store.Update(IssuesState, keyOf(n), json.RawMessage(raw), rev)
if err == nil {
return is, nil
}
if !errors.Is(err, stdio.ErrChanged) {
return nil, fmt.Errorf("issue %d could not be written (%v); nothing was changed", n, err)
}
t.Sleep(jitter(time.Duration(i+1) * 5 * time.Millisecond))
}
return nil, fmt.Errorf("issue %d was changed by another caller %d times while this change was made; nothing was "+
"changed. Try again", n, issueAttempts)
}
// Get is one issue whole.
func (t *Tracker) Get(n uint64) (*Issue, error) {
is, _, err := read(t.Store, n)
@@ -451,7 +414,7 @@ func (t *Tracker) move(by string, a MoveAsk) (*Moved, error) {
var from string
var changed []string
var work workOutcome
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: EventMoved, By: by}, func(is *Issue) error {
changed = nil
from = is.Status
at := t.now()
@@ -497,6 +460,11 @@ func (t *Tracker) move(by string, a MoveAsk) (*Moved, error) {
return err
}
is.Status = to
if to == StatusFixing && from != StatusFixing && is.FixingFrom == nil {
// The first move to fixing starts the fix time; none after it restarts it (ADR 0313 §1).
start := at
is.FixingFrom = &start
}
if (to == StatusResolved || to == StatusWontfix) && from != to {
is.Closed = &at
} else if to != StatusResolved && to != StatusWontfix {
@@ -610,7 +578,7 @@ func (t *Tracker) note(by string, n uint64, text string) (*Noted, error) {
return nil, err
}
note := Note{By: by, At: t.now(), Text: text}
is, err := t.change(n, func(is *Issue) error {
is, err := t.change(n, Act{Op: EventNoted, By: by}, func(is *Issue) error {
is.Notes = append(is.Notes, note)
touch(is, by, note.At)
return nil
@@ -680,7 +648,7 @@ func (t *Tracker) link(by string, a LinkAsk) (*Linked, error) {
}
}
at := t.now()
is, err := t.change(a.Number, func(is *Issue) error {
is, err := t.change(a.Number, Act{Op: EventLinked, By: by}, func(is *Issue) error {
for i, l := range is.Links {
if l.Kind == kind && l.Target == target {
if !a.Remove {
@@ -727,6 +695,10 @@ type ListAsk struct {
Ready *bool
// Flag keeps only the issues delivered and not resolved, or the abandoned ones (novox/hq issue 483).
Flag string
// Search ranks the issues whose title, report or notes hold its words, best first (novox/hq ADR 0300 §5).
Search string
// Before keeps only the issues numbered below it: the cursor that pages the newest-first list.
Before uint64
}
// Row is one issue in a list: short enough to read in a chat.
@@ -755,12 +727,15 @@ type Row struct {
// Listed is the answer to a list.
type Listed struct {
Rows []Row `json:"rows"`
Matched int `json:"matched"`
Said string `json:"said"`
Rows []Row `json:"rows"`
Matched int `json:"matched"`
// Next is the before that continues the list where this page ends; 0 when nothing more matches.
Next uint64 `json:"next,omitempty"`
Said string `json:"said"`
}
// List is the issues that match, newest first.
// List is the issues that match, newest first, or ranked by a search: from memory once it is loaded on start
// (novox/hq ADR 0300 §5).
func (t *Tracker) List(a ListAsk) (*Listed, error) {
status := strings.TrimSpace(a.Status)
if status != "" && !known(status) {
@@ -795,6 +770,19 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
return nil, err
}
}
search := ""
if strings.TrimSpace(a.Search) != "" {
if search, err = singleLine("search", a.Search, MostPhrase); err != nil {
return nil, err
}
if len(tokens(search)) == 0 {
return nil, fmt.Errorf("search %q holds no word of two letters or digits", search)
}
if a.Before != 0 {
return nil, errors.New("before pages the newest-first list; a search is ranked best first, so give limit, " +
"or more words, instead")
}
}
limit := a.Limit
if limit <= 0 {
limit = 50
@@ -806,10 +794,30 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
}
now, staleAfter := t.now(), t.workerStaleAfter()
var numbers []uint64
for n := range all {
numbers = append(numbers, n)
if search != "" {
var hits []hit
if v := t.view(); v.ready() {
hits = v.search(search)
} else {
x := newIndex()
for _, is := range all {
x.add(is)
}
hits = x.search(search)
}
for _, h := range hits {
if all[h.Number] != nil {
numbers = append(numbers, h.Number)
}
}
} else {
for n := range all {
if a.Before == 0 || n < a.Before {
numbers = append(numbers, n)
}
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] > numbers[j] })
}
sort.Slice(numbers, func(i, j int) bool { return numbers[i] > numbers[j] })
out := &Listed{Rows: []Row{}}
lower := strings.ToLower(phrase)
for _, n := range numbers {
@@ -849,13 +857,23 @@ func (t *Tracker) List(a ListAsk) (*Listed, error) {
out.Rows = append(out.Rows, row)
}
}
below := ""
if a.Before != 0 {
below = fmt.Sprintf(" below %d", a.Before)
}
switch {
case out.Matched == 0:
out.Said = "No issue matches."
out.Said = "No issue matches" + below + "."
case search != "" && out.Matched > len(out.Rows):
out.Said = fmt.Sprintf("%d issues match the search; the best %d are shown, best first.", out.Matched, len(out.Rows))
case search != "":
out.Said = fmt.Sprintf("%d issues match the search, best first.", out.Matched)
case out.Matched > len(out.Rows):
out.Said = fmt.Sprintf("%d issues match; the newest %d are shown.", out.Matched, len(out.Rows))
out.Next = out.Rows[len(out.Rows)-1].Number
out.Said = fmt.Sprintf("%d issues match%s; the newest %d are shown. The next page is before %d.", out.Matched,
below, len(out.Rows), out.Next)
default:
out.Said = fmt.Sprintf("%d issues match, newest first.", out.Matched)
out.Said = fmt.Sprintf("%d issues match%s, newest first.", out.Matched, below)
}
return out, nil
}
@@ -106,19 +106,19 @@ func TestAnIssuesKeyIsNeverOverwrittenAndTheFaultIsSaid(t *testing.T) {
func TestANumberTakenAndNotWrittenIsSaidAndNeverHandedOutAgain(t *testing.T) {
h := newHarness(t)
h.open(t, "first")
h.mem.fail[IssuesState] = errors.New("the bus did not answer")
h.log.failNext = 1
// One failure is tried again and succeeds: no gap.
if got := h.open(t, "second"); got != 366 {
t.Fatalf("got %d", got)
}
// Failing every time leaves a gap, said.
failing := &failingStore{memory: h.mem, failCreates: createAttempts}
h.Store = failing
// Failing every time spends the number, said (novox/hq ADR 0300 §2).
h.log.failNext = createAttempts
_, err := h.Open(OpenAsk{Title: "third", Report: "r"})
if err == nil || !strings.Contains(err.Error(), "may have been written") || !strings.Contains(err.Error(), "get 367") {
t.Fatalf("a write that never landed: %v", err)
if err == nil || !strings.Contains(err.Error(), "367 is spent") || !strings.Contains(err.Error(), "did not answer") {
t.Fatalf("an append that never landed: %v", err)
}
h.Store = h.mem
// A bucket that takes no write after the log took the issue: the issue is opened, the bucket folds it later.
h.mem.fail[IssuesState] = errors.New("the bus did not answer")
if got := h.open(t, "fourth"); got != 368 {
t.Fatalf("got %d, want 368: a number is never handed out twice", got)
}
@@ -189,17 +189,30 @@ func (u unreadable) Get(state, key string) (*Entry, error) {
return u.memory.Get(state, key)
}
// An open whose write may have landed never says nothing was opened (the review of 2026-10-10).
// unreadableLog loses every append's answer and cannot be read back.
type unreadableLog struct{ *fakeLog }
func (u unreadableLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
_, _ = u.fakeLog.AppendAt(key, value, expect)
return 0, errors.New("no answer in 30 s")
}
func (u unreadableLog) Read(uint64, string, int) (*stdio.LogPage, error) {
return nil, errors.New("no answer in 30 s")
}
// An open whose write may have landed never says nothing was opened (the review of 2026-10-10), in the bucket
// or in the log.
func TestAnOpenWhoseWriteMayHaveLandedNeverSaysNothingWasOpened(t *testing.T) {
h := newHarness(t)
h.Store = lostEvery{h.mem}
if o, err := h.Open(OpenAsk{Title: "t", Report: "r"}); err != nil || o.Number != 365 {
t.Fatalf("a write that landed, every answer lost: %v %v", o, err)
}
h.Store = unreadable{h.mem}
h.Store = h.mem
h.Log = unreadableLog{h.log}
_, err := h.Open(OpenAsk{Title: "t", Report: "r"})
if err == nil || strings.Contains(err.Error(), "nothing was opened") || !strings.Contains(err.Error(), "get 366") {
t.Fatalf("a taken key that cannot be read back: %v", err)
t.Fatalf("an append whose answer was lost and cannot be read back: %v", err)
}
}
@@ -434,7 +447,7 @@ func TestANoteThatWouldMakeTheIssueTooLargeIsRefusedAndNothingChanges(t *testing
for i := 0; i < 40 && err == nil; i++ {
_, err = h.Note(n, note)
}
if err == nil || !strings.Contains(err.Error(), "256 KiB") {
if err == nil || !strings.Contains(err.Error(), "252 KiB") {
t.Fatalf("no refusal: %v", err)
}
e, _ := h.mem.Get(IssuesState, keyOf(n))
@@ -305,7 +305,7 @@ func TestTheSameEventTwiceJudgesOnceAndFinishesWhatIsMissing(t *testing.T) {
o2 := h2.open(t, "the counter skips a number")
n2 := h2.open(t, "a number is skipped")
kept := &Verdict{Verdict: VerdictDuplicate, Issue: o2, Certain: true, Reason: "the same", Severity: "minor", Needs: "none", By: "the agent"}
if _, err := h2.change(n2, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
if _, err := h2.change(n2, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
t.Fatal(err)
}
s2 := withJudge(h2, &stubJudge{err: errors.New("must not be asked")})
@@ -690,7 +690,7 @@ func TestATriagedIssueAPersonResolvedIsNotTouchedAgain(t *testing.T) {
older := h.open(t, "the counter skips a number")
newer := h.open(t, "a number is skipped")
kept := &Verdict{Verdict: VerdictDuplicate, Issue: older, Certain: true, Reason: "same"}
if _, err := h.change(newer, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
if _, err := h.change(newer, Act{Op: EventTriaged, By: GroomBy}, func(i *Issue) error { i.Triage = kept; return nil }); err != nil {
t.Fatal(err)
}
resolve(t, h, newer)
+291
View File
@@ -0,0 +1,291 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"math"
"sort"
"strings"
"sync"
"unicode"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// memoryView is every folded issue in memory, kept current by watching the module's own bucket, with a full-text
// index of title, report and notes (novox/hq ADR 0300 §5). Until it is loaded on start, the tracker reads
// the bucket as before.
type memoryView struct {
mu sync.RWMutex
loaded bool
issues map[uint64]*Issue
index *index
}
func newMemoryView() *memoryView { return &memoryView{issues: map[uint64]*Issue{}, index: newIndex()} }
// view is the tracker's issues in memory.
func (t *Tracker) view() *memoryView {
t.heldOnce.Do(func() { t.held = newMemoryView() })
return t.held
}
// load holds these issues and rebuilds the index from them. An issue the watch handed over meanwhile is kept
// where it was folded later than the one read.
func (h *memoryView) load(all map[uint64]*Issue) {
h.mu.Lock()
defer h.mu.Unlock()
merged := map[uint64]*Issue{}
for n, is := range all {
merged[n] = is
}
for n, is := range h.issues {
if old := merged[n]; old == nil || old.Folded < is.Folded {
merged[n] = is
}
}
h.issues = merged
h.index = newIndex()
for _, is := range merged {
h.index.add(is)
}
h.loaded = true
}
func (h *memoryView) ready() bool {
h.mu.RLock()
defer h.mu.RUnlock()
return h.loaded
}
// put holds an issue unless what is held was folded later. Before the load it is held for the load to merge.
func (h *memoryView) put(is *Issue) {
h.mu.Lock()
defer h.mu.Unlock()
old := h.issues[is.Number]
if old != nil && old.Folded > is.Folded {
return
}
h.issues[is.Number] = is
if h.loaded {
if old != nil {
h.index.remove(old.Number)
}
h.index.add(is)
}
}
func (h *memoryView) drop(n uint64) {
h.mu.Lock()
defer h.mu.Unlock()
delete(h.issues, n)
if h.loaded {
h.index.remove(n)
}
}
// all is every issue held, each a shallow copy: a caller may set an issue's fields without changing what is held,
// but its lists (notes, links, fixes and the rest) are shared with what is held, so a caller reads them and never
// changes them in place. Every caller in this module only reads; a write goes through change, which reads the
// bucket afresh. A deep copy per list would cost a JSON round trip of every issue on every list.
func (h *memoryView) all() map[uint64]*Issue {
h.mu.RLock()
defer h.mu.RUnlock()
out := make(map[uint64]*Issue, len(h.issues))
for n, is := range h.issues {
c := *is
out[n] = &c
}
return out
}
func (h *memoryView) search(q string) []hit {
h.mu.RLock()
defer h.mu.RUnlock()
return h.index.search(q)
}
// remember holds an issue the tracker just wrote, so a list right after a write shows it before the watch
// hands it back.
func (t *Tracker) remember(is *Issue) {
if is == nil {
return
}
c, err := clone(is)
if err != nil {
return
}
t.view().put(c)
}
// Watched takes one change of the module's own bucket into memory: a put held unless memory holds a later
// fold, a delete dropped (the next start folds it back from the log).
func (t *Tracker) Watched(c stdio.StateChange) error {
n, ok := numberOfKey(c.Key)
if !ok {
return nil
}
if c.Op != "put" {
t.view().drop(n)
return nil
}
var is Issue
if err := json.Unmarshal(c.Value, &is); err != nil {
return fmt.Errorf("issue %d is kept in a form this module cannot read: %v", n, err)
}
t.view().put(&is)
return nil
}
// readAll is every issue the bucket holds, by number, each read by its key.
func (t *Tracker) readAll() (map[uint64]*Issue, error) {
keys, err := t.Store.Keys(IssuesState)
if err != nil {
return nil, fmt.Errorf("the issues cannot be listed: %v", err)
}
out := map[uint64]*Issue{}
for _, k := range keys {
n, ok := numberOfKey(k)
if !ok {
continue
}
is, _, err := read(t.Store, n)
if errors.Is(err, ErrNoIssue) {
continue
}
if err != nil {
return nil, err
}
out[n] = is
}
return out, nil
}
// allIssues is every issue kept, by number: from memory once it is loaded, from the bucket before.
func (t *Tracker) allIssues() (map[uint64]*Issue, error) {
if v := t.view(); v.ready() {
return v.all(), nil
}
return t.readAll()
}
// index is a full-text index of the issues' title, report and notes, ranked by BM25. Written here rather than
// taken from a library (bleve): the tracker holds a few thousand issues, the index is rebuilt on every start in
// well under a second, and its whole need — words in, issues ranked out — is a hundred lines, against a
// dependency of dozens of modules for a bundle that has two.
type index struct {
// postings is, for each word, how often it stands in each issue, the title's words counted titleWeight
// times.
postings map[string]map[uint64]int
// lengths is each issue's words, weighted as postings counts them; total their sum.
lengths map[uint64]int
words map[uint64][]string
total int
}
const titleWeight = 3
func newIndex() *index {
return &index{postings: map[string]map[uint64]int{}, lengths: map[uint64]int{}, words: map[uint64][]string{}}
}
// tokens is a text's words: lower case, letters and digits, two characters at least.
func tokens(s string) []string {
var out []string
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) }) {
if len([]rune(w)) >= 2 {
out = append(out, w)
}
}
return out
}
func (x *index) add(is *Issue) {
counts := map[string]int{}
for _, w := range tokens(is.Title) {
counts[w] += titleWeight
}
body := []string{is.Report}
for _, n := range is.Notes {
body = append(body, n.Text)
}
for _, w := range tokens(strings.Join(body, "\n")) {
counts[w]++
}
length := 0
words := make([]string, 0, len(counts))
for w, c := range counts {
p := x.postings[w]
if p == nil {
p = map[uint64]int{}
x.postings[w] = p
}
p[is.Number] = c
length += c
words = append(words, w)
}
x.lengths[is.Number] = length
x.words[is.Number] = words
x.total += length
}
func (x *index) remove(n uint64) {
for _, w := range x.words[n] {
if p := x.postings[w]; p != nil {
delete(p, n)
if len(p) == 0 {
delete(x.postings, w)
}
}
}
x.total -= x.lengths[n]
delete(x.lengths, n)
delete(x.words, n)
}
// hit is one issue a search found, and how well it matched.
type hit struct {
Number uint64
Score float64
}
// search ranks every issue holding at least one of the query's words by BM25: a word few issues hold weighs
// more, a word standing often weighs more with diminishing return, a long issue weighs each word less. Equal
// scores rank the newer first.
func (x *index) search(q string) []hit {
docs := len(x.lengths)
if docs == 0 {
return nil
}
const k1, b = 1.2, 0.75
avg := float64(x.total) / float64(docs)
scores := map[uint64]float64{}
seen := map[string]bool{}
for _, w := range tokens(q) {
if seen[w] {
continue
}
seen[w] = true
p := x.postings[w]
if len(p) == 0 {
continue
}
idf := math.Log(1 + (float64(docs)-float64(len(p))+0.5)/(float64(len(p))+0.5))
for n, tf := range p {
f := float64(tf)
scores[n] += idf * f * (k1 + 1) / (f + k1*(1-b+b*float64(x.lengths[n])/avg))
}
}
out := make([]hit, 0, len(scores))
for n, s := range scores {
out = append(out, hit{n, s})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Score != out[j].Score {
return out[i].Score > out[j].Score
}
return out[i].Number > out[j].Number
})
return out
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-issues
go 1.25.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.51.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf h1:ED/+RszKwm86U7H/NZRbS4b5LzQ8n7rubqKmeFqFtvM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261011091942-4cf45b5ca0cf/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac h1:jnZ/n+fFMo7VQPNNXK1gjStHIJPw3SPbzPxsf/PpZqw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012082113-9747f3e1b1ac/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+8 -2
View File
@@ -17,7 +17,8 @@
"groom",
"set",
"tracking",
"reserve"
"reserve",
"fix-times"
]
}
],
@@ -36,7 +37,8 @@
"groom",
"set",
"tracking",
"reserve"
"reserve",
"fix-times"
]
}
],
@@ -61,11 +63,15 @@
"claude-code.claude_code_judge",
"seat:mesh-delivery.checks",
"seat:operator-channel.notify",
"gitea.gitea_pull_reviews",
"records.records_status",
"records.records_list",
"records.records_read",
"mesh-knowledge.knowledge_highest"
],
"logs": [
"changes"
],
"state": [
{
"name": "issues",
+31 -9
View File
@@ -23,7 +23,7 @@ operator, it is accepted only on the operator's Approve, given on a channel that
| `reserve` | `kind`, `for` | the next decision or research number, asked of the issue tracker (`issue-tracker.reserve`); spent whether or not its record is accepted |
| `propose` | the record whole | proposed, or refused naming every rule it breaks; an accepted record changes only by a progressive insight (`insight`). Each version proposed is then reviewed by the knowledge base itself (below) |
| `import` | `dry-run`, `commit` | hq's decision records and research efforts at the commit the records module stands at, each whole at its own number |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again; refused for a record whose clauses another record changed or links by number |
`ask`, `get`, `holds` and `history` stay within `bound` bytes (4096 unless asked for, at most 65536) and say what they cut
and how to read the rest. The module's own tool `knowledge_health` answers its health (below).
@@ -78,7 +78,7 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- 25 on the structure: kind, reserved number, title, topic, date, deciders, research status, at least one
clause, clause numbering, bold lead, clause size, the Context, Consequences and "How it is checked" sections,
a research effort's overview, link kind, link target, link fit, no link to itself, what it extends, what it
supersedes or changes is accepted, a changed clause is in force, a superseded record cited beside its
supersedes or changes is accepted, a changed clause is in force or changed (a drafted one too), a superseded record cited beside its
superseder, a graduated effort says what it became, and the links in its text resolve;
- 4 on the prose: no word the glossary retired, no daemon's line, a deferral names its tracking issue, and a
progressive insight in the dated, marked form.
@@ -99,13 +99,21 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
the record, the records it links and the rules, and answers `approve` or `changes` with findings. The rules it sees
name every kind of link `propose` accepts, from the one list `propose` holds a link to, and the record it sees
shows its state in the knowledge base apart from a research effort's own status (hq issue 492). The background
work asks it at once, before the next step of any drafting, one judgement at a time.
work asks it at once, before the next step of any drafting, one judgement at a time. Every judgement asks a bound
sized to its question: 25 seconds and 2 more per KiB, at most 120 (hq issue 508: an eleven-clause record was never
reviewed in 25). A judgement not done within one bus call is answered pending with a ticket, and collected from
the same node with `claude-code.claude_code_judged`.
- On `changes` the record stays proposed, the findings kept with the version they judged (`get` shows them).
Its author revises it by proposing it again, and that version is reviewed again.
- A judgement that cannot be had (the judge unreachable, or an answer that is not `approve`, or `changes` naming
no finding) leaves the record proposed and says so on it (`get`'s `review`). It is never taken as an
approval. It is asked again after 10, then 20 minutes; after 3 tries the record says its author proposes it
again to have it reviewed again, and the module is unhealthy, naming it, so the mesh raises a condition.
approval. It is owed: its try count and when it is asked again (10, then 20 minutes later) are kept with the
version in the store and said in its history, so the background work takes it up at that time, also after a
restart of the tool runner (hq issue 497). After 3 tries the record says it was the last and its author
proposes it again to have it reviewed again; the operator is told once on the operator's channel (keyed by
record and version, cleared once the record is proposed again, accepted, withdrawn or superseded, and sent
again at the next step when it could not be sent). The module's health names the record and stays healthy:
the record waits on its author, not on the module, so it never holds the module's own rollouts (hq issue 508).
An ask of the operator that cannot be made counts as a try the same way.
- A review whose outcome the database refused to keep is held back for that record alone (10, then 20
minutes, 3 tries, logged as given up), so it never blocks the records behind it.
@@ -124,8 +132,16 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- `partial`: both ends of a change agree;
- `rests-on-live`: no accepted record extends a superseded one.
A failure is kept as a finding and makes the module unhealthy, so the mesh raises a condition naming the rule
and the records. The acceptance stands.
A failure is kept as a finding and told to the operator once, as a warning naming the rule and the records
(`operator-channel.notify`, keyed `knowledge.finding.<rule>.<records>`, sent again at the next step when it
could not be sent, and cleared once the rule holds for them again). It is a fault of the records, which their
authors repair, not of the module: the module's health names it and stays healthy, so it never holds the
module's own rollouts (the operator's decision of 2026-10-12 on hq issue 504). The acceptance stands.
`cited` is also judged ahead, in the review: a decision that names no home and that no accepted (or
superseded) record links is not judged and not accepted. Its review, by the knowledge base itself, asks changes
naming the rule (hq issue 504). It is not refused at propose, because no record can link a number only
reserved: a record citing a new one is proposed after it and accepted, and the new one is proposed again.
## The import
@@ -162,6 +178,12 @@ the older clause marked `changed`. When every prose record of a topic has been t
operator is told once (`operator-channel.notify`, a warning) with the counts. `return` sends a record back to be
drafted again.
A drafted clause may be changed by a later record (`changes 0281 §9`, novox/hq issue 494, ADR 0321): on its acceptance the
clause is marked `changed` with the change named, keeps its number and stays `drafted`, so it is still never served
as in force, and `get` says both. A redraft numbers a record's clauses anew, so `return` refuses a record any of
whose clauses another record changed or links by number, served or proposed, naming each link: a link never comes
to point at another clause.
## Where it keeps its records
In a database in the store: the module requires a `postgres-database` provision named `knowledge`, as
@@ -173,14 +195,14 @@ the consolidated schema for a fresh database is `schema.sql`. A test holds the t
- **the bundle**: the verbs, `knowledge_health`, the operator's answers heard, and once a minute (and at once when
a record is proposed) the router's record read, reservations never proposed withdrawn, the reviews of proposals,
and the drafting worker. It never changes the schema, and
and the drafting worker; and it tells the whole-set findings. It never changes the schema, and
refuses to serve while the schema is behind.
- **`prepare`** (run once): brings the schema to this version.
- **`keep`** (judged by its unit): judges the whole record set every five minutes and keeps the findings.
- **`watch`** (judged by `knowledge_health`, run as the machine's account, the one the runtime runs the bundle
as, so the tool can read what it writes): every minute, reads whether the database serves and what the
findings say, and leaves its verdict in `${dir:state}/health.json`. That is the file the health tool answers
from. A verdict older than three minutes is unhealthy.
from. A verdict older than three minutes is unhealthy; a whole-set finding is named in it but leaves it healthy.
## Settings

Some files were not shown because too many files have changed in this diff Show More