Compare commits

..
Author SHA1 Message Date
jochen 46c46c955b betterbird: an account's own aliases and identities that do not send by SMTP share no outgoing server, and the pins name their vendors on purpose (review 36, review 35 of mesh-controller #251)
mesh/delivery-group group fix/345-reminder-test rejected: a member's own check failed
mesh/delivery stopped: the pull request closed unmerged
mesh/merge-gate fail: builds betterbird, sent nowhere; no bus step; 1 wait(s) for a person; a manifest the change touches fails the module check: modules/b…
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 24
The shared-server check refused an adopted account for its own alias, and
counted Owl's, local folders' and feeds' identities as users of the default
server, though they never send by SMTP. The controller now judges an
upstream pin's host like any name, so each pin declares its vendor.
2026-10-12 13:25:58 +02:00
jochen 1ca26c944a betterbird: withhold account values in the user.js answer, never write a shared outgoing server, and say a newer Owl as a fact (review 34, hq issue 345, ADR 0317)
mesh/merge-gate fail: builds betterbird → shanks; no bus step; a manifest the change touches fails the module check: modules/betterbird/module.json: this i…
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 24
mesh/delivery-group group fix/345-reminder-test rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
The user.js answer carried declared addresses and hosts (ADR 0316 §16). An
adopted identity's outgoing server may be shared with an account the module
does not own, whose sending the declared values would change at every start:
refused with the remedy, since ADR 0317 §2 leaves the identity's server and
the lists as they are; a recorded adoption follows the identity's current
server. Owl installed by policy cannot be removed, so a newer Owl than the
pin is a fact whose remedy is moving the pin.
2026-10-12 13:19:49 +02:00
jochen 96e1792881 betterbird: install Betterbird from its pinned release, ship Owl by policy, and declare the accounts without credentials (hq issue 345, ADR 0316, ADR 0317)
mesh/merge-gate fail: builds betterbird → shanks; no bus step; a manifest the change touches fails the module check: modules/betterbird/module.json: this i…
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 24
mesh/delivery-group group fix/345-reminder-test rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A module that cannot set Betterbird up is of little use (the operator,
2026-10-12). The module now declares the community package absent and
unpacks the pinned upstream release where that package had it, so the same
profile opens; places the enterprise policy that turns Betterbird's updater
off and force-installs Owl from the xpi the mesh pinned by its vendor's
digest (never in a repository); and takes the operator's accounts from the
trusted setting accounts, without any password or token: an account the
profile already holds is adopted under its own keys, a new one is created
under the module's mesh_ keys, an ambiguous one is refused with the remedy,
and the account lists are only written into. betterbird_accounts says the
profile's accounts by count and proposes the setting built from them for the
operator's Approve. Every new build waits for the operator's push. The
reminder test is visibly a test (MESH TEST band, event name) and its
description says it opens a window on the operator's screen.
2026-10-12 12:59:39 +02:00
jochen cb9b66e628 betterbird: keep the reminder test's update check off so no About window takes the screen (hq issue 345)
A fresh profile has never checked for updates, so Betterbird checked at the
test's start, found a newer build published and opened its About window on the
operator's screen instead of leaving the reminder to show. The test's own
preferences now also come last in its user.js, so nothing declared or carried
overrides them.
2026-10-12 12:59:39 +02:00
114 changed files with 781 additions and 12114 deletions
-2
View File
@@ -19,5 +19,3 @@ modules/mesh-issues/cmd/mesh-issues/mesh-issues
modules/mounts/cmd/mounts/mounts
modules/mesh-knowledge/cmd/mesh-knowledge/mesh-knowledge
modules/mesh-knowledge/mesh-knowledge
modules/mesh-reads/mesh-reads
modules/mesh-reads/cmd/mesh-reads/mesh-reads
+1 -1
View File
@@ -1,4 +1,4 @@
67
69
The long-running resources of this catalogue that do not say how they are ready (`health`, novox/hq ADR 0240
rule 8), as `module check` counts them. It may only go down: merge-check.sh fails a change that raises it, and
one that lowers it writes the new number on the first line. From 2026-11-18, or once it is 0, a long-running
+4 -22
View File
@@ -1,8 +1,7 @@
# artifact-store-tools
Tools that say what the mesh's artifact store holds, and what the controller's records say of it
(novox/hq ADR 0251 §1–3, to-be 51). It changes nothing on its machine; its only resources are its state
directory and the file the mesh writes there from its setting `older-store-names`.
(novox/hq ADR 0251 §1–3, to-be 51). It declares no resources and changes nothing on its machine.
## Why a module beside the store
@@ -19,7 +18,7 @@ on that machine.
|---|---|---|
| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size |
| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) |
| `artifact_store_references` | r | what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_references` | r | what the controller's records say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` |
### How the store is read
@@ -44,7 +43,6 @@ it lists. The store's collector removes every blob no manifest marks, so these n
| state | what | removable |
|---|---|---|
| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no |
| `named-by-a-definition` | no record names it, and a definition in the catalogue pins it by digest, under the store's current name or an older one; `why` names the module (hq issue 323) | no |
| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no |
| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` |
| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive |
@@ -58,20 +56,6 @@ its own right; the controller lets an index go with the platform manifests only
counted as the index's. Two indexes naming one platform give it the more keeping state. An index no
record names leaves its platform manifests unrecorded with it.
**What the definitions pin** (hq issue 323). A definition may pin an image of the store the mesh did not
make: pushed before the mesh built it, and pinned under the store's current name or a name it had before.
No record of the controller names it, yet a machine runs it, so `artifact_store_references` reads every
definition the catalogue holds (`mesh-catalog.catalog_modules`, then `catalog_module` for each, eight at
a time and within twenty seconds) and finds every `<name>/<repository>@sha256:…` in any of its strings,
under each of the store's names: `artifact-store://`, the store's address as the controller answers it,
and the names it had before. Those are the mesh's setting `older-store-names` (space-separated, `none` by
default, set with the controller's `settings`), which the mesh writes to the module's `store-names` file: a
catalogue names no installation. When they cannot be read, the answer's first line says so. A manifest no record
names that a definition pins is `named-by-a-definition`; one the records name keeps the records' state. A
definition read at its commit is kept, so the next call asks only for the list. Definitions that could
not be read are listed under `definitions.unread`, and the answer's first line says that what they pin
reads as unrecorded until they are.
**Unrecorded copies of upstream images** are listed in `unrecorded_mirrors`: each manifest at the top of
a repository only the build's mirror writes (`upstream/<host>/<path>`, or a module's `<module>/on-<argument>`
from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the
@@ -104,8 +88,6 @@ with slashes, a cut listing refused), manifests read with one absent or unreadab
digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks),
every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept
manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked,
the fall-back when the references let go of cannot be carried, a manifest a definition pins under the
store's current or an older name, a definition not read said first, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs and the
catalogue's two tools.
the fall-back when the references let go of cannot be carried, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs.
Any command other than the two read-only scripts fails the test.
@@ -7,15 +7,12 @@ import (
// The states a manifest the store holds can be in, as the records see it (novox/hq to-be 51).
const (
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
// StateNamedByDefinition is a manifest no record names that a definition pins by digest, under the
// store's current name or an older one (novox/hq issue 323): a machine may run it, so it is kept.
StateNamedByDefinition = "named-by-a-definition"
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
StateUnrecorded = "unrecorded"
collectionRemovesThese = "eligible and holder-of-eligible-archive, through the controller's collect; nothing else"
)
@@ -48,8 +45,8 @@ var mirrorRepository = regexp.MustCompile(`^(upstream/[a-z0-9][a-z0-9._/-]*|[a-z
// keeping orders the states a platform manifest may take from the indexes naming it: the most keeping
// wins, so a platform two indexes name is as kept as the more kept of them.
var keeping = map[string]int{
StateKept: 0, StateNamedByDefinition: 1, StateNamedDocument: 2, StateHolderKept: 3, StateEligible: 4,
StateHolderEligible: 5, StateCollectedPresent: 6,
StateKept: 0, StateNamedDocument: 1, StateHolderKept: 2, StateEligible: 3, StateHolderEligible: 4,
StateCollectedPresent: 5,
}
// recordIndex finds a record by repository and digest.
@@ -67,9 +64,8 @@ func indexRecords(r *Records) recordIndex {
return idx
}
// Classify gives every manifest the store holds one state. A manifest no record names that a definition
// pins is named by a definition; defs may be nil, when the definitions were not read.
func Classify(v *View, recs *Records, defs *Definitions) []Entry {
// Classify gives every manifest the store holds one state.
func Classify(v *View, recs *Records) []Entry {
idx := indexRecords(recs)
var out []Entry
for _, name := range v.L.RepoNames() {
@@ -103,12 +99,6 @@ func Classify(v *View, recs *Records, defs *Definitions) []Entry {
e.State = StateNamedDocument
}
}
if e.State == StateUnrecorded && defs != nil {
for _, p := range defs.Pins[Key(name, d)] {
e.State = StateNamedByDefinition
e.Why = append(e.Why, p.why())
}
}
out = append(out, e)
}
}
@@ -1,343 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"regexp"
"sort"
"strings"
"sync"
"time"
)
// What the definitions pin in the store (novox/hq issue 323). A definition may pin an image of the store by
// digest that the mesh did not make: one pushed before the mesh built it, pinned under the store's current
// name or under a name the store had before. No record of the controller names such a manifest, and yet a
// machine runs it, so it is said to be named by a definition and never unrecorded.
// NamesFileEnv names the file the mesh writes from the setting older-store-names: a line `older=` with the
// names the store had before, as a definition may still write them — host names, or host:port, separated by
// spaces — or `none`. A catalogue names no installation, so the names are the mesh's setting, not the
// manifest's.
const NamesFileEnv = "MESH_ARTIFACT_STORE_NAMES_FILE"
// CatalogueModules and CatalogueModule are the catalogue's tools these definitions are read through.
const (
CatalogueModules = "mesh-catalog.catalog_modules"
CatalogueModule = "mesh-catalog.catalog_module"
)
// DefinitionsTimeout bounds reading every definition, below the runtime's thirty-second call limit; what
// was read is kept, so a call that runs out reads only the rest the next time.
const DefinitionsTimeout = 20 * time.Second
// definitionReaders is how many definitions are asked at once.
const definitionReaders = 8
// StoreNames are the names a definition may pin an image of the store under: the scheme the mesh writes
// (artifact-store://), the store's address now, and the names it had before.
type StoreNames struct {
Current []string
Older []string
}
// NamesOf is the store's names: its address as the controller answers it, and the older ones given.
func NamesOf(address, older string) StoreNames {
n := StoreNames{Current: []string{"artifact-store://"}}
if a := strings.Trim(strings.TrimSpace(address), "/"); a != "" {
n.Current = append(n.Current, a+"/")
}
for _, o := range strings.FieldsFunc(older, func(r rune) bool { return r == ',' || r == ' ' || r == '\n' || r == '\t' }) {
if o = strings.Trim(o, "/"); o != "" && o != "none" {
n.Older = append(n.Older, o+"/")
}
}
return n
}
// OlderNames reads the older names from the file the mesh writes, and says why when it cannot.
func OlderNames() (string, string) {
path := os.Getenv(NamesFileEnv)
if path == "" {
return "", NamesFileEnv + " is not set, so no older name of the store was read"
}
raw, err := os.ReadFile(path)
if err != nil {
return "", "the store's older names could not be read: " + err.Error()
}
for _, line := range strings.Split(string(raw), "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "older="); ok {
return v, ""
}
}
return "", path + " holds no line older=, so no older name of the store was read"
}
// Pin is one definition pinning a manifest of the store by digest.
type Pin struct {
Module string `json:"module"`
// Name is the store's name it is pinned under, as written: "artifact-store://", an address, or an older name.
Name string `json:"name"`
Older bool `json:"older,omitempty"`
}
// pinned is one pin found in a manifest: where in the store, and under which name.
type pinned struct {
Repository, Digest, Name string
Older bool
}
var pinShapes sync.Map // name -> *regexp.Regexp
// pinShape is `<name><repository>@sha256:<hex>`, the name standing on its own: not the tail of a longer host.
func pinShape(name string) *regexp.Regexp {
if re, ok := pinShapes.Load(name); ok {
return re.(*regexp.Regexp)
}
re := regexp.MustCompile(`(?:^|[^A-Za-z0-9.-])` + regexp.QuoteMeta(name) +
`([a-z0-9][a-z0-9._/-]*)@(sha256:[0-9a-f]{64})`)
pinShapes.Store(name, re)
return re
}
// PinsIn finds every image of the store a manifest pins by digest, in any string it holds: a resource's
// image, a build's base, a file's content.
func PinsIn(manifest json.RawMessage, names StoreNames) []pinned {
var doc any
if json.Unmarshal(manifest, &doc) != nil {
return nil
}
seen := map[string]bool{}
var out []pinned
var walk func(any)
walk = func(x any) {
switch v := x.(type) {
case map[string]any:
for _, e := range v {
walk(e)
}
case []any:
for _, e := range v {
walk(e)
}
case string:
for _, group := range []struct {
names []string
older bool
}{{names.Current, false}, {names.Older, true}} {
for _, name := range group.names {
for _, m := range pinShape(name).FindAllStringSubmatch(v, -1) {
p := pinned{Repository: strings.Trim(m[1], "/"), Digest: m[2], Name: name, Older: group.older}
if k := p.Repository + "@" + p.Digest + " " + p.Name; !seen[k] {
seen[k] = true
out = append(out, p)
}
}
}
}
}
}
walk(doc)
return out
}
// Definitions is what the catalogue's definitions pin in the store, and what of them could not be read.
type Definitions struct {
// Pins are by Key(repository, digest).
Pins map[string][]Pin
// Read is how many definitions were read; Unread says, per module, why its definition was not.
Read int
Unread map[string]string
// NotAsked says why the catalogue could not be asked at all, when it could not.
NotAsked string
// NamesNotRead says why the store's older names could not be read, when they could not.
NamesNotRead string
Names StoreNames
}
// Catalogue reads the definitions through the catalogue's tools, and keeps each definition it read by its
// module and commit, which never change together.
type Catalogue struct {
Ask Asking
mu sync.Mutex
kept map[string][]pinned // "<module>@<commit>|<names>" -> its pins
}
type catalogueEntry struct {
Module string `json:"module"`
Commit string `json:"commit"`
}
// Read asks the catalogue for every module and reads each definition, bounded by DefinitionsTimeout.
func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
defs := &Definitions{Pins: map[string][]Pin{}, Unread: map[string]string{}, Names: names}
if c == nil || c.Ask == nil {
defs.NotAsked = "this bundle has no way to ask the catalogue"
return defs
}
raw, err := c.Ask(CatalogueModules, map[string]any{})
if err != nil {
defs.NotAsked = "the catalogue's modules could not be asked: " + err.Error()
return defs
}
answer, output, ok := answerOf(raw)
if !ok {
defs.NotAsked = "the catalogue refused its modules: " + lastLine(output)
return defs
}
var list struct {
Modules []catalogueEntry `json:"modules"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &list); err != nil || list.Error != "" || list.Modules == nil {
defs.NotAsked = "the catalogue's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
return defs
}
c.mu.Lock()
if c.kept == nil {
c.kept = map[string][]pinned{}
}
c.mu.Unlock()
nameKey := strings.Join(names.Current, ",") + ";" + strings.Join(names.Older, ",")
ctx, cancel := context.WithTimeout(ctx, DefinitionsTimeout)
defer cancel()
type result struct {
module string
pins []pinned
why string
}
results := make(chan result, len(list.Modules))
work := make(chan catalogueEntry)
var wg sync.WaitGroup
for i := 0; i < definitionReaders; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for m := range work {
pins, why := c.definition(m, names, nameKey)
results <- result{m.Module, pins, why}
}
}()
}
go func() {
defer close(work)
for _, m := range list.Modules {
select {
case work <- m:
case <-ctx.Done():
return
}
}
}()
go func() { wg.Wait(); close(results) }()
got := map[string]bool{}
collect:
for {
select {
case r, open := <-results:
if !open {
break collect
}
got[r.module] = true
if r.why != "" {
defs.Unread[r.module] = r.why
continue
}
defs.Read++
for _, p := range r.pins {
k := Key(p.Repository, p.Digest)
defs.Pins[k] = append(defs.Pins[k], Pin{Module: r.module, Name: p.Name, Older: p.Older})
}
case <-ctx.Done():
break collect
}
}
for _, m := range list.Modules {
if !got[m.Module] {
defs.Unread[m.Module] = fmt.Sprintf("not read within %d s", int(DefinitionsTimeout/time.Second))
}
}
for k := range defs.Pins {
sort.Slice(defs.Pins[k], func(i, j int) bool { return defs.Pins[k][i].Module < defs.Pins[k][j].Module })
}
return defs
}
// definition reads one module's definition, or answers it from what was read before.
func (c *Catalogue) definition(m catalogueEntry, names StoreNames, nameKey string) ([]pinned, string) {
key := m.Module + "@" + m.Commit + "|" + nameKey
if m.Commit != "" {
c.mu.Lock()
pins, had := c.kept[key]
c.mu.Unlock()
if had {
return pins, ""
}
}
args := map[string]any{"module": m.Module}
if m.Commit != "" {
args["commit"] = m.Commit
}
raw, err := c.Ask(CatalogueModule, args)
if err != nil {
return nil, "could not be asked: " + err.Error()
}
answer, output, ok := answerOf(raw)
if !ok {
return nil, "refused: " + lastLine(output)
}
var one struct {
Module *struct {
Manifest json.RawMessage `json:"manifest"`
} `json:"module"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &one); err != nil || one.Error != "" || one.Module == nil || len(one.Module.Manifest) == 0 {
return nil, "not readable: " + firstOf(one.Error, errText(err), "no manifest in the answer")
}
pins := PinsIn(one.Module.Manifest, names)
if m.Commit != "" {
c.mu.Lock()
c.kept[key] = pins
c.mu.Unlock()
}
return pins, ""
}
// why says which definitions name a manifest, and under which of the store's names.
func (p Pin) why() string {
if p.Older {
return fmt.Sprintf("a definition names it: %s, under %s, an older name of the store", p.Module, shownName(p.Name))
}
return "a definition names it: " + p.Module
}
func firstOf(s ...string) string {
for _, x := range s {
if strings.TrimSpace(x) != "" {
return x
}
}
return ""
}
func errText(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// shownName is a name of the store as a person reads it: a host without the slash a pin puts after it, the
// scheme as written.
func shownName(name string) string {
if strings.HasSuffix(name, "://") {
return name
}
return strings.TrimSuffix(name, "/")
}
@@ -1,259 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// A definition may pin an image of the store the mesh did not make, under a name the store had before
// (novox/hq issue 323): three apps the anchor runs are pinned so. No record names them; a definition does.
const oldName = "registry-api.example.org"
// container is a definition whose one container runs image.
func container(image string) string {
return fmt.Sprintf(`{"module":"x","resources":[{"id":"server","type":"container","image":%q}]}`, image)
}
func TestAnImageADefinitionPinsUnderAnOlderNameOfTheStoreIsNamedByADefinition(t *testing.T) {
v := view(t, world())
names := NamesOf("store:5000", oldName)
defs := &Definitions{Pins: map[string][]Pin{}}
for module, manifest := range map[string]string{
// m-stranger: no record names it; the invoicing app pins it under the store's older name.
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
// m-old: the records say eligible; a definition pinning it does not change what the records say.
"website": container("store:5000/app/server@" + d("m-old")),
} {
for _, p := range PinsIn(json.RawMessage(manifest), names) {
defs.Pins[Key(p.Repository, p.Digest)] = append(defs.Pins[Key(p.Repository, p.Digest)], Pin{Module: module, Name: p.Name, Older: p.Older})
}
}
got := stateByName(Classify(v, withFill(records()), defs))
stranger := got["app/server "+d("m-stranger")]
if stranger.State != StateNamedByDefinition {
t.Fatalf("a manifest a definition pins under an older name reads %q, want %q", stranger.State, StateNamedByDefinition)
}
if want := []string{"a definition names it: invoicing, under " + oldName + ", an older name of the store"}; !reflect.DeepEqual(stranger.Why, want) {
t.Errorf("why %q, want %q", stranger.Why, want)
}
if s := got["app/server "+d("m-old")].State; s != StateEligible {
t.Errorf("a recorded manifest a definition pins reads %q: the records decide what the mesh made", s)
}
// Without the definitions it is what it was: unrecorded.
if s := stateByName(Classify(v, withFill(records()), nil))["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("without definitions %q", s)
}
}
func TestPinsAreFoundInEveryStringAndOnlyUnderTheStoresNames(t *testing.T) {
names := NamesOf("store:5000", oldName+", older.example/")
manifest := `{"resources":[
{"type":"container","image":"` + oldName + `/team/www@` + d("www") + `"},
{"type":"file","content":"IMAGE=older.example/team/tool@` + d("tool") + `\n"},
{"type":"container","image":"artifact-store://mod/art@` + d("built") + `"},
{"type":"container","image":"store:5000/mod/art@` + d("addressed") + `"},
{"type":"container","image":"not` + oldName + `/team/www@` + d("lookalike") + `"},
{"type":"container","image":"docker.io/library/nginx@` + d("upstream") + `"},
{"type":"container","image":"` + oldName + `/team/www:latest"}],
"build":{"on":{"base":"` + oldName + `/team/www@` + d("www") + `"}}}`
var got []string
for _, p := range PinsIn(json.RawMessage(manifest), names) {
got = append(got, fmt.Sprintf("%s@%s %s %v", p.Repository, p.Digest, p.Name, p.Older))
}
want := map[string]bool{
"team/www@" + d("www") + " " + oldName + "/ true": true,
"team/tool@" + d("tool") + " older.example/ true": true,
"mod/art@" + d("built") + " artifact-store:// false": true,
"mod/art@" + d("addressed") + " store:5000/ false": true,
}
if len(got) != len(want) {
t.Fatalf("found %d pins, want %d (one per image and name; not a longer host's tail, not another registry, not a tag): %v", len(got), len(want), got)
}
for _, g := range got {
if !want[g] {
t.Errorf("found %s", g)
}
}
}
func TestAPlatformTakesTheStateOfAnIndexADefinitionNames(t *testing.T) {
v := mirrorWorld()
defs := &Definitions{Pins: map[string][]Pin{
Key("route-proxy/on-go_base", d("i-pre")): {{Module: "route-proxy", Name: oldName + "/", Older: true}},
}}
got := stateByName(Classify(v, withFill(mirrorRecords()), defs))
for _, name := range []string{"i-pre", "p-pre"} {
if s := got["route-proxy/on-go_base "+d(name)].State; s != StateNamedByDefinition {
t.Errorf("%s reads %q, want %q", name, s, StateNamedByDefinition)
}
}
if e := got["route-proxy/on-go_base "+d("p-pre")]; e.Index != d("i-pre") {
t.Errorf("p-pre takes its state from %q", e.Index)
}
if s := got["route-proxy/on-go_base "+d("p-lost")].State; s != StateUnrecorded {
t.Errorf("a platform nothing names reads %q", s)
}
for _, um := range UnrecordedMirrors(v, Classify(v, withFill(mirrorRecords()), defs)) {
if strings.Contains(um.Reference, hexOf(d("i-pre"))) {
t.Errorf("a copy a definition names is offered to the mirrors verb: %v", um)
}
}
}
// olderNames writes the file the mesh writes from the setting older-store-names.
func olderNames(t *testing.T, names string) {
t.Helper()
path := filepath.Join(t.TempDir(), "store-names")
if err := os.WriteFile(path, []byte("# written by the mesh\nolder="+names+"\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(NamesFileEnv, path)
}
func TestTheOlderNamesAreReadFromTheFileTheMeshWrites(t *testing.T) {
olderNames(t, oldName+" older.example")
got, why := OlderNames()
if why != "" || !reflect.DeepEqual(NamesOf("", got).Older, []string{oldName + "/", "older.example/"}) {
t.Errorf("read %q (%s)", got, why)
}
olderNames(t, "none")
if got, why := OlderNames(); why != "" || len(NamesOf("", got).Older) != 0 {
t.Errorf("none read as %q (%s)", got, why)
}
t.Setenv(NamesFileEnv, filepath.Join(t.TempDir(), "absent"))
if _, why := OlderNames(); why == "" {
t.Error("an absent file was not said")
}
}
func TestOlderNamesNotReadAreSaidFirst(t *testing.T) {
t.Setenv(NamesFileEnv, "")
a := &asked{definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger"))}}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, NamesFileEnv) {
t.Errorf("first said %q", w)
}
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q without the older names", s)
}
}
// references is one artifact_store_references tool, kept across calls as the runtime keeps the bundle.
func references(t *testing.T, a *asked) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
if tl.Name == "artifact_store_references" {
return tl.Run
}
}
t.Fatal("no artifact_store_references")
return nil
}
func manifestsOf(t *testing.T, out any) map[string]Entry {
t.Helper()
entries, ok := out.(map[string]any)["manifests"].([]Entry)
if !ok {
t.Fatalf("no manifests in %v", out)
}
return stateByName(entries)
}
func TestReferencesReadTheDefinitionsUnderTheStoresOlderNames(t *testing.T) {
olderNames(t, oldName)
a := &asked{definitions: map[string]string{
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
"grafana": container("grafana/grafana@" + d("upstream")),
}}
run := references(t, a)
out, err := run(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
e := manifestsOf(t, out)["app/server "+d("m-stranger")]
if e.State != StateNamedByDefinition || len(e.Why) != 1 || !strings.Contains(e.Why[0], "invoicing") || !strings.Contains(e.Why[0], oldName) {
t.Errorf("m-stranger: %+v", e)
}
said := out.(map[string]any)["definitions"].(map[string]any)
if said["read"] != 2 || !reflect.DeepEqual(said["older_store_names"], []string{oldName}) ||
!reflect.DeepEqual(said["store_names"], []string{"artifact-store://", "store:5000"}) {
t.Errorf("definitions said %v", said)
}
if w := out.(map[string]any)["said"].([]string)[0]; strings.Contains(w, "could not be read") {
t.Errorf("a warning with every definition read: %s", w)
}
// A definition read at its commit is kept: the next call asks only for the list.
before := 0
for _, c := range a.calls {
if c == CatalogueModule {
before++
}
}
if _, err := run(map[string]any{"repository": "app/server"}); err != nil {
t.Fatal(err)
}
after := 0
for _, c := range a.calls {
if c == CatalogueModule {
after++
}
}
if before != 2 || after != 2 {
t.Errorf("asked catalog_module %d times, then %d: each definition at its commit is read once", before, after)
}
}
func TestADefinitionNotReadIsSaidFirstAndLeavesWhatItPinsUnrecorded(t *testing.T) {
olderNames(t, oldName)
a := &asked{
definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger")), "grafana": container("x")},
failing: map[string]bool{"invoicing": true},
}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q with its definition unread", s)
}
if w := m["said"].([]string)[0]; !strings.Contains(w, "1 definitions could not be read") {
t.Errorf("first said %q", w)
}
if u := m["definitions"].(map[string]any)["unread"].(map[string]string); u["invoicing"] == "" || len(u) != 1 {
t.Errorf("unread %v", u)
}
// The catalogue not answering at all: the records are still answered, and the warning stands first.
a = &asked{noCatalogue: true}
out, err = references(t, a)(map[string]any{})
if err != nil {
t.Fatal(err)
}
m = out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, "the definitions could not be read") || !strings.Contains(w, "no responders") {
t.Errorf("first said %q", w)
}
if m["definitions"].(map[string]any)["not_asked"] == nil {
t.Errorf("definitions %v", m["definitions"])
}
}
func TestNoCatalogueIsSaidAndNothingIsAsked(t *testing.T) {
defs := (&Catalogue{}).Read(context.Background(), NamesOf("", ""))
if defs.NotAsked == "" || defs.Read != 0 {
t.Errorf("%+v", defs)
}
}
@@ -24,7 +24,7 @@ func main() {
UID: os.Getuid(),
}
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools.
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask}, &Catalogue{Ask: stdio.Ask})); err != nil {
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask})); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -65,7 +65,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
for i := range recs.References {
fill(&recs.References[i])
}
got := stateByName(Classify(v, recs, nil))
got := stateByName(Classify(v, recs))
up := "upstream/docker.io/library/golang "
for name, want := range map[string]string{
up + d("p-amd"): StateEligible, // only the eligible index names it
@@ -92,7 +92,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
if len(mirrors) != 2 || !got["route-proxy/on-go_base "+d("i-pre")].Mirror || !got["route-proxy/on-go_base "+d("p-lost")].Mirror {
t.Errorf("offered as unrecorded copies: %v", mirrors)
}
um := UnrecordedMirrors(v, Classify(v, recs, nil))
um := UnrecordedMirrors(v, Classify(v, recs))
if len(um) != 2 || um[0].Reference != "artifact-store://route-proxy/on-go_base@"+d("i-pre") || um[0].OnlyTheir != 1+1+1000 {
t.Errorf("unrecorded copies: %+v", um)
}
@@ -104,7 +104,7 @@ func TestLettingAnIndexGoTakesItsOwnPlatformsAndSaysWhatStaysAndNamesOne(t *test
for i := range recs.References {
fill(&recs.References[i])
}
entries := Classify(v, recs, nil)
entries := Classify(v, recs)
old := "artifact-store://upstream/docker.io/library/golang@" + d("i-old")
gone := LetGoKeys(v, []string{old}, Spared(v, entries))
up := "upstream/docker.io/library/golang"
@@ -215,7 +215,7 @@ func TestTheMarkIsTheCollectorsIncludingAnIndexAndSharedBlobs(t *testing.T) {
func TestEveryStateIsGiven(t *testing.T) {
v := view(t, world())
got := map[string]string{}
for _, e := range Classify(v, withFill(records()), nil) {
for _, e := range Classify(v, withFill(records())) {
got[e.Repository+" "+e.Digest] = e.State
}
want := map[string]string{
@@ -239,7 +239,7 @@ func TestALetGoReferenceStillPresentIsSaidSo(t *testing.T) {
v := view(t, world())
r := records()
r.References = append(r.References, Record{Reference: "artifact-store://app/server@" + d("m-stranger"), State: "collected"})
for _, e := range Classify(v, withFill(r), nil) {
for _, e := range Classify(v, withFill(r)) {
if e.Digest == d("m-stranger") && e.State != StateCollectedPresent {
t.Errorf("state %s", e.State)
}
@@ -270,7 +270,7 @@ func TestBytesFreedCountASharedBlobAsKept(t *testing.T) {
func TestOnlyTheirBytesLeaveOutWhatAnotherStateMarks(t *testing.T) {
v := view(t, world())
entries := Classify(v, withFill(records()), nil)
entries := Classify(v, withFill(records()))
sums := Summarise(v, entries, entries)
el := sums[StateEligible]
if el.OnlyTheirBytes != 500+v.L.Blobs[d("m-old")] {
@@ -284,11 +284,6 @@ type asked struct {
calls []string
args []map[string]any
fail map[string]bool
// definitions are the catalogue's modules and their manifests; failing, the modules whose
// definition cannot be read; noCatalogue, the catalogue not answering at all.
definitions map[string]string
failing map[string]bool
noCatalogue bool
}
func (a *asked) ask(key string, body any) (json.RawMessage, error) {
@@ -304,31 +299,6 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
switch key {
case "seat:mesh-controller.artifacts":
answer = records()
case CatalogueModules:
if a.noCatalogue {
return nil, fmt.Errorf("no responders")
}
ms := []map[string]any{}
for name := range a.definitions {
ms = append(ms, map[string]any{"module": name, "commit": "c-" + name, "path": "modules/" + name})
}
answer = map[string]any{"modules": ms}
case CatalogueModule:
name, _ := args["module"].(string)
if a.failing[name] {
return nil, fmt.Errorf("no answer")
}
if args["commit"] != "c-"+name {
return nil, fmt.Errorf("asked %s at %v", name, args["commit"])
}
manifest, has := a.definitions[name]
if !has {
answer = map[string]any{"error": "the catalogue holds no " + name}
break
}
// The catalogue's tool answers through the runtime as the protocol's content list.
raw, _ := json.Marshal(map[string]any{"module": map[string]any{"module": name, "manifest": json.RawMessage(manifest)}})
return json.Marshal(map[string]any{"content": []map[string]any{{"type": "text", "text": string(raw)}}})
case "seat:mesh-controller.collect":
if args["confirm"] == "true" {
answer = CollectAnswer{LetGo: []string{"artifact-store://app/server@" + d("m-old")}}
@@ -345,7 +315,7 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
for _, tl := range Tools(s, Controller{Ask: a.ask}) {
if tl.Name == name {
return tl.Run
}
@@ -416,7 +386,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
t.Fatal(err)
}
var served []string
for _, tl := range Tools(&Store{}, Controller{}, &Catalogue{}) {
for _, tl := range Tools(&Store{}, Controller{}) {
if !strings.HasPrefix(tl.Name, "artifact_store_") || tl.Description == "" || tl.Run == nil {
t.Errorf("tool %q", tl.Name)
}
@@ -428,7 +398,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
if !reflect.DeepEqual(served, listed) {
t.Fatalf("served %v, manifest %v", served, listed)
}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect", CatalogueModules, CatalogueModule}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect"}
if !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes %v", m.Invokes)
}
@@ -13,7 +13,7 @@ import (
// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect
// only reads too, unless it is a real run, and then it asks the controller, which decides and deletes.
func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
func Tools(s *Store, c Controller) []stdio.Tool {
ctx := context.Background()
repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (<module>/<artifact>)"}
return []stdio.Tool{
@@ -53,9 +53,7 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
{
Name: "artifact_store_references",
Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " +
"names it, or one of the five most recent builds of a module the mesh holds), named by a definition (no " +
"record names it, and a definition in the catalogue pins it by digest, under the store's current name or " +
"an older one: the module is said), the holder of a kept archive, " +
"names it, or one of the five most recent builds of a module the mesh holds), the holder of a kept archive, " +
"eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " +
"a named document the controller keeps under a tag, or unrecorded — no record names it. A platform manifest " +
"takes the state of the recorded index naming it. Counted and sized by " +
@@ -69,19 +67,11 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
if err != nil {
return nil, err
}
// The definitions are read while the store is: each bounded below the runtime's call limit.
read := make(chan *Definitions, 1)
go func() {
older, why := OlderNames()
defs := cat.Read(ctx, NamesOf(recs.Store, older))
defs.NamesNotRead = why
read <- defs
}()
v, err := s.View(ctx)
if err != nil {
return nil, err
}
return References(v, recs, <-read, repo)
return References(v, recs, repo)
},
},
{
@@ -126,7 +116,7 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
if err != nil {
return map[string]any{"controller": answer, "bytes_not_worked_out": "the records could not be read: " + err.Error()}, nil
}
return Collected(v, answer, dry, Classify(v, recs, nil)), nil
return Collected(v, answer, dry, Classify(v, recs)), nil
},
},
}
@@ -235,11 +225,11 @@ func Usage(v *View, top int) map[string]any {
}
// References answers artifact_store_references.
func References(v *View, recs *Records, defs *Definitions, only string) (map[string]any, error) {
func References(v *View, recs *Records, only string) (map[string]any, error) {
if only != "" && v.L.Repos[only] == nil {
return nil, fmt.Errorf("the store holds no repository %q", only)
}
entries := Classify(v, recs, defs)
entries := Classify(v, recs)
byRepo := map[string][]Entry{}
for _, e := range entries {
byRepo[e.Repository] = append(byRepo[e.Repository], e)
@@ -253,7 +243,7 @@ func References(v *View, recs *Records, defs *Definitions, only string) (map[str
}
whole := Summarise(v, entries, entries)
said := []string{}
for _, state := range []string{StateKept, StateNamedByDefinition, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
for _, state := range []string{StateKept, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
if s := whole[state]; s != nil {
said = append(said, fmt.Sprintf("%s: %d manifests marking %s, %s of it marked by nothing else", state, s.Manifests, human(s.Bytes), human(s.OnlyTheirBytes)))
}
@@ -286,57 +276,9 @@ func References(v *View, recs *Records, defs *Definitions, only string) (map[str
if u := unreadNote(v); u != nil {
out["unread"] = u
}
out["definitions"] = definitionsSaid(defs)
if warn := definitionsWarning(defs); warn != "" {
// Said first: what it leaves out is a manifest a machine may run, read as unrecorded.
out["said"] = append([]string{warn}, said...)
}
return out, nil
}
// definitionsSaid is how the definitions were read: how many, under which names, and what was not read.
func definitionsSaid(defs *Definitions) map[string]any {
if defs == nil {
return map[string]any{"read": 0, "not_asked": "the definitions were not read"}
}
var current, older []string
for _, n := range defs.Names.Current {
current = append(current, shownName(n))
}
for _, n := range defs.Names.Older {
older = append(older, shownName(n))
}
out := map[string]any{"read": defs.Read, "store_names": nonNil(current), "older_store_names": nonNil(older)}
if defs.NotAsked != "" {
out["not_asked"] = defs.NotAsked
}
if defs.NamesNotRead != "" {
out["older_store_names_not_read"] = defs.NamesNotRead
}
if len(defs.Unread) > 0 {
out["unread"] = defs.Unread
}
return out
}
// definitionsWarning says, when any definition was not read, that a manifest it pins reads as unrecorded.
func definitionsWarning(defs *Definitions) string {
switch {
case defs == nil:
return ""
case defs.NotAsked != "":
return "the definitions could not be read (" + defs.NotAsked + "), so a manifest a definition pins " +
"and no record names reads as unrecorded here: ask again before reading unrecorded as unused"
case defs.NamesNotRead != "":
return defs.NamesNotRead + ", so a manifest a definition pins under an older name of the store and no " +
"record names reads as unrecorded here"
case len(defs.Unread) > 0:
return fmt.Sprintf("%d definitions could not be read (listed in definitions.unread), so a manifest one "+
"of them pins and no record names reads as unrecorded here: ask again — what was read is kept", len(defs.Unread))
}
return ""
}
// Collected answers artifact_store_collect: the controller's answer, and what the store's collector frees.
func Collected(v *View, a *CollectAnswer, dry bool, entries []Entry) map[string]any {
refs := a.LetGo
+2 -28
View File
@@ -3,9 +3,7 @@
"version": "1",
"invokes": [
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect",
"mesh-catalog.catalog_modules",
"mesh-catalog.catalog_module"
"seat:mesh-controller.collect"
],
"tools": [
"artifact_store_repositories",
@@ -13,29 +11,6 @@
"artifact_store_references",
"artifact_store_collect"
],
"settings": {
"older-store-names": {
"kind": "preference",
"default": "none",
"why": "the names the artifact store had before, as a definition may still pin an image under them: host names or host:port, separated by spaces; none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323)"
}
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "store-names",
"type": "file",
"path": "${dir:state}/store-names",
"mode": "0600",
"trusted": false,
"content": "# The names the artifact store had before, from the setting older-store-names, written by the mesh.\n# artifact_store_references reads every definition under these too (novox/hq issue 323).\n# Replaced on every push; change the setting, never this file.\nolder=${setting:older-store-names}\n"
}
],
"build": {
"artifacts": [
{
@@ -49,8 +24,7 @@
"artifact-store-tools"
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry",
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names"
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry"
}
}
]
+4 -75
View File
@@ -18,14 +18,13 @@ issue 345.
| Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole |
| the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole |
| Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole |
| the module's defaults (`install/defaults-pref/mesh-defaults.js`) and the spelling dictionaries' default (`vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its two files |
| the spelling dictionaries' default (`install/defaults-pref/vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its one file |
| the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole |
| the start | `/usr/local/bin/betterbird` | owned, whole |
| the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole |
| the module's directory, with its `user.js` (setting `prefs`), `accounts.json` (setting `accounts`) and the record of adopted and created accounts | `~/.config/betterbird/` | owned, whole |
| the profile's `user.js`, placed by the tools | the profile | the module's, or never touched |
| the account lists in the profile's `prefs.js` | the profile | written into: the module's keys only |
| the profile's own values of the preferences the module manages, in `prefs.js` | the profile | cleared only by `betterbird_config_apply` with apply, Betterbird quit |
| the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) |
**The community package `betterbird-bin` is declared absent** (ADR 0180), before anything is placed in
@@ -57,15 +56,7 @@ honours) and installs Owl into every profile from `file:///opt/betterbird/owl/ow
`force_installed`, with `updates_disabled`. A profile that already runs another Owl keeps it: the policy
installs the pinned one where Owl is missing. A newer Owl in the profile (g14 runs 1.5.7) is said as a
fact by `betterbird_status`; Owl installed by policy cannot be removed in Betterbird, so the remedy is
moving the pin. An older one is a finding of `betterbird_check`, with the same remedy.
**Betterbird never asks to be the default client.** The policy locks `mail.shell.checkDefaultClient` off
(`Preferences`, `Status: locked`): Betterbird's own default (`all-thunderbird.js`) is to ask at every start,
with a window to make it the default for mail, news, feeds and calendar files, while that preference is true
(`messenger.js`). Betterbird 153 has no policy of its own for it (its `policies-schema.json`), and its
`Preferences` policy accepts the `mail.` prefix. A lock, not a default: the window's own "always check" box
writes the preference into the profile, which wins over a default; a lock wins over the profile and greys the
box out in the settings. `betterbird_check` says when Betterbird would ask (novox/hq issue 345). **Owl's licence stays the person's**: the mesh holds no licence, key or payment (ADR 0316 §9).
moving the pin. An older one is a finding of `betterbird_check`, with the same remedy. **Owl's licence stays the person's**: the mesh holds no licence, key or payment (ADR 0316 §9).
**No start of its own.** Betterbird is started by the operator from the launcher. The start
(`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the
@@ -74,65 +65,6 @@ starts Betterbird either way. `betterbird_restart` starts it as the transient us
`mesh-betterbird`, so its output then goes to the journal.
## The configuration this module ships
Betterbird's configuration comes in three layers (novox/hq issue 345). The operator approved the set on
2026-10-12. It is the laptop's setup, with the start page locked off, reminders missed while asleep shown at the
next start, and chat off.
| layer | file | who changes it | holds |
|---|---|---|---|
| **locked** | `/opt/betterbird/distribution/policies.json` (`install/distribution/`) | only a reviewed catalogue change | what must never differ on a node; Betterbird's settings show it greyed out |
| **default** | `/opt/betterbird/defaults/pref/mesh-defaults.js` (`install/defaults-pref/`) | a catalogue change; or the operator in Betterbird's settings, where the change lands in the profile's `prefs.js` | the operator's setup: what a fresh node gets without anyone touching it |
| **node** | the setting `prefs`, written to `~/.config/betterbird/user.js` and placed in the profile's `user.js` | the operator, per node, through the settings layers | a node's exceptions, such as a font size for a larger screen |
A locked value wins over everything. A node layer (`user.js`) wins over the profile's `prefs.js`, and
`prefs.js` wins over a default.
**Locked** (the policy): Betterbird's updater off (`DisableAppUpdate`); telemetry off (`DisableTelemetry`);
add-ons change only through the mesh's pins (`ExtensionUpdate` false); and three preferences locked off:
`mail.shell.checkDefaultClient` (the ask to be the default client), `mail.inappnotifications.enabled` (the
vendor's in-app notices), and `mailnews.start_page.enabled` (the start page, which otherwise loads from the
vendor at every start). The `Preferences` policy accepts only some prefixes. `privacy.` and `toolkit.` are not
among them, so such a preference can only be a default.
**Default** (`mesh-defaults.js`, each line commented with why):
- display: compact density (`mail.uidensity` 0), 14-point interface text, and message bodies not darkened;
- layout: the vertical view with multi-line rows, the list as cards, threads grouped;
- no confirmation on Shift+Delete;
- new mail: no pop-up, no sound, no tray icon, and alerts through the desktop's notifier if a node turns them on;
- calendar: the day from 09:00 to 20:00, deleting without asking, the node's time zone, reminders shown, and
reminders missed while asleep shown at the next start;
- chat off;
- remote content blocked, and Do Not Track sent.
A line `// keeps Betterbird's default: <name>` names a preference the module manages without a value of its own.
For these the operator's setup is Betterbird's own default: the theme, system colours, always-underlined
links, manual spam marking, and the composer's attachment pane.
**Drift.** A default does not beat a value the profile saved in `prefs.js`. That value may be a choice made in
the settings window, or one the profile already held. `betterbird_config` shows each managed preference through
the layers, and flags **drift**: a saved value that beats the module's default without a node layer saying so.
`betterbird_check` reports it. There are two remedies:
- `betterbird_config_apply` with `apply` takes the drifting values out of `prefs.js`, so the module's value
holds at the next start. It acts only while Betterbird is quit (no main process, and no `lock` in the profile),
because Betterbird rewrites `prefs.js` when it quits, and it says what it cleared. Without `apply` it lists
what it would clear.
- **The theme is the exception.** Betterbird keeps the active theme in its add-on database (`extensions.json`),
and `extensions.activeThemeID` only mirrors it. So the tools read the theme from the add-on database, and the
module's value is `default-theme@mozilla.org`. Another theme is drift whose remedy is to enable the default
theme in the add-on manager. `betterbird_config_apply` never clears it, and `keep` refuses it.
**Never an account.** A name the tools withhold (a credential's, an account's, a server's), or an account's own
preference, is never managed, even when the defaults file lists it. `betterbird_config` leaves it out and names it
in `refused`, `betterbird_config_apply` never clears it, and `betterbird_check` says to take it out of the file.
- `betterbird_config` with `keep` proposes, through the controller's settings verb, a node layer of `prefs` that
holds the profile's value for this machine. The operator approves it on a channel that proves who answers.
The tool never writes `user.js` itself. The setting is one string, so the layer carries what `prefs` says on
this machine today, with the kept values added.
## The user.js: how a preference is declared
Betterbird reads `user.js` in its profile when it starts, and a preference set there wins over the
@@ -223,13 +155,11 @@ and key.
| tool | does |
|---|---|
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where (the mesh's release, or the community package before the switch), its version and build beside the newest published build (Betterbird's own `versions.txt`); the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect); the declared accounts by count and, per place, adopted, created or refused with their keys; the add-ons in the profile (ids, versions); the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (the policy's lock, user.js, prefs.js or the default), and the default itself with the file that gives it. Betterbird's own defaults are read from `omni.ja` (`greprefs.js`, `defaults/pref/*.js`) and the installation's `defaults/pref`, the module's from its files there. Of several files, the one first by name holds, as Gecko reads them. A preference compiled into Betterbird is said as built in. Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, and of `mailnews.*` only a name that holds a server, account, identity or tag); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked |
| `betterbird_config` (r; a with `keep`) | each managed preference: its lock, the module's default (or Betterbird's, where the module keeps that), the node layer, the profile's saved value, the value in effect, and drift; with `keep`, a node layer holding the profile's values proposed for this machine |
| `betterbird_config_apply` (r; a with `apply`) | the drifting values listed; with `apply`, while Betterbird is quit, taken out of `prefs.js`, each said with the value it had and the one that now holds |
| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (user.js, prefs.js or the default). Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, …); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked |
| `betterbird_user_js` (r; a with `apply`) | the declared user.js against the profile's; `apply` places it |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked (the value after `password=`, `token:` and the like, and the token after `Bearer`/`Basic`). A Betterbird the module did not start prints elsewhere, and the answer says where |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js where allowed, starts it as `mesh-betterbird`. **Refused while a composer window is open** (X instance `Msgcompose`, whatever its class), because the unsent message would be lost, unless `discard_unsent` is true, which only the operator's yes justifies. It answers within the runtime's 30 s: a start not yet seen after 4 s is said in `note`, and `betterbird_status` shows it |
| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed, and the module's defaults with it; no managed preference drifting from them; no ask at the start to be the default mail client; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read |
| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts by count and kind (declarable, Owl's, others); with `propose`, the setting `accounts` built from them and proposed to the operator (ADR 0317 §9) |
| `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) |
@@ -295,7 +225,6 @@ outside the mesh. With this version, a push to a node (each waits for the operat
| `/usr/local` | the start, the launcher entry, the tools' program |
| `~/.config/betterbird/` | `user.js`, `accounts.json` (no account until the setting `accounts` is approved) |
| the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` |
| the configuration | the policy's locks at Betterbird's next start; the module's defaults wherever the profile saved no value of its own; a saved value is drift, said by `betterbird_check`, until `betterbird_config_apply` or a node layer |
## Relies on
@@ -263,15 +263,11 @@ func readPrefs(raw []byte) map[string]string {
var (
passwordSecret = regexp.MustCompile(`(?i)(password|passwd|secret|token|oauth|auth|cookie|credential|signon|apikey|api_key|privatekey|private_key|\.key$)`)
accountPrefix = []string{"mail.server.", "mail.identity.", "mail.smtpserver.", "mail.smtp.", "mail.account.",
"mail.accountmanager.", "ldap_2.", "extensions.owl.", "calendar.registry.", "mail.root.",
"mail.accountmanager.", "ldap_2.", "extensions.owl.", "calendar.registry.", "mail.root.", "mailnews.",
"services.sync.", "mail.displayname", "mail.default_", "mail.last_msg", "messenger.account.", "mail.cloud_files."}
// allowedExtensions are the only extensions.* preferences answered: the add-ons' own preferences can
// hold anything, and a theme's id is what a window's look depends on.
allowedExtensions = map[string]bool{"extensions.activeThemeID": true}
// mailnewsAccount is a mailnews.* name that holds a server, an account, an identity or the operator's own
// tags: only those are withheld of mailnews.*, so a setting such as mailnews.start_page.enabled or
// mailnews.default_view_flags is answered (novox/hq issue 345).
mailnewsAccount = regexp.MustCompile(`(?i)^mailnews\..*(server|account|ident|smtp|ldap|imap|pop3|nntp|tags\.|email|addr)`)
)
// accountOwned are the preferences of an account and of the account lists: the setting accounts' alone, so
@@ -292,24 +288,16 @@ func withheld(name string) string {
return "it names an account, a server, a calendar or a person"
}
}
if mailnewsAccount.MatchString(name) {
return "it names an account, a server, a calendar or a person"
}
return ""
}
// Pref is one preference as the profile holds it.
type Pref struct {
Name string `json:"name"`
// Value is what Betterbird uses: the policy's lock, else user.js's at the start, else prefs.js's, else the
// default, when a defaults file gives it.
Value string `json:"value,omitempty"`
From string `json:"from,omitempty"`
// Default is the default Betterbird starts with, and the file that gives it: Betterbird's own (omni.ja), or
// the module's (novox/hq issue 345). Absent for a preference compiled into Betterbird.
Default string `json:"default,omitempty"`
DefaultFrom string `json:"default_from,omitempty"`
Withheld string `json:"withheld,omitempty"`
// Value is what Betterbird uses: user.js's at the start, else prefs.js's. Absent: the default.
Value string `json:"value,omitempty"`
From string `json:"from,omitempty"`
Withheld string `json:"withheld,omitempty"`
}
// mostPrefs bounds how many preferences one answer carries.
@@ -331,9 +319,7 @@ func (m *Machine) Prefs(names []string) (map[string]any, error) {
}
prefsJS, _ := readBounded(filepath.Join(p.path, "prefs.js"))
userJS, _ := readBounded(filepath.Join(p.path, "user.js"))
fromPrefs, fromUser := readPrefs(prefsJS), statementsOf(userJS)
own, ownErr := m.defaults(true)
locked := m.locks()
fromPrefs, fromUser := readPrefs(prefsJS), readPrefs(userJS)
all := map[string]bool{}
for n := range fromPrefs {
all[n] = true
@@ -368,40 +354,25 @@ func (m *Machine) Prefs(names []string) (map[string]any, error) {
break
}
pr := Pref{Name: n}
d, hasDefault := own[n]
v, from := fromPrefs[n], "prefs.js"
if l, ok := locked[n]; ok {
v, from = l, "locked by the module's policy"
} else if u, ok := fromUser[n]; ok {
if u, ok := fromUser[n]; ok {
v, from = u, "user.js (wins at the start)"
} else if _, ok := fromPrefs[n]; !ok {
switch {
case hasDefault:
v, from = d.Value, "not set: "+d.From
default:
v, from = "", "not set: Betterbird's built-in default (in no defaults file)"
}
from = "not set: Betterbird's default"
v = ""
}
pr.From = from
if hasDefault {
pr.DefaultFrom = d.From
}
if why := withheld(n); why != "" {
pr.Withheld = why
} else if allowedExtensions[n] {
pr.Value = v
pr.Default = d.Value
} else {
pr.Value = m.maskValue(v)
pr.Default = m.maskValue(d.Value)
}
out = append(out, pr)
}
}
answer := map[string]any{"profile": p, "prefs": out}
if ownErr != nil {
answer["defaults_problem"] = ownErr.Error()
}
if cut {
answer["cut"] = fmt.Sprintf("the first %d; ask for a narrower prefix", mostPrefs)
}
@@ -878,8 +849,7 @@ type CheckAnswer struct {
}
// Check verifies what the module promises: Betterbird installed, one running in the session, its
// profile found, the module's user.js declared, valid, placed and in effect, its output read, and no ask at its
// start to be the default mail client (novox/hq issue 345).
// profile found, the module's user.js declared, valid, placed and in effect, and its output read.
func (m *Machine) Check() (CheckAnswer, error) {
a := CheckAnswer{Findings: []Finding{}}
add := func(what, do string) { a.Findings = append(a.Findings, Finding{what, do}) }
@@ -905,35 +875,6 @@ func (m *Machine) Check() (CheckAnswer, error) {
if in.From == "the mesh" && !strings.HasPrefix(in.Policy, "placed:") {
add("the enterprise policy is "+in.Policy, "push the module to this node")
}
if in.From == "the mesh" && !exists(m.path(meshDefaults)) {
add("the module's defaults are not placed ("+meshDefaults+")", "push the module to this node")
}
if c, _, err := m.config(); err == nil {
var cleared []string
for _, cp := range c.Prefs {
switch {
case cp.Drift == "":
case cp.Remedy != "":
add(cp.Name+" drifts from the module's: "+cp.Drift, cp.Remedy)
default:
cleared = append(cleared, cp.Name)
}
}
if len(cleared) > 0 {
add(fmt.Sprintf("%d managed preference(s) drift from the module's defaults: the profile saved its own value and no "+
"node layer says so (%s)", len(cleared), strings.Join(cleared, ", ")),
"betterbird_config shows each; to keep a value on this node, betterbird_config with keep proposes a node "+
"layer; otherwise quit Betterbird and call betterbird_config_apply with apply")
}
if len(c.Refused) > 0 {
add(fmt.Sprintf("the module's defaults list %d preference(s) the tools never manage (%s)", len(c.Refused), strings.Join(c.Refused, "; ")),
"take them out of install/defaults-pref/mesh-defaults.js in a reviewed catalogue change: an account is declared in the setting accounts")
}
}
if why := m.defaultClientAsk(profilePath); why != "" && in.From != "not installed" {
add("Betterbird asks at its start to be the default mail client: "+why,
"push the module to this node: its policy locks "+defaultClientPref+" off (novox/hq issue 345)")
}
switch {
case in.From != "the mesh":
case in.Owl == nil || in.Owl.Pinned == "":
@@ -1,652 +0,0 @@
package main
// The configuration the module ships (novox/hq issue 345), in three layers:
//
// - LOCKED: the enterprise policy (/opt/betterbird/distribution/policies.json). Its Preferences entries with
// Status locked win over everything, and Betterbird's settings show them greyed out.
// - DEFAULT: the module's defaults file (/opt/betterbird/defaults/pref/mesh-defaults.js), the operator's
// setup. Betterbird reads it as a default, so a value the profile saved in prefs.js wins over it: that is
// drift, unless a node layer says so.
// - NODE: the setting prefs, which the node-engine writes to ~/.config/betterbird/user.js and the tools place
// in the profile's user.js, where it wins over prefs.js at every start.
//
// betterbird_config shows each managed preference through the layers and flags the drift; betterbird_config_apply
// clears a drifting value from prefs.js while Betterbird is not running; keep proposes a node layer that holds the
// profile's value for this node instead, through the controller's settings verb, never by writing user.js.
import (
"archive/zip"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
)
const (
// prefDir is the installation's own defaults directory; the module places its files there.
prefDir = installDir + "/defaults/pref"
// meshDefaults is the module's DEFAULT layer.
meshDefaults = prefDir + "/mesh-defaults.js"
// omniJa is Betterbird's packed application, holding its own default preferences.
omniJa = installDir + "/omni.ja"
)
// moduleFiles are the module's files in the installation's defaults directory: their values are the module's
// defaults, not Betterbird's.
var moduleFiles = map[string]bool{"mesh-defaults.js": true, "vendor-prefs.js": true}
var (
// defaultLine is one pref(…) or sticky_pref(…) line of a defaults file; a comment may follow it.
defaultLine = regexp.MustCompile(`^\s*(?:sticky_)?pref\(\s*"((?:[^"\\]|\\.)*)"\s*,\s*(.*?)\s*\)\s*;`)
// keepsDefault is a line of the module's defaults naming a preference it manages without a value.
keepsDefault = regexp.MustCompile(`^//\s*keeps Betterbird's default:\s*([A-Za-z0-9_.@{}\-]+)\s*$`)
)
// builtIn are the defaults compiled into Betterbird of the preferences the module keeps at Betterbird's default,
// which no defaults file gives: so a value saved equal to them is not drift.
var builtIn = map[string]string{
themePref: `"` + defaultTheme + `"`,
"browser.display.use_system_colors": "false",
"layout.css.always_underline_links": "false",
}
const (
themePref = "extensions.activeThemeID"
defaultTheme = "default-theme@mozilla.org"
// themeRemedy is how the theme comes back: Betterbird keeps the active theme in its add-on database
// (extensions.json), and the preference only mirrors it (XPIDatabase addonChanged), so clearing it changes
// nothing and betterbird_config_apply never touches it.
themeRemedy = "enable the default theme in Betterbird's add-on manager (Add-ons and Themes, Themes): the active " +
"theme is kept in the add-on database, and its preference only mirrors it, so clearing prefs.js would not change it"
)
// activeTheme is the theme the profile's add-on database marks active, or "" when it cannot be read.
func activeTheme(profile string) string {
raw, err := readBounded(filepath.Join(profile, "extensions.json"))
if err != nil {
return ""
}
var ext struct {
Addons []struct {
ID string `json:"id"`
Type string `json:"type"`
Active bool `json:"active"`
} `json:"addons"`
}
if json.Unmarshal(raw, &ext) != nil {
return ""
}
for _, a := range ext.Addons {
if a.Type == "theme" && a.Active {
return a.ID
}
}
return ""
}
// unmanageable says why a name in the module's defaults is never managed by these tools, or "": a credential's,
// an account's or a server's preference, which betterbird_config never answers and betterbird_config_apply never
// clears, whatever the defaults file lists.
func unmanageable(name string) string {
if why := withheld(name); why != "" {
return why
}
for _, p := range accountOwned {
if strings.HasPrefix(name, p) {
return "an account's preference, the setting accounts' alone"
}
}
return ""
}
// answer is a value as these tools answer it: masked, but for the theme's id.
func (m *Machine) answer(name, v string) string {
if allowedExtensions[name] {
return v
}
return m.maskValue(v)
}
// nodeName is the machine this runtime serves, as the mesh names it: a variable so a test answers.
var nodeName = func() string { return os.Getenv("MESH_NODE") }
// shipped is the module's DEFAULT layer as placed on the machine.
type shipped struct {
values map[string]string
// keeps are the managed names whose module value is Betterbird's own default.
keeps map[string]bool
// order is every managed name, in the file's order.
order []string
}
// parseShipped reads the module's defaults file.
func parseShipped(raw []byte) shipped {
s := shipped{values: map[string]string{}, keeps: map[string]bool{}}
for _, line := range strings.Split(string(raw), "\n") {
if mm := keepsDefault.FindStringSubmatch(strings.TrimSpace(line)); mm != nil {
if !s.keeps[mm[1]] {
s.keeps[mm[1]] = true
s.order = append(s.order, mm[1])
}
continue
}
if mm := defaultLine.FindStringSubmatch(line); mm != nil {
if _, seen := s.values[mm[1]]; !seen {
s.order = append(s.order, mm[1])
}
s.values[mm[1]] = mm[2]
}
}
return s
}
// shippedDefaults reads the module's defaults file from the installation.
func (m *Machine) shippedDefaults() (shipped, error) {
raw, err := readBounded(m.path(meshDefaults))
if err != nil {
return shipped{values: map[string]string{}, keeps: map[string]bool{}}, err
}
return parseShipped(raw), nil
}
// DefaultValue is a default preference and the file that gives it.
type DefaultValue struct {
Value string
From string
// Module says the module's files give it, not Betterbird's.
Module bool
}
// readDefaultsInto reads the pref(…) lines of the files of one place, in Gecko's order: it reads a place's
// files last-first by name (Preferences.cpp, pref_ReadDefaultPrefs), so of a name in several files the value
// of the file first by name holds. A later place overrides an earlier one.
func readDefaultsInto(out map[string]DefaultValue, files map[string][]byte, label func(string) (string, bool)) {
names := make([]string, 0, len(files))
for n := range files {
names = append(names, n)
}
sort.Strings(names)
place := map[string]DefaultValue{}
for i := len(names) - 1; i >= 0; i-- {
from, module := label(names[i])
for _, line := range strings.Split(string(files[names[i]]), "\n") {
if mm := defaultLine.FindStringSubmatch(line); mm != nil {
place[mm[1]] = DefaultValue{Value: mm[2], From: from, Module: module}
}
}
}
for k, v := range place {
out[k] = v
}
}
// mostDefaultsFile bounds one defaults file read from omni.ja.
const mostDefaultsFile = 4 << 20
// defaults reads every default preference Betterbird starts with: greprefs.js and defaults/pref/*.js in
// omni.ja, then the installation's defaults/pref/*.js, where the module's files are. A preference compiled into
// Betterbird (a static preference) is in no file, and is absent here. withModule false leaves the module's own
// files out, so what remains is Betterbird's. The error says what could not be read.
func (m *Machine) defaults(withModule bool) (map[string]DefaultValue, error) {
out := map[string]DefaultValue{}
var problems []string
if r, err := zip.OpenReader(m.path(omniJa)); err != nil {
problems = append(problems, omniJa+" is unreadable: "+err.Error())
} else {
gre, app := map[string][]byte{}, map[string][]byte{}
for _, f := range r.File {
switch {
case f.Name == "greprefs.js":
case strings.HasPrefix(f.Name, "defaults/pref/") && strings.HasSuffix(f.Name, ".js") && !strings.Contains(f.Name[len("defaults/pref/"):], "/"):
default:
continue
}
rc, err := f.Open()
if err != nil {
continue
}
b, _ := io.ReadAll(io.LimitReader(rc, mostDefaultsFile))
rc.Close()
if f.Name == "greprefs.js" {
gre[f.Name] = b
} else {
app[filepath.Base(f.Name)] = b
}
}
r.Close()
readDefaultsInto(out, gre, func(n string) (string, bool) { return "Betterbird's default (omni.ja " + n + ")", false })
readDefaultsInto(out, app, func(n string) (string, bool) {
return "Betterbird's default (omni.ja defaults/pref/" + n + ")", false
})
}
installed := map[string][]byte{}
entries, _ := os.ReadDir(m.path(prefDir))
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".js") || (moduleFiles[e.Name()] && !withModule) {
continue
}
if b, err := readBounded(m.path(filepath.Join(prefDir, e.Name()))); err == nil {
installed[e.Name()] = b
}
}
readDefaultsInto(out, installed, func(n string) (string, bool) {
if moduleFiles[n] {
return "the module's default (" + prefDir + "/" + n + ")", true
}
return "Betterbird's default (" + prefDir + "/" + n + ")", false
})
if len(problems) > 0 {
return out, errors.New(strings.Join(problems, "; "))
}
return out, nil
}
// policyPrefs are the Preferences entries of the placed policy: name to its value as a preference file writes
// it, and whether it is locked. An unreadable policy has none.
type policyPref struct {
Value string
Locked bool
}
func policyPrefsOf(raw []byte) map[string]policyPref {
var p struct {
Policies struct {
Preferences map[string]struct {
Value any `json:"Value"`
Status string `json:"Status"`
} `json:"Preferences"`
} `json:"policies"`
}
out := map[string]policyPref{}
if json.Unmarshal(raw, &p) != nil {
return out
}
for name, e := range p.Policies.Preferences {
out[name] = policyPref{Value: literal(e.Value), Locked: e.Status == "locked"}
}
return out
}
// locks are the preferences the placed policy locks: name to value.
func (m *Machine) locks() map[string]string {
raw, err := readBounded(m.path(policyFile))
if err != nil {
return map[string]string{}
}
out := map[string]string{}
for name, p := range policyPrefsOf(raw) {
if p.Locked {
out[name] = p.Value
}
}
return out
}
// literal is a JSON value as a preference file writes it.
func literal(v any) string {
switch x := v.(type) {
case bool:
return strconv.FormatBool(x)
case float64:
return strconv.FormatFloat(x, 'f', -1, 64)
case string:
return quoted(x)
case nil:
return ""
}
b, _ := json.Marshal(v)
return string(b)
}
// statementsOf are the user_pref(…) statements of a user.js, several to a line allowed: name to value.
func statementsOf(raw []byte) map[string]string {
out := map[string]string{}
for _, line := range strings.Split(string(raw), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
if mm := prefLine.FindStringSubmatch(st); mm != nil {
out[mm[1]] = mm[2]
}
}
}
return out
}
// ConfigPref is one managed preference through the layers.
type ConfigPref struct {
Name string `json:"name"`
// Layer is locked (the policy), default (the module's defaults file) or "Betterbird's default" (a managed
// preference whose module value is Betterbird's own).
Layer string `json:"layer"`
Locked string `json:"locked,omitempty"`
ModuleDefault string `json:"module_default,omitempty"`
BetterbirdDefault string `json:"betterbird_default,omitempty"`
// NodeLayer is what the setting prefs on this machine says (the module's user.js).
NodeLayer string `json:"node_layer,omitempty"`
// Profile is what the profile saved in its prefs.js.
Profile string `json:"profile,omitempty"`
Effective string `json:"effective"`
From string `json:"effective_from"`
// Drift says how the profile's own value beats the module's without a node layer saying so.
Drift string `json:"drift,omitempty"`
// Remedy is how drift that clearing prefs.js cannot fix is fixed (the theme).
Remedy string `json:"remedy,omitempty"`
// module is the value the module means: its default, or Betterbird's.
module string
}
// ConfigAnswer is what betterbird_config answers.
type ConfigAnswer struct {
Profile string `json:"profile"`
Defaults string `json:"module_defaults"`
Prefs []ConfigPref `json:"prefs"`
// Drifting are the names that drift.
Drifting []string `json:"drifting"`
// Refused are names the module's defaults list that the tools never manage: an account's or a credential's.
Refused []string `json:"refused,omitempty"`
// Proposed is the controller's answer to a node layer proposed with keep.
Proposed string `json:"proposed,omitempty"`
Note string `json:"note,omitempty"`
}
// config reads every managed preference through the layers: the policy's locks, the module's defaults, the
// node layer (the declared user.js), the profile's prefs.js and user.js.
func (m *Machine) config() (ConfigAnswer, Profile, error) {
a := ConfigAnswer{Prefs: []ConfigPref{}, Drifting: []string{}}
p, err := m.profile()
if err != nil {
return a, p, err
}
a.Profile = p.Dir
sh, serr := m.shippedDefaults()
if serr != nil {
a.Defaults = "not placed: " + meshDefaults + " is missing; the module's push places it"
} else {
a.Defaults = fmt.Sprintf("placed: %s, %d preference(s) with a value and %d kept at Betterbird's default",
meshDefaults, len(sh.values), len(sh.keeps))
}
locked := m.locks()
own, _ := m.defaults(false)
declaredRaw, _ := readBounded(m.home(declared))
node := statementsOf(declaredRaw)
prefsRaw, _ := readBounded(filepath.Join(p.path, "prefs.js"))
saved := readPrefs(prefsRaw)
userRaw, _ := readBounded(filepath.Join(p.path, "user.js"))
placed := statementsOf(userRaw)
var names []string
for n := range locked {
names = append(names, n)
}
sort.Strings(names)
for _, n := range sh.order {
if _, isLocked := locked[n]; !isLocked {
names = append(names, n)
}
}
theme := activeTheme(p.path)
for _, n := range names {
if why := unmanageable(n); why != "" {
a.Refused = append(a.Refused, n+": "+why)
continue
}
c := ConfigPref{Name: n, NodeLayer: node[n], Profile: saved[n]}
if n == themePref && theme != "" {
// The add-on database says which theme is active; the preference only mirrors it.
c.Profile = quoted(theme)
}
bb, hasBB := own[n]
if hasBB {
c.BetterbirdDefault = bb.Value
} else if v, ok := builtIn[n]; ok {
c.BetterbirdDefault = v
bb.From = "Betterbird's built-in default"
}
switch v, isLocked := locked[n]; {
case isLocked:
c.Layer, c.Locked, c.module = "locked", v, v
case sh.keeps[n]:
c.Layer = "Betterbird's default"
c.module = c.BetterbirdDefault
default:
c.Layer, c.ModuleDefault, c.module = "default", sh.values[n], sh.values[n]
}
switch {
case c.Locked != "":
c.Effective, c.From = c.Locked, "locked by the module's policy"
case placed[n] != "":
c.Effective, c.From = placed[n], "the profile's user.js (wins at the start)"
case c.Profile != "":
c.Effective, c.From = c.Profile, "the profile's prefs.js"
case c.ModuleDefault != "":
c.Effective, c.From = c.ModuleDefault, "the module's default"
case c.BetterbirdDefault != "":
c.Effective, c.From = c.BetterbirdDefault, bb.From
default:
c.From = "Betterbird's built-in default (in no defaults file)"
}
if why := drift(c); why != "" {
c.Drift = why
if n == themePref {
c.Remedy = themeRemedy
}
a.Drifting = append(a.Drifting, n)
}
c.NodeLayer, c.Profile, c.Effective = m.answer(n, c.NodeLayer), m.answer(n, c.Profile), m.answer(n, c.Effective)
a.Prefs = append(a.Prefs, c)
}
return a, p, nil
}
// drift says how the profile's saved value beats the module's, or "": a value in prefs.js that is not the
// module's, where no lock holds and no node layer names the preference.
func drift(c ConfigPref) string {
if c.Locked != "" || c.NodeLayer != "" || c.Profile == "" || c.Profile == c.module {
return ""
}
want := c.module
if want == "" {
want = "Betterbird's built-in default"
}
where := "the profile's prefs.js holds"
if c.Name == themePref {
where = "the profile's active theme is"
}
return fmt.Sprintf("%s %s, which beats the module's %s, and no node layer says so", where, c.Profile, want)
}
// Config is what betterbird_config answers; with keep it proposes a node layer of the setting prefs that holds
// the profile's own values of the named managed preferences on this machine (novox/hq issue 345). The module
// never writes user.js for it: the operator approves the layer, and the next push writes it.
func (m *Machine) Config(keep []string) (ConfigAnswer, error) {
a, p, err := m.config()
if err != nil {
return a, err
}
a.Note = "drift is a value the profile saved that beats the module's default without a node layer saying so: " +
"betterbird_config_apply clears it (Betterbird quit), or keep proposes a node layer that holds it for this machine"
if len(keep) == 0 {
return a, nil
}
by := map[string]ConfigPref{}
for _, c := range a.Prefs {
by[c.Name] = c
}
prefsRaw, _ := readBounded(filepath.Join(p.path, "prefs.js"))
saved := readPrefs(prefsRaw)
kept := map[string]string{}
for _, n := range keep {
c, managed := by[n]
switch {
case !managed:
return a, fmt.Errorf("nothing was proposed: %s is not a preference the module manages (betterbird_config lists them)", n)
case c.Locked != "":
return a, fmt.Errorf("nothing was proposed: %s is locked by the module's policy; a lock changes only in the catalogue", n)
case n == themePref:
return a, fmt.Errorf("nothing was proposed: a node layer cannot choose the theme; the add-on manager does (%s)", themeRemedy)
case saved[n] == "":
return a, fmt.Errorf("nothing was proposed: the profile saved no value of its own for %s", n)
}
kept[n] = saved[n]
}
declaredRaw, err := readBounded(m.home(declared))
if err != nil {
return a, fmt.Errorf("nothing was proposed: the module's user.js is not on this machine (~/%s)", declared)
}
node := nodeName()
if node == "" {
return a, errors.New("nothing was proposed: this runtime does not say which machine it serves (MESH_NODE)")
}
// The setting is one string, so a node layer replaces it whole: it carries what the setting says on this
// machine today, with the kept values in place of any the same names had.
var parts []string
for _, line := range strings.Split(string(declaredRaw), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
if mm := prefLine.FindStringSubmatch(st); mm != nil {
if _, replaced := kept[mm[1]]; !replaced {
parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), mm[2]))
}
}
}
}
for _, n := range keep {
if v, ok := kept[n]; ok {
parts = append(parts, fmt.Sprintf("user_pref(%s, %s);", quoted(n), v))
delete(kept, n)
}
}
value := strings.Join(parts, " ")
if err := validUserJS(mark + ")\n" + value + "\n"); err != nil {
return a, fmt.Errorf("nothing was proposed: the layer would be refused: %v", err)
}
body, _ := json.Marshal(map[string]string{"prefs": value})
raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "node": node,
"values": string(body), "propose": "true"})
if err != nil {
return a, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
a.Proposed = m.maskValue(said)
if !ok {
return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed)
}
a.Note = "proposed for " + node + ": the operator approves it on a channel that proves who answers; the next " +
"push writes it, and it is placed at Betterbird's next start from the launcher (or betterbird_restart). " +
"The layer carries the whole setting prefs as it stands on this machine, since a layer replaces it whole"
return a, nil
}
// Cleared is one value betterbird_config_apply took out of prefs.js.
type Cleared struct {
Name string `json:"name"`
Was string `json:"was"`
// Now is the value that holds once Betterbird starts again.
Now string `json:"now"`
}
// ConfigApplyAnswer is what betterbird_config_apply answers.
type ConfigApplyAnswer struct {
Profile string `json:"profile"`
Applied bool `json:"applied"`
Clears []Cleared `json:"clears"`
// NotCleared is drift that clearing prefs.js cannot fix, each with how it is fixed.
NotCleared []string `json:"not_cleared,omitempty"`
Note string `json:"note"`
}
// ConfigApply takes the drifting values out of the profile's prefs.js, so the module's value holds at the next
// start. Without apply it lists them. It acts only while Betterbird is not running, because Betterbird
// rewrites prefs.js when it quits. Every other line of prefs.js is kept as it was.
func (m *Machine) ConfigApply(apply bool) (ConfigApplyAnswer, error) {
c, p, err := m.config()
a := ConfigApplyAnswer{Profile: c.Profile, Clears: []Cleared{}}
if err != nil {
return a, err
}
if strings.HasPrefix(c.Defaults, "not placed") {
return a, fmt.Errorf("the module's defaults are %s: nothing to bring the profile back to", c.Defaults)
}
drifting := map[string]bool{}
for _, cp := range c.Prefs {
if cp.Drift != "" && cp.Remedy != "" {
a.NotCleared = append(a.NotCleared, cp.Name+": "+cp.Remedy)
continue
}
if cp.Drift != "" {
drifting[cp.Name] = true
now := m.answer(cp.Name, cp.module)
if now == "" {
now = "Betterbird's built-in default"
}
a.Clears = append(a.Clears, Cleared{Name: cp.Name, Was: cp.Profile, Now: now})
}
}
if !apply {
a.Note = "a dry run: these values would be taken out of prefs.js. Quit Betterbird, then call with apply"
return a, nil
}
if ps := m.mainProcs(); len(ps) > 0 {
return a, fmt.Errorf("Betterbird is running (pid %d): it rewrites prefs.js when it quits, so nothing was cleared; quit it, then apply again", ps[0].PID)
}
// The profile's own lock is Gecko's sign that a Betterbird holds it, whichever installation started it.
if _, err := os.Lstat(filepath.Join(p.path, "lock")); err == nil {
return a, errors.New("the profile is locked (its lock file is there): a Betterbird holds it, or one ended " +
"without quitting; nothing was cleared. Quit Betterbird (or start and quit it once), then apply again")
}
if len(drifting) == 0 {
a.Applied, a.Note = true, "nothing in prefs.js drifts: it was left as it was"
return a, nil
}
target := filepath.Join(p.path, "prefs.js")
raw, err := readBounded(target)
if err != nil {
return a, err
}
st, err := os.Stat(target)
if err != nil {
return a, err
}
lines := strings.SplitAfter(string(raw), "\n")
var kept strings.Builder
for _, l := range lines {
if mm := prefLine.FindStringSubmatch(strings.TrimRight(l, "\n")); mm != nil && drifting[mm[1]] {
continue
}
kept.WriteString(l)
}
tmp := target + ".mesh-new"
if err := writeSynced(tmp, []byte(kept.String()), st.Mode().Perm()); err != nil {
os.Remove(tmp)
return a, err
}
if err := os.Rename(tmp, target); err != nil {
os.Remove(tmp)
return a, err
}
a.Applied = true
a.Note = fmt.Sprintf("cleared %d value(s) from prefs.js; the module's values hold from Betterbird's next start", len(a.Clears))
return a, nil
}
// writeSynced writes a file and syncs it to disk before it is renamed into place, so a crash never leaves an
// empty prefs.js.
func writeSynced(path string, data []byte, mode os.FileMode) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, mode)
if err != nil {
return err
}
if _, err := f.Write(data); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
return f.Close()
}
@@ -1,520 +0,0 @@
package main
import (
"archive/zip"
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// The configuration the module ships (novox/hq issue 345): the policy's locks, the module's defaults file, and the
// tools that show the layers, flag drift, clear it and propose a node layer.
// shippedDefaultsWant is the DEFAULT layer the operator approved on 2026-10-12: the laptop's setup, with
// reminders missed while asleep shown at the next start and chat off.
var shippedDefaultsWant = map[string]string{
"mail.uidensity": "0",
"mail.uifontsize": "14",
"mail.dark-reader.enabled": "false",
"mail.pane_config.dynamic": "2",
"mail.pane_config.multiline_all": "true",
"mail.threadpane.listview": "0",
"mailnews.default_view_flags": "1",
"mail.warn_on_shift_delete": "false",
"mail.biff.show_alert": "false",
"mail.biff.play_sound": "false",
"mail.biff.show_tray_icon": "false",
"mail.biff.use_system_alert": "true",
"calendar.view.daystarthour": "9",
"calendar.view.dayendhour": "20",
"calendar.item.promptDelete": "false",
"calendar.timezone.useSystemTimezone": "true",
"calendar.alarms.show": "true",
"calendar.alarms.showmissed": "true",
"mail.chat.enabled": "false",
"mailnews.message_display.disable_remote_image": "true",
"privacy.donottrackheader.enabled": "true",
}
var shippedKeepsWant = []string{"extensions.activeThemeID", "browser.display.use_system_colors",
"layout.css.always_underline_links", "mail.spam.manualMark", "mail.compose.show_attachment_pane"}
func repoFile(t *testing.T, rel string) []byte {
t.Helper()
raw, err := os.ReadFile(filepath.Join("..", "..", rel))
if err != nil {
t.Fatal(err)
}
return raw
}
// withDefaults places the module's defaults file as the push does.
func (f *fake) withDefaults() {
f.t.Helper()
f.write(meshDefaults, string(repoFile(f.t, "install/defaults-pref/mesh-defaults.js")))
}
// omni places a packed application holding the given files.
func (f *fake) omni(files map[string]string) {
f.t.Helper()
full := filepath.Join(f.Root, omniJa)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
f.t.Fatal(err)
}
out, err := os.Create(full)
if err != nil {
f.t.Fatal(err)
}
w := zip.NewWriter(out)
for name, body := range files {
e, _ := w.Create(name)
e.Write([]byte(body))
}
w.Close()
out.Close()
}
// betterbirdsOwn is a small omni.ja: two files that contradict each other, as all-thunderbird.js and
// mailnews.js do in the pinned release.
func (f *fake) betterbirdsOwn() {
f.omni(map[string]string{
"greprefs.js": `pref("gfx.webrender.software", false);` + "\n",
"defaults/pref/all-thunderbird.js": `pref("offline.autoDetect", true);` + "\n" +
`pref("mail.threadpane.listview", 1);` + "\n" + `pref("mail.uidensity", 1);` + "\n" +
`pref("mail.shell.checkDefaultClient", true);` + "\n",
"defaults/pref/mailnews.js": `pref("offline.autoDetect", false);` + "\n" +
`pref("mail.biff.show_alert", true); // a comment after it` + "\n" + `pref("mail.spam.manualMark", false);` + "\n" +
`pref("mailnews.start_page.enabled", true);` + "\n",
"defaults/pref/calendar.js": `pref("calendar.alarms.showmissed", false);` + "\n",
"chrome/other.js": `pref("not.a.default", 1);` + "\n",
})
}
func stopBetterbird(t *testing.T, f *fake) {
t.Helper()
for _, pid := range []string{"4000", "4001"} {
if err := os.RemoveAll(filepath.Join(f.Root, "proc", pid)); err != nil {
t.Fatal(err)
}
}
}
func TestTheModuleShipsTheApprovedLocksAndDefaults(t *testing.T) {
m, _ := readManifest(t)
if a := artifact(t, m, "defaults"); a["from"] != "install/defaults-pref" {
t.Fatalf("the defaults file ships through the defaults artifact: %v", a)
}
if r := resource(t, m, "spelling"); r["path"] != prefDir || r["artifact"] != "defaults" {
t.Fatalf("placed in Betterbird's own defaults directory: %v", r)
}
raw := repoFile(t, "install/distribution/policies.json")
if why := policyProblem(raw); why != "" {
t.Fatal(why)
}
var p struct {
Policies map[string]json.RawMessage `json:"policies"`
}
if err := json.Unmarshal(raw, &p); err != nil {
t.Fatal(err)
}
for k, want := range map[string]string{"DisableAppUpdate": "true", "DisableTelemetry": "true", "ExtensionUpdate": "false"} {
if string(p.Policies[k]) != want {
t.Errorf("%s is %s, not %s", k, p.Policies[k], want)
}
}
prefs := policyPrefsOf(raw)
for _, n := range []string{"mail.shell.checkDefaultClient", "mail.inappnotifications.enabled", "mailnews.start_page.enabled"} {
if prefs[n] != (policyPref{Value: "false", Locked: true}) {
t.Errorf("%s is locked off: %+v", n, prefs[n])
}
}
// Betterbird's Preferences policy accepts only these prefixes (Policies.sys.mjs); privacy. and toolkit. are not.
allowed := []string{"accessibility.", "app.update.", "browser.", "calendar.", "chat.", "datareporting.policy.", "dom.",
"extensions.", "general.autoScroll", "general.smoothScroll", "geo.", "gfx.", "intl.", "layers.", "layout.", "mail.",
"mailnews.", "media.", "network.", "pdfjs.", "places.", "print.", "signon.", "spellchecker.", "ui.", "widget."}
for n := range prefs {
ok := false
for _, a := range allowed {
ok = ok || strings.HasPrefix(n, a)
}
if !ok {
t.Errorf("the policy cannot lock %s: its prefix is not one Betterbird accepts", n)
}
}
text := string(repoFile(t, "install/defaults-pref/mesh-defaults.js"))
sh := parseShipped([]byte(text))
if !reflect.DeepEqual(sh.values, shippedDefaultsWant) {
t.Errorf("the defaults are the approved ones:\n got %v\nwant %v", sh.values, shippedDefaultsWant)
}
var keeps []string
for _, n := range sh.order {
if sh.keeps[n] {
keeps = append(keeps, n)
}
}
if !reflect.DeepEqual(keeps, shippedKeepsWant) {
t.Errorf("the preferences kept at Betterbird's default: %v", keeps)
}
lines := strings.Split(text, "\n")
for i, l := range lines {
if defaultLine.MatchString(l) && (i == 0 || !strings.HasPrefix(lines[i-1], "// ")) {
t.Errorf("line %d has no comment saying why: %s", i+1, l)
}
if t2 := strings.TrimSpace(l); t2 != "" && !strings.HasPrefix(t2, "//") && !defaultLine.MatchString(l) {
t.Errorf("line %d is neither a comment nor a pref(…) line: %s", i+1, l)
}
}
for n := range sh.values {
if withheld(n) != "" || passwordSecret.MatchString(n) {
t.Errorf("%s is an account's or a credential's preference, never a default of the module", n)
}
if _, locked := prefs[n]; locked {
t.Errorf("%s is both locked and a default", n)
}
}
}
// T2: Betterbird's own defaults are values read from omni.ja, in Gecko's order, and the module's beside them.
func TestPrefsSayTheDefaultAndTheFileThatGivesIt(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
f.betterbirdsOwn()
links(t, nil)
got, err := f.Prefs([]string{"offline.autoDetect", "mail.threadpane.listview", "mail.uidensity", "gfx.webrender.software",
"mailnews.start_page.enabled", "layout.css.always_underline_links", "not.a.default"})
if err != nil {
t.Fatal(err)
}
by := map[string]Pref{}
for _, p := range got["prefs"].([]Pref) {
by[p.Name] = p
}
if p := by["offline.autoDetect"]; p.Value != "true" || p.Default != "true" || !strings.Contains(p.DefaultFrom, "all-thunderbird.js") {
t.Errorf("of two files, the one first by name holds, as Gecko reads them: %+v", p)
}
if p := by["mail.threadpane.listview"]; p.Value != "0" || !strings.Contains(p.From, "the module's default") {
t.Errorf("the module's default is over Betterbird's: %+v", p)
}
if p := by["gfx.webrender.software"]; p.Value != "false" || !strings.Contains(p.From, "greprefs.js") {
t.Errorf("%+v", p)
}
if p := by["mailnews.start_page.enabled"]; p.Value != "false" || p.From != "locked by the module's policy" || p.Default != "true" {
t.Errorf("a locked preference says its lock and Betterbird's default under it: %+v", p)
}
if p := by["layout.css.always_underline_links"]; p.Value != "" || !strings.Contains(p.From, "built-in") {
t.Errorf("a preference in no file is said as such: %+v", p)
}
if p := by["not.a.default"]; p.Default != "" {
t.Errorf("only defaults/pref and greprefs.js are defaults: %+v", p)
}
}
// T4: of mailnews.* only what names an account, a server or the operator's tags is withheld.
func TestMailnewsIsWithheldOnlyWhereItNamesAnAccount(t *testing.T) {
for name, held := range map[string]bool{
"mailnews.start_page.enabled": false,
"mailnews.default_view_flags": false,
"mailnews.message_display.disable_remote_image": false,
"mailnews.tags.$label1.tag": true,
"mailnews.smtp.loglevel": true,
"mailnews.reply_to_self_check_all_ident": true,
"mailnews.auto_config.fetchFromISP.sendEmailAddress": true,
"mailnews.oauth.useSchemeRedirect": true,
"mail.server.server1.hostname": true,
} {
if got := withheld(name) != ""; got != held {
t.Errorf("%s: withheld %v, want %v", name, got, held)
}
}
}
// T1: every managed preference through the layers, and drift where the profile's own value beats the module's
// default without a node layer saying so.
func TestTheConfigShowsTheLayersAndTheDrift(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
f.betterbirdsOwn()
links(t, nil)
f.declare(`user_pref("mail.uifontsize", 16);`)
f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+
`user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");`+"\n"+
`user_pref("mail.uifontsize", 16);`+"\n"+
`user_pref("mail.uidensity", 0);`+"\n"+
`user_pref("mail.biff.show_alert", true);`+"\n"+
`user_pref("mail.spam.manualMark", false);`+"\n"+
`user_pref("mailnews.start_page.enabled", true);`+"\n")
c, err := f.Config(nil)
if err != nil {
t.Fatal(err)
}
by := map[string]ConfigPref{}
for _, p := range c.Prefs {
by[p.Name] = p
}
if len(c.Prefs) != 3+len(shippedDefaultsWant)+len(shippedKeepsWant) || !strings.HasPrefix(c.Defaults, "placed") {
t.Fatalf("every lock, default and kept default: %d, %s", len(c.Prefs), c.Defaults)
}
if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID", "mail.biff.show_alert"}) {
t.Fatalf("drift: %v", c.Drifting)
}
if p := by["mail.biff.show_alert"]; p.Layer != "default" || p.ModuleDefault != "false" || p.BetterbirdDefault != "true" ||
p.Profile != "true" || p.Effective != "true" || p.From != "the profile's prefs.js" || !strings.Contains(p.Drift, "beats the module's false") {
t.Errorf("%+v", p)
}
if p := by["extensions.activeThemeID"]; p.Layer != "Betterbird's default" || !strings.Contains(p.Drift, `"default-theme@mozilla.org"`) ||
!strings.Contains(p.Remedy, "add-on manager") || p.Profile != `"thunderbird-compact-dark@mozilla.org"` {
t.Errorf("another theme is drift whose remedy is the add-on manager, its id answered whole: %+v", p)
}
if p := by["mail.spam.manualMark"]; p.Drift != "" || p.BetterbirdDefault != "false" {
t.Errorf("a saved value equal to Betterbird's default is not drift: %+v", p)
}
if p := by["mail.uifontsize"]; p.Drift != "" || p.NodeLayer != "16" {
t.Errorf("a node layer that says so is not drift: %+v", p)
}
if p := by["mailnews.start_page.enabled"]; p.Layer != "locked" || p.Effective != "false" || p.Drift != "" {
t.Errorf("a lock wins over the profile: %+v", p)
}
if p := by["calendar.alarms.showmissed"]; p.Effective != "true" || p.From != "the module's default" {
t.Errorf("%+v", p)
}
// T5: the check says the drift, with both remedies.
ch, err := f.Check()
if err != nil {
t.Fatal(err)
}
found := false
for _, x := range ch.Findings {
found = found || (strings.Contains(x.What, "1 managed preference(s) drift") && strings.Contains(x.What, "mail.biff.show_alert") &&
strings.Contains(x.Do, "betterbird_config_apply") && strings.Contains(x.Do, "keep"))
}
theme := false
for _, x := range ch.Findings {
theme = theme || (strings.HasPrefix(x.What, "extensions.activeThemeID drifts") && strings.Contains(x.Do, "add-on manager") &&
!strings.Contains(x.Do, "betterbird_config_apply"))
}
if !found || !theme {
t.Fatalf("the check reports the drift: %+v", ch.Findings)
}
os.Remove(filepath.Join(f.Root, meshDefaults))
ch, _ = f.Check()
found = false
for _, x := range ch.Findings {
found = found || strings.Contains(x.What, "the module's defaults are not placed")
}
if !found {
t.Fatalf("defaults missing on a node of the mesh's release is a finding: %+v", ch.Findings)
}
}
// T3: the drifting values are cleared from prefs.js only while Betterbird is not running, and nothing else.
func TestConfigApplyClearsTheDriftOnlyWhileBetterbirdIsQuit(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
f.betterbirdsOwn()
links(t, nil)
f.declare("// none")
prefs := "// Mozilla User Preferences\n" +
`user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");` + "\n" +
`user_pref("mail.identity.id1.useremail", "someone@example.org");` + "\n" +
`user_pref("mail.biff.show_alert", true);` + "\n" +
`user_pref("mail.uidensity", 0);` + "\n"
f.write(profileDir+"/prefs.js", prefs)
dry, err := f.ConfigApply(false)
if err != nil || dry.Applied || len(dry.Clears) != 1 || len(dry.NotCleared) != 1 {
t.Fatalf("a dry run lists and changes nothing: %+v %v", dry, err)
}
if f.read(profileDir+"/prefs.js") != prefs {
t.Fatal("a dry run wrote prefs.js")
}
if _, err := f.ConfigApply(true); err == nil || !strings.Contains(err.Error(), "running") {
t.Fatalf("refused while Betterbird runs: %v", err)
}
if f.read(profileDir+"/prefs.js") != prefs {
t.Fatal("prefs.js changed while Betterbird runs")
}
stopBetterbird(t, f)
f.write(profileDir+"/lock", "")
if _, err := f.ConfigApply(true); err == nil || !strings.Contains(err.Error(), "locked") {
t.Fatalf("refused while the profile's own lock is there: %v", err)
}
if err := os.Remove(filepath.Join(f.Root, profileDir, "lock")); err != nil {
t.Fatal(err)
}
a, err := f.ConfigApply(true)
if err != nil || !a.Applied {
t.Fatalf("%+v %v", a, err)
}
want := "// Mozilla User Preferences\n" +
`user_pref("extensions.activeThemeID", "thunderbird-compact-dark@mozilla.org");` + "\n" +
`user_pref("mail.identity.id1.useremail", "someone@example.org");` + "\n" +
`user_pref("mail.uidensity", 0);` + "\n"
if got := f.read(profileDir + "/prefs.js"); got != want {
t.Fatalf("only the drifting lines go:\n%s", got)
}
said := map[string]Cleared{}
for _, c := range a.Clears {
said[c.Name] = c
}
if c := said["mail.biff.show_alert"]; c.Was != "true" || c.Now != "false" {
t.Errorf("it says what it cleared and what holds now: %+v", c)
}
if _, cleared := said["extensions.activeThemeID"]; cleared || len(a.NotCleared) != 1 || !strings.Contains(a.NotCleared[0], "add-on manager") {
t.Errorf("the theme is never cleared, and said with its remedy: %+v", a)
}
raw, _ := json.Marshal(a)
if strings.Contains(string(raw), "example.org") {
t.Fatalf("no account value leaves: %s", raw)
}
if c, _ := f.Config(nil); !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID"}) {
t.Fatalf("once applied, only the theme drifts, until the add-on manager changes it: %v", c.Drifting)
}
os.Remove(filepath.Join(f.Root, meshDefaults))
if _, err := f.ConfigApply(true); err == nil || !strings.Contains(err.Error(), "not placed") {
t.Fatalf("without the module's defaults there is nothing to come back to: %v", err)
}
}
// T6: keeping a value on one node is a node layer proposed through the controller, never a write to user.js.
func TestKeepProposesANodeLayerThroughTheController(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
links(t, nil)
f.declare(`user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 12);`)
f.write(profileDir+"/prefs.js", `user_pref("mail.uifontsize", 16);`+"\n"+`user_pref("mail.biff.show_alert", true);`+"\n")
oldNode, oldAsk := nodeName, asker
t.Cleanup(func() { nodeName, asker = oldNode, oldAsk })
nodeName = func() string { return "shanks" }
var verb string
var args map[string]any
asker = func(v string, a any) (json.RawMessage, error) {
verb, args = v, a.(map[string]any)
return json.RawMessage(`{"ok": true, "output": "proposed: p-1"}`), nil
}
before := f.read(testHome + "/.config/betterbird/user.js")
c, err := f.Config([]string{"mail.uifontsize", "mail.biff.show_alert"})
if err != nil {
t.Fatal(err)
}
if verb != "seat:mesh-controller.settings" || args["node"] != "shanks" || args["propose"] != "true" || args["module"] != "betterbird" {
t.Fatalf("%s %v", verb, args)
}
var values map[string]string
if err := json.Unmarshal([]byte(args["values"].(string)), &values); err != nil {
t.Fatal(err)
}
want := `user_pref("calendar.alarms.playsound", false); user_pref("mail.uifontsize", 16); user_pref("mail.biff.show_alert", true);`
if values["prefs"] != want {
t.Fatalf("the layer keeps what the setting says and the kept values:\n got %s\nwant %s", values["prefs"], want)
}
if c.Proposed != "proposed: p-1" || f.read(testHome+"/.config/betterbird/user.js") != before {
t.Fatalf("proposed, and user.js untouched: %+v", c)
}
for _, bad := range [][]string{{"mailnews.start_page.enabled"}, {"some.other.pref"}, {"mail.uidensity"}} {
verb = ""
if _, err := f.Config(bad); err == nil || verb != "" {
t.Errorf("%v: a lock, an unmanaged name or a value the profile did not save is refused: %v", bad, err)
}
}
nodeName = func() string { return "" }
if _, err := f.Config([]string{"mail.uifontsize"}); err == nil || !strings.Contains(err.Error(), "MESH_NODE") {
t.Fatalf("%v", err)
}
}
// The review of pull request 253: whether Betterbird asks to be the default client is decided from the policy's
// own Preferences entry, never from what else the policy lacks.
func TestTheDefaultClientAskIsDecidedFromThePreferencesEntry(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
f.write(policyFile, `{"policies": {"ExtensionSettings": {}}}`)
if why := f.defaultClientAsk(filepath.Join(f.Root, profileDir)); why == "" {
t.Fatal("a policy lacking the updater and the lock still asks")
}
f.write(policyFile, `{"policies": {"Preferences": {"mail.shell.checkDefaultClient": {"Value": false, "Status": "locked"}}}}`)
if why := f.defaultClientAsk(filepath.Join(f.Root, profileDir)); why != "" {
t.Fatalf("the lock holds whatever else the policy lacks: %s", why)
}
}
// The review of pull request 256: the theme is the add-on database's, and a saved default is not drift.
func TestTheThemeIsTheAddOnDatabasesAndASavedDefaultIsNotDrift(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
f.withDefaults()
links(t, nil)
f.write(profileDir+"/prefs.js", `user_pref("browser.display.use_system_colors", false);`+"\n"+
`user_pref("layout.css.always_underline_links", false);`+"\n")
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"thunderbird-compact-dark@mozilla.org","type":"theme","active":true,"location":"app-builtin"},`+
`{"id":"default-theme@mozilla.org","type":"theme","active":false,"location":"app-builtin"}]}`)
c, err := f.Config(nil)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(c.Drifting, []string{"extensions.activeThemeID"}) {
t.Fatalf("the active theme from the add-on database, with no preference saved, is drift; a saved default is not: %v", c.Drifting)
}
f.write(profileDir+"/prefs.js", `user_pref("extensions.activeThemeID", "default-theme@mozilla.org");`+"\n")
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"default-theme@mozilla.org","type":"theme","active":true,"location":"app-builtin"}]}`)
if c, _ := f.Config(nil); len(c.Drifting) != 0 {
t.Fatalf("a profile back on the default theme does not drift: %v", c.Drifting)
}
if _, err := f.Config([]string{"extensions.activeThemeID"}); err == nil || !strings.Contains(err.Error(), "add-on manager") {
t.Fatalf("a node layer cannot choose the theme: %v", err)
}
}
// The review of pull request 256: a credential's or an account's name in the defaults file is never managed.
func TestAnAccountsNameInTheDefaultsIsNeverManaged(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
f.write(meshDefaults, string(repoFile(t, "install/defaults-pref/mesh-defaults.js"))+
`pref("mail.identity.id1.useremail", "x");`+"\n"+"// keeps Betterbird's default: mail.server.server1.hostname\n"+
`pref("mail.smtpservers", "");`+"\n")
prefs := `user_pref("mail.identity.id1.useremail", "someone@example.org");` + "\n" +
`user_pref("mail.server.server1.hostname", "imap.example.org");` + "\n" +
`user_pref("mail.smtpservers", "smtp1");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
c, err := f.Config(nil)
if err != nil {
t.Fatal(err)
}
if len(c.Refused) != 3 || len(c.Drifting) != 0 {
t.Fatalf("refused, never managed: %+v", c)
}
raw, _ := json.Marshal(c)
if strings.Contains(string(raw), "example.org") {
t.Fatalf("no account value is answered: %s", raw)
}
stopBetterbird(t, f)
if _, err := f.ConfigApply(true); err != nil {
t.Fatal(err)
}
if f.read(profileDir+"/prefs.js") != prefs {
t.Fatal("prefs.js keeps every account's value")
}
ch, _ := f.Check()
found := false
for _, x := range ch.Findings {
found = found || (strings.Contains(x.What, "3 preference(s) the tools never manage") && strings.Contains(x.Do, "mesh-defaults.js"))
}
if !found {
t.Fatalf("the check says so: %+v", ch.Findings)
}
}
@@ -30,10 +30,6 @@ const (
// owlFile is Owl's xpi as the mesh placed it, and the URL the policy installs it from.
owlFile = installDir + "/owl/owl.xpi"
owlID = "owl@beonex.com"
// defaultClientPref is Betterbird's ask at its start to be the default mail, news, feed and calendar
// client (Thunderbird's all-thunderbird.js: pref("mail.shell.checkDefaultClient", true); messenger.js opens
// systemIntegrationDialog.xhtml while it is true). The module's policy locks it off (novox/hq issue 345).
defaultClientPref = "mail.shell.checkDefaultClient"
// preparation is the tools' program as the module places it for its start (betterbird-tools prepare).
preparation = "/usr/local/lib/mesh-betterbird/betterbird-tools"
// versionsURL is where Betterbird publishes its newest build per major version: what its own updater
@@ -162,7 +158,7 @@ func newestBuild(version, build string) (string, bool) {
return "unknown: " + versionsURL + " names no build for " + major, false
}
// policyState says whether the module's policy is placed and does what ADR 0316 §5 and §7 and issue 345 ask.
// policyState says whether the module's policy is placed and does what ADR 0316 §5 and §7 say.
func (m *Machine) policyState() string {
raw, err := readBounded(m.path(policyFile))
if err != nil {
@@ -171,25 +167,14 @@ func (m *Machine) policyState() string {
if why := policyProblem(raw); why != "" {
return "placed, and " + why
}
return "placed: Betterbird's updater, its telemetry and add-on updates off, Owl installed into every profile from " +
"the mesh's xpi with its updates off, and the ask to be the default mail client, the vendor's in-app notices " +
"and the start page locked off"
return "placed: Betterbird's updater off, Owl installed into every profile from the mesh's xpi, its updates off"
}
// lockedOff are the preferences the module's policy locks to false (novox/hq issue 345), each with what it is.
var lockedOff = []struct{ Name, What string }{
{defaultClientPref, "the ask to be the default mail client"},
{"mail.inappnotifications.enabled", "the vendor's in-app notices"},
{"mailnews.start_page.enabled", "the start page, loaded from the vendor at every start"},
}
// policyProblem is everything the policy fails to say of what ADR 0316 and issue 345 ask, or "".
// policyProblem is what the policy fails to say of what ADR 0316 asks, or "".
func policyProblem(raw []byte) string {
var p struct {
Policies struct {
DisableAppUpdate bool `json:"DisableAppUpdate"`
DisableTelemetry bool `json:"DisableTelemetry"`
ExtensionUpdate *bool `json:"ExtensionUpdate"`
DisableAppUpdate bool `json:"DisableAppUpdate"`
ExtensionSettings map[string]struct {
Mode string `json:"installation_mode"`
URL string `json:"install_url"`
@@ -200,57 +185,18 @@ func policyProblem(raw []byte) string {
if err := json.Unmarshal(raw, &p); err != nil {
return "it is not JSON: " + err.Error()
}
var problems []string
if !p.Policies.DisableAppUpdate {
problems = append(problems, "it does not turn Betterbird's updater off (DisableAppUpdate)")
}
if !p.Policies.DisableTelemetry {
problems = append(problems, "it does not turn telemetry off (DisableTelemetry)")
}
if p.Policies.ExtensionUpdate == nil || *p.Policies.ExtensionUpdate {
problems = append(problems, "it leaves add-ons updating themselves (ExtensionUpdate)")
}
owl, ok := p.Policies.ExtensionSettings[owlID]
switch {
case !p.Policies.DisableAppUpdate:
return "it does not turn Betterbird's updater off (DisableAppUpdate)"
case !ok || owl.Mode != "force_installed":
problems = append(problems, "it does not install Owl into every profile (force_installed)")
return "it does not install Owl into every profile (force_installed)"
case owl.URL != "file://"+owlFile:
problems = append(problems, "it installs Owl from "+owl.URL+", not from the mesh's xpi")
return "it installs Owl from " + owl.URL + ", not from the mesh's xpi"
case !owl.Disabled:
problems = append(problems, "it leaves Owl's updates on")
return "it leaves Owl's updates on"
}
prefs := policyPrefsOf(raw)
for _, l := range lockedOff {
if pr, ok := prefs[l.Name]; !ok || pr.Value != "false" || !pr.Locked {
problems = append(problems, "it does not lock "+l.What+" off (Preferences "+l.Name+": false, locked)")
}
}
return strings.Join(problems, "; ")
}
// defaultClientAsk says why Betterbird asks at its start to be the default mail client, or "" when it does
// not: the placed policy's own Preferences entry locks the ask off, else the profile's user.js (which wins at
// the start) or prefs.js turns it off. Betterbird's own default is to ask. Decided from the entry itself, never
// from what else the policy lacks.
func (m *Machine) defaultClientAsk(profile string) string {
if v, ok := m.locks()[defaultClientPref]; ok && v == "false" {
return ""
}
if profile != "" {
for _, f := range []string{"user.js", "prefs.js"} {
b, err := readBounded(filepath.Join(profile, f))
if err != nil {
continue
}
if v, ok := statementsOf(b)[defaultClientPref]; ok {
if v == "false" {
return ""
}
return "the profile's " + f + " sets " + defaultClientPref + " to " + v + " and no policy locks it off"
}
}
}
return "no policy locks " + defaultClientPref + " off and Betterbird's default is to ask"
return ""
}
// xpiVersion is the version an add-on's manifest.json says, or "" when it cannot be read.
@@ -172,60 +172,3 @@ func TestTheUserJSDescriptionSaysAdoption(t *testing.T) {
}
}
}
// Betterbird never asks at its start to be the default mail client: the module's policy locks
// mail.shell.checkDefaultClient off, and the check says so when it is on (novox/hq issue 345).
func TestTheDefaultClientAskIsLockedOffAndTheCheckSaysWhenItIsOn(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
asks := func() string {
t.Helper()
c, err := f.Check()
if err != nil {
t.Fatal(err)
}
for _, x := range c.Findings {
if strings.Contains(x.What, "asks at its start") {
return x.What + " | " + x.Do
}
}
return ""
}
if got := asks(); got != "" {
t.Fatalf("the module's policy locks the ask off, yet: %s", got)
}
unlocked := `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`
if why := policyProblem([]byte(unlocked)); !strings.Contains(why, defaultClientPref) {
t.Fatalf("a policy that leaves the ask on is said: %q", why)
}
f.write(policyFile, unlocked)
if got := asks(); !strings.Contains(got, "default mail client") || !strings.Contains(got, "push the module") {
t.Fatalf("the ask on is a finding with its remedy: %q", got)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "Preferences": {"mail.shell.checkDefaultClient": {"Value": false, "Status": "default"}},
"ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed", "install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`)
if why := policyProblem(mustRead(t, f, policyFile)); !strings.Contains(why, "locked") {
t.Fatalf("a default the profile can override is not a lock: %q", why)
}
f.write(profileDir+"/prefs.js", `user_pref("mail.shell.checkDefaultClient", true);`+"\n")
if got := asks(); got == "" {
t.Fatal("a profile that turned the ask back on over an unlocked default is a finding")
}
f.write(policyFile, unlocked)
f.write(profileDir+"/prefs.js", `user_pref("mail.shell.checkDefaultClient", false);`+"\n")
if got := asks(); got != "" {
t.Fatalf("a profile with the ask off does not ask: %s", got)
}
}
func mustRead(t *testing.T, f *fake, p string) []byte {
t.Helper()
raw, err := os.ReadFile(filepath.Join(f.Root, p))
if err != nil {
t.Fatal(err)
}
return raw
}
@@ -49,12 +49,10 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_prefs",
Description: "Named preferences of Betterbird's profile: each one's value and where it comes from (the " +
"module's policy, which locks it; user.js, which wins at the start; prefs.js; or the default), and the " +
"default itself with the file that gives it: Betterbird's own, read from omni.ja and the installation's " +
"defaults/pref, or the module's defaults (mesh-defaults.js). A name ending in . or * is a prefix (at most " +
"200 answered). A preference whose name can hold a credential, or that names an account, a server, a " +
"calendar or a person, is answered without its value. (r)",
Description: "Named preferences of Betterbird's profile: each one's value and where it comes from (user.js, " +
"which wins at the start; prefs.js; or Betterbird's default). A name ending in . or * is a prefix " +
"(at most 200 answered). A preference whose name can hold a credential, or that names an account, a " +
"server, a calendar or a person, is answered without its value. (r)",
Input: map[string]any{
"names": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "preference names or prefixes, e.g. [\"calendar.alarms.\", \"extensions.activeThemeID\"]"},
@@ -67,45 +65,6 @@ func tools() []stdio.Tool {
return machine.Prefs(names)
},
},
{
Name: "betterbird_config",
Description: "The configuration the module ships, preference by preference: the value its policy locks, " +
"its default (mesh-defaults.js, or Betterbird's own where the module keeps that), the node layer (the " +
"setting prefs on this machine), the profile's own saved value and the value in effect. It flags drift: " +
"a value the profile saved that beats the module's default without a node layer saying so. With keep " +
"(names), it proposes a node layer of the setting prefs for this machine holding the profile's values of " +
"those preferences, through the controller's settings verb, for the operator's Approve; it never writes " +
"user.js. (r; a with keep)",
Input: map[string]any{
"keep": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "managed preferences whose profile value this machine keeps: proposed as a node layer, e.g. [\"mail.uifontsize\"]"},
},
Run: func(args map[string]any) (any, error) {
keep, err := texts(args, "keep")
if err != nil {
return nil, err
}
return machine.Config(keep)
},
},
{
Name: "betterbird_config_apply",
Description: "Bring the profile back to the module's configuration: takes the drifting values " +
"(betterbird_config) out of the profile's prefs.js, so the module's defaults hold at Betterbird's next " +
"start, and says what it cleared, each with the value it had and the one that now holds. Without apply " +
"it lists them only. With apply it acts only while Betterbird is not running, because Betterbird " +
"rewrites prefs.js when it quits; every other line of prefs.js is kept. (r; a with apply)",
Input: map[string]any{
"apply": map[string]any{"type": "boolean", "description": "clear the drifting values (default false: a dry run)"},
},
Run: func(args map[string]any) (any, error) {
apply, err := flag(args, "apply", false)
if err != nil {
return nil, err
}
return machine.ConfigApply(apply)
},
},
{
Name: "betterbird_user_js",
Description: "The module's user.js — the preferences the module declares (setting prefs), then the declared " +
@@ -176,8 +135,7 @@ func tools() []stdio.Tool {
{
Name: "betterbird_check",
Description: "Check what the module promises: Betterbird installed from the mesh's pinned release and not " +
"behind the newest published build; the enterprise policy placed and the module's defaults with it; no " +
"managed preference drifting from them; Owl at its pin and active; one " +
"behind the newest published build; the enterprise policy placed; Owl at its pin and active; one " +
"Betterbird runs; its profile is found; the module's user.js on the machine, valid, placed and in " +
"effect; the declared accounts in the profile's list, none refused, each signed in once; its output " +
"read. Answers ok and each finding with what to do, never an address, host or name. (r)",
@@ -1,85 +0,0 @@
// Written by the mesh (module betterbird, novox/hq issue 345): the operator's setup of Betterbird, taken from
// the laptop's profile and approved by the operator on 2026-10-12. Betterbird reads this file at its start as
// a default: a value the profile saved in its prefs.js (a choice made in Betterbird's settings window) wins
// over it, a node's layer of the setting prefs (user.js) wins over both, and the policy's locks
// (distribution/policies.json) win over everything. betterbird_config shows the layers and the drift;
// betterbird_config_apply brings a profile back to these values. Change it in the catalogue, never here.
//
// A line "keeps Betterbird's default" names a preference the module manages without writing a value: the
// operator's setup is Betterbird's own default there, and a value the profile saved is drift.
// --- Theme and display ---
// keeps Betterbird's default: extensions.activeThemeID
// The default theme (default-theme@mozilla.org) follows the desktop's dark mode; compactness comes from
// mail.uidensity, not a theme. Betterbird keeps the active theme in its add-on database, so another theme is
// changed back in the add-on manager, never by clearing prefs.js.
// keeps Betterbird's default: browser.display.use_system_colors
// Betterbird's own colours, which the default theme is drawn for.
// keeps Betterbird's default: layout.css.always_underline_links
// Links are underlined only where the message's own style says so.
// Compact density: more of the message list fits on a laptop screen (Betterbird's default is 1, normal).
pref("mail.uidensity", 0);
// 14-point interface text, the operator's choice; a larger desk screen sets its own size in a node layer.
pref("mail.uifontsize", 14);
// Message bodies are shown as their sender wrote them, not darkened.
pref("mail.dark-reader.enabled", false);
// --- Layout ---
// Vertical view: folders, the message list and the message side by side (Betterbird's default is 0, classic).
pref("mail.pane_config.dynamic", 2);
// Each row of the message list on several lines, so sender, subject and date all show in the narrow column.
pref("mail.pane_config.multiline_all", true);
// The message list as cards (Betterbird's default is 1, the table).
pref("mail.threadpane.listview", 0);
// Threads grouped in every folder by default (Betterbird's default is 0, unthreaded).
pref("mailnews.default_view_flags", 1);
// --- Mail handling and compose ---
// No confirmation before Shift+Delete removes a message for good.
pref("mail.warn_on_shift_delete", false);
// keeps Betterbird's default: mail.spam.manualMark
// Marking a message as junk by hand does nothing more than mark it.
// keeps Betterbird's default: mail.compose.show_attachment_pane
// The composer shows its attachment pane only when a message has attachments.
// --- Notifications ---
// No pop-up for new mail: the operator reads mail when they choose to.
pref("mail.biff.show_alert", false);
// No sound for new mail.
pref("mail.biff.play_sound", false);
// No tray icon for new mail.
pref("mail.biff.show_tray_icon", false);
// If a node's layer turns alerts back on, they go through the desktop's notification daemon.
pref("mail.biff.use_system_alert", true);
// --- Calendar ---
// The calendar's working day starts at 09:00 (Betterbird's default is 08:00).
pref("calendar.view.daystarthour", 9);
// and ends at 20:00 (Betterbird's default is 17:00).
pref("calendar.view.dayendhour", 20);
// A calendar item is deleted without asking.
pref("calendar.item.promptDelete", false);
// The calendar follows the node's time zone, because the laptop travels.
pref("calendar.timezone.useSystemTimezone", true);
// Reminders show in their window (Betterbird's default, made explicit: the reminder is why this module exists).
pref("calendar.alarms.show", true);
// Reminders missed while the laptop slept show at the next start (Betterbird's default is false).
pref("calendar.alarms.showmissed", true);
// --- Chat ---
// No chat button: the operator uses no chat account in Betterbird (Betterbird's default is true).
pref("mail.chat.enabled", false);
// --- Privacy ---
// No remote content in a message unless its sender is allowed (Betterbird's default, made explicit).
pref("mailnews.message_display.disable_remote_image", true);
// Ask sites not to track. A default only: the policy cannot lock a privacy. preference.
pref("privacy.donottrackheader.enabled", true);
@@ -1,28 +1,12 @@
{
"policies": {
"DisableAppUpdate": true,
"DisableTelemetry": true,
"ExtensionUpdate": false,
"ExtensionSettings": {
"owl@beonex.com": {
"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi",
"updates_disabled": true
}
},
"Preferences": {
"mail.shell.checkDefaultClient": {
"Value": false,
"Status": "locked"
},
"mail.inappnotifications.enabled": {
"Value": false,
"Status": "locked"
},
"mailnews.start_page.enabled": {
"Value": false,
"Status": "locked"
}
}
}
}
-2
View File
@@ -21,8 +21,6 @@
"tools": [
"betterbird_status",
"betterbird_prefs",
"betterbird_config",
"betterbird_config_apply",
"betterbird_user_js",
"betterbird_log",
"betterbird_restart",
+1 -9
View File
@@ -70,15 +70,7 @@
"restart-on": [
"agent-env"
],
"network": "host",
"health": {
"kind": "exec",
"command": "/usr/local/bin/mesh-builder ready",
"interval": "10s",
"timeout": "5s",
"looks": 3,
"grace": "60s"
}
"network": "host"
}
],
"build": {
+8 -18
View File
@@ -45,38 +45,28 @@ base64-encoded. The Go bundle `cmd/forge` serves the commit, and the reads an ag
| `gitea_search_code` | r | the lines of a ref's text files matching a literal string or a regular expression, optionally under a path |
**Where a commit may go** is the rule `gitea_api` already holds for a file written (novox/hq ADR 0266):
never the repository's default branch, never a branch a protection rule names. A rule covers a branch as the
forge itself decides it (Gitea 1.27, novox/hq issue 539): a plain name covers that branch in any case; a glob
(`*`, `**`, `?`, `[abc]`, `[!abc]`, `{a,b}`, `\` escaping) covers case-sensitively with `/` as separator; a glob
the forge cannot compile covers its own text. `cmd/forge/rules.go` and `guard.ts` hold this one rule. The rules are read before anything is sent; rules that cannot be read refuse the
never the repository's default branch, never a branch a protection rule names (a glob rule covers by the same
matcher as `guard.ts`). The rules are read before anything is sent; rules that cannot be read refuse the
commit. Nothing about merging changes: a merge still needs an independent review at the head and both
required checks green, through `gitea_merge_pull_request`.
**How the commit is made.** `files` become one call to the forge's multi-file contents endpoint: a path the
ref holds is updated, one it does not is created, and an update or delete carries the blob read just
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. The blobs are read at
the commit the branch (or, for a new branch, its base) was read at, never at its name. A `patch` goes to the
forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than the
file. Neither endpoint takes a commit to build on, only a branch (Gitea 1.27), so the parent answered is the
one the forge says it made the commit on; when the branch moved between the read and the commit, the answer
says so. One commit carries at most 200 files, 4 MiB per file and 16 MiB together, or a patch of 4 MiB. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
before, so a file someone changed meanwhile is refused by the forge rather than overwritten. A `patch` goes
to the forge's diff endpoint, which applies it; for a small change to a large file this is far shorter than
the file. Either way the commit is the forge's own, made as the forge's admin account, so its sha is not the
local commit's: the merge check judges the head on the forge. After the write the branch is read back, and a
branch not at the commit the forge named is a failure, never a success.
**Code search** has no endpoint on the forge, so the ref's archive (`/archive/{ref}.tar.gz`, at most 64 MiB)
is read once and searched in memory; a binary file is passed over, a file over 4 MiB is named under
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found. A
search that reaches the call's deadline stops there and answers what it found, with `stopped_at_deadline`.
`skipped`, and at most `max` matches (default 100) are answered, with `truncated` when more were found.
It reaches the forge through the same HTTP client as the npm-registry bundle (`internal/forgehttp`), and one
tool call takes at most 50 seconds.
It reaches the forge as the npm-registry bundle does (below), and one tool call takes at most 50 seconds.
`go test ./cmd/forge` runs it against a fake forge: the files of one commit created, updated and deleted on
a new branch from its base; an existing branch committed on; the default branch, a protected branch and
unreadable rules refused before anything is sent; a patch sent to the forge; a branch read back at another
commit failing; a branch or base that moved under the commit said, with the parent the forge used; a file
changed under the commit refused; a commit over its size limits refused; protection globs matched as the
forge matches them; a code search stopped at its deadline; every malformed ask refused; and each read
summarised.
commit failing; every malformed ask refused; and each read summarised.
## The package registry's retention (novox/hq ADR 0251 §4, to-be 51)
+1 -69
View File
@@ -49,23 +49,6 @@ export interface GiteaIssue {
body?: string;
}
/**
* **The branch a pull request was opened from, also once it is deleted** (novox/hq issue 578). The forge says it as
* `head.ref` while the branch exists; once it is gone — the merge tool deletes it after the merge by default —
* `head.ref` reads `refs/pull/<n>/head`, and only `head.label` still names the branch. A delivery group is the pull
* requests sharing a head branch name (novox/hq ADR 0249), and the controller finds a merge's group, and joins the
* group's merges in one batch, by the head its `pull.merged` names (ADR 0276 decision 5, ADR 0306 decision 2). With
* the pull ref it found no group: on 2026-10-12 mesh-catalog #251's walk was cut at once and went before
* mesh-controller #271's, which the group's order put first.
*/
export function headBranchOf(head: { ref?: unknown; label?: unknown } | null | undefined): string | undefined {
const said = (v: unknown): string | undefined => (typeof v === "string" && v !== "" ? v : undefined);
const ref = said(head?.ref);
const label = said(head?.label);
if (ref === undefined || /^refs\/pull\/\d+\/head$/.test(ref)) return label ?? ref;
return ref;
}
/** A pull request, trimmed to the fields a reviewer or an event body needs. */
export interface GiteaPull {
number: number;
@@ -358,57 +341,6 @@ export class GiteaClient {
return out;
}
/** **Every open pull request of a repository**, page by page (novox/hq ADR 0323 §4): what a trunk's move puts out of
* date, however many are open, whatever their update time (a pass announces from listOpenPullsSince). Bounded at
* 20 pages of 50; a repository with more is said, and the rest are left to the merge tool's own comparison. */
async listOpenPulls(owner: string, repo: string): Promise<GiteaPull[]> {
const out: GiteaPull[] = [];
for (let page = 1; page <= 20; page++) {
const batch = await this.listPullRequests(owner, repo, { state: "open", limit: "50", page: String(page) });
out.push(...batch);
if (batch.length < 50) return out;
}
console.error(`[gitea] ${owner}/${repo} has more than 1000 open pull requests; only the first 1000 are read`);
return out;
}
/** **The open pull requests of a repository updated since a moment**, newest first, page by page (novox/hq issue
* 579): what a pass announces new heads from. The forge sorts `recentupdate` by update time, newest first, so the
* paging stops at the first page that reaches back past `since`; with no moment — a sweep or the first pass —
* every page is read. Bounded like listOpenPulls at 20 pages of 50. A pull request seen twice, pushed up a page
* while the pages were read, is answered once. */
async listOpenPullsSince(owner: string, repo: string, since: string): Promise<GiteaPull[]> {
const out: GiteaPull[] = [];
const seen = new Set<number>();
const floor = since ? Date.parse(since) : NaN;
for (let page = 1; page <= 20; page++) {
const batch = await this.listPullRequests(owner, repo, { state: "open", sort: "recentupdate", limit: "50", page: String(page) });
for (const p of batch) {
if (seen.has(p.number)) continue;
seen.add(p.number);
out.push(p);
}
if (batch.length < 50) return out;
const oldest = batch[batch.length - 1]?.updated_at;
if (!Number.isNaN(floor) && oldest && Date.parse(oldest) < floor) return out;
}
console.error(`[gitea] ${owner}/${repo} has more than 1000 open pull requests updated since ${since || "ever"}; only the newest 1000 are read`);
return out;
}
/** The commit a branch is at now: a trunk's head (novox/hq ADR 0323). */
async branchHead(owner: string, repo: string, branch: string): Promise<string> {
const b = await this.request<any>(`${repoPath(owner, repo)}/branches/${pathSegments("branch", branch)}`);
const sha = String(b?.commit?.id ?? "");
if (!/^[0-9a-f]{40,64}$/i.test(sha)) throw new Error(`the forge named no commit for ${owner}/${repo}'s ${branch}`);
return sha;
}
/** One repository, as the forge holds it: its clone URL above all, for an announcement. */
async getRepo(owner: string, repo: string): Promise<GiteaRepo> {
return GiteaClient.mapRepo(await this.request<any>(repoPath(owner, repo)));
}
async getPullRequest(owner: string, repo: string, index: number): Promise<GiteaPull> {
indexOf(index);
return GiteaClient.mapPull(await this.request<any>(`${repoPath(owner, repo)}/pulls/${index}`));
@@ -714,7 +646,7 @@ export class GiteaClient {
merge_commit_sha: p.merge_commit_sha ?? undefined,
merged_at: p.merged_at ?? undefined,
user: p.user?.login,
head: headBranchOf(p.head),
head: p.head?.ref,
head_sha: p.head?.sha ?? undefined,
base: p.base?.ref,
updated_at: p.updated_at ?? undefined,
+16 -90
View File
@@ -6,7 +6,6 @@ import (
"errors"
"fmt"
"net/http"
"net/url"
"strings"
"unicode/utf8"
)
@@ -53,14 +52,8 @@ type CommittedFile struct {
Operation string `json:"operation"`
}
// What one commit carries at most (novox/hq issue 539): files, one file's content, one patch, and every
// file's content together, in bytes as committed (content_base64 decoded). Larger work is several commits.
const (
maxFiles = 200
maxFileBytes = 4 << 20
maxPatchBytes = 4 << 20
maxCommitBytes = 16 << 20
)
// maxFiles is the most files one commit carries.
const maxFiles = 200
// readCommitAsk reads the tool's arguments.
func readCommitAsk(args map[string]any) (CommitAsk, error) {
@@ -116,11 +109,8 @@ func readCommitAsk(args map[string]any) (CommitAsk, error) {
return a, refuse("give files or patch, not both: one commit carries one of them. Nothing was sent")
case len(a.Files) > maxFiles:
return a, refuse("%d files is more than the %d one commit carries. Nothing was sent", len(a.Files), maxFiles)
case len(a.Patch) > maxPatchBytes:
return a, refuse("the patch is %s, more than the %s one commit carries: split it into several commits. Nothing was sent", mib(len(a.Patch)), mib(maxPatchBytes))
}
seen := map[string]bool{}
total := 0
for i, fc := range a.Files {
if strings.HasPrefix(fc.Path, "/") {
return a, refuse("files[%d].path %q is relative to the repository's root, without a leading slash. Nothing was sent", i, fc.Path)
@@ -144,35 +134,15 @@ func readCommitAsk(args map[string]any) (CommitAsk, error) {
if fc.Content != nil && !utf8.ValidString(*fc.Content) {
return a, refuse("files[%d].content (%s) is not UTF-8 text: give it as content_base64. Nothing was sent", i, fc.Path)
}
size := 0
switch {
case fc.Content != nil:
size = len(*fc.Content)
case fc.ContentBase64 != nil:
if base64.StdEncoding.DecodedLen(len(*fc.ContentBase64)) > maxFileBytes+3 {
size = base64.StdEncoding.DecodedLen(len(*fc.ContentBase64))
break
}
raw, err := base64.StdEncoding.DecodeString(*fc.ContentBase64)
if err != nil {
if fc.ContentBase64 != nil {
if _, err := base64.StdEncoding.DecodeString(*fc.ContentBase64); err != nil {
return a, refuse("files[%d].content_base64 (%s) is not base64: %v. Nothing was sent", i, fc.Path, err)
}
size = len(raw)
}
if size > maxFileBytes {
return a, refuse("files[%d] (%s) is %s, more than the %s one file of a commit may be. Nothing was sent", i, fc.Path, mib(size), mib(maxFileBytes))
}
total += size
}
if total > maxCommitBytes {
return a, refuse("the files are %s together, more than the %s one commit carries: split them into several commits. Nothing was sent", mib(total), mib(maxCommitBytes))
}
return a, nil
}
// mib says a size in bytes in MiB, to a tenth.
func mib(n int) string { return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20)) }
// targetRefused says why a commit may not go on a branch: the default branch, or one a protection rule names.
func targetRefused(target, defaultBranch string, rules []Protection) error {
if target == defaultBranch {
@@ -202,7 +172,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
return nil, err
}
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
if err := f.getJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.DefaultBranch == "" {
@@ -210,7 +180,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
}
// Read the rules, or refuse: a branch the rules could not be read for is never written (fails closed).
var rules []Protection
if err := f.GetJSON(ctx, repo+"/branch_protections", &rules); err != nil {
if err := f.getJSON(ctx, repo+"/branch_protections", &rules); err != nil {
return nil, fmt.Errorf("the protection rules of %s/%s could not be read, so the commit was not judged and nothing was sent: %w", a.Owner, a.Repo, err)
}
if err := targetRefused(a.Branch, r.DefaultBranch, rules); err != nil {
@@ -222,13 +192,9 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
if err != nil {
return nil, err
}
// The files are judged at the commit just read, never at a branch name that could move between two
// reads (novox/hq issue 539): a file the branch changes meanwhile is refused by the forge, whose update
// and delete carry the blob read here.
var ref string
ref := a.Branch // where the files are read and the commit goes
if existing != nil {
out.Parent = existing.Commit.ID
ref = existing.Commit.ID
if a.Base != "" {
out.Said = append(out.Said, fmt.Sprintf("%s exists, at %s: base %s was not used", a.Branch, short(existing.Commit.ID), a.Base))
}
@@ -238,6 +204,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
if out.Base == "" {
out.Base = r.DefaultBranch
}
ref = out.Base
b, err := f.branch(ctx, repo, out.Base)
if err != nil {
return nil, err
@@ -246,6 +213,8 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
return nil, refuse("%s does not exist, and its base %s is not a branch of %s/%s. Nothing was sent", a.Branch, out.Base, a.Owner, a.Repo)
}
out.Parent = b.Commit.ID
// The files are judged at the commit just read; a file the base changes meanwhile is refused by the
// forge, whose update and delete carry the blob read here.
ref = b.Commit.ID
}
@@ -258,14 +227,14 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
on["branch"] = a.Branch
}
// The forge takes no commit to build on, only a branch (Gitea 1.27: neither /contents nor /diffpatch
// carries a last commit), so the parent is read from what it made, never assumed (novox/hq issue 539).
var made struct {
Commit madeCommit `json:"commit"`
Commit struct {
SHA string `json:"sha"`
} `json:"commit"`
}
if a.Patch != "" {
on["content"] = a.Patch
if err := f.SendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
if err := f.sendJSON(ctx, http.MethodPost, repo+"/diffpatch", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the patch: %w", err)
}
} else {
@@ -297,7 +266,7 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
out.Files = append(out.Files, CommittedFile{Path: fc.Path, Operation: op["operation"].(string)})
}
on["files"] = ops
if err := f.SendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
if err := f.sendJSON(ctx, http.MethodPost, repo+"/contents", on, &made); err != nil {
return nil, fmt.Errorf("the forge did not take the commit: %w", err)
}
}
@@ -305,25 +274,6 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
return nil, fmt.Errorf("the forge answered the commit on %s without saying the commit it made: read the branch (gitea_list_branches) before trying again", a.Branch)
}
out.Commit = made.Commit.SHA
read := out.Parent
parent, err := f.parentOf(ctx, repo, made.Commit)
if err != nil {
return nil, fmt.Errorf("the forge made %s on %s, but which commit it went on could not be read: %w", out.Commit, a.Branch, err)
}
out.Parent = parent
if parent != read {
moved := "branch " + a.Branch
if out.Created {
moved = "base " + out.Base
}
how := "each file it changes was checked against " + short(read) + " by its blob, so none of them changed in between"
if a.Patch != "" {
how = "the forge applied the patch there, and it refuses a patch whose context changed"
}
out.Said = append(out.Said, fmt.Sprintf("the %s moved from %s to %s between the read and the commit, so the "+
"commit went on %s: %s. What else changed in between: gitea_compare %s...%s",
moved, short(read), short(parent), short(parent), how, short(read), short(parent)))
}
// Evidence over assertion: the branch is read back, and it must be at the commit the forge said.
after, err := f.branch(ctx, repo, a.Branch)
@@ -344,30 +294,6 @@ func commit(ctx context.Context, f *Forge, a CommitAsk) (*Committed, error) {
return out, nil
}
// madeCommit is the commit the forge answers a write with.
type madeCommit struct {
SHA string `json:"sha"`
Parents []struct {
SHA string `json:"sha"`
} `json:"parents"`
}
// parentOf is the commit the forge made its commit on: from its answer, or read from the commit when the
// answer does not say it.
func (f *Forge) parentOf(ctx context.Context, repo string, made madeCommit) (string, error) {
if len(made.Parents) == 0 {
var read madeCommit
if err := f.GetJSON(ctx, repo+"/git/commits/"+url.PathEscape(made.SHA)+"?stat=false&verification=false&files=false", &read); err != nil {
return "", err
}
made = read
}
if len(made.Parents) != 1 || made.Parents[0].SHA == "" {
return "", fmt.Errorf("the forge says %s has %d parents, not one", made.SHA, len(made.Parents))
}
return made.Parents[0].SHA, nil
}
// blobAt is the blob sha of a file at a ref, "" when there is no file there.
func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, error) {
seg, err := segments("file path", path)
@@ -379,7 +305,7 @@ func (f *Forge) blobAt(ctx context.Context, repo, path, ref string) (string, err
SHA string `json:"sha"`
}
var list []any
body, err := f.Do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
body, err := f.do(ctx, http.MethodGet, repo+"/contents/"+seg+"?ref="+queryEscape(ref), nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return "", nil
+154 -11
View File
@@ -1,43 +1,156 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"regexp"
"strings"
"gitea/internal/forgehttp"
"time"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the npm-registry bundle beside it).
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account,
// which the vault delivered into a file this module owns (novox/hq ADR 0086), by basic authentication — the
// same way the npm-registry bundle beside it does. The password is read when a call is made and never logged,
// answered or put in an error.
// Forge is the forge's HTTP interface.
type Forge struct {
forgehttp.Client
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
HTTP *http.Client
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
c, missing := forgehttp.FromEnv()
f := &Forge{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
}
var missing []string
for _, kv := range [][2]string{{"MESH_GITEA_URL", f.URL}, {"MESH_GITEA_ADMIN_USER", f.User},
{"MESH_GITEA_ADMIN_PASSWORD_FILE", f.PasswordFile}} {
if kv[1] == "" {
missing = append(missing, kv[0])
}
}
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
}
c.Timeout = deadline
return &Forge{Client: c}, nil
return f, nil
}
// errNotFound is the forge answering 404.
var errNotFound = forgehttp.ErrNotFound
var errNotFound = errors.New("not found")
// statusError is an answer the forge gave that is not a success, said without the request's credential.
type statusError struct {
method, path string
status int
said string
}
func (e *statusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
}
func (f *Forge) client() *http.Client {
if f.HTTP != nil {
return f.HTTP
}
return &http.Client{Timeout: 50 * time.Second}
}
func (f *Forge) password() (string, error) {
raw, err := os.ReadFile(f.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
// maxBody is the largest JSON answer read from the forge; maxArchive the largest archive code search reads.
const (
maxBody = forgehttp.MaxBody
maxBody = 32 << 20
maxArchive = 64 << 20
)
// do sends one request and answers the body, read up to limit; 404 is errNotFound. A body the limit cut is
// refused, never read as if it were whole.
func (f *Forge) do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
password, err := f.password()
if err != nil {
return nil, err
}
var body io.Reader
if payload != nil {
raw, err := json.Marshal(payload)
if err != nil {
return nil, err
}
body = bytes.NewReader(raw)
}
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, body)
if err != nil {
return nil, err
}
req.SetBasicAuth(f.User, password)
req.Header.Set("Accept", "application/json")
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
res, err := f.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
if err != nil {
return nil, err
}
if int64(len(got)) > limit {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(got))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &statusError{method, path, res.StatusCode, said}
}
return got, nil
}
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
return f.sendJSON(ctx, http.MethodGet, path, nil, into)
}
func (f *Forge) sendJSON(ctx context.Context, method, path string, payload, into any) error {
body, err := f.do(ctx, method, path, payload, maxBody)
if err != nil {
return err
}
if into == nil {
return nil
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
}
return nil
}
// Refused is a refusal of policy or of an argument, in the words the caller sees.
type Refused struct{ msg string }
@@ -91,6 +204,36 @@ func (p Protection) name() string {
return p.BranchName
}
// ruleCovers is whether a protection rule's name — a branch, or a glob of them (`*`, `**`, `?`) — covers a
// branch: the TypeScript bundle's ruleCovers (guard.ts), which gitea_api's file writes are judged by.
func ruleCovers(rule, branch string) bool {
if rule == branch {
return true
}
if !strings.ContainsAny(rule, "*?[") {
return false
}
var re strings.Builder
re.WriteString("^")
for i := 0; i < len(rule); i++ {
c := rule[i]
switch {
case c == '*' && i+1 < len(rule) && rule[i+1] == '*':
re.WriteString(".*")
i++
case c == '*':
re.WriteString("[^/]*")
case c == '?':
re.WriteString("[^/]")
default:
re.WriteString(regexp.QuoteMeta(string(c)))
}
}
re.WriteString("$")
ok, err := regexp.MatchString(re.String(), branch)
return err == nil && ok
}
// Branch is one branch with the commit it points at.
type Branch struct {
Name string `json:"name"`
@@ -106,7 +249,7 @@ func (f *Forge) branch(ctx context.Context, repo, name string) (*Branch, error)
return nil, err
}
var b Branch
if err := f.GetJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
if err := f.getJSON(ctx, repo+"/branches/"+seg, &b); err != nil {
if errors.Is(err, errNotFound) {
return nil, nil
}
+14 -175
View File
@@ -13,12 +13,9 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"testing"
"gitea/internal/forgehttp"
)
// fakeForge is the forge's HTTP interface as far as these tools use it.
@@ -40,16 +37,6 @@ type fakeForge struct {
comments map[string][]map[string]any
commits []map[string]any
dir []map[string]any
// moves: a branch moved by someone else right after it is first read — to a commit, with its tree.
moves map[string]move
noParents bool // the forge's answer to a write does not say the parents
parentsBroken bool // and GET git/commits fails
parents map[string]string // commit -> parent, as GET git/commits says it
}
type move struct {
to string
tree map[string]string
}
func newFake(t *testing.T) *fakeForge {
@@ -75,7 +62,7 @@ func (f *fakeForge) serve() *Forge {
}
srv := httptest.NewServer(http.HandlerFunc(f.handle))
f.t.Cleanup(srv.Close)
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}}
return &Forge{URL: srv.URL, User: "mesh-admin", PasswordFile: pw}
}
func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
@@ -105,11 +92,6 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
return
}
reply(map[string]any{"name": name, "commit": map[string]any{"id": c}})
if m, ok := f.moves[name]; ok {
delete(f.moves, name)
f.trees[m.to] = m.tree
f.branches[name] = m.to
}
case r.Method == "GET" && (path == repo+"/contents" || path == repo+"/contents/docs"):
reply(f.dir)
case r.Method == "GET" && strings.HasPrefix(path, repo+"/contents/"):
@@ -166,28 +148,19 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
if !f.staleRead {
f.branches[target] = made
}
reply(map[string]any{"commit": f.made(made, from)})
reply(map[string]any{"commit": map[string]any{"sha": made}})
case r.Method == "POST" && path == repo+"/diffpatch":
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
f.patches = append(f.patches, body)
f.next++
made := fmt.Sprintf("p%d", f.next)
from := f.branches[body["branch"].(string)]
target := body["branch"].(string)
if nb, ok := body["new_branch"].(string); ok && nb != "" {
target = nb
}
f.branches[target] = made
reply(map[string]any{"commit": f.made(made, from)})
case r.Method == "GET" && strings.HasPrefix(path, repo+"/git/commits/"):
sha := strings.TrimPrefix(path, repo+"/git/commits/")
p, ok := f.parents[sha]
if !ok || f.parentsBroken {
http.NotFound(w, r)
return
}
reply(map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": p}}})
reply(map[string]any{"commit": map[string]any{"sha": made}})
case r.Method == "GET" && path == repo+"/commits":
f.writes = append(f.writes, map[string]any{"query": r.URL.RawQuery})
reply(f.commits)
@@ -207,18 +180,6 @@ func (f *fakeForge) handle(w http.ResponseWriter, r *http.Request) {
}
}
// made is the forge's answer about a commit it made on a parent, with the parents unless noParents.
func (f *fakeForge) made(sha, parent string) map[string]any {
if f.parents == nil {
f.parents = map[string]string{}
}
f.parents[sha] = parent
if f.noParents {
return map[string]any{"sha": sha}
}
return map[string]any{"sha": sha, "parents": []any{map[string]any{"sha": parent}}}
}
func filesArg(files ...map[string]any) []any {
out := make([]any, len(files))
for i, f := range files {
@@ -398,27 +359,17 @@ func TestACommitAskThatCannotBeReadIsRefused(t *testing.T) {
}
}
// The forge's own answers (Gitea 1.27's Match), one table for this bundle and guard.ts alike (novox/hq issue 539).
// `forge` is the forge's answer, which this port must give; `guard` is where guard.ts deliberately covers more.
func TestRuleCoversAnswersAsTheForgeDoes(t *testing.T) {
raw, err := os.ReadFile("../../test/rule-cases.json")
if err != nil {
t.Fatal(err)
}
var cases []struct {
Rule string `json:"rule"`
Branch string `json:"branch"`
Forge bool `json:"forge"`
}
if err := json.Unmarshal(raw, &cases); err != nil {
t.Fatal(err)
}
if len(cases) < 100 {
t.Fatalf("only %d cases", len(cases))
}
for _, c := range cases {
if got := ruleCovers(c.Rule, c.Branch); got != c.Forge {
t.Errorf("%q covers %q: %v, the forge says %v", c.Rule, c.Branch, got, c.Forge)
func TestRuleCoversIsTheGuardsRule(t *testing.T) {
for _, c := range []struct {
rule, branch string
want bool
}{
{"main", "main", true}, {"main", "mainx", false}, {"release/*", "release/1.0", true},
{"release/*", "release/a/b", false}, {"release/**", "release/a/b", true}, {"v?", "v1", true},
{"a.b", "axb", false},
} {
if got := ruleCovers(c.rule, c.branch); got != c.want {
t.Errorf("%s covers %s: %v", c.rule, c.branch, got)
}
}
}
@@ -630,115 +581,3 @@ func TestToolsAreTheOnesTheReadmeNamesAndTheManifestBuilds(t *testing.T) {
t.Error("the manifest builds no cmd/forge bundle")
}
}
// novox/hq issue 539: the forge takes a branch, never a commit to build on, so the answer's parent is the one
// the forge made the commit on, and a branch that moves between the read and the commit is said.
func TestAnExistingBranchMovedUnderTheCommitIsSaidAndItsParentIsTheOneTheForgeUsed(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "a on fix\n", "other.md": "x\n"}}}
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "a fixed\n"})}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c2" || f.branches["fix/1-old"] != got.Commit {
t.Fatalf("parent %s, want c2 (the commit the forge built on): %+v", got.Parent, got)
}
said := strings.Join(got.Said, " ")
if !strings.Contains(said, "moved from c1 to c2") || !strings.Contains(said, "after c2") {
t.Fatalf("the move was not said: %v", got.Said)
}
}
func TestAnExistingBranchsFilesAreJudgedAtTheCommitReadNotAtWhereTheBranchMovedTo(t *testing.T) {
f := newFake(t)
// Someone else changes docs/a.md on the branch between the read and the commit: that change is not
// overwritten by a blob read at the branch's new head.
f.moves = map[string]move{"fix/1-old": {to: "c2", tree: map[string]string{"README.md": "hello\n", "docs/a.md": "theirs\n"}}}
_, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/1-old",
"files": filesArg(map[string]any{"path": "docs/a.md", "content": "mine\n"})}))
if err == nil || !strings.Contains(err.Error(), "sha does not match") {
t.Fatalf("a file changed under the commit was overwritten: %v", err)
}
if f.trees["c2"]["docs/a.md"] != "theirs\n" || f.branches["fix/1-old"] != "c2" {
t.Fatalf("the branch was changed: %v", f.branches)
}
}
func TestAPatchOnABaseThatMovedNamesTheCommitTheForgeAppliedItTo(t *testing.T) {
f := newFake(t)
f.moves = map[string]move{"main": {to: "c9", tree: map[string]string{"README.md": "hello\n"}}}
patch := "diff --git a/README.md b/README.md\n--- a/README.md\n+++ b/README.md\n@@ -1 +1 @@\n-hello\n+hi\n"
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/6", "patch": patch}))
if err != nil {
t.Fatal(err)
}
if got.Parent != "c9" || !strings.Contains(strings.Join(got.Said, " "), "base main moved from c0 to c9") {
t.Fatalf("%+v", got)
}
}
func TestAParentTheAnswerDoesNotSayIsReadFromTheCommit(t *testing.T) {
f := newFake(t)
f.noParents = true
got, err := commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/7",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err != nil || got.Parent != "c0" {
t.Fatalf("%v %+v", err, got)
}
// A parent that cannot be read is a failure, said with the commit that was made.
f = newFake(t)
f.noParents, f.parentsBroken = true, true
_, err = commit(context.Background(), f.serve(), ask(t, map[string]any{"branch": "fix/8",
"files": filesArg(map[string]any{"path": "x", "content": "x"})}))
if err == nil || !strings.Contains(err.Error(), "the forge made n1 on fix/8, but which commit it went on could not be read") {
t.Fatalf("%v", err)
}
}
func TestACommitLargerThanItsLimitsIsRefusedBeforeAnythingIsSent(t *testing.T) {
big := strings.Repeat("x", maxFileBytes+1)
quarter := strings.Repeat("y", maxFileBytes)
many := filesArg()
for i := 0; i < maxCommitBytes/maxFileBytes+1; i++ {
many = append(many, map[string]any{"path": fmt.Sprintf("f%d", i), "content": quarter})
}
for name, args := range map[string]map[string]any{
"one file": {"files": filesArg(map[string]any{"path": "x", "content": big})},
"one file b64": {"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(big))})},
"the patch": {"patch": "diff --git a/x b/x\n" + big},
"all files": {"files": many},
} {
full := map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b"}
for k, v := range args {
full[k] = v
}
_, err := readCommitAsk(full)
if err == nil || !strings.Contains(err.Error(), "MiB") || !strings.Contains(err.Error(), "Nothing was sent") {
t.Errorf("%s: %v", name, err)
}
}
// At the limit is taken.
if _, err := readCommitAsk(map[string]any{"owner": "acme", "repo": "app", "message": "m", "branch": "b",
"files": filesArg(map[string]any{"path": "x", "content_base64": base64.StdEncoding.EncodeToString([]byte(quarter))})}); err != nil {
t.Errorf("a file at the limit was refused: %v", err)
}
}
func TestCodeSearchStopsAtItsDeadlineAndSaysSo(t *testing.T) {
gz := archive(t, map[string]string{"a.md": "forge\n", "b.md": "forge\n"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
out := &Found{Matches: []Match{}}
if err := searchArchive(ctx, gz, "", regexp.MustCompile("forge"), 100, out); err != nil {
t.Fatal(err)
}
if !out.Stopped || out.Files != 0 {
t.Fatalf("a search past its deadline went on: %+v", out)
}
// Within its deadline, the same archive is searched whole.
out = &Found{Matches: []Match{}}
if err := searchArchive(context.Background(), gz, "", regexp.MustCompile("forge"), 100, out); err != nil || out.Stopped || len(out.Matches) != 2 {
t.Fatalf("%v %+v", err, out)
}
}
+1 -3
View File
@@ -53,9 +53,7 @@ func Tools(forge func() (*Forge, error)) []stdio.Tool {
"independent review approved and whose required checks passed (gitea_merge_pull_request). Carries " +
"either files — each {path, content (text) | content_base64 | delete: true}, created or updated as " +
"the branch holds them — or patch, a unified diff as git diff writes it, which the forge applies. " +
"At most 200 files, 4 MiB each and 16 MiB together, or a 4 MiB patch. Answers the commit it made and " +
"the parent the forge made it on, read back from the branch, and says when the branch moved under " +
"it. Replaces git commit and git " +
"Answers the commit it made and its parent, read back from the branch. Replaces git commit and git " +
"push of a feature branch. (a)",
Input: map[string]any{
"type": "object",
+6 -6
View File
@@ -72,7 +72,7 @@ func history(ctx context.Context, f *Forge, owner, name, ref, path string, limit
q.Set("verification", "false")
q.Set("files", "false")
var got []forgeCommit
if err := f.GetJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
if err := f.getJSON(ctx, repo+"/commits?"+q.Encode(), &got); err != nil {
return nil, err
}
h := &History{Repository: owner + "/" + name, Ref: ref, Path: path, Page: page, Commits: []CommitLine{}, More: len(got) >= limit}
@@ -121,7 +121,7 @@ func compare(ctx context.Context, f *Forge, owner, name, base, head string) (*Co
Total int `json:"total_commits"`
Commits []forgeCommit `json:"commits"`
}
if err := f.GetJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
if err := f.getJSON(ctx, repo+"/compare/"+b+"..."+h, &got); err != nil {
return nil, err
}
c := &Comparison{Repository: owner + "/" + name, Base: base, Head: head, Total: got.Total, Commits: []CommitLine{}, Files: []FileLine{}}
@@ -195,7 +195,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
SHA string `json:"sha"`
} `json:"head"`
}
if err := f.GetJSON(ctx, pr, &pull); err != nil {
if err := f.getJSON(ctx, pr, &pull); err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no pull request %d", owner, name, number)
}
@@ -218,7 +218,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
var all []forgeReview
for p := 1; ; p++ {
var batch []forgeReview
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
if err := f.getJSON(ctx, fmt.Sprintf("%s/reviews?limit=%d&page=%d", pr, page, p), &batch); err != nil {
return nil, err
}
all = append(all, batch...)
@@ -240,7 +240,7 @@ func reviews(ctx context.Context, f *Forge, owner, name string, number int) (*Re
CommitID string `json:"commit_id"`
Resolver *forgeUser `json:"resolver"`
}
if err := f.GetJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
if err := f.getJSON(ctx, fmt.Sprintf("%s/reviews/%d/comments", pr, r.ID), &cs); err != nil {
return nil, fmt.Errorf("review %d's line comments could not be read: %w", r.ID, err)
}
for _, c := range cs {
@@ -292,7 +292,7 @@ func listDir(ctx context.Context, f *Forge, owner, name, path, ref string) (*Lis
if ref != "" {
at += "?ref=" + queryEscape(ref)
}
body, err := f.Do(ctx, "GET", at, nil, maxBody)
body, err := f.do(ctx, "GET", at, nil, maxBody)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no %q at %s", owner, name, path, orDefault(ref))
-116
View File
@@ -1,116 +0,0 @@
package main
import (
"errors"
"regexp"
"strings"
)
// ruleCovers is whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
// models/git/protected_branch.go Match and modules/glob, novox/hq issue 539): a name without a glob character
// covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any
// run within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]`
// one character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge
// cannot compile covers its own text exactly. The TypeScript bundle's ruleCovers (guard.ts) is the same rule.
func ruleCovers(rule, branch string) bool {
if !strings.ContainsAny(rule, `*?\[]{}`) {
return strings.EqualFold(rule, branch)
}
re, err := globRegexp(rule)
if err != nil {
return rule == branch
}
return re.MatchString(branch)
}
// globRegexp is the forge's glob as a regular expression, compiled as modules/glob compiles it with `/` as
// the separator.
func globRegexp(glob string) (*regexp.Regexp, error) {
g := []rune(glob)
var b strings.Builder
pos := 0
var compile func(sub bool) error
chars := func() error {
b.WriteByte('[')
if pos < len(g) && g[pos] == '!' {
pos++
b.WriteByte('^')
}
for pos < len(g) {
c := g[pos]
pos++
switch {
case c == ']':
b.WriteByte(']')
return nil
case c == '\\':
if pos >= len(g) {
return errors.New("unterminated character class escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
default:
b.WriteRune(c)
}
}
return errors.New("unterminated character class")
}
compile = func(sub bool) error {
for pos < len(g) {
c := g[pos]
pos++
if sub && c == '}' {
b.WriteByte(')')
return nil
}
switch c {
case '*':
if pos < len(g) && g[pos] == '*' {
pos++
b.WriteString(".*")
} else {
b.WriteString("[^/]*")
}
case '?':
b.WriteString("[^/]")
case '+':
b.WriteString(`\+`)
case '[':
if err := chars(); err != nil {
return err
}
case '{':
b.WriteByte('(')
if err := compile(true); err != nil {
return err
}
case ',':
if sub {
b.WriteByte('|')
} else {
b.WriteByte(',')
}
case '\\':
if pos >= len(g) {
return errors.New("no character to escape")
}
b.WriteByte('\\')
b.WriteRune(g[pos])
pos++
case '.', '^', '$', '(', ')', '|':
b.WriteByte('\\')
b.WriteRune(c)
default:
b.WriteRune(c)
}
}
return nil
}
b.WriteByte('^')
if err := compile(false); err != nil {
return nil, err
}
b.WriteByte('$')
return regexp.Compile(b.String())
}
+7 -42
View File
@@ -32,7 +32,6 @@ type Found struct {
Files int `json:"files_searched"`
Skipped []string `json:"skipped,omitempty"`
Truncated bool `json:"truncated"`
Stopped bool `json:"stopped_at_deadline,omitempty"`
Said []string `json:"said"`
}
@@ -70,7 +69,7 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
ref := a.Ref
if ref == "" {
var r Repo
if err := f.GetJSON(ctx, repo, &r); err != nil {
if err := f.getJSON(ctx, repo, &r); err != nil {
return nil, err
}
if r.Empty || r.DefaultBranch == "" {
@@ -88,7 +87,7 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
return nil, err
}
}
body, err := f.Do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
body, err := f.do(ctx, "GET", repo+"/archive/"+seg+".tar.gz", nil, maxArchive)
if err != nil {
if errors.Is(err, errNotFound) {
return nil, refuse("%s/%s has no ref %s", a.Owner, a.Repo, ref)
@@ -96,15 +95,11 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
return nil, err
}
out := &Found{Repository: a.Owner + "/" + a.Repo, Ref: ref, Query: a.Query, Matches: []Match{}}
if err := searchArchive(ctx, body, prefix, re, a.Max, out); err != nil {
if err := searchArchive(body, prefix, re, a.Max, out); err != nil {
return nil, fmt.Errorf("the archive of %s at %s could not be read: %w", out.Repository, ref, err)
}
said := fmt.Sprintf("%d matches in %d files searched", len(out.Matches), out.Files)
switch {
case out.Stopped:
said = fmt.Sprintf("the search stopped at its deadline of %s after %d files, with %d matches: the rest of the "+
"archive was not searched. Narrow with path", deadline, out.Files, len(out.Matches))
case out.Truncated:
if out.Truncated {
said = fmt.Sprintf("the first %d matches, and more were found: narrow with path or the query", len(out.Matches))
}
out.Said = append(out.Said, said)
@@ -115,32 +110,18 @@ func searchCode(ctx context.Context, f *Forge, a SearchAsk) (*Found, error) {
}
// searchArchive reads a gzipped tar the forge made — every path under one top directory — and collects the
// lines that match, from text files under the prefix. It stops at the call's deadline (novox/hq issue 539):
// what it found by then is kept, and out.Stopped says the rest was not searched.
func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
// lines that match, from text files under the prefix.
func searchArchive(gz []byte, prefix string, re *regexp.Regexp, max int, out *Found) error {
zr, err := gzip.NewReader(bytes.NewReader(gz))
if err != nil {
return err
}
tr := tar.NewReader(ctxReader{ctx, zr})
stopped := func(err error) bool {
if ctx.Err() != nil && (err == nil || errors.Is(err, ctx.Err())) {
out.Stopped = true
return true
}
return false
}
tr := tar.NewReader(zr)
for {
if stopped(nil) {
return nil
}
h, err := tr.Next()
if err == io.EOF {
return nil
}
if stopped(err) {
return nil
}
if err != nil {
return err
}
@@ -159,9 +140,6 @@ func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Reg
continue
}
content, err := io.ReadAll(io.LimitReader(tr, maxFileSearched+1))
if stopped(err) {
return nil
}
if err != nil {
return err
}
@@ -191,16 +169,3 @@ func searchArchive(ctx context.Context, gz []byte, prefix string, re *regexp.Reg
}
}
}
// ctxReader is a reader that ends when its context does: decompressing is the long part of a search.
type ctxReader struct {
ctx context.Context
r io.Reader
}
func (c ctxReader) Read(p []byte) (int, error) {
if err := c.ctx.Err(); err != nil {
return 0, err
}
return c.r.Read(p)
}
+104 -20
View File
@@ -2,33 +2,45 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"time"
"gitea/internal/forgehttp"
)
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin account
// (internal/forgehttp, shared with the forge bundle beside it). Basic authentication, so this bundle mints
// nothing and keeps nothing: the TypeScript bundle's token is its own.
// The forge as this bundle reaches it: its own HTTP interface on the machine, as the forge's admin
// account, which the vault delivered into a file this module owns (novox/hq ADR 0086). Basic
// authentication, so this bundle mints nothing and keeps nothing: the TypeScript bundle's token is its
// own. The password is read when a call is made and never logged, answered or put in an error.
// Forge is the forge's HTTP interface.
type Forge struct {
forgehttp.Client
Owner string // the package registry's owner, as the seat serves it
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
Owner string // the package registry's owner, as the seat serves it
HTTP *http.Client
}
// ForgeFromEnv is the forge as the manifest hands it to this bundle.
func ForgeFromEnv() (*Forge, error) {
c, missing := forgehttp.FromEnv()
f := &Forge{Client: c, Owner: os.Getenv("MESH_NPM_OWNER")}
if f.Owner == "" {
missing = append(missing, "MESH_NPM_OWNER")
f := &Forge{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
Owner: os.Getenv("MESH_NPM_OWNER"),
}
var missing []string
for name, v := range map[string]string{"MESH_GITEA_URL": f.URL, "MESH_GITEA_ADMIN_USER": f.User,
"MESH_GITEA_ADMIN_PASSWORD_FILE": f.PasswordFile, "MESH_NPM_OWNER": f.Owner} {
if v == "" {
missing = append(missing, name)
}
}
if len(missing) > 0 {
return nil, fmt.Errorf("the forge cannot be reached: %s not set by the manifest", strings.Join(missing, ", "))
@@ -37,14 +49,86 @@ func ForgeFromEnv() (*Forge, error) {
}
// errNotFound is the forge answering 404.
var errNotFound = forgehttp.ErrNotFound
var errNotFound = errors.New("not found")
// statusError is an answer the forge gave that is not a success, said without the request's credential.
type statusError struct {
method, path string
status int
said string
}
func (e *statusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.status, e.method, e.path, e.said)
}
func (f *Forge) client() *http.Client {
if f.HTTP != nil {
return f.HTTP
}
return &http.Client{Timeout: 20 * time.Second}
}
func (f *Forge) password() (string, error) {
raw, err := os.ReadFile(f.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
// maxBody is the largest answer read from the forge.
const maxBody = forgehttp.MaxBody
const maxBody = 32 << 20
// do sends one request without a body and answers the body; 404 is errNotFound.
// do sends one request and answers the body; 404 is errNotFound.
func (f *Forge) do(ctx context.Context, method, path string) ([]byte, error) {
return f.Do(ctx, method, path, nil, maxBody)
password, err := f.password()
if err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, method, f.URL+path, nil)
if err != nil {
return nil, err
}
req.SetBasicAuth(f.User, password)
req.Header.Set("Accept", "application/json")
res, err := f.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
// One byte past the cap is read, so a body the cap cut is told from one that fits it exactly: a
// cut body is refused, never parsed as if it were whole.
body, err := io.ReadAll(io.LimitReader(res.Body, maxBody+1))
if err != nil {
return nil, err
}
if len(body) > maxBody {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, maxBody>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, errNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(body))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &statusError{method, path, res.StatusCode, said}
}
return body, nil
}
func (f *Forge) getJSON(ctx context.Context, path string, into any) error {
body, err := f.do(ctx, http.MethodGet, path)
if err != nil {
return err
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s is not readable: %w", path, err)
}
return nil
}
// PackageVersion is one version of one package, as the forge lists it.
@@ -62,7 +146,7 @@ func (f *Forge) Packages(ctx context.Context) ([]PackageVersion, error) {
for page := 1; page <= 1000; page++ {
var batch []PackageVersion
path := fmt.Sprintf("/api/v1/packages/%s?type=npm&page=%d&limit=%d", url.PathEscape(f.Owner), page, pageSize)
if err := f.GetJSON(ctx, path, &batch); err != nil {
if err := f.getJSON(ctx, path, &batch); err != nil {
return nil, err
}
all = append(all, batch...)
@@ -82,7 +166,7 @@ func (f *Forge) Size(ctx context.Context, name, version string) (int64, error) {
var files []struct {
Size int64 `json:"size"`
}
if err := f.GetJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
if err := f.getJSON(ctx, f.versionPath(name, version)+"/files", &files); err != nil {
return 0, err
}
var total int64
@@ -98,7 +182,7 @@ func (f *Forge) DistTags(ctx context.Context, name string) (map[string]string, e
DistTags map[string]string `json:"dist-tags"`
}
path := fmt.Sprintf("/api/packages/%s/npm/%s", url.PathEscape(f.Owner), url.PathEscape(name))
if err := f.GetJSON(ctx, path, &meta); err != nil {
if err := f.getJSON(ctx, path, &meta); err != nil {
return nil, err
}
return meta.DistTags, nil
@@ -124,7 +208,7 @@ func (f *Forge) Repositories(ctx context.Context) ([]Repository, error) {
var found struct {
Data []Repository `json:"data"`
}
if err := f.GetJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
if err := f.getJSON(ctx, fmt.Sprintf("/api/v1/repos/search?page=%d&limit=%d", page, pageSize), &found); err != nil {
return nil, err
}
all = append(all, found.Data...)
@@ -155,7 +239,7 @@ func (f *Forge) Files(ctx context.Context, repo Repository) ([]string, error) {
}
path := fmt.Sprintf("/api/v1/repos/%s/git/trees/%s?recursive=true&page=%d&per_page=1000",
repoPath(repo.FullName), url.PathEscape(repo.DefaultBranch), page)
if err := f.GetJSON(ctx, path, &tree); err != nil {
if err := f.getJSON(ctx, path, &tree); err != nil {
return nil, err
}
for _, e := range tree.Tree {
@@ -15,8 +15,6 @@ import (
"sync"
"testing"
"time"
"gitea/internal/forgehttp"
)
// fakeForge answers the forge's interface from maps, and records every delete.
@@ -40,7 +38,7 @@ func (f *fakeForge) serve() *Forge {
if err := os.WriteFile(file, []byte(f.password+"\n"), 0o600); err != nil {
f.t.Fatal(err)
}
return &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
return &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
}
var (
@@ -503,7 +501,7 @@ func TestAnAnswerTheCapCutIsRefused(t *testing.T) {
if err := os.WriteFile(file, []byte("x\n"), 0o600); err != nil {
t.Fatal(err)
}
f := &Forge{Client: forgehttp.Client{URL: srv.URL, User: "admin", PasswordFile: file}, Owner: "acme"}
f := &Forge{URL: srv.URL, User: "admin", PasswordFile: file, Owner: "acme"}
if _, err := f.do(context.Background(), http.MethodGet, "/big"); err == nil || !strings.Contains(err.Error(), "not read whole") {
t.Fatalf("a cut answer: %v", err)
}
+1 -23
View File
@@ -18,7 +18,7 @@
// restart, when the check was opened is read from the forge's own history of the gate.
import { CHECK_CONTEXT, pendingWords } from "./pulls.js";
import type { CommitStatus, HeadCheck } from "./client.js";
import type { CommitStatus } from "./client.js";
/** The build seat's `started`, as mesh-controller's link.BuildStart publishes it. */
export interface BuildStarted {
@@ -37,8 +37,6 @@ export interface GateForge {
gateOf(owner: string, repo: string, sha: string): Promise<{ state: string; updated_at?: string } | undefined>;
/** The head's mesh/merge-gate statuses, every one the forge keeps, newest first (novox/hq issue 456). */
gateHistory(owner: string, repo: string, sha: string): Promise<GateStatus[]>;
/** The head's statuses by context, the newest of each, with their words (novox/hq ADR 0323). */
headChecks(owner: string, repo: string, sha: string): Promise<Record<string, HeadCheck>>;
}
/** One status of a head's gate as the forge keeps it: its state, its words, and when it was set. */
@@ -176,26 +174,6 @@ export class GateWriter {
});
}
/**
* **A trunk's move puts the head's verdict out of date** (novox/hq ADR 0323 §4): its merge check statuses are put back
* to pending, with the reason, in their turn after every write of the head already asked for — a verdict in flight
* lands first. What is written is decided in the turn, from the statuses the forge holds then (`decide`, trunk.ts's
* outOfDateStatuses): a check on its way, or a verdict already judged against the trunk's head, is left as it is. It
* opens no check — nothing is asked — so a verdict of a check asked before it keeps its weight. Answers what it wrote;
* a write that fails fails the caller, and the move is weighed again on the next pass.
*/
outOfDate(owner: string, repo: string, sha: string,
decide: (checks: Record<string, HeadCheck>) => CommitStatus[] | Promise<CommitStatus[]>, target?: string): Promise<CommitStatus[]> {
const h = this.head(owner, repo, sha);
const written: CommitStatus[] = [];
return this.turn(h, async () => {
for (const s of await decide(await this.forge.headChecks(owner, repo, sha))) {
await this.forge.setCommitStatus(owner, repo, sha, target ? { ...s, target_url: target } : s);
written.push(s);
}
}).then(() => written);
}
/**
* A build machine took an ask of this head: said while no verdict was heard, and only for the check now pending —
* a start from before the pending words were set (a run before a recheck, the stream's history) is not this check's.
+13 -98
View File
@@ -164,106 +164,21 @@ export function judgeApiCall(method: string, rawPath: string, body: unknown): Ap
};
}
/**
* Whether a protection rule's name covers a branch, as the forge itself decides it (Gitea 1.27,
* models/git/protected_branch.go Match and modules/glob; novox/hq issue 539): a name without a glob character
* covers the branch of that name in any case; any other name is a glob with `/` as its separator — `*` any run
* within a segment, `**` any run across them, `?` one character of a segment, `[abc]`, `[a-z]` and `[!abc]` one
* character, `{a,b}` either, `\` the next character itself — matched case-sensitively; a glob the forge cannot
* compile covers its own text exactly. The forge bundle's ruleCovers (cmd/forge/rules.go) is the same rule, and
* both are held to one table of answers (test/rule-cases.json).
*
* The forge hands the compiled glob to Go's regexp, which reads some things JavaScript's reads otherwise: a
* backslash before a letter or digit (`\d`, `\w`, `\pL`, `\b`, `\A` are classes or anchors there), a class
* that opens with `]` (`[]a]` holds `]` and `a` there, an empty class here) or with `^` (a negation there),
* and a POSIX class (`[[:digit:]]`).
* A rule using any of them is taken to cover every branch: refusing a write the forge would take is safe,
* taking one it guards is not.
*/
/** Whether a protection rule's name — a branch, or a glob of them (`*`, `**`) — covers a branch. */
export function ruleCovers(rule: string, branch: string): boolean {
if (!/[*?\\[\]{}]/.test(rule)) return rule.toLowerCase() === branch.toLowerCase();
if (goReadsOtherwise(rule)) return true;
const re = globRegExp(rule);
return re ? re.test(branch) : rule === branch;
}
/** Whether a glob holds what Go's regexp reads otherwise than JavaScript's (see ruleCovers): fail closed. */
function goReadsOtherwise(rule: string): boolean {
return /\\[\p{L}\p{N}]/u.test(rule) || /\[!?[\]^]/.test(rule) || /\[[^\]]*\[:/.test(rule);
}
/** The forge's glob as a regular expression, compiled as its modules/glob compiles it with `/` as separator. */
function globRegExp(glob: string): RegExp | null {
const g = Array.from(glob);
let out = "";
let pos = 0;
// A character as itself: escaped when it is syntax to a regular expression, and only then — the `u` flag
// refuses any other escape, and `-` may be escaped only inside a class.
const lit = (c: string) => c.replace(/[.*+?^${}()|[\]\\/]/g, "\\$&");
const litInClass = (c: string) => c.replace(/[\]\\[^-]/g, "\\$&");
const chars = (): boolean => {
out += "[";
if (g[pos] === "!") {
pos++;
out += "^";
}
while (pos < g.length) {
const c = g[pos++];
if (c === "]") {
out += "]";
return true;
}
if (c === "\\") {
if (pos >= g.length) return false;
out += litInClass(g[pos++]);
} else out += c === "[" || c === "^" ? "\\" + c : c;
}
return false;
};
const compile = (sub: boolean): boolean => {
while (pos < g.length) {
const c = g[pos++];
if (sub && c === "}") {
out += ")";
return true;
}
switch (c) {
case "*":
if (g[pos] === "*") {
pos++;
out += ".*";
} else out += "[^/]*";
break;
case "?":
out += "[^/]";
break;
case "[":
if (!chars()) return false;
break;
case "{":
out += "(";
if (!compile(true)) return false;
break;
case ",":
out += sub ? "|" : ",";
break;
case "\\":
if (pos >= g.length) return false;
out += lit(g[pos++]);
break;
default:
out += lit(c);
}
}
// An alternative left open is a glob the forge cannot compile.
return !sub;
};
if (!compile(false)) return null;
try {
return new RegExp(`^${out}$`, "u");
} catch {
return null;
if (rule === branch) return true;
if (!/[*?[]/.test(rule)) return false;
let re = "";
for (let i = 0; i < rule.length; i++) {
const c = rule[i];
if (c === "*" && rule[i + 1] === "*") {
re += ".*";
i++;
} else if (c === "*") re += "[^/]*";
else if (c === "?") re += "[^/]";
else re += c.replace(/[.+^${}()|[\]\\]/g, "\\$&");
}
return new RegExp(`^${re}$`).test(branch);
}
/** Refuses a file written to the default branch or to a branch a protection rule names. */
+35 -71
View File
@@ -17,13 +17,8 @@
// Consumes node-build-agent.started — a build machine took an ask — and says, on the head a pending check is of,
// which machine builds it since when (novox/hq issue 446). This module is the one writer of mesh/merge-gate.
//
// **A trunk's move puts every open verdict into it out of date** (novox/hq ADR 0323 §4, trunk.ts): each pass reads the
// head of every trunk an open pull request merges into, and when one moved, the merge check statuses of every open pull
// request into it go back to pending with the reason. Nothing is run for it. A verdict that lands judged against a trunk
// commit that is no longer the trunk's head is said out of date in place of its pass or failure.
//
// Serves gitea_commit_status and gitea_merge_pull_request from this bundle, not the tools', so a recheck's pending and
// the merge tool's ask of a check reach that one writer (novox/hq issue 456, ADR 0323).
// Serves gitea_commit_status from this bundle, not the tools', so a recheck's pending reaches that one writer
// (novox/hq issue 456).
//
// Every pull request the forge holds is announced, whatever its repository: the controller holds the
// module graph and decides what is checked — a repository is never asked to opt in.
@@ -39,12 +34,10 @@
import { emit, on } from "@novox/mesh-sdk/events";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { commitStatusTool } from "./delivery.js";
import { mergeTool } from "./merge.js";
import { GiteaClient, movedSince } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, commentFor, headsToAnnounce, statusesFor, type Announced, type Checked,
type RepoCheckFacts, reposToLook } from "./pulls.js";
import { GateWriter, clonedAs, type BuildStarted } from "./gate.js";
import { announcementOf, keepTook, lateVerdictComment, outOfDateWords, sameCommit, tookFile, weighTrunks } from "./trunk.js";
// Without a way to a token — configured, or mintable with the admin account (token.ts) — there is
// nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints
@@ -252,15 +245,42 @@ async function pollPulls(client: GiteaClient): Promise<void> {
}
if (sweep) lastSweep = Date.now();
for (const repo of reposToLook(await client.listAllRepos(), floor, updated, sweep)) {
// Every open pull request updated since the floor, paged (novox/hq issue 579): the newest 20 alone left a head
// beyond them unannounced. A sweep and the first pass read every page.
const open = await client.listOpenPullsSince(repo.owner, repo.name, sweep ? "" : floor);
const open = await client.listPullRequests(repo.owner, repo.name, { state: "open", sort: "recentupdate", limit: "20" });
for (const pull of headsToAnnounce(repo.full_name, open, heads)) {
const key = `${repo.full_name}#${pull.number}`;
const fresh = primedPulls || (!!pull.updated_at && pull.updated_at > dayAgo);
if (fresh) {
// The announcement the merge tool makes too when it asks a check (trunk.ts, novox/hq ADR 0323 §5).
await emit("pull.updated", await announcementOf(client, repo.owner, repo.name, repo.clone_url, pull));
const files = await client.listPullFiles(repo.owner, repo.name, pull.number);
const head = String(pull.head_sha);
// What the controller maps the change onto the mesh's module graph with (novox/hq ADR 0237 as
// amended): which changed directories hold a module at the head — the merge's rule (issue 278) —
// and whether the head holds the repository's own merge-check.sh. Not said when it could not be
// read; the controller then reads the change as touching everything built from the repository.
let moduleDirs: string[] | null = null;
let mergeCheck: boolean | null = null;
try {
if (!files.truncated) moduleDirs = await client.moduleDirsAt(repo.owner, repo.name, head, files.paths);
mergeCheck = await client.holdsFile(repo.owner, repo.name, head, "merge-check.sh");
} catch (err) {
console.error(`[gitea] ${key}: what the head holds could not be read — ${err instanceof Error ? err.message : err}`);
}
await emit("pull.updated", {
owner: repo.owner,
repo: repo.name,
number: pull.number,
title: pull.title,
body: pull.body,
base: pull.base,
head: pull.head,
head_sha: pull.head_sha,
clone_url: repo.clone_url,
html_url: pull.html_url,
paths: files.paths,
paths_truncated: files.truncated,
removed: files.removed,
...(moduleDirs ? { module_dirs: moduleDirs, module_dirs_said: true } : {}),
...(mergeCheck !== null ? { merge_check: mergeCheck, merge_check_said: true } : {}),
});
// Said, so an operator knows the mesh was asked to check it.
console.log(`[gitea] announced ${key} at ${String(pull.head_sha).slice(0, 8)} to be checked before it merges`);
// Pending until the verdict comes, so the pull request says a check is running rather than nothing — and
@@ -270,52 +290,17 @@ async function pollPulls(client: GiteaClient): Promise<void> {
heads[key] = String(pull.head_sha);
changed = true;
}
if (await trunksMoved(client, repo.owner, repo.name)) changedTrunks = true;
}
if (!primedPulls || changed) keepHeads();
if (changedTrunks) keepTrunks();
changedTrunks = false;
primedPulls = true;
lastPullLook = began;
}
// **The trunks' heads, as this module last read them** (novox/hq ADR 0323 §4), by `owner/repo:branch`: kept beside the
// other records, so a move made while this module was away is weighed on its first pass back. With no record — the
// first pass after this rolls out — a trunk's head is taken as it is and nothing is put out of date: the verdicts
// then standing are weighed on the trunk's next move, and the merge tool compares commits whatever this holds.
const trunksRecord = process.env.MESH_GITEA_STATE_DIR ? join(process.env.MESH_GITEA_STATE_DIR, "trunks-seen.json") : null;
let trunks: Record<string, string> = {};
let changedTrunks = false;
if (trunksRecord && existsSync(trunksRecord)) {
try {
trunks = (JSON.parse(readFileSync(trunksRecord, "utf8")) as { trunks: Record<string, string> }).trunks ?? {};
} catch {
// An unreadable record is no record: the next pass takes each trunk's head as it is.
}
}
function keepTrunks(): void {
if (!trunksRecord) return;
mkdirSync(join(trunksRecord, ".."), { recursive: true });
const tmp = trunksRecord + ".tmp";
writeFileSync(tmp, JSON.stringify({ trunks }));
renameSync(tmp, trunksRecord);
}
/** Whether a trunk of one repository moved, weighed with this module's forge and its one writer (trunk.ts). */
function trunksMoved(client: GiteaClient, owner: string, name: string): Promise<boolean> {
return weighTrunks(client, gates, trunks, owner, name);
}
// **The verdict, set where the pull request shows it.** Every verdict is the head commit's status; one
// that is not a pass also leaves the check's own account as a comment, so the reason is read where the
// change is reviewed. An error — the check could not run — is the forge's `error`, never a success.
async function setVerdict(client: GiteaClient, event: { body: unknown }): Promise<void> {
const c = (event.body ?? {}) as Checked;
if (!c.group && c.owner && c.repo && c.took) {
// How long it took, for the merge tool to say how long the next one may (novox/hq ADR 0323 §5).
keepTook(tookFile(process.env.MESH_GITEA_STATE_DIR), `${c.owner}/${c.repo}`,
{ took: c.took, at: new Date().toISOString(), head: c.commit });
}
if (c.group) {
// A delivery group's composed check (novox/hq ADR 0239): its verdict is the group owner's to say, on each
// member's head, as mesh/delivery-group — never this head's merge gate.
@@ -333,26 +318,6 @@ async function setVerdict(client: GiteaClient, event: { body: unknown }): Promis
target = pull?.html_url || undefined;
base = pull?.base || undefined;
}
// **A verdict judged against a trunk commit that is no longer the trunk's head is out of date when it lands**
// (novox/hq ADR 0323 §4): a check asked before the trunk moved. It is said so, pending with the reason, in place of
// its pass or failure; a trunk that cannot be read lets the verdict stand, and the merge tool compares the commits.
const trunk = base ?? c.plan?.base;
if (c.trunk && trunk) {
const head = await client.branchHead(c.owner, c.repo, trunk).catch(() => undefined);
if (head && !sameCommit(c.trunk, head)) {
const description = outOfDateWords(c.trunk, head);
for (const context of [CHECK_CONTEXT, ...(c["repo-check"] ? [REPO_CHECK_CONTEXT] : [])]) {
await client.setCommitStatus(c.owner, c.repo, c.commit, { state: "pending", context, description,
...(target ? { target_url: target } : {}) });
}
// Its account is kept and shown all the same (ADR 0323 §7): what it judged, and that it is out of date.
const said = commentFor(c, await repoCheckFacts(client, c, trunk));
if (c.number) await client.addComment(c.owner, c.repo, c.number, lateVerdictComment(c, said, description));
console.log(`[gitea] ${c.owner}/${c.repo}#${c.number ?? "?"} at ${c.commit.slice(0, 8)}: merge check ${c.verdict}, ` +
`${description}`);
return;
}
}
const facts = await repoCheckFacts(client, c, base ?? c.plan?.base);
for (const status of statusesFor(c, facts)) {
await client.setCommitStatus(c.owner, c.repo, c.commit, target ? { ...status, target_url: target } : status);
@@ -385,7 +350,6 @@ const gates = new GateWriter({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gateOf: async (owner, repo, sha) => (await (gitea as GiteaClient).headChecks(owner, repo, sha))[CHECK_CONTEXT],
gateHistory: (owner, repo, sha) => (gitea as GiteaClient).contextHistory(owner, repo, sha, CHECK_CONTEXT),
headChecks: (owner, repo, sha) => (gitea as GiteaClient).headChecks(owner, repo, sha),
});
// **The status tool is served here, beside the writer** (novox/hq issue 456): the runtime starts each bundle as its own
@@ -394,7 +358,7 @@ const gates = new GateWriter({
registerModuleTools("gitea", () => (gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
}), mergeTool(gitea, { gates })] : []));
})] : []));
// **A build machine took an ask** (the build seat's `started`, novox/hq ADR 0157): said on the head it checks, when
// the head is an open pull request's this module announced. Any other build — a branch, a merge's — says nothing.
@@ -1,151 +0,0 @@
// Package forgehttp is the forge's HTTP interface as this module's Go bundles reach it (cmd/forge,
// cmd/npm-registry): the forge on its own machine, as its admin account, whose password the vault delivered
// into a file this module owns (novox/hq ADR 0086), by basic authentication. The password is read when a call
// is made and never logged, answered or put in an error. One copy for both bundles (novox/hq issue 539).
package forgehttp
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
// MaxBody is the largest JSON answer read from the forge.
const MaxBody = 32 << 20
// Client is the forge's HTTP interface.
type Client struct {
URL string // e.g. http://127.0.0.1:3000
User string
PasswordFile string
HTTP *http.Client // when nil, a client with Timeout
Timeout time.Duration // one request's limit when HTTP is nil; 20 s when zero
}
// FromEnv is the forge as the manifest hands it to a bundle, and the names of the variables it left unset.
func FromEnv() (Client, []string) {
c := Client{
URL: strings.TrimRight(os.Getenv("MESH_GITEA_URL"), "/"),
User: os.Getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: os.Getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"),
}
var missing []string
for _, kv := range [][2]string{{"MESH_GITEA_URL", c.URL}, {"MESH_GITEA_ADMIN_USER", c.User},
{"MESH_GITEA_ADMIN_PASSWORD_FILE", c.PasswordFile}} {
if kv[1] == "" {
missing = append(missing, kv[0])
}
}
return c, missing
}
// ErrNotFound is the forge answering 404.
var ErrNotFound = errors.New("not found")
// StatusError is an answer the forge gave that is not a success, said without the request's credential.
type StatusError struct {
Method, Path string
Status int
Said string
}
func (e *StatusError) Error() string {
return fmt.Sprintf("the forge answered %d to %s %s: %s", e.Status, e.Method, e.Path, e.Said)
}
func (c *Client) client() *http.Client {
if c.HTTP != nil {
return c.HTTP
}
timeout := c.Timeout
if timeout <= 0 {
timeout = 20 * time.Second
}
return &http.Client{Timeout: timeout}
}
func (c *Client) password() (string, error) {
raw, err := os.ReadFile(c.PasswordFile)
if err != nil {
// The path is the manifest's, not a secret; the content never appears.
return "", fmt.Errorf("the forge's admin password cannot be read from its file: %w", err)
}
return strings.TrimSpace(string(raw)), nil
}
// Do sends one request, with payload as its JSON body when not nil, and answers the body, read up to limit;
// 404 is ErrNotFound. One byte past the limit is read, so a body the limit cut is told from one that fits it
// exactly: a cut body is refused, never read as if it were whole.
func (c *Client) Do(ctx context.Context, method, path string, payload any, limit int64) ([]byte, error) {
password, err := c.password()
if err != nil {
return nil, err
}
var body io.Reader
if payload != nil {
raw, err := json.Marshal(payload)
if err != nil {
return nil, err
}
body = bytes.NewReader(raw)
}
req, err := http.NewRequestWithContext(ctx, method, c.URL+path, body)
if err != nil {
return nil, err
}
req.SetBasicAuth(c.User, password)
req.Header.Set("Accept", "application/json")
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
res, err := c.client().Do(req)
if err != nil {
return nil, fmt.Errorf("the forge did not answer %s %s: %w", method, path, err)
}
defer res.Body.Close()
got, err := io.ReadAll(io.LimitReader(res.Body, limit+1))
if err != nil {
return nil, err
}
if int64(len(got)) > limit {
return nil, fmt.Errorf("the forge's answer to %s %s is larger than %d MiB and was not read whole", method, path, limit>>20)
}
switch {
case res.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("%s %s: %w", method, path, ErrNotFound)
case res.StatusCode >= 300:
said := strings.TrimSpace(string(got))
if len(said) > 300 {
said = said[:300] + "…"
}
return nil, &StatusError{method, path, res.StatusCode, said}
}
return got, nil
}
// GetJSON reads one answer into into.
func (c *Client) GetJSON(ctx context.Context, path string, into any) error {
return c.SendJSON(ctx, http.MethodGet, path, nil, into)
}
// SendJSON sends one request and reads its answer into into, when into is not nil.
func (c *Client) SendJSON(ctx context.Context, method, path string, payload, into any) error {
body, err := c.Do(ctx, method, path, payload, MaxBody)
if err != nil {
return err
}
if into == nil {
return nil
}
if err := json.Unmarshal(body, into); err != nil {
return fmt.Errorf("the forge's answer to %s %s is not readable: %w", method, path, err)
}
return nil
}
-183
View File
@@ -1,183 +0,0 @@
// The forge's merge tool, `gitea_merge_pull_request` (novox/hq ADR 0266, issue 381, ADR 0323).
//
// **Served from the watcher's bundle, beside the merge check's one writer** (index.ts, gate.ts; novox/hq issue 456):
// when a verdict is judged against an earlier trunk commit than the trunk's head, the tool asks the head's check and
// puts its statuses back to pending through that writer, so where the check runs is said and no second check is asked.
// Each bundle the runtime serves is its own process; from the tools' bundle the pending bypassed the writer's chain.
import { emit } from "@novox/mesh-sdk/events";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus, GiteaClient } from "./client.js";
import { mergeableUnder, newestByContext } from "./checks.js";
import { MERGE_METHODS, Refused, checkName } from "./guard.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT } from "./pulls.js";
import { announcementOf, askedWords, outOfDateWords, tookFile, tookWords, twelve, verdictAtTrunk } from "./trunk.js";
/** A head named for a merge: at least 12 hex characters of its sha (novox/hq issue 381), up to a sha-256 commit's 64. */
const REVIEWED_HEAD = /^[0-9a-f]{12,64}$/;
/** What the merge tool reaches beside the forge: the merge check's one writer, how an event is said, the module's
* state directory, and the clock. */
export interface MergeDeps {
gates: { rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> };
emit?: (type: string, body: unknown) => Promise<void>;
stateDir?: string;
now?: () => Date;
}
export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition {
const deps = { ...given, emit: given.emit ?? emit, stateDir: given.stateDir ?? process.env.MESH_GITEA_STATE_DIR,
now: given.now ?? (() => new Date()) };
return {
name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head " +
"and every status its base branch's protection requires succeeded there (mesh/merge-gate, mesh/repo-check), " +
"judged against the trunk's head as it is now, optionally deleting the source branch afterwards. Refused " +
"otherwise, with nothing merged: no tool merges past a check, and none merges a head that moved after it was " +
"reviewed. A verdict judged against an earlier trunk commit is refused, naming both, and the head's check against " +
"the trunk's head is asked once, with how long the last check of the repository took: ask again at the same head " +
"once it passes — the tool never merges by itself.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
number: { type: "number", description: "the PR number" },
head: { type: "string", description: "the commit you reviewed, as its sha or at least its first 12 characters: " +
"refused unless it is the pull request's head" },
method: { type: "string", description: "merge (default) | fast-forward-only | squash | rebase | rebase-merge. " +
"Only merge and fast-forward-only can be read back to prove the head you named is what landed" },
delete_branch: { type: "boolean", description: "delete the source branch after merge (default true)" },
},
run: async (args) => {
const owner = checkName("owner", args.owner);
const repo = checkName("name", args.repo);
const number = Number(args.number);
const method = args.method ? String(args.method) : "merge";
if (!MERGE_METHODS.has(method)) throw new Refused(`${method} is not a merge method: ${[...MERGE_METHODS].join(", ")}`);
const deleteBranch = args.delete_branch === undefined ? true : Boolean(args.delete_branch);
// **Only a head whose required statuses all succeeded** (novox/hq ADR 0266): read here rather than left
// to the forge, whose administrator — the account this tool's token is minted for — may merge past a
// rule that lets it. A rule that cannot be read is no rule passed.
// **Only the head that was reviewed** (novox/hq issue 381): mesh-controller#197 was reviewed at 065cfa0d,
// force-pushed to 3ced96f, checked green there, and merged at 3ced96f, because this tool merged whatever the
// head was and nothing told it which head the review had approved. The caller names it; a head that moved
// since is refused, and the forge is handed the same head, so a push between this read and the merge is
// refused by the forge too.
const named = args.head === undefined ? "" : String(args.head).trim().toLowerCase();
if (!named) {
throw new Refused(`${owner}/${repo}#${number} is not merged: name the head you reviewed (head), so that only ` +
`that commit can land (novox/hq issue 381). Nothing was merged`);
}
// Twelve characters, not seven: an author who force-pushes can make a commit whose first seven match the
// reviewed one in seconds, and the review would then merge it. Twelve cannot be matched that way.
if (!REVIEWED_HEAD.test(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: head ${named} does not name one commit — give its ` +
`sha, or at least its first 12 characters. Nothing was merged`);
}
const pull = await gitea.getPullRequest(owner, repo, number);
if (pull.merged) throw new Refused(`${owner}/${repo}#${number} is already merged`);
if (!pull.head_sha || !pull.base) throw new Refused(`${owner}/${repo}#${number} names no head commit or base branch`);
if (!pull.head_sha.toLowerCase().startsWith(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its head is ${pull.head_sha.slice(0, 12)}, not the ` +
`${named} you named. If you reviewed an earlier head, review the one it has now, then name it. Nothing was merged`);
}
const rule = await gitea.branchProtection(owner, repo, pull.base);
// Only into a branch whose protection requires the checks: a pull request whose base was moved to a
// branch nothing guards would merge past every check (novox/hq ADR 0266).
if (!rule || !rule.enable_status_check || !(rule.status_check_contexts ?? []).filter(Boolean).length) {
throw new Refused(`${owner}/${repo}#${number} merges into ${pull.base}, which no protection rule requiring ` +
`the checks guards: a tool merges only into such a branch (novox/hq ADR 0266). Nothing was merged`);
}
// **The verdict must be judged against the trunk's head** (novox/hq ADR 0323 §5, issue 341): a check judges the
// head merged into a named trunk commit, and a merge of another trunk is not the one that check judged. Compared
// here at the moment the merge is asked, not left to the statuses the trunk's move resets a pass later. When they
// differ nothing is merged, and the head's check against the trunk's head is asked once — the announcement a new
// head makes — never again while one is on its way. The caller asks again at the reviewed head once it passes.
const trunkHead = await gitea.branchHead(owner, repo, pull.base);
const checks = await gitea.headChecks(owner, repo, pull.head_sha);
const at = verdictAtTrunk(checks, trunkHead);
if (!at.current) {
const judged = at.judged ? `was judged against trunk ${twelve(at.judged)}` : "names no trunk commit it was judged against";
const took = tookWords(tookFile(deps.stateDir), `${owner}/${repo}`);
const head = twelve(pull.head_sha), now = twelve(trunkHead);
if (at.onItsWay) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}. Its check is already asked and not yet answered (${checks[CHECK_CONTEXT]?.description}), so it is not ` +
`asked again; ${took}. Ask again at ${head} once both statuses pass: this tool never merges by itself ` +
`(novox/hq ADR 0323). A check that never answers is asked again with mesh-delivery.recheck. Nothing was merged`);
}
// Pending first, so a second ask finds the check on its way; then the head is announced as a new head is. An
// announcement that cannot be made puts the statuses back to out of date, so the next ask asks again rather
// than wait for ever on a check nobody asked.
const contexts = [CHECK_CONTEXT, ...(checks[REPO_CHECK_CONTEXT] ? [REPO_CHECK_CONTEXT] : [])];
// Through the module's one writer of the merge check (gate.ts, novox/hq issue 456): the pending opens a new
// check of the head, so where it runs is said ("building on …") and its verdict is not weighed as older.
const say = async (description: string) => {
for (const context of contexts) {
await deps.gates.rechecked(owner, repo, String(pull.head_sha), { state: "pending", context, description,
...(pull.html_url ? { target_url: pull.html_url } : {}) });
}
};
await say(askedWords(deps.now(), trunkHead));
try {
const cloneURL = (await gitea.getRepo(owner, repo)).clone_url;
await deps.emit("pull.updated", await announcementOf(gitea, owner, repo, cloneURL, pull));
} catch (err) {
await say(outOfDateWords(at.judged, trunkHead)).catch(() => {});
throw new Error(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}, and its check could not be asked: ${err instanceof Error ? err.message : err}. Nothing was merged`);
}
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}. The check of head ${head} merged into trunk ${now} was asked; ${took}. Ask again at ${head} once both ` +
`statuses pass: this tool never merges by itself (novox/hq ADR 0323). In a busy repository another merge may ` +
`move the trunk again first; this is then refused and asked once more. A check that never answers is asked ` +
`again with mesh-delivery.recheck. Nothing was merged`);
}
const statuses = newestByContext((await gitea.combinedStatus(owner, repo, pull.head_sha)).statuses);
const verdict = mergeableUnder(pull.base, rule, statuses);
if (verdict.mergeable !== true) {
throw new Refused(`${owner}/${repo}#${number} is not merged: ${verdict.says}. A pull request merges once ` +
`its checks succeed; no tool merges past them (novox/hq ADR 0266)`);
}
await gitea.mergePullRequest(owner, repo, number, method, deleteBranch, pull.head_sha);
// The merge commit only exists now; answered so the caller can follow what is built from it.
// pull.merged is the events entrypoint's to announce (index.ts), once, within its poll.
// From here the merge has happened. Nothing below may read as if it had not: a read-back that fails or cannot
// be done is said beside merged: true, and only a proven other head is an error — one that says the merge happened.
const answer: Record<string, unknown> = { merged: true, number, method, head: pull.head_sha, deleted_branch: deleteBranch };
let landed: string | undefined;
try {
const merged = await gitea.getPullRequest(owner, repo, number);
answer.merge_commit_sha = merged.merge_commit_sha;
// What landed, read back from the commit the merge made rather than from the forge's record of the pull
// request, which is the very thing a race would leave behind. A merge commit's second parent is the head it
// joined; a fast-forward lands the head itself. A squash or a rebase makes new commits that name no head.
if (!merged.merge_commit_sha) {
answer.read_back = "the forge named no merge commit, so what landed cannot be read back";
} else if (method === "merge") {
const parents = await gitea.commitParents(owner, repo, merged.merge_commit_sha);
if (parents.length === 2) landed = parents[1];
else answer.read_back = `the merge commit has ${parents.length} parent(s), not two: what landed cannot be read back`;
} else if (method === "fast-forward-only") {
// That the forge names the head itself as a fast-forward's merge commit is assumed, not seen on this forge:
// a different sha is said, not raised as another head having landed.
if (merged.merge_commit_sha === pull.head_sha) landed = merged.merge_commit_sha;
else answer.read_back = `the forge named ${merged.merge_commit_sha.slice(0, 12)} as this fast-forward's commit, ` +
`not the head: that the head landed could not be confirmed`;
} else {
answer.read_back = `a ${method} makes new commits that name no head: what landed cannot be read back`;
}
} catch (err) {
answer.read_back = `the merge happened, but reading back what landed failed: ${(err as Error).message}`;
}
// Loud, not a field beside merged: true that a caller reading only that would miss. The merge cannot be taken
// back by this tool; what must not happen is that it reads as the merge that was asked for.
if (landed !== undefined && landed !== pull.head_sha) {
throw new Error(`${owner}/${repo}#${number} WAS merged (${answer.merge_commit_sha}), but the forge merged ` +
`${landed.slice(0, 12)}, not the ${pull.head_sha.slice(0, 12)} it was handed and you reviewed: a commit nobody ` +
`reviewed has landed. Read it before anything is built from it (novox/hq issue 381)`);
}
if (landed !== undefined) answer.landed = landed;
return answer;
},
};
}
+2 -22
View File
@@ -59,12 +59,6 @@ export interface Checked {
plan?: ChangePlan;
/** Set on a delivery group's composed check (novox/hq ADR 0239): not this head's merge gate. */
group?: string;
/** The trunk commit the head was merged into and judged as, twelve characters (novox/hq ADR 0323 §3); absent from a
* controller whose build seat judged the head alone. Beside are the other repositories' commits the check read,
* by directory; took how long it ran. */
trunk?: string;
beside?: Record<string, string>;
took?: string;
}
/** A change plan, as the controller says it (mesh-controller internal/link, ChangePlan). */
@@ -205,25 +199,17 @@ export function statusFor(c: Checked): CommitStatus {
const description = builds(c.plan)
? described(gate.verdict, `${c.plan!.summary}; ${gate.summary}`)
: described(gate.verdict, gate.summary, gate.modules, gate.dependents);
return judgedFirst(c, { state: stateOf(gate.verdict), context: CHECK_CONTEXT, description });
return { state: stateOf(gate.verdict), context: CHECK_CONTEXT, description };
}
/** Every status a verdict sets: the gate's, and the repository's own check's when it was said. */
export function statusesFor(c: Checked, facts: RepoCheckFacts = {}): CommitStatus[] {
const out = [statusFor(c)];
const repo = c["repo-check"];
if (repo) out.push(judgedFirst(c, repoStatus(repo, facts)));
if (repo) out.push(repoStatus(repo, facts));
return out;
}
/** **A status names what its verdict judged, first** (novox/hq ADR 0323 §3): "head 1a2b3c4d5e6f merged into trunk
* 7a8b9c0d1e2f — pass: …", so the forge's 140 characters never clip it away; the trunk's move is weighed by it. A
* verdict that names no trunk commit is said as before. */
function judgedFirst(c: Checked, status: CommitStatus): CommitStatus {
if (!c.trunk) return status;
return { ...status, description: clipped(`head ${c.commit.slice(0, 12)} merged into trunk ${c.trunk.slice(0, 12)} — ${status.description}`) };
}
/** The comment a verdict leaves on its pull request, with the check's own account: the change plan of a
* change that builds something (novox/hq ADR 0238), and why, when the gate is not a pass or the
* repository's own check failed or could not run. A repository with no merge-check.sh, touching nothing,
@@ -234,12 +220,6 @@ export function commentFor(c: Checked, facts: RepoCheckFacts = {}): string | nul
const wrong = repoWrong(repo, facts);
if (gate.verdict === "pass" && !wrong && !builds(c.plan)) return null;
const lines = [`**Merge check** at \`${c.commit.slice(0, 8)}\``, ""];
if (c.trunk) {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and what was read beside it.
const beside = Object.entries(c.beside ?? {}).map(([d, sha]) => `${d} \`${sha}\``).join(", ");
lines.splice(1, 0, `Judged: head \`${c.commit.slice(0, 12)}\` merged into trunk \`${c.trunk.slice(0, 12)}\`` +
(beside ? `; read beside it: ${beside}` : ""));
}
lines.push(`- \`${CHECK_CONTEXT}\`: **${saidAs(gate.verdict).toUpperCase()}** — ${gate.summary}` +
(gate.modules?.length ? ` (modules: ${gate.modules.join(", ")}` +
(gate.dependents?.length ? `; built after them: ${gate.dependents.join(", ")}` : "") + ")" : ""));
-44
View File
@@ -93,47 +93,3 @@ test("a pull request or issue number that is not one is refused before the forge
server.close();
}
});
// **A merge names the branch it came from, also once that branch is deleted** (novox/hq issue 578). On 2026-10-12
// the merge tool merged mesh-controller #271 and mesh-catalog #251, one delivery group, and deleted each branch. The
// forge then read each pull request's head.ref as `refs/pull/<n>/head`, and `pull.merged` named that as the head: the
// controller asked mesh-delivery for a group of that name, found none, and cut #251's walk at once, before #271's,
// which the group's order put first.
test("a merged pull request whose branch was deleted is announced with its branch, not its pull ref", async () => {
const { GiteaClient, headBranchOf } = await import("../client.ts");
const group = "feat/341-a-merge-check-judges-the-merge";
// As the forge answered for #251 after its merge.
const merged = {
number: 251, title: "A verdict goes out of date when its trunk moves", state: "closed", merged: true,
merged_at: "2026-10-12T16:11:29Z", merge_commit_sha: "23ea0f431faf91aa83e7b0297e0a9e442180f0c7",
head: { label: group, ref: "refs/pull/251/head", sha: "41b6e10077efdef694b519879fcbaeaf69d472b1" },
base: { label: "main", ref: "main" }, html_url: "http://fake/novox/mesh-catalog/pulls/251",
};
const server = createServer((req, res) => {
const url = req.url ?? "";
const send = (v: unknown) => res.end(JSON.stringify(v));
if (url.startsWith("/api/v1/repos/novox/mesh-catalog/pulls/251")) return send(merged);
if (url.startsWith("/api/v1/repos/novox/mesh-catalog/pulls?")) return send([merged]);
res.statusCode = 404;
res.end("{}");
});
await new Promise<void>((r) => server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(server.address() as any).port}`, "t");
try {
// What the announcer reads each pass, and what `pull.merged` carries as its head.
const listed = await client.listPullRequests("novox", "mesh-catalog", { state: "closed", sort: "recentupdate", limit: "20" });
assert.deepEqual(listed.map((p) => p.head), [group], "the pass names the group's branch, never refs/pull/251/head");
assert.equal((await client.getPullRequest("novox", "mesh-catalog", 251)).head, group);
} finally {
server.close();
}
// While the branch exists the forge says it as ref, and that is what is named; a label alone is the branch too.
assert.equal(headBranchOf({ label: "feat/x", ref: "feat/x" }), "feat/x");
assert.equal(headBranchOf({ label: "feat/x" }), "feat/x");
assert.equal(headBranchOf({ ref: "feat/x" }), "feat/x");
// A branch whose own name only looks like a pull ref's start is still a branch.
assert.equal(headBranchOf({ label: "y", ref: "refs/pulled" }), "refs/pulled");
// No label to read: the pull ref is all the forge said, and it is kept rather than made up.
assert.equal(headBranchOf({ ref: "refs/pull/7/head" }), "refs/pull/7/head");
assert.equal(headBranchOf(undefined), undefined);
});
+9 -130
View File
@@ -7,14 +7,10 @@
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { mkdtempSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { mergeTool } from "../dist/merge.js";
import { commitStatusTool } from "../dist/delivery.js";
import { GateWriter } from "../dist/gate.js";
import { judgeApiCall, judgeProtectionChange, ruleCovers, routedPath } from "../dist/guard.js";
@@ -24,11 +20,7 @@ interface Sent { method: string; path: string; body: any }
const state = {
sent: [] as Sent[],
asked: [] as string[],
statuses: [] as { context: string; state: string; description?: string; updated_at?: string }[],
// The trunk's head (novox/hq ADR 0323): what the merge tool compares the verdict's trunk commit with.
trunk: "",
// What the tools said on the bus.
emitted: [] as { type: string; body: any }[],
statuses: [] as { context: string; state: string }[],
rules: [] as any[],
lastCommit: "2026-01-01T00:00:00Z",
// The pull request's head; the head the merge commit joined (the same, unless a test says otherwise) — the forge's
@@ -43,21 +35,13 @@ const state = {
commitsFail: 0,
};
const REVIEWED = "abc1234def5678901234567890abcdef12345678";
const TRUNK = "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b";
const MOVED = "3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d";
// A verdict judged against a trunk commit, as both statuses say it (novox/hq ADR 0323 §3).
const judgedAt = (trunk: string, state = "success") => [
{ context: "mesh/merge-gate", state, description: `head ${REVIEWED.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: x` },
{ context: "mesh/repo-check", state, description: `head ${REVIEWED.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: y` }];
const protectedMain = () => ({ rule_name: "main", enable_push: false, enable_status_check: true,
status_check_contexts: ["mesh/merge-gate", "mesh/repo-check"], block_admin_merge_override: true });
beforeEach(() => {
state.sent = [];
state.asked = [];
state.statuses = judgedAt(TRUNK);
state.trunk = TRUNK;
state.emitted = [];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "success" }];
state.rules = [protectedMain()];
state.head = REVIEWED;
state.landed = "";
@@ -88,15 +72,6 @@ const server = createServer(async (req, res) => {
const repo = "/repos/novox/mesh-catalog";
if (method === "GET" && path === repo) return json(res, 200, { full_name: "novox/mesh-catalog", default_branch: "main" });
if (method === "GET" && path === `${repo}/branch_protections`) return json(res, 200, state.rules);
if (method === "GET" && path === `${repo}/branches/main`) return json(res, 200, { name: "main", commit: { id: state.trunk } });
if (method === "GET" && path === `${repo}/pulls/7/files`) {
return json(res, 200, url.searchParams.get("page") === "1" ? [{ filename: "modules/gitea/x.ts", status: "modified" }] : []);
}
if (method === "POST" && path === `${repo}/statuses/${state.head}`) {
// Kept as the forge keeps them: the newest of a context first.
state.statuses = [{ ...body, updated_at: new Date().toISOString() }, ...state.statuses.filter((s) => s.context !== body.context)];
return json(res, 201, body);
}
if (method === "GET" && path === "/user") return json(res, 200, { login: "mesh-admin" });
if (method === "GET" && path === "/repos/mesh-admin/scratch/branch_protections") return json(res, 200, []);
if (method === "GET" && /^\/repos\/[^/]+\/[^/]+\/commits$/.test(path)) {
@@ -126,19 +101,11 @@ after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
// gitea_commit_status is served from the watcher's bundle, beside the merge check's one writer (novox/hq issue 456).
const stateDir = mkdtempSync(join(tmpdir(), "gitea-tools-"));
// The merge check's one writer, as the watcher's bundle holds it: what the merge tool's pending goes through.
const writer = new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [], headChecks: async () => ({}) });
const throughTheWriter: string[] = [];
const gates = { rechecked: (o: string, r: string, sha: string, s: any) => {
throughTheWriter.push(`${sha.slice(0, 12)} ${s.context} ${s.state}`);
return writer.rechecked(o, r, sha, s);
} };
const tools = [...getGiteaTools(client), commitStatusTool({
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s), gates: writer,
}), mergeTool(client, { gates, stateDir, emit: async (type, body) => { state.emitted.push({ type, body }); },
now: () => new Date(2026, 9, 12, 15, 1) })];
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gates: new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [] }),
})];
const tool = (name: string) => tools.find((t) => t.name === name)!;
const writes = () => state.sent.map((s) => `${s.method} ${s.path}`);
@@ -179,25 +146,6 @@ test("gitea_api refuses a file written to the default branch or one a rule guard
assert.deepEqual(writes(), ["PUT /repos/novox/mesh-catalog/contents/modules/sudo/module.json"]);
});
test("a protection rule covers a branch as the forge decides it: classes, alternatives, a plain name in any case (issue 539)", async () => {
// One table with the forge bundle (cmd/forge/rules.go): `forge` is the forge's answer; `guard` is where this
// guard deliberately covers more — a rule Go's regexp reads otherwise than JavaScript's — and never less.
const cases = JSON.parse(readFileSync(new URL("./rule-cases.json", import.meta.url), "utf8")) as
Array<{ rule: string; branch: string; forge: boolean; guard?: boolean }>;
assert.ok(cases.length >= 100);
for (const c of cases) {
assert.ok(c.guard === undefined || (c.guard && !c.forge), `${c.rule}: the guard may only cover more than the forge`);
assert.equal(ruleCovers(c.rule, c.branch), c.guard ?? c.forge, `${c.rule} covers ${c.branch}`);
}
const put = (branch: string) => tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/contents/README.md",
method: "PUT", body: { content: "e30=", sha: "x", branch } });
state.rules.push({ rule_name: "release-[ab]" }, { rule_name: "{stable,lts}/*" });
await assert.rejects(put("release-a"), /protection rule release-\[ab\]/);
await assert.rejects(put("lts/2"), /protection rule \{stable,lts\}\/\*/);
await assert.rejects(put("MAIN"), /protection rule main/);
assert.deepEqual(writes(), []);
});
test("gitea_api still writes what an author writes", async () => {
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/issues/3/comments", method: "POST", body: { body: "hi" } });
await tool("gitea_api").run({ path: "/repos/novox/mesh-catalog/pulls/7/reviews", method: "POST", body: { event: "COMMENT" } });
@@ -221,15 +169,15 @@ test("gitea_branch_protection_set only strengthens", async () => {
test("gitea_merge_pull_request merges only a head whose required statuses all succeeded, and only that head", async () => {
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7,
head: REVIEWED });
state.statuses = [judgedAt(TRUNK)[0], judgedAt(TRUNK, "failure")[1]];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "failure" }];
await assert.rejects(merge(), /not merged: .*mesh\/repo-check \(failure\)/);
state.statuses = [judgedAt(TRUNK)[0]];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }];
await assert.rejects(merge(), /mesh\/repo-check \(missing\)/);
await assert.rejects(tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7,
head: REVIEWED, method: "manually-merged" }), /not a merge method/);
assert.deepEqual(writes(), []);
// The operator said yes and the checks passed: the flow that stays.
state.statuses = judgedAt(TRUNK);
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "success" }];
await merge();
const sent = state.sent.find((s) => s.path.endsWith("/pulls/7/merge"))!;
assert.equal(sent.body.force_merge, false);
@@ -237,75 +185,6 @@ test("gitea_merge_pull_request merges only a head whose required statuses all su
});
// **The merge tool refuses a verdict whose trunk commit is not the trunk's head, and asks the check itself** (novox/hq
// ADR 0323 §5, issue 341): node-engine #66 merged at 00:54 on a verdict of 00:49 that judged its head alone, after #65
// had moved the trunk. Both commits are named; one check is asked, the announcement a new head makes, and not a second
// while it is on its way; the refusal says how long the last check of the repository took; and once the verdict judges
// the trunk's head and passes, the merge goes through — asked again, never by itself.
test("gitea_merge_pull_request refuses a verdict judged against an earlier trunk, and asks the check once", async () => {
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED });
const { keepTook, tookFile } = await import("../dist/trunk.js");
keepTook(tookFile(stateDir), "novox/mesh-catalog", { took: "4m31s", at: "2026-10-12T14:49:00Z" });
state.trunk = MOVED; // another merge moved the trunk after the verdict
await assert.rejects(merge(), (err: Error) => {
assert.match(err.message, /is not merged: its verdict was judged against trunk 7a8b9c0d1e2f, and main is now 3c4d5e6f7a8b/);
assert.match(err.message, /check of head abc1234def56 merged into trunk 3c4d5e6f7a8b was asked/);
assert.match(err.message, /the last check of novox\/mesh-catalog took 4m31s/);
assert.match(err.message, /never merges by itself/);
return true;
});
assert.equal(state.merged, false);
assert.equal(state.emitted.length, 1, "one check asked");
assert.equal(state.emitted[0].type, "pull.updated");
assert.equal(state.emitted[0].body.head_sha, REVIEWED);
assert.equal(state.emitted[0].body.base, "main");
assert.deepEqual(state.emitted[0].body.paths, ["modules/gitea/x.ts"]);
// The announcement names the trunk's head, for a verdict no build judges (ADR 0323 §3).
assert.equal(state.emitted[0].body.base_sha, MOVED);
// Its pending went through the merge check's one writer (novox/hq issue 456), not around it.
assert.deepEqual(throughTheWriter.splice(0), ["abc1234def56 mesh/merge-gate pending", "abc1234def56 mesh/repo-check pending"]);
const gate = state.statuses.find((s) => s.context === "mesh/merge-gate")!;
assert.equal(gate.state, "pending");
assert.match(gate.description!, /^waiting in the build queue since 15:01: asked by the merge tool against trunk 3c4d5e6f7a8b/);
assert.equal(state.statuses.find((s) => s.context === "mesh/repo-check")!.state, "pending");
// Asked again while that check is on its way: refused, and nothing is asked a second time.
await assert.rejects(merge(), /already asked and not yet answered .*not asked again; the last check of novox\/mesh-catalog took 4m31s.*mesh-delivery\.recheck/);
assert.equal(state.emitted.length, 1, "no second check while one is on its way");
state.statuses = [{ ...gate, description: "building on ace since 15:02" }, ...state.statuses.slice(1)];
await assert.rejects(merge(), /not asked again/);
assert.equal(state.emitted.length, 1);
// A verdict put out of date by the trunk's move is asked too, once.
state.statuses = judgedAt(TRUNK).map((s) => ({ ...s, state: "pending",
description: "out of date: judged against trunk 7a8b9c0d1e2f, the trunk is now 3c4d5e6f7a8b; judged again on a recheck or when it is merged" }));
await assert.rejects(merge(), /was judged against trunk 7a8b9c0d1e2f, and main is now 3c4d5e6f7a8b\. The check .* was asked/);
assert.equal(state.emitted.length, 2);
// A verdict that names no trunk commit — a build seat from before the rule — is not taken as the trunk's head.
state.statuses = [{ context: "mesh/merge-gate", state: "success", description: "pass: x" },
{ context: "mesh/repo-check", state: "success", description: "pass: y" }];
await assert.rejects(merge(), /names no trunk commit it was judged against, and main is now 3c4d5e6f7a8b/);
assert.equal(state.emitted.length, 3);
assert.equal(state.merged, false);
// An ask that cannot be announced puts the statuses back to out of date, so the next ask asks again.
state.statuses = judgedAt(TRUNK);
const failing = mergeTool(client, { gates, stateDir, emit: async () => { throw new Error("the bus is away"); } });
await assert.rejects(failing.run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED }),
/its check could not be asked: the bus is away\. Nothing was merged/);
assert.match(state.statuses.find((s) => s.context === "mesh/merge-gate")!.description!, /^out of date: /);
await assert.rejects(merge(), /was asked/);
assert.equal(state.emitted.length, 4, "asked again after the announcement failed");
// The check judged the merge into the trunk's head and passed: asked again at the reviewed head, it merges.
state.statuses = judgedAt(MOVED);
const done = (await merge()) as Record<string, unknown>;
assert.equal(done.merged, true);
assert.equal(state.emitted.length, 4, "a merge asks no check");
});
// A merge lands only the head that was reviewed (novox/hq issue 381): mesh-controller#197 was reviewed at one head,
// force-pushed, checked green at the new one, and merged there, because the tool merged whatever the head was.
test("gitea_merge_pull_request merges only the head it is named, and refuses one that is not the head", async () => {
-92
View File
@@ -1,92 +0,0 @@
import assert from "node:assert/strict";
import { test } from "node:test";
import { createServer } from "node:http";
// **Every open pull request's new head is announced, however many are open** (novox/hq issue 579). A pass asked the
// forge for the 20 most recently updated open pull requests of a repository, so a new head on the 21st or later was
// not announced and its merge check never started. A pass now pages through the open pull requests updated since its
// last look — every page on a sweep or the first pass — and stops at the first page that reaches back past the floor.
const HOUR = 3600_000;
const T0 = Date.parse("2026-10-12T12:00:00Z");
const sha = (n: number, v: string) => `${v}${String(n).padStart(4, "0")}`.padEnd(40, "0");
/** A forge holding `count` open pull requests of novox/hq, #n updated n hours after T0 (#count the newest), paged and
* sorted the way gitea answers `sort=recentupdate`: by update time, newest first. */
function forge(count: number, heads: Record<number, string>) {
const asked: URLSearchParams[] = [];
const server = createServer((req, res) => {
const url = new URL(req.url ?? "", "http://x");
if (url.pathname !== "/api/v1/repos/novox/hq/pulls") {
res.statusCode = 404;
return res.end("{}");
}
asked.push(url.searchParams);
const limit = Number(url.searchParams.get("limit") ?? 30);
const page = Number(url.searchParams.get("page") ?? 1);
const all = Array.from({ length: count }, (_, i) => count - i).map((n) => ({
number: n, state: "open", base: { ref: "main" }, head: { ref: `b${n}`, sha: heads[n] ?? sha(n, "a") },
updated_at: new Date(T0 + n * HOUR).toISOString(), html_url: `h/${n}`,
}));
res.end(JSON.stringify(all.slice((page - 1) * limit, page * limit)));
});
return { server, asked };
}
async function serve(count: number, heads: Record<number, string>) {
const { GiteaClient } = await import("../client.ts");
const f = forge(count, heads);
await new Promise<void>((r) => f.server.listen(0, r));
const client = new GiteaClient(`http://127.0.0.1:${(f.server.address() as any).port}`, "t");
return { client, ...f };
}
test("a new head beyond the newest 20 open pull requests is announced, exactly once", async () => {
const { headsToAnnounce } = await import("../pulls.ts");
// 25 open, every head announced before; #3 — the 23rd most recently updated — got a new head.
const announced: Record<string, string> = {};
for (let n = 1; n <= 25; n++) announced[`novox/hq#${n}`] = sha(n, "a");
const { client, server, asked } = await serve(25, { 3: sha(3, "b") });
try {
// The fault: the newest 20 alone do not hold it.
const newest = await client.listPullRequests("novox", "hq", { state: "open", sort: "recentupdate", limit: "20" });
assert.deepEqual(headsToAnnounce("novox/hq", newest, announced), [], "the newest 20 miss #3's new head");
asked.length = 0;
const said: string[] = [];
for (let pass = 0; pass < 2; pass++) {
// The sweep, as index.ts's pollPulls asks: no floor, every page.
const open = await client.listOpenPullsSince("novox", "hq", "");
for (const p of headsToAnnounce("novox/hq", open, announced)) {
said.push(`#${p.number}@${p.head_sha}`);
announced[`novox/hq#${p.number}`] = String(p.head_sha);
}
}
assert.deepEqual(said, [`#3@${sha(3, "b")}`], "the new head past the newest 20, announced once over two passes");
assert.ok(asked.every((q) => q.get("state") === "open" && q.get("sort") === "recentupdate"), "asked newest first");
} finally {
server.close();
}
});
test("a pass pages until it reaches back past its floor, and no further", async () => {
// 130 open pull requests, #n updated n hours after T0: page 1 holds #130–#81, page 2 #80–#31, page 3 #30–#1.
const { client, server, asked } = await serve(130, {});
try {
// A floor between #60 and #61: page 2 reaches back past it, so page 3 is never asked.
const floor = new Date(T0 + 60.5 * HOUR).toISOString();
const open = await client.listOpenPullsSince("novox", "hq", floor);
assert.deepEqual(asked.map((q) => q.get("page")), ["1", "2"]);
assert.equal(open.length, 100);
assert.ok(open.some((p) => p.number === 61), "every one updated since the floor is read");
// A floor no page reaches past: the last short page ends it.
asked.length = 0;
assert.equal((await client.listOpenPullsSince("novox", "hq", "")).length, 130);
assert.deepEqual(asked.map((q) => q.get("page")), ["1", "2", "3"]);
// Every page holds fewer than 50 updated since a recent floor: one page.
asked.length = 0;
await client.listOpenPullsSince("novox", "hq", new Date(T0 + 129 * HOUR).toISOString());
assert.deepEqual(asked.map((q) => q.get("page")), ["1"]);
} finally {
server.close();
}
});
-285
View File
@@ -1,285 +0,0 @@
[
{"rule":"main","branch":"main","forge":true,"why":"a plain name covers its branch"},
{"rule":"main","branch":"mainx","forge":false},
{"rule":"main","branch":"MAIN","forge":true,"why":"a plain name covers in any case (EqualFold)"},
{"rule":"Main","branch":"main","forge":true},
{"rule":"release/*","branch":"release/1.0","forge":true,"why":"* stays within a segment"},
{"rule":"release/*","branch":"release/a/b","forge":false},
{"rule":"release/**","branch":"release/a/b","forge":true,"why":"** crosses segments"},
{"rule":"v?","branch":"v1","forge":true},
{"rule":"v?","branch":"v/","forge":false,"why":"? is not the separator"},
{"rule":"a.b","branch":"axb","forge":false,"why":". is itself"},
{"rule":"Feat/*","branch":"feat/x","forge":false,"why":"a glob is matched with case"},
{"rule":"release-[ab]","branch":"release-a","forge":true,"why":"a class"},
{"rule":"release-[ab]","branch":"release-c","forge":false},
{"rule":"release-[!ab]","branch":"release-c","forge":true,"why":"a negated class"},
{"rule":"release-[!ab]","branch":"release-a","forge":false},
{"rule":"v[0-9].x","branch":"v7.x","forge":true,"why":"a range"},
{"rule":"v[0-9].x","branch":"vA.x","forge":false},
{"rule":"{main,master}","branch":"master","forge":true,"why":"alternatives"},
{"rule":"{main,master}","branch":"mainline","forge":false},
{"rule":"rel/{a,b}/*","branch":"rel/b/1","forge":true},
{"rule":"rel/{a,b}/*","branch":"rel/c/1","forge":false},
{"rule":"a\\*","branch":"a*","forge":true,"why":"an escape"},
{"rule":"a\\*","branch":"ab","forge":false},
{"rule":"a+b","branch":"a+b","forge":true,"why":"+ is itself"},
{"rule":"a+b","branch":"aab","forge":false},
{"rule":"{open","branch":"{open","forge":true,"why":"a glob the forge cannot compile covers its own text"},
{"rule":"[open","branch":"[open","forge":true},
{"rule":"[open","branch":"o","forge":false},
{"rule":"release-\\d","branch":"release-1","forge":true,"why":"Go reads \\d as a digit"},
{"rule":"release-\\d","branch":"release-d","forge":false,"guard":true},
{"rule":"v\\w*","branch":"v2","forge":true},
{"rule":"main\\s","branch":"main","forge":false,"guard":true},
{"rule":"[]a]","branch":"a","forge":true,"why":"a class opening with ] holds ]"},
{"rule":"[]a]","branch":"]","forge":true},
{"rule":"[]a]","branch":"b","forge":false,"guard":true},
{"rule":"rel-[\\d]","branch":"rel-1","forge":true},
{"rule":"v[\\pL]","branch":"va","forge":true,"why":"\\p is a Unicode class"},
{"rule":"v\\pL","branch":"va","forge":true},
{"rule":"[[:alpha:]]","branch":"a","forge":true,"why":"a POSIX class"},
{"rule":"rel-[[:digit:]]","branch":"rel-1","forge":true},
{"rule":"rel-[[:digit:]]","branch":"rel-x","forge":false,"guard":true},
{"rule":"[^+]*","branch":"-[1","forge":true,"why":"a class opening with ^ is negated in Go"},
{"rule":"d[^](","branch":"d[^](","forge":true},
{"rule":"]?","branch":":,1},","forge":false},
{"rule":"Ab,","branch":"Ab,","forge":true},
{"rule":"\\:[}-|","branch":",{","forge":false},
{"rule":"^,*.d::","branch":"/x","forge":false},
{"rule":"","branch":"A*:","forge":false},
{"rule":"!|A{","branch":":.{]","forge":false},
{"rule":"","branch":"]éb","forge":false},
{"rule":"/{)w","branch":"/{)w","forge":true},
{"rule":"*^w]","branch":"}!.","forge":false},
{"rule":"*),^?*d","branch":"*),^?*d","forge":true},
{"rule":"A)^}!","branch":"","forge":false},
{"rule":"}^d?","branch":"1\\.","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"$?[","branch":"$?[","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"+","forge":true},
{"rule":"","branch":"},[","forge":false},
{"rule":"xA[d:a)","branch":"]1.!bx","forge":false},
{"rule":"^/}(","branch":"","forge":false},
{"rule":"-|x.x1?","branch":"-|x.x1?","forge":true},
{"rule":"x+)A]-A","branch":"x+)A]-A","forge":true},
{"rule":"x*a","branch":"x*a","forge":true},
{"rule":"$xb","branch":"}é","forge":false},
{"rule":"wAx?/a,","branch":"","forge":false},
{"rule":"[,","branch":"[,","forge":true},
{"rule":"}!","branch":"}!","forge":true},
{"rule":"*:|\\w","branch":"","forge":false,"guard":true},
{"rule":"","branch":"","forge":true},
{"rule":"é","branch":"é","forge":true},
{"rule":"..(","branch":"{\\{+","forge":false},
{"rule":"*|{","branch":"1.}-","forge":false},
{"rule":"!+,","branch":"}++-*","forge":false},
{"rule":"x\\({","branch":"*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-{+-","branch":"?-{+-","forge":true},
{"rule":"/-{é|$","branch":"b!.[","forge":false},
{"rule":"a-+dxA","branch":"1AA,xA","forge":false},
{"rule":"]|(","branch":"]|(","forge":true},
{"rule":"}","branch":"{!x{d","forge":false},
{"rule":"/{*](","branch":"!]","forge":false},
{"rule":"($+{-}|","branch":"/","forge":false},
{"rule":"(-b.-1}","branch":"*A*+,1","forge":false},
{"rule":"1[d|)wa","branch":"+}[!}}","forge":false},
{"rule":"w","branch":":.","forge":false},
{"rule":"?}\\*","branch":"","forge":false},
{"rule":"?$*","branch":"?$*","forge":true},
{"rule":"","branch":"!/{béa","forge":false},
{"rule":"","branch":"A1","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"!a$1/b{","branch":"!a$1/b{","forge":true},
{"rule":":dww/|d","branch":":dww/|d","forge":true},
{"rule":"!|*}","branch":"xéxé{b","forge":false},
{"rule":"1é||","branch":"a+.","forge":false},
{"rule":"/+],","branch":"!-\\[{é","forge":false},
{"rule":"a}(","branch":"a}(","forge":true},
{"rule":"|A?.","branch":"}.","forge":false},
{"rule":"d$a^d","branch":"*{","forge":false},
{"rule":"-ba$+A","branch":"-ba$+A","forge":true},
{"rule":"-.|,?a!","branch":"-.|,?a!","forge":true},
{"rule":"A","branch":"A","forge":true},
{"rule":"w","branch":"w","forge":true},
{"rule":".!1!+a","branch":".!1!+a","forge":true},
{"rule":"+$w:}","branch":"+$w:}","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":")]","branch":")]","forge":true},
{"rule":"a})^,}1","branch":"a})^,}1","forge":true},
{"rule":":[:!:","branch":":[:!:","forge":true},
{"rule":"w?w:]1!","branch":"\\.]-","forge":false},
{"rule":"*$/","branch":"*$/","forge":true},
{"rule":"é](","branch":"é](","forge":true},
{"rule":"x$a/","branch":"x$a/","forge":true},
{"rule":"*)b+[^","branch":"*)b+[^","forge":true},
{"rule":"/]/?","branch":"/]/?","forge":true},
{"rule":"{x,A","branch":"!Ab","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"-,","branch":"xé}A[.","forge":false},
{"rule":"w)Ad","branch":"w)Ad","forge":true},
{"rule":":??","branch":":??","forge":true},
{"rule":"x/xé|","branch":"x/xé|","forge":true},
{"rule":"","branch":"1!","forge":false},
{"rule":"!![é[","branch":"x","forge":false},
{"rule":"wb$","branch":"wb$","forge":true},
{"rule":")","branch":")","forge":true},
{"rule":"Aw:)(1","branch":"{b\\,","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"1*}?d","branch":"1*}?d","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"?-b.","branch":"?-b.","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"d","branch":",aé1","forge":false},
{"rule":"+","branch":"+{","forge":false},
{"rule":"]d","branch":"]}:/bA","forge":false},
{"rule":"*?+:|","branch":"!é.","forge":false},
{"rule":"}$d","branch":"}$d","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"a$+b)b","branch":"d}","forge":false},
{"rule":"?$[/A","branch":"","forge":false},
{"rule":"é){","branch":"é){","forge":true},
{"rule":"$(d","branch":"{é}","forge":false},
{"rule":"b(","branch":"b(","forge":true},
{"rule":"^^+]","branch":"^^+]","forge":true},
{"rule":",)","branch":",)","forge":true},
{"rule":"d-.A-","branch":"d-.A-","forge":true},
{"rule":"$","branch":"\\/","forge":false},
{"rule":"}+[\\|\\$","branch":"x[!b","forge":false},
{"rule":"(d1aé!é","branch":"A[-","forge":false},
{"rule":"x","branch":"1","forge":false},
{"rule":"x","branch":"dx","forge":false},
{"rule":"]|)]","branch":"]|)]","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"[.*?","branch":"[.*?","forge":true},
{"rule":"[?","branch":"[?","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"),/","branch":"a1,,1","forge":false},
{"rule":"}+","branch":"}+","forge":true},
{"rule":"$d\\,d","branch":"$d\\,d","forge":false},
{"rule":"d1","branch":"d1","forge":true},
{"rule":"-!^.:","branch":"-!^.:","forge":true},
{"rule":"x","branch":"+].A11","forge":false},
{"rule":"","branch":"-]","forge":false},
{"rule":"}]}.","branch":"","forge":false},
{"rule":"^{wx","branch":"]1a!","forge":false},
{"rule":"^","branch":"*]x,","forge":false},
{"rule":"(:x*x","branch":"","forge":false},
{"rule":"|[/*a/","branch":"}","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"1\\^","branch":"!é!x{","forge":false},
{"rule":"(.b,AA+","branch":"(.b,AA+","forge":true},
{"rule":"xx","branch":"","forge":false},
{"rule":"d)^):","branch":"/}!","forge":false},
{"rule":"|","branch":",é*]*[","forge":false},
{"rule":"d*d:*1","branch":"\\b,","forge":false},
{"rule":":]","branch":"[","forge":false},
{"rule":"([|!\\^é","branch":"([|!\\^é","forge":true},
{"rule":"$[w)x-*","branch":"","forge":false},
{"rule":")","branch":")","forge":true},
{"rule":".d^)?","branch":"x\\é","forge":false},
{"rule":"1|{w","branch":"1|{w","forge":true},
{"rule":"é$,(]$$","branch":"é$,(]$$","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"+","branch":"","forge":false},
{"rule":"]","branch":"","forge":false},
{"rule":"+:a:","branch":"+:a:","forge":true},
{"rule":"/*","branch":"d.}","forge":false},
{"rule":"|/ba1-","branch":"|/ba1-","forge":true},
{"rule":"","branch":"+","forge":false},
{"rule":"A","branch":"A","forge":true},
{"rule":",-!]","branch":",-!]","forge":true},
{"rule":"$*d!)","branch":"","forge":false},
{"rule":")é+x.d","branch":")é+x.d","forge":true},
{"rule":"éa$\\[1","branch":"1/,","forge":false},
{"rule":"}?1|]d","branch":"}?1|]d","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"-1.*+","branch":"-1.*+","forge":true},
{"rule":"Aw|{","branch":"Aw|{","forge":true},
{"rule":"adAAAa","branch":"A","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"d/^{:","branch":"d/^{:","forge":true},
{"rule":"é:(,^","branch":"é:(,^","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"!ébé({+","branch":"!ébé({+","forge":true},
{"rule":"x","branch":"x","forge":true},
{"rule":"|","branch":",:*","forge":false},
{"rule":"","branch":"","forge":true},
{"rule":"x|","branch":"x|","forge":true},
{"rule":"a","branch":"aaxéb,","forge":false},
{"rule":"/w-),1é","branch":"/w-),1é","forge":true},
{"rule":"|*)^,","branch":"|*)^,","forge":true},
{"rule":"){é.!b.","branch":"a-}:","forge":false},
{"rule":"(\\","branch":"(\\","forge":true},
{"rule":".w(\\","branch":"+","forge":false},
{"rule":"/d[","branch":"/d[","forge":true},
{"rule":"(}$x!}","branch":"a,bb","forge":false},
{"rule":"w{","branch":"w{","forge":true},
{"rule":"[+a\\A(+","branch":"[+a\\A(+","forge":true},
{"rule":"(1^{)bb","branch":"(1^{)bb","forge":true},
{"rule":"^1","branch":"é-}","forge":false},
{"rule":"*,+}1):","branch":"*,+}1):","forge":true},
{"rule":"[A[(.x,","branch":"]db!*","forge":false},
{"rule":"","branch":"Ad}","forge":false},
{"rule":".","branch":".","forge":true},
{"rule":"w!é{x","branch":"é.{","forge":false},
{"rule":"$\\^","branch":"bx]+","forge":false},
{"rule":"1)","branch":"+*d","forge":false},
{"rule":"x)}/])","branch":"]","forge":false},
{"rule":",a}.1}","branch":",a}.1}","forge":true},
{"rule":"-a{(","branch":"-a{(","forge":true},
{"rule":"1","branch":"1","forge":true},
{"rule":"","branch":"","forge":true},
{"rule":"}[é$","branch":"]d-!","forge":false},
{"rule":"xé{é+?","branch":"xé{é+?","forge":true},
{"rule":"/A1!","branch":"xAA\\::","forge":false},
{"rule":"1][b-.(","branch":"{/-b![","forge":false},
{"rule":"(d}(","branch":"(d}(","forge":true},
{"rule":"{*}","branch":"!ébd","forge":true},
{"rule":"()^^)","branch":"()^^)","forge":true},
{"rule":"1(1A{$","branch":"*,[","forge":false},
{"rule":"A:!^\\","branch":"A:!^\\","forge":true},
{"rule":"-]:b]","branch":"-]:b]","forge":true},
{"rule":",","branch":",","forge":true},
{"rule":"[-:]","branch":"[-:]","forge":false},
{"rule":"","branch":"é:b1:d","forge":false},
{"rule":"1\\!]?d","branch":"","forge":false},
{"rule":"","branch":"bb!]+","forge":false},
{"rule":"/da","branch":"b]d[","forge":false},
{"rule":",]","branch":",]","forge":true},
{"rule":"[","branch":"[","forge":true},
{"rule":"a]{A-","branch":"-+","forge":false},
{"rule":"A?A^1","branch":"A?A^1","forge":true},
{"rule":"\\-$","branch":"","forge":false},
{"rule":"+","branch":"*[}[","forge":false},
{"rule":",b?{dw","branch":",b?{dw","forge":true},
{"rule":"d[$b1","branch":"/","forge":false},
{"rule":"})|xAA","branch":"é,!","forge":false},
{"rule":"]--","branch":"]--","forge":true},
{"rule":"],/b","branch":"],/b","forge":true},
{"rule":"/","branch":"/","forge":true},
{"rule":"*aé","branch":"*aé","forge":true},
{"rule":"{","branch":"{","forge":true},
{"rule":":aé","branch":"","forge":false},
{"rule":".\\}","branch":".\\}","forge":false},
{"rule":"wA]*","branch":"\\]!d{+","forge":false},
{"rule":"a/1x$/","branch":"[1!:/A","forge":false},
{"rule":"[.wb}d.","branch":"-!*..a","forge":false},
{"rule":".[{()a","branch":"+}","forge":false},
{"rule":"]/!:,","branch":"{Aéa","forge":false},
{"rule":"é{ba[-","branch":"],,1d/","forge":false},
{"rule":"(w*","branch":"1}.!","forge":false},
{"rule":"{?","branch":"1","forge":false},
{"rule":")","branch":",+!","forge":false},
{"rule":":","branch":"{/+A][","forge":false},
{"rule":".1{)","branch":"a\\bA","forge":false},
{"rule":"w-1+x1-","branch":"*]1}bb","forge":false},
{"rule":"xb$$","branch":"éx:]-:","forge":false},
{"rule":"++$!?*$","branch":"bA","forge":false},
{"rule":"]{","branch":"[xd*x.","forge":false}
]
+1 -2
View File
@@ -414,8 +414,7 @@ test("the tools register once there is a way to a token, and the first call mint
[
"gitea_list_repos", "gitea_create_repo", "gitea_delete_repo",
"gitea_list_issues", "gitea_get_issue", "gitea_create_issue", "gitea_close_issue", "gitea_add_comment",
// gitea_merge_pull_request is served from the watcher's bundle, beside the merge check's one writer (ADR 0323).
"gitea_list_pull_requests", "gitea_get_pull_request", "gitea_create_pull_request",
"gitea_list_pull_requests", "gitea_get_pull_request", "gitea_create_pull_request", "gitea_merge_pull_request",
"gitea_close_pull_request",
"gitea_reopen_pull_request",
"gitea_update_pull_request",
-175
View File
@@ -1,175 +0,0 @@
// **A merge check's verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341): both statuses name
// the head and the trunk commit; a trunk's move puts the merge check of every open pull request into that trunk — and
// of none into another — back to pending with the reason, and runs nothing; a verdict that lands judged against a trunk
// that has moved since is said out of date.
import { test } from "node:test";
import assert from "node:assert/strict";
import { checkOnItsWay, lateVerdictComment, outOfDateStatuses, outOfDateWords, trunkNamed, verdictAtTrunk, weighTrunks } from "../dist/trunk.js";
import { GateWriter } from "../dist/gate.js";
import { statusesFor, commentFor } from "../dist/pulls.js";
const HEAD = "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
const TRUNK = "a2c2c35bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
const MOVED = "d28ca95ccccccccccccccccccccccccccccccccc";
test("both statuses say the head and the trunk commit the verdict judged, first, whatever the summary's length", () => {
const c = { owner: "novox", repo: "mesh-host", number: 66, commit: HEAD, verdict: "pass", summary: "x", id: "build-1",
trunk: TRUNK.slice(0, 12), beside: { "mesh-catalog": "0123456789ab", "mesh-lab": "ba9876543210" },
gate: { verdict: "pass", summary: "4 of 4 compose ".repeat(20) },
"repo-check": { verdict: "fail", summary: "its merge-check.sh failed: internal/store/store.go:145:2: Declared redeclared" } };
const [gate, repo] = statusesFor(c);
for (const s of [gate, repo]) {
assert.ok(s.description.startsWith("head dfb1415aaaaa merged into trunk a2c2c35bbbbb — "), s.description);
assert.ok(s.description.length <= 140, `${s.description.length} characters`);
assert.equal(trunkNamed(s.description), "a2c2c35bbbbb");
}
assert.equal(gate.state, "success");
assert.equal(repo.state, "failure");
const comment = commentFor(c)!;
assert.match(comment, /Judged: head `dfb1415aaaaa` merged into trunk `a2c2c35bbbbb`; read beside it: mesh-catalog `0123456789ab`, mesh-lab `ba9876543210`/);
// A verdict of a build seat from before the rule names no trunk, and is said as before.
const [old] = statusesFor({ ...c, trunk: undefined, gate: { verdict: "pass", summary: "every machine composes" } });
assert.equal(old.description, "pass: every machine composes");
assert.equal(trunkNamed(old.description), undefined);
});
test("a trunk's move puts an open head's merge check out of date, naming both commits, unless a check is on its way", () => {
const verdict = (trunk: string, state = "success") => ({
"mesh/merge-gate": { state, description: `head ${HEAD.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: x` },
"mesh/repo-check": { state, description: `head ${HEAD.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: y` } });
const put = outOfDateStatuses(verdict(TRUNK), MOVED);
assert.deepEqual(put.map((s) => [s.context, s.state]), [["mesh/merge-gate", "pending"], ["mesh/repo-check", "pending"]]);
assert.equal(put[0].description, "out of date: judged against trunk a2c2c35bbbbb, the trunk is now d28ca95ccccc; " +
"judged again on a recheck or when it is merged");
assert.equal(put[0].description, outOfDateWords(TRUNK, MOVED));
assert.equal(trunkNamed(put[0].description), "a2c2c35bbbbb", "an out-of-date status still names what it judged");
// A failure is out of date too: it judged another merge.
assert.equal(outOfDateStatuses(verdict(TRUNK, "failure"), MOVED).length, 2);
// Already judged against the trunk's head; a check on its way or already out of date; no check yet: left alone.
assert.deepEqual(outOfDateStatuses(verdict(MOVED), MOVED), []);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": { state: "pending", description: "building on ace since 14:02" } }, MOVED), []);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": { state: "pending", description: put[0].description } }, MOVED), []);
assert.deepEqual(outOfDateStatuses({}, MOVED), []);
// A head with no repository check of its own gets no pending one a verdict would never set.
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": verdict(TRUNK)["mesh/merge-gate"] }, MOVED).map((s) => s.context),
["mesh/merge-gate"]);
// A verdict naming no trunk, of a build seat from before the rule, judged an earlier trunk.
assert.match(outOfDateStatuses({ "mesh/merge-gate": { state: "success", description: "pass: x" } }, MOVED)[0].description,
/judged against a trunk it does not name, the trunk is now d28ca95ccccc/);
assert.ok(checkOnItsWay({ state: "pending", description: "waiting in the build queue since 14:00" }));
assert.ok(checkOnItsWay({ state: "pending", description: "checking again: a person asked" }));
assert.ok(!checkOnItsWay({ state: "pending", description: put[0].description }));
assert.deepEqual(verdictAtTrunk(verdict(MOVED), MOVED), { current: true, onItsWay: false, judged: "d28ca95ccccc" });
assert.equal(verdictAtTrunk(verdict(TRUNK), MOVED).current, false);
});
/** A forge holding open pull requests into two branches, each head's statuses, and what was written. */
function aForge(pulls: { number: number; base: string; head: string }[], heads: Record<string, string>) {
const statuses = new Map<string, Record<string, { state: string; description?: string }>>();
const writes: string[] = [];
const verdict = (trunk: string) => ({
"mesh/merge-gate": { state: "success", description: `head x merged into trunk ${trunk.slice(0, 12)} — pass` },
"mesh/repo-check": { state: "success", description: `head x merged into trunk ${trunk.slice(0, 12)} — pass` } });
for (const p of pulls) statuses.set(p.head, verdict(heads[p.base]));
const forge = {
listOpenPulls: async () => pulls.map((p) => ({ number: p.number, title: "t", state: "open", merged: false, base: p.base,
head: "feat", head_sha: p.head, html_url: `http://forge/pulls/${p.number}` })),
branchHead: async (_o: string, _r: string, branch: string) => heads[branch],
setCommitStatus: async (_o: string, _r: string, sha: string, s: { context: string; state: string; description: string }) => {
writes.push(`${sha} ${s.context} ${s.state}`);
statuses.set(sha, { ...statuses.get(sha), [s.context]: { state: s.state, description: s.description } });
},
gateOf: async () => undefined,
gateHistory: async () => [],
headChecks: async (_o: string, _r: string, sha: string) => statuses.get(sha) ?? {},
};
return { forge, writes, statuses };
}
test("a push to a trunk puts every open pull request into it out of date — twenty-five of them — and none into another", async () => {
const heads: Record<string, string> = { main: TRUNK, release: "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" };
const pulls = Array.from({ length: 25 }, (_, i) => ({ number: i + 1, base: "main", head: `${String(i).padStart(2, "0")}`.repeat(20) }));
pulls.push({ number: 99, base: "release", head: "9".repeat(40) });
const { forge, writes, statuses } = aForge(pulls, heads);
const gates = new GateWriter(forge);
const trunks: Record<string, string> = {};
// First sight: each trunk is taken as it is, and nothing is put out of date.
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), true);
assert.deepEqual(writes, []);
assert.equal(trunks["novox/mesh-host:main"], TRUNK);
// Nothing moved: nothing read past the trunks, nothing written.
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), false);
heads.main = MOVED;
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), true);
assert.equal(writes.length, 50, "both statuses of each of the 25 pull requests into main");
for (const p of pulls.filter((p) => p.base === "main")) {
const s = statuses.get(p.head)!;
assert.equal(s["mesh/merge-gate"].state, "pending");
assert.match(s["mesh/merge-gate"].description!, /judged against trunk a2c2c35bbbbb, the trunk is now d28ca95ccccc/);
assert.equal(s["mesh/repo-check"].state, "pending");
}
assert.equal(statuses.get("9".repeat(40))!["mesh/merge-gate"].state, "success", "a pull request into another branch is left");
assert.equal(trunks["novox/mesh-host:main"], MOVED);
// Weighed again, nothing more: the move was taken once, and the out-of-date statuses are left as they are.
writes.length = 0;
heads.main = "f".repeat(40);
await weighTrunks(forge, gates, trunks, "novox", "mesh-host");
assert.deepEqual(writes, [], "a check already out of date is not written again");
});
test("a trunk that cannot be read is weighed again on the next pass, never recorded half", async () => {
const heads: Record<string, string> = { main: TRUNK };
const { forge } = aForge([{ number: 1, base: "main", head: "1".repeat(40) }], heads);
const trunks: Record<string, string> = { "novox/mesh-host:main": TRUNK };
heads.main = MOVED;
const failing = { ...forge, setCommitStatus: async () => { throw new Error("the forge is away"); } };
assert.equal(await weighTrunks(failing, new GateWriter(failing), trunks, "novox", "mesh-host"), false);
assert.equal(trunks["novox/mesh-host:main"], TRUNK, "not recorded: the next pass weighs it again");
});
test("the trunk's move takes its turn after a verdict in flight, and weighs what the forge holds then", async () => {
const heads: Record<string, string> = { main: TRUNK };
const { forge, statuses } = aForge([{ number: 1, base: "main", head: HEAD }], heads);
const gates = new GateWriter(forge);
// A verdict judged against the trunk's new head, in flight when the move is weighed: it lands first, and is left.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
const verdictLanded = gates.verdict("novox", "mesh-host", HEAD, async () => {
await held;
statuses.set(HEAD, { "mesh/merge-gate": { state: "success", description: `head x merged into trunk ${MOVED.slice(0, 12)} — pass` } });
});
const moved = gates.outOfDate("novox", "mesh-host", HEAD, (checks) => outOfDateStatuses(checks, MOVED));
release();
await verdictLanded;
assert.deepEqual(await moved, [], "a verdict judged against the trunk's head is not put out of date");
assert.equal(statuses.get(HEAD)!["mesh/merge-gate"].state, "success");
});
test("a verdict judged against a trunk newer than the pass's first read is not put out of date", async () => {
const NEWER = "ffffffffffffffffffffffffffffffffffffffff";
const heads: Record<string, string> = { main: TRUNK };
const { forge, statuses } = aForge([{ number: 1, base: "main", head: HEAD }], heads);
const trunks: Record<string, string> = { "novox/mesh-host:main": TRUNK };
// The pass reads main at MOVED; by the head's turn the trunk moved again, and the head's check answered for it.
let reads = 0;
const moving = { ...forge, branchHead: async () => (reads++ === 0 ? MOVED : NEWER) };
statuses.set(HEAD, { "mesh/merge-gate": { state: "success", description: `head x merged into trunk ${NEWER.slice(0, 12)} — pass` } });
await weighTrunks(moving, new GateWriter(moving), trunks, "novox", "mesh-host");
assert.equal(statuses.get(HEAD)!["mesh/merge-gate"].state, "success", "a fresh verdict was put out of date");
});
test("a verdict that landed out of date keeps its account on the pull request", () => {
const words = outOfDateWords(TRUNK, MOVED);
const own = lateVerdictComment({ commit: HEAD, trunk: TRUNK.slice(0, 12), verdict: "fail" }, "**Merge check** at `dfb1415a`\n\nthe report", words);
assert.match(own, /the report/);
assert.match(own, /\*\*Out of date when it landed\*\*: out of date: judged against trunk a2c2c35bbbbb/);
const bare = lateVerdictComment({ commit: HEAD, trunk: TRUNK.slice(0, 12), verdict: "pass" }, null, words);
assert.match(bare, /head dfb1415aaaaa merged into trunk a2c2c35bbbbb, pass/);
});
+105 -2
View File
@@ -63,8 +63,6 @@ export function getGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return namesChecked(allGiteaTools(gitea));
}
// gitea_merge_pull_request is served from the watcher's bundle (merge.ts, index.ts), beside the merge check's one
// writer (novox/hq issue 456, ADR 0323).
function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return [
// ---- Repositories ----
@@ -265,6 +263,111 @@ function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
}),
}),
},
{
name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head " +
"and every status its base branch's protection requires succeeded there (mesh/merge-gate, mesh/repo-check), " +
"optionally deleting the source branch afterwards. Refused otherwise, with nothing merged: no tool merges past " +
"a check, and none merges a head that moved after it was reviewed.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
number: { type: "number", description: "the PR number" },
head: { type: "string", description: "the commit you reviewed, as its sha or at least its first 12 characters: " +
"refused unless it is the pull request's head" },
method: { type: "string", description: "merge (default) | fast-forward-only | squash | rebase | rebase-merge. " +
"Only merge and fast-forward-only can be read back to prove the head you named is what landed" },
delete_branch: { type: "boolean", description: "delete the source branch after merge (default true)" },
},
run: async (args) => {
const owner = String(args.owner);
const repo = String(args.repo);
const number = Number(args.number);
const method = args.method ? String(args.method) : "merge";
if (!MERGE_METHODS.has(method)) throw new Refused(`${method} is not a merge method: ${[...MERGE_METHODS].join(", ")}`);
const deleteBranch = args.delete_branch === undefined ? true : Boolean(args.delete_branch);
// **Only a head whose required statuses all succeeded** (novox/hq ADR 0266): read here rather than left
// to the forge, whose administrator — the account this tool's token is minted for — may merge past a
// rule that lets it. A rule that cannot be read is no rule passed.
// **Only the head that was reviewed** (novox/hq issue 381): mesh-controller#197 was reviewed at 065cfa0d,
// force-pushed to 3ced96f, checked green there, and merged at 3ced96f, because this tool merged whatever the
// head was and nothing told it which head the review had approved. The caller names it; a head that moved
// since is refused, and the forge is handed the same head, so a push between this read and the merge is
// refused by the forge too.
const named = args.head === undefined ? "" : String(args.head).trim().toLowerCase();
if (!named) {
throw new Refused(`${owner}/${repo}#${number} is not merged: name the head you reviewed (head), so that only ` +
`that commit can land (novox/hq issue 381). Nothing was merged`);
}
// Twelve characters, not seven: an author who force-pushes can make a commit whose first seven match the
// reviewed one in seconds, and the review would then merge it. Twelve cannot be matched that way.
if (!REVIEWED_HEAD.test(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: head ${named} does not name one commit — give its ` +
`sha, or at least its first 12 characters. Nothing was merged`);
}
const pull = await gitea.getPullRequest(owner, repo, number);
if (pull.merged) throw new Refused(`${owner}/${repo}#${number} is already merged`);
if (!pull.head_sha || !pull.base) throw new Refused(`${owner}/${repo}#${number} names no head commit or base branch`);
if (!pull.head_sha.toLowerCase().startsWith(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its head is ${pull.head_sha.slice(0, 12)}, not the ` +
`${named} you named. If you reviewed an earlier head, review the one it has now, then name it. Nothing was merged`);
}
const rule = await gitea.branchProtection(owner, repo, pull.base);
// Only into a branch whose protection requires the checks: a pull request whose base was moved to a
// branch nothing guards would merge past every check (novox/hq ADR 0266).
if (!rule || !rule.enable_status_check || !(rule.status_check_contexts ?? []).filter(Boolean).length) {
throw new Refused(`${owner}/${repo}#${number} merges into ${pull.base}, which no protection rule requiring ` +
`the checks guards: a tool merges only into such a branch (novox/hq ADR 0266). Nothing was merged`);
}
const statuses = newestByContext((await gitea.combinedStatus(owner, repo, pull.head_sha)).statuses);
const verdict = mergeableUnder(pull.base, rule, statuses);
if (verdict.mergeable !== true) {
throw new Refused(`${owner}/${repo}#${number} is not merged: ${verdict.says}. A pull request merges once ` +
`its checks succeed; no tool merges past them (novox/hq ADR 0266)`);
}
await gitea.mergePullRequest(owner, repo, number, method, deleteBranch, pull.head_sha);
// The merge commit only exists now; answered so the caller can follow what is built from it.
// pull.merged is the events entrypoint's to announce (index.ts), once, within its poll.
// From here the merge has happened. Nothing below may read as if it had not: a read-back that fails or cannot
// be done is said beside merged: true, and only a proven other head is an error — one that says the merge happened.
const answer: Record<string, unknown> = { merged: true, number, method, head: pull.head_sha, deleted_branch: deleteBranch };
let landed: string | undefined;
try {
const merged = await gitea.getPullRequest(owner, repo, number);
answer.merge_commit_sha = merged.merge_commit_sha;
// What landed, read back from the commit the merge made rather than from the forge's record of the pull
// request, which is the very thing a race would leave behind. A merge commit's second parent is the head it
// joined; a fast-forward lands the head itself. A squash or a rebase makes new commits that name no head.
if (!merged.merge_commit_sha) {
answer.read_back = "the forge named no merge commit, so what landed cannot be read back";
} else if (method === "merge") {
const parents = await gitea.commitParents(owner, repo, merged.merge_commit_sha);
if (parents.length === 2) landed = parents[1];
else answer.read_back = `the merge commit has ${parents.length} parent(s), not two: what landed cannot be read back`;
} else if (method === "fast-forward-only") {
// That the forge names the head itself as a fast-forward's merge commit is assumed, not seen on this forge:
// a different sha is said, not raised as another head having landed.
if (merged.merge_commit_sha === pull.head_sha) landed = merged.merge_commit_sha;
else answer.read_back = `the forge named ${merged.merge_commit_sha.slice(0, 12)} as this fast-forward's commit, ` +
`not the head: that the head landed could not be confirmed`;
} else {
answer.read_back = `a ${method} makes new commits that name no head: what landed cannot be read back`;
}
} catch (err) {
answer.read_back = `the merge happened, but reading back what landed failed: ${(err as Error).message}`;
}
// Loud, not a field beside merged: true that a caller reading only that would miss. The merge cannot be taken
// back by this tool; what must not happen is that it reads as the merge that was asked for.
if (landed !== undefined && landed !== pull.head_sha) {
throw new Error(`${owner}/${repo}#${number} WAS merged (${answer.merge_commit_sha}), but the forge merged ` +
`${landed.slice(0, 12)}, not the ${pull.head_sha.slice(0, 12)} it was handed and you reviewed: a commit nobody ` +
`reviewed has landed. Read it before anything is built from it (novox/hq issue 381)`);
}
if (landed !== undefined) answer.landed = landed;
return answer;
},
},
{
name: "gitea_close_pull_request",
description: "Close a pull request without merging it — one whose work landed elsewhere, or was abandoned.",
-250
View File
@@ -1,250 +0,0 @@
// **A merge check's verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341).
//
// The build seat checks a pull request's head merged into its trunk's head at that moment — the verdict's trunk
// commit — and the verdict names both: "head 1a2b3c4d5e6f merged into trunk 7a8b9c0d1e2f", in both statuses'
// descriptions. When the trunk moves, every open pull request's verdict into it is out of date:
//
// - **the forge's holder** (index.ts) hears the move on its own — it reads each trunk's head on every pass that
// looks at the repository, never through mesh-delivery (ADR 0239 §36) — and sets mesh/merge-gate and
// mesh/repo-check on the head of every open pull request into that trunk back to pending, with the reason.
// Nothing is run for it: a recheck, a new head, or the merge tool asks the check again;
// - **the merge tool** (tools/index.ts) compares the trunk commit the head's verdict names with the trunk's head at
// the moment it is asked, merges nothing when they differ, asks the head's check once — the same announcement a
// new head makes — and says how long the last check of that repository took. It never merges by itself after.
//
// Pure functions and two small files of the module's own state, so this is tested without a forge.
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type { CommitStatus, GiteaClient, GiteaPull, HeadCheck } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT } from "./pulls.js";
/** A commit as a verdict names it: its first twelve characters. */
export function twelve(sha: string | undefined): string {
return String(sha ?? "").slice(0, 12).toLowerCase();
}
/** Whether two commits are one, however either is abbreviated (seven characters at least). */
export function sameCommit(a: string | undefined, b: string | undefined): boolean {
const x = String(a ?? "").toLowerCase(), y = String(b ?? "").toLowerCase();
if (x.length < 7 || y.length < 7) return false;
return x.length <= y.length ? y.startsWith(x) : x.startsWith(y);
}
/** What a verdict judged, as both statuses' descriptions begin (ADR 0323 §3). */
export function judgedWords(head: string, trunk: string): string {
return `head ${twelve(head)} merged into trunk ${twelve(trunk)}`;
}
/** The trunk commit a merge check's status names: the one its verdict judged ("merged into trunk …"), or, for a
* verdict put out of date, the one it had judged ("judged against trunk …"). Undefined when it names none, as a
* verdict of a build seat from before ADR 0323 does. */
export function trunkNamed(description: string | undefined): string | undefined {
const m = /(?:merged into|judged against) trunk ([0-9a-f]{7,64})\b/i.exec(description ?? "");
return m ? m[1].toLowerCase() : undefined;
}
/** A verdict out of date, as its statuses say it (ADR 0323 §4). */
export function outOfDateWords(judged: string | undefined, now: string): string {
const was = judged ? `judged against trunk ${twelve(judged)}` : "judged against a trunk it does not name";
return `out of date: ${was}, the trunk is now ${twelve(now)}; judged again on a recheck or when it is merged`;
}
/** Whether a pending gate says a check of the head is asked and not yet answered: waiting in the build queue,
* building on a machine (gate.ts), or asked again by a recheck (mesh-delivery). Such a check is not asked again. */
export function checkOnItsWay(gate: HeadCheck | undefined): boolean {
if (!gate || gate.state !== "pending") return false;
const d = gate.description ?? "";
return d.startsWith("waiting in the build queue") || d.startsWith("building on ") || d.startsWith("checking again");
}
/**
* The statuses a trunk's move puts back to pending on one open pull request's head, from what the forge holds on it
* now: none when it has no merge check yet, when a check is pending (on its way, or already out of date — its
* verdict is weighed against the trunk when it lands), or when its verdict judged the trunk's head. Otherwise the
* gate, and the repository's own check when the head has one — a pending repo-check a verdict will not set would
* stand for ever (novox/hq issue 305's rule).
*/
export function outOfDateStatuses(checks: Record<string, HeadCheck>, trunkHead: string): CommitStatus[] {
const gate = checks[CHECK_CONTEXT];
if (!gate || gate.state === "pending") return [];
const judged = trunkNamed(gate.description);
if (judged && sameCommit(judged, trunkHead)) return [];
const description = outOfDateWords(judged, trunkHead);
const out: CommitStatus[] = [{ state: "pending", context: CHECK_CONTEXT, description }];
if (checks[REPO_CHECK_CONTEXT]) out.push({ state: "pending", context: REPO_CHECK_CONTEXT, description });
return out;
}
/** Where the merge tool stands with a head's verdict and the trunk's head (ADR 0323 §5). */
export interface VerdictAtTrunk {
/** The verdict judged the trunk's head: the required statuses decide, as before. */
current: boolean;
/** A check of the head is asked and not yet answered: nothing is asked again. */
onItsWay: boolean;
/** The trunk commit the verdict names, when it names one. */
judged?: string;
}
export function verdictAtTrunk(checks: Record<string, HeadCheck>, trunkHead: string): VerdictAtTrunk {
const gate = checks[CHECK_CONTEXT];
const judged = trunkNamed(gate?.description);
if (checkOnItsWay(gate)) return { current: false, onItsWay: true, judged };
return { current: !!gate && gate.state !== "pending" && !!judged && sameCommit(judged, trunkHead), onItsWay: false, judged };
}
/** The pending words the merge tool's ask of a check is said with: in the build queue from now, as an announced head's. */
export function askedWords(at: Date, trunkHead: string): string {
const two = (n: number) => String(n).padStart(2, "0");
return `waiting in the build queue since ${two(at.getHours())}:${two(at.getMinutes())}: asked by the merge tool ` +
`against trunk ${twelve(trunkHead)}`;
}
/** The comment a verdict that landed out of date leaves (ADR 0323 §7: kept and shown): its own account when it has
* one, else what it judged and said, and that it is out of date. */
export function lateVerdictComment(c: { commit: string; trunk?: string; verdict: string }, account: string | null,
outOfDate: string): string {
const own = account ?? `**Merge check** at \`${c.commit.slice(0, 8)}\`: ${judgedWords(c.commit, c.trunk ?? "")}, ${c.verdict}`;
return `${own}\n\n**Out of date when it landed**: ${outOfDate}`;
}
// --- a trunk's move, weighed --------------------------------------------------------------------------------------
/** What weighing a trunk's move reads of the forge. */
export interface TrunkForge {
listOpenPulls(owner: string, repo: string): Promise<GiteaPull[]>;
branchHead(owner: string, repo: string, branch: string): Promise<string>;
}
/** What it writes with: the merge check's one writer (gate.ts), which decides in the head's own turn. */
export interface OutOfDateWriter {
outOfDate(owner: string, repo: string, sha: string,
decide: (checks: Record<string, HeadCheck>) => CommitStatus[] | Promise<CommitStatus[]>, target?: string): Promise<CommitStatus[]>;
}
/**
* Every trunk of one repository that an open pull request merges into, read; for one that moved since it was last read
* (`trunks`, by `owner/repo:branch`, which this keeps current), the merge check of every open pull request into it put
* out of date (ADR 0323 §4). A trunk first seen is recorded as it is and nothing is put out of date. Bounded however
* many are open: one listing of the open pull requests (paged), one read of each trunk, then per open pull request into
* a trunk that moved one read and at most two writes, one at a time. A trunk that could not be weighed whole is not
* recorded, so the next pass weighs it again. Answers whether the record changed.
*/
export async function weighTrunks(forge: TrunkForge, gates: OutOfDateWriter, trunks: Record<string, string>, owner: string,
name: string): Promise<boolean> {
let changed = false;
try {
const open = (await forge.listOpenPulls(owner, name)).filter((p) => p.state === "open" && !!p.head_sha && !!p.base);
for (const base of new Set(open.map((p) => String(p.base)))) {
const key = `${owner}/${name}:${base}`.toLowerCase();
const head = await forge.branchHead(owner, name, base);
const before = trunks[key];
if (before && sameCommit(before, head)) continue;
if (before) {
let put = 0;
for (const pull of open.filter((p) => p.base === base)) {
// The trunk is read again in the head's own turn: a verdict judged against a head newer than the pass's first
// read — the trunk moved again meanwhile, and a check answered — is not put out of date.
const written = await gates.outOfDate(owner, name, String(pull.head_sha),
async (checks) => outOfDateStatuses(checks, await forge.branchHead(owner, name, base)), pull.html_url || undefined);
if (written.length) put++;
}
console.log(`[gitea] ${owner}/${name}'s ${base} moved from ${twelve(before)} to ${twelve(head)}: ${put} open pull ` +
`request(s) into it put out of date, nothing run (novox/hq ADR 0323)`);
}
trunks[key] = head;
changed = true;
}
} catch (err) {
console.error(`[gitea] ${owner}/${name}: whether a trunk moved could not be weighed, so it is weighed again on the next ` +
`pass — ${err instanceof Error ? err.message : err}`);
}
return changed;
}
// --- how long the last check of a repository took -------------------------------------------------------------
/** How long one repository's last check took, and when it was heard: kept by the watcher, read by the merge tool. */
export interface Took {
took: string;
at: string;
head?: string;
}
/** The file the watcher keeps each repository's last check's duration in, beside its other records. */
export function tookFile(stateDir: string | undefined): string | null {
return stateDir ? `${stateDir.replace(/\/+$/, "")}/checks-took.json` : null;
}
function readTookAll(file: string | null): Record<string, Took> {
if (!file || !existsSync(file)) return {};
try {
return JSON.parse(readFileSync(file, "utf8")) as Record<string, Took>;
} catch {
return {};
}
}
/** Keep how long a repository's check took. A record that cannot be written is said, never fatal to the verdict. */
export function keepTook(file: string | null, repository: string, took: Took): void {
if (!file || !took.took) return;
try {
const all = readTookAll(file);
all[repository.toLowerCase()] = took;
mkdirSync(dirname(file), { recursive: true });
writeFileSync(file + ".tmp", JSON.stringify(all));
renameSync(file + ".tmp", file);
} catch (err) {
console.error(`[gitea] how long ${repository}'s check took could not be kept — ${err instanceof Error ? err.message : err}`);
}
}
/** How long a repository's last check took, in words for the merge tool's refusal. */
export function tookWords(file: string | null, repository: string): string {
const t = readTookAll(file)[repository.toLowerCase()];
if (!t) return `how long a check of ${repository} takes is not known yet: no verdict of it was heard since this was kept`;
return `the last check of ${repository} took ${t.took} (heard ${t.at})`;
}
// --- the announcement of a head, shared by the watcher and the merge tool ---------------------------------------
/** `pull.updated` for one open pull request's head: what the controller asks the build seat to check (novox/hq to-be
* 45 §9). The merge tool's ask is this same announcement (ADR 0323 §5). */
export async function announcementOf(client: GiteaClient, owner: string, name: string, cloneURL: string | undefined,
pull: GiteaPull): Promise<Record<string, unknown>> {
const files = await client.listPullFiles(owner, name, pull.number);
const head = String(pull.head_sha);
// What the controller maps the change onto the mesh's module graph with (novox/hq ADR 0237 as amended): which
// changed directories hold a module at the head — the merge's rule (issue 278) — and whether the head holds the
// repository's own merge-check.sh. Not said when it could not be read; the controller then reads the change as
// touching everything built from the repository.
let moduleDirs: string[] | null = null;
let mergeCheck: boolean | null = null;
try {
if (!files.truncated) moduleDirs = await client.moduleDirsAt(owner, name, head, files.paths);
mergeCheck = await client.holdsFile(owner, name, head, "merge-check.sh");
} catch (err) {
console.error(`[gitea] ${owner}/${name}#${pull.number}: what the head holds could not be read — ${err instanceof Error ? err.message : err}`);
}
// The base branch's head as the head is announced: the trunk commit a verdict no build judges is said against
// (novox/hq ADR 0323 §3). Not said when it cannot be read; that verdict then names none.
const baseSHA = pull.base ? await client.branchHead(owner, name, String(pull.base)).catch(() => undefined) : undefined;
return {
owner,
repo: name,
number: pull.number,
...(baseSHA ? { base_sha: baseSHA } : {}),
title: pull.title,
body: pull.body,
base: pull.base,
head: pull.head,
head_sha: pull.head_sha,
clone_url: cloneURL,
html_url: pull.html_url,
paths: files.paths,
paths_truncated: files.truncated,
removed: files.removed,
...(moduleDirs ? { module_dirs: moduleDirs, module_dirs_said: true } : {}),
...(mergeCheck !== null ? { merge_check: mergeCheck, merge_check_said: true } : {}),
};
}
@@ -51,10 +51,6 @@ type ChecksAnswer struct {
Statuses []CheckStatus `json:"statuses"`
// Verdict is the merge check's verdict whole, as this owner heard it for this commit.
Verdict *Verdict `json:"verdict,omitempty"`
// Judged is what the verdict judged: the head merged into the trunk commit (novox/hq ADR 0323 §3); OutOfDate
// why it no longer counts, when its trunk has moved since.
Judged string `json:"judged,omitempty"`
OutOfDate string `json:"out_of_date,omitempty"`
// Merge is whether the branch's protection lets the commit merge; Mergeable nil when it is not known.
Merge ForgeMerge `json:"merge"`
// Forge is why the forge could not be read, when it could not.
@@ -69,11 +65,6 @@ type ChecksOf struct {
Since string `json:"since"`
Waits string `json:"waits,omitempty"`
Group string `json:"group,omitempty"`
// Class, Opened and MergedAt are what the issue tracker measures a fix's time by (novox/hq ADR 0313): the
// class of what it moves, when its pull request was first heard, and when it merged.
Class string `json:"class,omitempty"`
Opened string `json:"opened,omitempty"`
MergedAt string `json:"merged_at,omitempty"`
}
// Checks is the `checks` verb: a repository and a pull request's number, or a repository and a commit.
@@ -113,16 +104,9 @@ func (h *Holder) Checks(repository string, number int, commit string) (*ChecksAn
}
if d != nil {
a.Delivery = &ChecksOf{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), Waits: d.waits,
Group: d.Group, Class: fixClass(&d.Delivery)}
if !d.opened.IsZero() {
a.Delivery.Opened = d.opened.UTC().Format(time.RFC3339)
}
if !d.MergedAt.IsZero() {
a.Delivery.MergedAt = d.MergedAt.UTC().Format(time.RFC3339)
}
Group: d.Group}
if d.Check != nil && sameCommit(d.Commit, a.Commit) {
a.Verdict = d.Check
a.Judged, a.OutOfDate = d.Check.Judged(), d.outOfDate()
}
}
@@ -155,7 +139,6 @@ type checksDelivery struct {
Delivery
waits string
groupSaid string
opened time.Time
}
// deliveryOf is the delivery of a commit — its head, or the commit it merged as — else, for a pull request,
@@ -184,7 +167,7 @@ func (h *Holder) deliveryOf(repository string, number int, commit string) *check
if found == nil {
return nil
}
c := &checksDelivery{Delivery: *found, waits: h.waitsFor(found), opened: h.openedOf(found)}
c := &checksDelivery{Delivery: *found, waits: h.waitsFor(found)}
if found.Check != nil {
v := *found.Check
c.Check = &v
@@ -278,12 +261,6 @@ func (a *ChecksAnswer) says() string {
default:
merge = "NOT mergeable — " + merge
}
if a.Judged != "" {
what += " (" + a.Judged + ")"
}
if a.OutOfDate != "" {
merge += "; its verdict is " + a.OutOfDate
}
return what + ": " + statuses + "; " + merge
}
@@ -451,7 +451,7 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
State: Ready, Since: t0, Created: t0.Add(-time.Hour), Plan: planMoving(b, "b"), Check: &Verdict{Gate: "pass"}}
g := &Group{ID: "feat/k", Members: []string{aID, bID}, Order: []string{aID, bID}, Closed: true,
OrderOf: map[string]string{aID: aHead, bID: bHead},
Check: &GroupCheck{Asked: "check-old", At: t0, Commits: c.commits}}
Check: &GroupCheck{Asked: "check-1", At: t0, Commits: c.commits}}
for _, d := range []*Delivery{da, db} {
_ = store.PutDelivery(d)
}
@@ -463,7 +463,7 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
if s := w.state(aID); s != Held {
t.Fatalf("before its composed verdict it is %s", s)
}
w.h.Checked(CheckedEvent{Group: "feat/k", ID: "check-old", Verdict: c.verdict, Summary: "composed: " + c.verdict})
w.h.Checked(CheckedEvent{Group: "feat/k", ID: "check-1", Verdict: c.verdict, Summary: "composed: " + c.verdict})
w.settleAll()
if s := w.state(aID); s != c.want {
t.Fatalf("its composed check %s, and it is %s (%s), want %s", c.name, s, w.delivery(aID).HeldWhy, c.want)
@@ -471,10 +471,6 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
if c.want == Delivering && !containsString(ctl.delivers, "plan-a") {
t.Fatalf("lifted, its walk was not let go: %v", ctl.delivers)
}
// A composed check of other heads is asked again for these heads (novox/hq ADR 0323 §6).
if c.name == "passed for other heads" && !containsString(ctl.checks, "feat/k:"+aID+","+bID) {
t.Fatalf("the composed check was not asked again for the heads that merged: %v", ctl.checks)
}
})
}
}
@@ -14,10 +14,8 @@ import (
// the forge's holder, asked through its tools). Owed, never lost: kept with the delivery before anything is
// tried, and tried again until done.
// transitionEvent is what `mesh-delivery.transition` says: no secret, no address. opened is when its pull
// request was first heard (zero when not known); with its class and its merge's moment it is what the issue
// tracker measures a fix's time by (novox/hq ADR 0313 §2, §4).
func transitionEvent(d *Delivery, t Transition, opened time.Time) map[string]any {
// transitionEvent is what `mesh-delivery.transition` says: no secret, no address.
func transitionEvent(d *Delivery, t Transition) map[string]any {
e := map[string]any{"id": d.ID, "repository": d.Repository, "commit": d.Commit, "from": t.From, "to": t.To,
"event": t.Event, "why": t.Why, "at": t.At}
if t.By != "" {
@@ -38,57 +36,9 @@ func transitionEvent(d *Delivery, t Transition, opened time.Time) map[string]any
if d.Plan != nil {
e["plan"] = d.Plan.Summary
}
if c := fixClass(d); c != "" {
e["class"] = c
}
if !opened.IsZero() {
e["opened"] = opened.UTC()
}
if !d.MergedAt.IsZero() {
e["merged_at"] = d.MergedAt.UTC()
}
return e
}
// fixClass is the class of what a delivery moves (novox/hq ADR 0313 §2): its plan's, as the planner said it;
// none for a plan that moves no module, from a planner that did not say a class; else its walk's class once its
// delivery time is known; empty when nothing says it.
func fixClass(d *Delivery) string {
switch {
case d.Plan != nil && d.Plan.Class != "":
return d.Plan.Class
case d.Plan != nil && len(d.Plan.Moved) == 0 && len(d.Plan.Dependents) == 0 && len(d.Plan.New) == 0:
return classNone
case d.Time != nil:
return d.Time.Class
}
return ""
}
// classNone is the class of a delivery that moves no module: nothing for a walk to move, and no fix budget.
const classNone = "none"
// openedOf is when a delivery's pull request was first heard: the earliest head of it this owner holds, never
// one made from the forge's word at its merge; zero when none is held or it has no pull request. Under the lock.
func (h *Holder) openedOf(d *Delivery) time.Time {
if d.Number <= 0 {
return time.Time{}
}
var first time.Time
for _, c := range h.deliveries {
if c.FromForge || c.Number != d.Number || !strings.EqualFold(c.Repository, d.Repository) || c.Created.IsZero() {
continue
}
if first.IsZero() || c.Created.Before(first) {
first = c.Created
}
}
if first.IsZero() && !d.FromForge {
first = d.Created
}
return first
}
// noteLine is one transition as the commit's note keeps it.
func noteLine(d *Delivery, t Transition) string {
by := ""
@@ -154,9 +104,6 @@ func statusOf(d *Delivery) (string, string) {
case Superseded:
return "warning", clip("superseded: "+why, 140)
case Stopped:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its change reached the machines (novox/hq issue 480)
}
return "error", clip("stopped: "+why, 140)
case Retired:
return "success", "history: merged before this owner delivered anything, never walked"
@@ -177,9 +124,6 @@ const viewMarker = "<!-- mesh-delivery:view -->"
func ViewBody(d *Delivery, g *Group) string {
var b strings.Builder
fmt.Fprintf(&b, "%s\n**Delivery** `%s` — **%s** since %s\n\n", viewMarker, d.ID, d.State, d.Since.UTC().Format(time.RFC3339))
if d.coveredStop() {
fmt.Fprintf(&b, "**Covered** after its stop: a later merge that contains it was delivered: %s\n\n", d.CoveredBy)
}
if d.HeldWhy != "" && d.State == Held {
fmt.Fprintf(&b, "Held for a person: %s — `mesh-delivery.release` with why lets it go on.\n\n", d.HeldWhy)
}
@@ -334,16 +278,6 @@ func (h *Holder) Flush() {
d.Owed = owed(d.Owed)
h.keep(d)
}
// Counted for delivery_status, so a window's writes can be judged against its tries (novox/hq issue 407).
h.tried.last = h.Now()
switch {
case err == nil:
h.tried.done++
case dropped != "":
h.tried.givenUp++
default:
h.tried.failed++
}
if dropped != "" {
h.refuse(fmt.Errorf("%s: its owed %s given up — %s: %v", j.id, j.e.Kind, dropped, err))
}
+15 -205
View File
@@ -67,16 +67,6 @@ type Holder struct {
// leaveRefused is, by walk, what the controller answered when it refused both to leave a stopped merge out and
// to stop the walk whole.
leaveRefused map[string]string
// tried counts the owed effects Flush tried since this owner started, by how each ended (novox/hq issue 407).
tried owedTries
}
// owedTries is how many owed effects were tried since this owner started, and when the last was: three numbers
// and a time, however many were tried. Not kept: a restarted owner counts from its start, and says since when, so a
// window's count is the difference of two readings in the same run (novox/hq issue 407).
type owedTries struct {
done, failed, givenUp int
last time.Time
}
// unmatched is a walk no delivery is known for yet: its merge not heard, or merged before this owner held.
@@ -172,10 +162,9 @@ func (h *Holder) refuse(err error) {
// moved records what a delivery's transitions owe outside, and keeps it.
func (h *Holder) moved(d *Delivery, ts []Transition) {
now := h.Now()
opened := h.openedOf(d)
for _, t := range ts {
h.Logf("[mesh-delivery] %s: %s → %s (%s): %s", d.ID, orNothing(t.From), t.To, t.Event, t.Why)
d.Owed = append(d.Owed, Effect{Kind: EffectEmit, Line: string(mustJSON(transitionEvent(d, t, opened))), Since: now})
d.Owed = append(d.Owed, Effect{Kind: EffectEmit, Line: string(mustJSON(transitionEvent(d, t))), Since: now})
line := noteLine(d, t)
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: line, Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) && afterTheTrunk(t.To) {
@@ -280,8 +269,6 @@ type PullEvent struct {
ModuleDirs []string `json:"module_dirs,omitempty"`
ModuleDirsSaid bool `json:"module_dirs_said,omitempty"`
HeadChecks map[string]string `json:"head_checks,omitempty"`
// BaseSHA is the base branch's head when the head was announced (novox/hq ADR 0323).
BaseSHA string `json:"base_sha,omitempty"`
}
func (p PullEvent) repository() string { return strings.ToLower(p.Owner + "/" + p.Repo) }
@@ -299,22 +286,10 @@ func (h *Holder) PullUpdated(p PullEvent) {
h.mu.Lock()
defer h.mu.Unlock()
id := IDOf(p.repository(), p.HeadSHA)
now := h.Now()
if d := h.deliveries[id]; d != nil {
// Announced again: nothing new — but for a delivery whose verdict is out of date, the announcement is its
// check asked again against the trunk's head (the merge tool's ask, novox/hq ADR 0323 §5), and its bound
// runs from now (§7). The base head it was announced with is kept, for a check this owner asks after it.
if p.BaseSHA != "" && d.BaseSHA != p.BaseSHA {
d.BaseSHA = p.BaseSHA
h.keep(d)
}
if d.State == Proposed && d.outOfDate() != "" && d.AskedAgain == nil {
d.AskedAgain = &now
h.keep(d)
}
return
return // announced again: nothing new
}
replaced := ""
now := h.Now()
for _, old := range h.deliveries {
if old.Repository == p.repository() && old.Number == p.Number && !sameCommit(old.Commit, p.HeadSHA) &&
!old.State.Final() && old.MergedAs == "" {
@@ -322,14 +297,13 @@ func (h *Holder) PullUpdated(p PullEvent) {
h.settleAndKeep(old)
if old.Group != "" {
h.leaveGroup(old)
replaced = old.ID
}
}
}
d := &Delivery{ID: id, Repository: p.repository(), Commit: p.HeadSHA, Number: p.Number, Title: p.Title,
Base: p.Base, Branch: p.Head, HTMLURL: p.HTMLURL, CloneURL: p.CloneURL, Paths: p.Paths,
PathsTruncated: p.PathsTruncated, Removed: p.Removed, ModuleDirs: p.ModuleDirs, ModuleDirsSaid: p.ModuleDirsSaid,
After: AfterIn(p.Body), Created: now, BaseSHA: p.BaseSHA}
After: AfterIn(p.Body), Created: now}
t, err := Apply(d, EvAnnounced, false, Facts{Now: now, Why: fmt.Sprintf("%s#%d's head announced", d.Repository, d.Number)})
if err != nil {
h.refuse(err)
@@ -343,45 +317,7 @@ func (h *Holder) PullUpdated(p PullEvent) {
ts = append(ts, settle(d, Facts{Now: now})...)
}
h.moved(d, ts)
if !h.takePlaceOf(replaced, d) {
h.joinGroup(d)
}
}
// takePlaceOf puts a member's new head — a rebase, a push — in its group in the place of the head it superseded,
// when the group is closed: a member of it already reached the trunk, and a closed group takes no new member. The
// new head has its own check, and the group's composed check runs again for the new heads rather than holding the
// group for a person (novox/hq ADR 0323 §6, issue 406). An open group takes it as joinGroup does.
func (h *Holder) takePlaceOf(old string, d *Delivery) bool {
if old == "" {
return false
}
for _, g := range h.groups {
_, ordered := g.OrderOf[old]
if !g.Closed || !ordered && !containsString(g.Order, old) {
continue
}
if ordered {
delete(g.OrderOf, old)
g.OrderOf[d.ID] = d.Commit
}
if !containsString(g.Members, d.ID) {
g.Members = append(g.Members, d.ID)
sort.Strings(g.Members)
}
for i, id := range g.Order {
if id == old {
g.Order[i] = d.ID
}
}
d.Group = g.ID
g.Updated = h.Now()
h.keep(d)
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s: %s takes the place of %s, which it superseded", g.ID, d.ID, old)
return true
}
return false
h.joinGroup(d)
}
// decidedOnHead is the verdict the forge's statuses on a head hold, when the gate decided one: passed, warned
@@ -522,47 +458,6 @@ func (h *Holder) PullMerged(p PullEvent) {
}
h.moved(d, ts)
h.closeGroupOf(d)
h.trunkMoved(d, p.Base, p.MergeCommit, now)
}
// trunkMoved takes a merge into a trunk as that trunk's move (novox/hq ADR 0323 §7): every other open delivery into
// it keeps the trunk's new head, and a ready one whose verdict names the commit before goes back to proposed. A
// group's composed check goes out of date when the merge is not one of its members and a member still open merges
// into that trunk (§6); the group then reads checking. Nothing is asked: a recheck or the merge tool judges again.
func (h *Holder) trunkMoved(merged *Delivery, base, commit string, now time.Time) {
if base == "" || commit == "" {
return
}
for _, o := range h.deliveries {
if o == merged || o.Repository != merged.Repository || o.Base != base || o.MergedAs != "" || o.State.Final() ||
sameCommit(o.TrunkNow, commit) {
continue
}
// A check asked again against the trunk before this move judged that trunk: it is out of date too.
o.TrunkNow, o.AskedAgain = commit, nil
h.settleAndKeep(o)
}
for _, g := range h.groups {
if g.ID == merged.Group || g.Check == nil || g.Check.Verdict == "" || g.Check.OutOfDate != "" {
continue
}
for _, m := range h.membersOf(g) {
if m.Repository != merged.Repository || m.Base != base || m.MergedAs != "" || m.State.Final() {
continue
}
judged := g.Check.Trunks[m.ID]
if judged == "" {
judged = "?"
}
g.Check.OutOfDate = fmt.Sprintf("composed check out of date: %s was judged against trunk %s, the trunk is now %s "+
"(%s merged)", m.ID, shortOf(judged, 12), shortOf(commit, 12), merged.ID)
g.Check.OutOfDateAt = now
g.Updated = now
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s: %s", g.ID, g.Check.OutOfDate)
break
}
}
}
// closeGroupOf closes a delivery's group once it reached the trunk: a delivering group takes no new member.
@@ -634,13 +529,7 @@ type CheckedEvent struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
} `json:"members,omitempty"`
// Trunk is the trunk commit the head was merged into and judged as, Beside the other repositories' commits the
// check read, and Took how long it ran (novox/hq ADR 0323 §3).
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
}
// Checked takes a verdict: a head's own, or a group's composed check.
@@ -655,18 +544,6 @@ func (h *Holder) Checked(c CheckedEvent) {
return
}
g.Check.Verdict, g.Check.Summary = c.Verdict, c.Summary
// Each member's head and the trunk commit it was merged into (novox/hq ADR 0323 §6).
for _, m := range c.Members {
for id := range g.Check.Commits {
if d := h.deliveries[id]; d != nil && strings.EqualFold(d.Repository, m.Owner+"/"+m.Repo) &&
sameCommit(d.Commit, m.Commit) && m.Trunk != "" {
if g.Check.Trunks == nil {
g.Check.Trunks = map[string]string{}
}
g.Check.Trunks[id] = m.Trunk
}
}
}
g.Updated = now
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s composed: %s — %s", g.ID, c.Verdict, c.Summary)
@@ -687,23 +564,11 @@ func (h *Holder) Checked(c CheckedEvent) {
}
h.moved(d, []Transition{t})
}
if d.State == Rejected && c.Trunk != "" && d.Check != nil && !sameCommit(d.Check.Trunk, c.Trunk) {
// Judged again against another trunk commit — the merge tool's ask of a verdict out of date, or a check asked
// elsewhere (novox/hq ADR 0323 §5, §7): taken as the recheck it is.
t, err := Apply(d, EvRecheck, true, Facts{Now: now, By: byOwner, Why: fmt.Sprintf("judged again against trunk %s "+
"(it was rejected against trunk %s)", shortOf(c.Trunk, 12), shortOf(d.Check.Trunk, 12))})
if err != nil {
h.refuse(err)
return
}
h.moved(d, []Transition{t})
}
if d.State != Proposed {
h.Logf("[mesh-delivery] %s is %s: a verdict for it now is history (%s)", d.ID, d.State, c.Verdict)
return
}
v := &Verdict{ID: c.ID, On: c.On, At: now, Gate: c.Verdict, Summary: c.Summary, Report: c.Report,
Head: shortOf(c.Commit, 12), Trunk: c.Trunk, Beside: c.Beside, Took: c.Took}
v := &Verdict{ID: c.ID, On: c.On, At: now, Gate: c.Verdict, Summary: c.Summary, Report: c.Report}
if c.Gate != nil {
v.Gate, v.Summary = c.Gate.Verdict, c.Gate.Summary
v.Modules, v.Dependents = c.Gate.Modules, c.Gate.Dependents
@@ -714,14 +579,10 @@ func (h *Holder) Checked(c CheckedEvent) {
if v.Gate == "" {
v.Gate = "error"
}
d.Check, d.AskedAgain = v, nil
d.Check = v
if c.Plan != nil {
d.Plan = c.Plan
}
if why := d.outOfDate(); why != "" {
// Kept and shown, and it does not count: the delivery stays proposed (ADR 0323 §7).
h.Logf("[mesh-delivery] %s: its verdict %s is %s", d.ID, c.ID, why)
}
h.settleAndKeep(d)
}
@@ -1092,16 +953,12 @@ func (h *Holder) plan() []func() {
// grace, it is stopped saying so — never left waiting for a start that cannot come.
for _, d := range h.deliveries {
ended := d.endedUnstarted()
if d.coveredStop() && !d.CoverTold {
h.tellCovered(d, now) // covered before this holder told it, or before a restart (novox/hq issue 480)
continue
}
if strandedUncovered(d) {
// Stopped and final, but still read for a later merge that delivered its modules: until one did,
// `stalled` keeps listing it (review of novox/hq issue 474).
var ask []string
if d.CoveredBy, ask = h.coverOf(d); d.CoveredBy != "" {
h.tellCovered(d, now)
h.keep(d)
}
for _, in := range ask {
if a := h.askContains(d, in, now); a != nil {
@@ -1246,25 +1103,6 @@ func (h *Holder) laterOnItsWay(d *Delivery) string {
return ""
}
// tellCovered tells the forge that a stop is covered since, once: its status says covered and by which merge, its view
// says so, and its note keeps the line on its head and its merge commit (novox/hq issue 480). No transition: the stop
// stays final, and the bus's transition event is not said again.
func (h *Holder) tellCovered(d *Delivery, now time.Time) {
d.CoverTold = true
line := oneLine(fmt.Sprintf("%s delivery %s: stopped, %s", now.UTC().Format(time.RFC3339), d.ID, d.coveredStopSaid()))
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: line, Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) {
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.MergedAs, Line: line, Since: now})
}
if d.shownOnItsPullRequest() {
d.owe(Effect{Kind: EffectView, Since: now})
state, desc := statusOf(d)
d.owe(Effect{Kind: EffectStatus, Commit: d.Commit, Context: "mesh/delivery", State: state, Line: desc, Since: now})
}
h.Logf("[mesh-delivery] %s: stopped, %s", d.ID, d.coveredStopSaid())
h.keep(d)
}
// strandedUncovered is whether a delivery stopped because its ended walk delivered nothing is still uncovered: a
// merged change that never reached the machines, which `stalled` keeps listing until a later delivered merge
// carries its modules (review of novox/hq issue 474).
@@ -1570,19 +1408,14 @@ func (h *Holder) planGroup(g *Group, members []*Delivery, now time.Time) []func(
})
return asks
}
// The composed check, once every member is ready and the order holds. Asked again for new heads, even once the
// group is closed: a member rebased or pushed is a new head, and the group waits for its composed check rather
// than being held for a person (novox/hq ADR 0323 §6, issue 406). One out of date is asked again once a member
// was judged again after it went out of date: a recheck, or the merge tool's ask.
// The composed check, once every member is ready and the order holds.
allReady := true
for _, d := range members {
// In a closed group a member on the trunk counts — merged and on its way, or already delivered (reached).
if d.State != Ready && !(g.Closed && d.MergedAs != "" && (!d.State.Final() || d.State.Reached())) {
if d.State != Ready {
allReady = false
}
}
again := g.Check != nil && (!sameMembers(g.Check.Commits, commits) || g.Check.OutOfDate != "" && rejudgedSince(members, g.Check.OutOfDateAt))
if allReady && len(g.Cycle) == 0 && (g.Check == nil && !g.Closed || again) {
if allReady && len(g.Cycle) == 0 && !g.Closed && (g.Check == nil || !sameMembers(g.Check.Commits, commits)) {
ms := h.inOrder(g, members)
id := g.ID
asks = append(asks, func() {
@@ -1704,7 +1537,7 @@ func (h *Holder) turnsOf(g *Group, members []*Delivery, waits map[string]carryWa
}
// Composed for these very heads: a verdict on another set of heads is no verdict on these.
composed := g.Check != nil && (g.Check.Verdict == "pass" || g.Check.Verdict == "warning") &&
sameMembers(g.Check.Commits, commitsOf(members)) && g.Check.OutOfDate == ""
sameMembers(g.Check.Commits, commitsOf(members))
// A member's turn registered, the members after it that the same walk carries take theirs too: one walk is let
// go once, for all of them (ADR 0276 decision 5). A member on another walk, or not at its turn, waits.
var asks []func()
@@ -1790,11 +1623,9 @@ func (h *Holder) turnsOf(g *Group, members []*Delivery, waits map[string]carryWa
}
return asks
case d.State == Published:
// Asked and not answered is not "did not pass": it waits, published, for the answer (issue 316). Nor is a
// composed check of other heads — a member rebased or pushed — or one out of date: it runs again for these
// heads, and the member waits for it (novox/hq ADR 0323 §6, issue 406).
if !composed && d.HeldWhy == "" && g.Check != nil && (g.Check.Verdict == "" || g.Check.OutOfDate != "" ||
!sameMembers(g.Check.Commits, commitsOf(members))) {
// Asked and not answered is not "did not pass": it waits, published, for the answer (issue 316).
if !composed && d.HeldWhy == "" && g.Check != nil && g.Check.Verdict == "" &&
sameMembers(g.Check.Commits, commitsOf(members)) {
return asks
}
if !composed && d.HeldWhy == "" {
@@ -2315,23 +2146,12 @@ func GroupState(g *Group, members []*Delivery) (string, string) {
return "delivered", "every member is delivered"
case g.Closed && n > 0 && ended == n:
return "stopped", supersededWhy + strings.Join(uncovered, ", ")
case g.Closed && g.Check != nil && (g.Check.OutOfDate != "" || g.Check.Verdict == "" ||
!sameMembers(g.Check.Commits, commitsOf(members))):
// A member rebased or pushed once the group was delivering, or its composed check out of date: it is composed
// again for these heads, and the group is checking until then, never held (novox/hq ADR 0323 §6, issue 406).
if g.Check.OutOfDate != "" {
return "checking", g.Check.OutOfDate
}
return "checking", "composed again for its members' new heads"
case g.Closed:
return "delivering", fmt.Sprintf("%d of %d delivered", reached, n)
case len(g.Cycle) > 0:
return "rejected", cycleSaid(g)
case count[Rejected] > 0:
return "rejected", "a member's own check failed"
case g.Check != nil && g.Check.OutOfDate != "":
// Out of date counts as outstanding (ADR 0239 §18, novox/hq ADR 0323 §7): checking, saying why.
return "checking", g.Check.OutOfDate
case g.Check != nil && g.Check.Verdict != "" && g.Check.Verdict != "pass" && g.Check.Verdict != "warning":
return "rejected", "composed together: " + g.Check.Summary
case count[Ready] == n && g.Check != nil && sameMembers(g.Check.Commits, commitsOf(members)) &&
@@ -2397,13 +2217,3 @@ func cycleSaid(g *Group) string {
}
return why
}
// rejudgedSince is whether a member was judged again after a moment: a verdict of its own heard since.
func rejudgedSince(members []*Delivery, at time.Time) bool {
for _, d := range members {
if d.Check != nil && d.Check.At.After(at) {
return true
}
}
return false
}
@@ -1,463 +0,0 @@
package main
import (
"strings"
"testing"
"time"
)
// **A verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341). The check judges the head
// merged into a named trunk commit; when another merge moves that trunk, a ready delivery goes back to proposed,
// a rejected one stays rejected, and nothing is asked. A verdict for an old trunk is kept and shown and does not
// count, so the table reaches a fixed point. A group's composed check goes out of date on a merge that is not one of
// its members, and runs again for new heads instead of holding the group.
const (
trunk0 = "a2c2c35000000000000000000000000000000000"
merge1 = "d28ca95000000000000000000000000000000000"
)
// judged is a verdict of a head merged into a trunk commit.
func judged(repo string, number int, sha, gate, trunk string) CheckedEvent {
c := verdict(repo, number, sha, gate)
c.Trunk = shortOf(trunk, 12)
c.ID = "check-of-" + sha + "-at-" + shortOf(trunk, 8)
c.Took = "4m12s"
c.Beside = map[string]string{"mesh-catalog": "0123456789ab"}
plan := *c.Plan
plan.Head, plan.Trunk = sha, c.Trunk
c.Plan = &plan
return c
}
func TestATrunkMoveSendsAReadyDeliveryBackToProposed(t *testing.T) {
const host = "novox/mesh-host"
const h64, h66, h67 = "6464646464646464", "dfb1415dfb1415df", "6767676767676767"
w := newWorld(t)
for _, p := range []PullEvent{pr(host, 64, h64, "a"), pr(host, 66, h66, "b"), pr(host, 67, h67, "c")} {
w.h.PullUpdated(p)
}
w.h.Checked(judged(host, 64, h64, "pass", trunk0))
w.h.Checked(judged(host, 66, h66, "pass", trunk0))
w.h.Checked(judged(host, 67, h67, "fail", trunk0))
id66, id67 := IDOf(host, h66), IDOf(host, h67)
if s := w.state(id66); s != Ready {
t.Fatalf("#66 checked green against the trunk is %s", s)
}
asked := len(w.ctl.checks)
// #64 merges: the trunk moves to its merge commit.
w.h.PullMerged(merged(host, 64, h64, merge1))
w.settleAll()
d := w.delivery(id66)
if d.State != Proposed {
t.Fatalf("#66's verdict names trunk %s, the trunk is now %s, and it is %s", trunk0[:12], merge1[:12], d.State)
}
last := d.Transitions[len(d.Transitions)-1]
if last.Event != EvTrunkMoved || !strings.Contains(last.Why, trunk0[:12]) || !strings.Contains(last.Why, merge1[:12]) {
t.Fatalf("its transition was %s, %q: not trunk moved with both commits", last.Event, last.Why)
}
if d.Check == nil || d.Check.Trunk != trunk0[:12] {
t.Fatalf("its out-of-date verdict was not kept to be shown: %+v", d.Check)
}
if s := w.state(id67); s != Rejected {
t.Fatalf("a rejected delivery went to %s on the trunk's move", s)
}
if len(w.ctl.checks) != asked {
t.Fatalf("the trunk's move asked a check: %v", w.ctl.checks[asked:])
}
// Out of date and not asked: no bound, so stalled does not list it, however long it waits.
w.later(5 * time.Hour)
w.settleAll()
for _, l := range w.h.Stalled() {
if l.ID == id66 {
t.Fatalf("stalled lists a proposed delivery whose check is out of date and not asked: %+v", l)
}
}
if len(w.ctl.checks) != asked {
t.Fatalf("an out-of-date check was asked by this owner: %v", w.ctl.checks[asked:])
}
// A late verdict for the old trunk is kept, does not count, and moves nothing: the table's fixed point.
before := len(w.delivery(id66).Transitions)
late := judged(host, 66, h66, "pass", trunk0)
late.ID = "check-late"
w.h.Checked(late)
w.settleAll()
d = w.delivery(id66)
if d.State != Proposed || len(d.Transitions) != before || d.Check == nil || d.Check.ID != "check-late" {
t.Fatalf("a late verdict for an old trunk left it %s after %d transition(s), verdict %+v", d.State,
len(d.Transitions)-before, d.Check)
}
if ts := settle(d, Facts{Now: w.now}); len(ts) != 0 {
t.Fatalf("the table took %d more transition(s) on a verdict that does not count", len(ts))
}
// The check asked again judges the head merged into the trunk's head: it counts.
w.h.Checked(judged(host, 66, h66, "pass", merge1))
w.settleAll()
if s := w.state(id66); s != Ready {
t.Fatalf("a verdict judged against the trunk's head left it %s", s)
}
}
// A rejected delivery judged again against another trunk — the merge tool's ask — is taken as its recheck.
func TestARejectedDeliveryJudgedAgainstANewTrunkIsRecheckedByTheVerdict(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "fail", trunk0))
id := IDOf(repo, head)
if s := w.state(id); s != Rejected {
t.Fatalf("it is %s", s)
}
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
w.h.Checked(judged(repo, 9, head, "pass", merge1))
w.settleAll()
d := w.delivery(id)
if d.State != Ready || d.Check.Trunk != merge1[:12] {
t.Fatalf("judged green against the trunk's head it is %s with %+v", d.State, d.Check)
}
}
// **A verdict names the trunk commit it judged** in `checks` and `show`, and the kept plan records it (ADR 0323 §3).
func TestChecksAndShowNameTheHeadAndTheTrunkCommit(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "pass", trunk0))
id := IDOf(repo, head)
a, err := w.h.Checks(repo, 0, head)
if err != nil {
t.Fatal(err)
}
want := "head " + head[:12] + " merged into trunk " + trunk0[:12]
if a.Judged != want || a.Verdict == nil || a.Verdict.Head != head[:12] || a.Verdict.Trunk != trunk0[:12] {
t.Fatalf("checks says %q, verdict %+v; want %q", a.Judged, a.Verdict, want)
}
if !strings.Contains(a.Says, want) {
t.Fatalf("checks' words do not say what was judged: %s", a.Says)
}
if a.Verdict.Beside["mesh-catalog"] != "0123456789ab" || a.Verdict.Took != "4m12s" {
t.Fatalf("checks does not name the other repositories' commits or how long it took: %+v", a.Verdict)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
d := shown.(map[string]any)["delivery"].(*Delivery)
if d.Check.Head != head[:12] || d.Check.Trunk != trunk0[:12] {
t.Fatalf("show gives the verdict %+v", d.Check)
}
if d.Plan == nil || d.Plan.Trunk != trunk0[:12] || d.Plan.Head != head {
t.Fatalf("the kept delivery plan records %+v", d.Plan)
}
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
if a, _ = w.h.Checks(repo, 0, head); !strings.Contains(a.OutOfDate, merge1[:12]) {
t.Fatalf("checks does not say the verdict is out of date: %+v", a)
}
}
// **A delivery group's composed check follows the same rule** (ADR 0323 §6, §7): a merge that is not one of its
// members into a member's trunk puts it out of date and the group reads checking; its own members merging in its
// order do not; and it runs again once a member is judged again.
func TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1"
idc, idk := IDOf(ctl, hc), IDOf(cat, hk)
start := func(t *testing.T) *world {
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
if g == nil || g.Check == nil {
t.Fatalf("no composed check was asked: %+v", g)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed",
Members: []struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
}{{Owner: "novox", Repo: "mesh-controller", Commit: hc, Trunk: trunk0[:12]},
{Owner: "novox", Repo: "mesh-catalog", Commit: hk, Trunk: trunk0[:12]}}})
w.settleAll()
if state, _ := GroupState(g, w.h.membersOf(g)); state != "ready" {
t.Fatalf("the group composed green is %s", state)
}
if g.Check.Trunks[idc] != trunk0[:12] {
t.Fatalf("the composed check names its members' trunks %v", g.Check.Trunks)
}
return w
}
t.Run("a merge that is not a member", func(t *testing.T) {
w := start(t)
asked := len(w.ctl.checks)
w.h.PullMerged(merged(ctl, 5, "5555555555555555", merge1))
w.settleAll()
g := w.h.groups["feat/g"]
state, why := GroupState(g, w.h.membersOf(g))
if state != "checking" || g.Check.OutOfDate == "" {
t.Fatalf("the group is %s (%s) after a merge that is not its member moved a member's trunk", state, why)
}
for _, head := range []string{ctl + "@" + hc, cat + "@" + hk} {
s := w.forge.statuses[head+" "+ctxGroup]
if !strings.HasPrefix(s, "pending ") || !strings.Contains(s, "out of date") {
t.Fatalf("mesh/delivery-group on %s says %q, not pending with the reason", head, s)
}
}
if len(w.ctl.checks) != asked {
t.Fatalf("the trunk's move asked a check: %v", w.ctl.checks[asked:])
}
// The controller's member judged again against the trunk's head: the composed check runs again.
w.later(time.Minute)
w.h.Checked(judged(ctl, 1, hc, "pass", merge1))
w.settleAll()
if len(w.ctl.checks) == asked || !strings.HasPrefix(w.ctl.checks[len(w.ctl.checks)-1], "feat/g:") {
t.Fatalf("the composed check was not asked again once a member was judged again: %v", w.ctl.checks)
}
})
t.Run("its own member merging", func(t *testing.T) {
w := start(t)
mc := merged(ctl, 1, hc, merge1)
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.settleAll()
g := w.h.groups["feat/g"]
if g.Check.OutOfDate != "" {
t.Fatalf("the group's own member merging put its composed check out of date: %s", g.Check.OutOfDate)
}
if s := w.state(idk); s != Ready {
t.Fatalf("the catalogue's member, in another repository, is %s", s)
}
})
}
// **After a member is rebased or pushed, the composed check runs again for the new heads** and the group reads
// checking, not held (ADR 0323 §6, issue 406's second symptom): its first member merged, closing the group, and then
// its second was rebased just before its own merge.
func TestARebasedMemberIsComposedAgainNotHeld(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk, hc2 = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1", "c2c2c2c2c2c2c2c2"
idc, idk, idc2 := IDOf(ctl, hc), IDOf(cat, hk), IDOf(ctl, hc2)
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
if len(g.Order) != 2 || g.Order[0] != idk {
t.Fatalf("the group's order is %v: the catalogue's member first is what this replays", g.Order)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed"})
w.settleAll()
// The catalogue's member merges first: the group is closed.
mk := merged(cat, 2, hk, merge1)
mk.Head = "feat/g"
w.h.PullMerged(mk)
w.h.WalkMoved(w.ctl.put(aWalk("plan-cat", cat, merge1, true)))
w.settleAll()
if !g.Closed {
t.Fatal("the group is not closed once a member merged")
}
// The controller's member is rebased just before its merge: a new head, with its own check.
w.h.PullUpdated(pr(ctl, 1, hc2, "feat/g"))
w.settleAll()
if d := w.delivery(idc2); d.Group != "feat/g" {
t.Fatalf("the rebased head did not take its place in its closed group: %q", d.Group)
}
if containsString(g.Members, idc) || !containsString(g.Members, idc2) || len(g.Order) != 2 || g.Order[1] != idc2 {
t.Fatalf("the group's members are %v in the order %v", g.Members, g.Order)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
w.h.Checked(judged(ctl, 1, hc2, "pass", trunk0))
w.settleAll()
asked := w.ctl.checks[len(w.ctl.checks)-1]
if !strings.HasPrefix(asked, "feat/g:") || !strings.Contains(asked, idc2) || !strings.Contains(asked, idk) {
t.Fatalf("the composed check was not asked again for the new heads: %v", w.ctl.checks)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
for _, id := range []string{idk, idc2} {
if d := w.delivery(id); d.State == Held {
t.Fatalf("%s is held for a person: %s", id, d.HeldWhy)
}
}
// Its verdict for the new heads: the rebased member merges and takes its turn.
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed again"})
mc := merged(ctl, 1, hc2, "e0e0e0e0e0e0e0e0")
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.h.WalkMoved(w.ctl.put(aWalk("plan-ctl", ctl, "e0e0e0e0e0e0e0e0", true)))
w.settleAll()
if d := w.delivery(idc2); d.State == Held {
t.Fatalf("the rebased member, composed green for the new heads, is held: %s", d.HeldWhy)
}
}
// A head whose merge check the forge's holder put out of date is not a check asked and not answered: `stalled` does
// not list it, however long it waits — rejected, or with no delivery here — until its check is asked (ADR 0323 §7).
func TestAnOutOfDateCheckOnTheForgeIsNotSaidUnanswered(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b9c0d"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "fail", trunk0))
words := outOfDateWords(trunk0, merge1)
pending := func(description, at string) map[string]HeadCheck {
return map[string]HeadCheck{ctxGate: {State: "pending", Description: description, UpdatedAt: at},
ctxRepo: {State: "pending", Description: description, UpdatedAt: at}}
}
w.forge.open = []OpenPull{{Owner: "novox", Repo: "mesh-lab", Number: 9, Base: "main", HeadSHA: head,
Required: []string{ctxGate, ctxRepo}, Statuses: []string{ctxGate, ctxRepo}, Checks: pending(words, "2026-10-06T12:01:00Z")}}
w.h.SeeOpenPulls()
w.later(3 * time.Hour)
w.h.SeeOpenPulls()
for _, l := range w.h.Stalled() {
if strings.HasPrefix(l.ID, repo) {
t.Fatalf("an out-of-date check nobody asked is said stalled: %+v", l)
}
}
// Asked by the merge tool: pending anew, and its bound runs from then.
w.forge.open[0].Checks = pending("waiting in the build queue since 15:01: asked by the merge tool", "2026-10-06T15:01:00Z")
w.h.SeeOpenPulls()
w.later(2 * time.Hour)
w.h.SeeOpenPulls()
said := false
for _, l := range w.h.Stalled() {
said = said || l.State == Unanswered
}
if !said {
t.Fatal("a check the merge tool asked and that never answered is not said")
}
}
// The merge tool's ask announces an out-of-date head again: its one-hour bound runs from then, and a check that never
// answers is said stalled (ADR 0323 §7).
func TestAnOutOfDateDeliveryAskedAgainIsBoundFromTheAsk(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "pass", trunk0))
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
id := IDOf(repo, head)
w.later(3 * time.Hour)
w.h.PullUpdated(pr(repo, 9, head, "x")) // the merge tool asked its check
w.later(30 * time.Minute)
for _, l := range w.h.Stalled() {
if l.ID == id {
t.Fatalf("listed half an hour after its check was asked again: %+v", l)
}
}
w.later(time.Hour)
said := false
for _, l := range w.h.Stalled() {
if l.ID == id {
said = true
if !strings.Contains(l.Says, "asked again") || !strings.Contains(l.H2, "operator") {
t.Fatalf("its stalled line says %+v", l)
}
}
}
if !said {
t.Fatal("a check asked again that never answered is not said")
}
// Its verdict judged against the trunk's head: ready, and nothing bounds it as proposed.
w.h.Checked(judged(repo, 9, head, "pass", merge1))
if d := w.delivery(id); d.State != Ready || d.AskedAgain != nil {
t.Fatalf("judged again it is %s, asked again %v", d.State, d.AskedAgain)
}
}
// The same, with the first member already delivered when the second is rebased: a delivered member counts for the
// composed check of the new heads, which is asked, and the rebased member is never held (review of ADR 0323 §6).
func TestARebasedMemberAfterAnotherWasDeliveredIsComposedAgain(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk, hc2 = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1", "c2c2c2c2c2c2c2c2"
idk, idc2 := IDOf(cat, hk), IDOf(ctl, hc2)
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed"})
w.settleAll()
// The catalogue's member merges and its walk is done: delivered.
mk := merged(cat, 2, hk, merge1)
mk.Head = "feat/g"
w.h.PullMerged(mk)
walk := w.ctl.put(aWalk("plan-cat", cat, merge1, true))
w.h.WalkMoved(walk)
w.settleAll()
walk = w.ctl.walk("plan-cat")
walk.State = walkDone
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
if s := w.state(idk); s != Delivered {
t.Fatalf("the catalogue's member is %s, not delivered", s)
}
// The controller's member is rebased, and judged.
w.h.PullUpdated(pr(ctl, 1, hc2, "feat/g"))
w.h.Checked(judged(ctl, 1, hc2, "pass", trunk0))
w.settleAll()
asked := w.ctl.checks[len(w.ctl.checks)-1]
if !strings.HasPrefix(asked, "feat/g:") || !strings.Contains(asked, idc2) {
t.Fatalf("the composed check was not asked again for the new heads: %v", w.ctl.checks)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed again"})
mc := merged(ctl, 1, hc2, "e0e0e0e0e0e0e0e0")
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.h.WalkMoved(w.ctl.put(aWalk("plan-ctl", ctl, "e0e0e0e0e0e0e0e0", true)))
w.settleAll()
if d := w.delivery(idc2); d.State == Held {
t.Fatalf("the rebased member is held: %s", d.HeldWhy)
}
}
// A check this owner asks — a recheck — sends the base head the forge last announced the head with, so the
// controller's verdict for a head no build judges names its trunk commit (ADR 0323 §3, review of mesh-controller#271).
func TestARecheckSendsTheAnnouncedBaseHead(t *testing.T) {
const repo = "novox/mesh-tools-go"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
p := pr(repo, 3, head, "x")
p.BaseSHA = trunk0
w.h.PullUpdated(p)
id := IDOf(repo, head)
if m := MemberOf(w.delivery(id)); m.BaseSHA != trunk0 {
t.Fatalf("the member sends base head %q", m.BaseSHA)
}
p.BaseSHA = merge1 // announced again, by the merge tool
w.h.PullUpdated(p)
if m := MemberOf(w.delivery(id)); m.BaseSHA != merge1 {
t.Fatalf("announced again, the member sends base head %q", m.BaseSHA)
}
if raw := string(mustJSON(MemberOf(w.delivery(id)))); !strings.Contains(raw, `"base_sha":"`+merge1+`"`) {
t.Fatalf("the member as the controller reads it: %s", raw)
}
}
@@ -1,95 +0,0 @@
package main
import (
"testing"
"time"
)
// novox/hq issue 403, ADR 0313: the issue tracker measures a fix's time from the move to fixing to delivered and
// splits it at the pull request's opening and its merge. A transition says what it reads them by: when the pull
// request was first heard (its first head, not a later one), when it merged, and the class of what it moves —
// core, leaf, or none for a change that moves no module. `checks` says the same for a fix the tracker hears of late.
// transitionsTo are the transition events said for a delivery that moved to a state.
func transitionsTo(w *world, id string, to State) []map[string]any {
var out []map[string]any
for _, e := range w.said {
if e["event-kind"] == "transition" && e["id"] == id && e["to"] == string(to) {
out = append(out, e)
}
}
return out
}
func TestATransitionSaysWhenItsPullRequestOpenedMergedAndItsClass(t *testing.T) {
w := newWorld(t)
repo := "novox/mesh-catalog"
opened := w.now
first := pr(repo, 9, "head9first", "fix/403-x", "modules/app/a.go")
w.h.PullUpdated(first)
w.later(3 * time.Minute)
second := pr(repo, 9, "head9second", "fix/403-x", "modules/app/a.go")
w.h.PullUpdated(second)
c := verdictMoving(repo, 9, "head9second", "app")
c.Plan.Class = "core"
w.h.Checked(c)
w.later(2 * time.Minute)
mergedWhen := w.now
w.h.PullMerged(mergedAt(second, "merge9aaaaaa", mergedWhen))
walk := aWalk("plan-merge9", repo, "merge9aaaaaa", false)
walk.Delivery = &WalkDelivery{Merges: []WalkMerge{{Repository: repo, Commit: "merge9aaaaaa"}}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
walk = w.ctl.walk(walk.ID)
walk.State, walk.Revision = walkDone, walk.Revision+1
walk.Modules = map[string]*WalkModule{"app": {State: "built"}}
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
id := IDOf(repo, "head9second")
if s := w.state(id); s != Delivered {
t.Fatalf("the merge is %s, not delivered", s)
}
said := transitionsTo(w, id, Delivered)
if len(said) == 0 {
t.Fatalf("no transition to delivered was said: %v", w.said)
}
e := said[len(said)-1]
if e["opened"] != opened.UTC().Format(time.RFC3339) {
t.Errorf("opened %v, want the first head's %s (not the second head's)", e["opened"], opened.Format(time.RFC3339))
}
if e["merged_at"] != mergedWhen.UTC().Format(time.RFC3339) {
t.Errorf("merged_at %v, want %s", e["merged_at"], mergedWhen.Format(time.RFC3339))
}
if e["class"] != "core" {
t.Errorf("class %v, want the plan's core", e["class"])
}
a, err := w.h.Checks(repo, 9, "")
if err != nil {
t.Fatal(err)
}
if a.Delivery == nil || a.Delivery.Opened != opened.UTC().Format(time.RFC3339) || a.Delivery.Class != "core" ||
a.Delivery.MergedAt != mergedWhen.UTC().Format(time.RFC3339) {
t.Errorf("checks says %+v", a.Delivery)
}
}
// A plan that moves no module is class none, so the tracker gives its fix no budget; a plan from a controller that
// says no class falls back to its walk's class once known, and says nothing before.
func TestAFixClassIsNoneWhenItMovesNothing(t *testing.T) {
for _, c := range []struct {
why string
d Delivery
want string
}{
{"the planner's class", Delivery{Plan: &DeliveryPlan{Moved: []string{"app"}, Class: "leaf"}}, "leaf"},
{"a plan that moves nothing, no class said", Delivery{Plan: planMoving("novox/x")}, "none"},
{"no class said, its walk's known", Delivery{Plan: planMoving("novox/x", "app"), Time: &DeliveryTime{Class: "core"}}, "core"},
{"no class said, nothing known", Delivery{Plan: planMoving("novox/x", "app")}, ""},
} {
if got := fixClass(&c.d); got != c.want {
t.Errorf("%s: %q, want %q", c.why, got, c.want)
}
}
}
@@ -1,105 +0,0 @@
package main
import (
"testing"
"time"
)
// triedSaid is delivery_status's count of owed effects tried (novox/hq issue 407), read as a test reads it.
func triedSaid(t *testing.T, w *world) map[string]any {
t.Helper()
got, ok := w.h.Status("listening")["owed-tried"].(map[string]any)
if !ok {
t.Fatalf("delivery_status says no owed-tried: %v", w.h.Status("listening"))
}
return got
}
func countOf(t *testing.T, m map[string]any, key string) int {
t.Helper()
n, ok := m[key].(int)
if !ok {
t.Fatalf("owed-tried has no count %q: %v", key, m)
}
return n
}
// novox/hq issue 407: delivery_status says how many owed effects were tried since this owner started, how
// many were done, failed to be tried again, or were given up, and when the last was tried — so a window's
// writes can be checked against the tries made in it (issue 326's rule) by reading it at both ends.
func TestDeliveryStatusCountsTheOwedEffectsTried(t *testing.T) {
w := newWorld(t)
start := w.now
tried := triedSaid(t, w)
if countOf(t, tried, "tried") != 0 || countOf(t, tried, "done") != 0 || countOf(t, tried, "failed") != 0 ||
countOf(t, tried, "given-up") != 0 {
t.Fatalf("a holder that tried nothing says %v", tried)
}
if tried["since"] != start.Format(time.RFC3339) {
t.Fatalf("its count is since %v, the holder started %s", tried["since"], start.Format(time.RFC3339))
}
if _, there := tried["last"]; there {
t.Fatalf("nothing tried, yet a last try is said: %v", tried)
}
// A head seen: its event and its note are owed and done.
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
tried = triedSaid(t, w)
done := countOf(t, tried, "done")
if done < len(w.said)+len(w.forge.notesOn(head)) || done == 0 {
t.Fatalf("said %d and noted %d, yet done counts %d", len(w.said), len(w.forge.notesOn(head)), done)
}
if countOf(t, tried, "tried") != done || countOf(t, tried, "failed") != 0 {
t.Fatalf("every try succeeded, yet %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
// The forge away: its effects fail and are counted failed; quiet passes try nothing more.
w.forge.down = true
w.later(time.Minute)
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.Flush()
tried = triedSaid(t, w)
failed := countOf(t, tried, "failed")
if failed == 0 {
t.Fatalf("the forge away failed nothing: %v", tried)
}
if countOf(t, tried, "tried") != countOf(t, tried, "done")+failed+countOf(t, tried, "given-up") {
t.Fatalf("tries do not add up: %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
before := countOf(t, tried, "tried")
w.h.Flush() // before each failure's next try: nothing is tried
if n := countOf(t, triedSaid(t, w), "tried"); n != before {
t.Fatalf("a pass before any next try tried %d", n-before)
}
// The forge back, past every next try: the failed effects are done.
w.forge.down = false
w.later(time.Hour)
w.h.Flush()
tried = triedSaid(t, w)
if countOf(t, tried, "tried") != before+failed || countOf(t, tried, "done") <= done {
t.Fatalf("the forge back: tried %d (was %d, %d failed), %v", countOf(t, tried, "tried"), before, failed, tried)
}
}
// The count is this run's: a restarted holder says it counts since its own start, from nothing.
func TestOwedTriedCountsSinceTheStart(t *testing.T) {
w := newWorld(t)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
if countOf(t, triedSaid(t, w), "tried") == 0 {
t.Fatal("nothing counted before the restart")
}
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Minute))
tried := triedSaid(t, again)
if countOf(t, tried, "tried") != 0 || tried["since"] != again.now.Format(time.RFC3339) {
t.Fatalf("a restarted holder says %v", tried)
}
}
@@ -1,48 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
)
// **An unanswered line says the state of the delivery it holds, as data** (novox/hq issue 451): for a delivery in
// ready or rejected, whose check answered once, the controller advises a recheck rather than the build queue, and
// it reads that state from the line, never from its words. A head with no delivery held says none.
func TestAnUnansweredLineNamesTheStateOfItsDelivery(t *testing.T) {
at := time.Date(2026, 10, 11, 0, 43, 0, 0, time.UTC)
for _, c := range []struct {
gate string
want State
}{{"pass", Ready}, {"fail", Rejected}} {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
id := heldIn(t, w, c.gate, c.want)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].ID != id || lines[0].State != Unanswered || lines[0].Held != c.want {
t.Fatalf("a %s delivery's unanswered line does not say its state as data: %+v", c.want, lines)
}
raw, _ := json.Marshal(lines[0])
if !strings.Contains(string(raw), `"held":"`+string(c.want)+`"`) {
t.Fatalf("the line the controller reads carries no held state: %s", raw)
}
}
// The same head with no delivery held here: unanswered, and no state to say.
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].State != Unanswered || lines[0].Held != "" {
t.Fatalf("a head with no delivery held says a state: %+v", lines)
}
if raw, _ := json.Marshal(lines[0]); strings.Contains(string(raw), `"held"`) {
t.Fatalf("a line with no delivery held carries a held field: %s", raw)
}
}
@@ -1,134 +0,0 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 480: a delivery stopped because its walk ended done before its word without building its modules
// (issue 474) is still read for a later merge that delivers them, and kept CoveredBy once one did. Its state stays
// stopped — a final state never moves, as for a covered superseded member (issue 442) — but the forge's status, its
// view, its note and `show` went on saying "stopped" for a change that reached every machine. Once covered they say
// covered, and by which merge. And while `stalled` lists such a stop, `deliveries` lists it too, past a day.
// strandedNight is theNight with #215 stopped, its walk past the grace with nothing later carrying build-agent.
func strandedNight(t *testing.T) (*world, string) {
w := theNight(t)
id := IDOf(ctl474, head215)
w.later(strandedAfter + 5*time.Minute)
w.settleAll()
if d := w.delivery(id); d.State != Stopped || d.Stranded == "" {
t.Fatalf("#215 is %s, stranded %q", d.State, d.Stranded)
}
return w, id
}
func TestAStopALaterMergeCoversReadsCoveredByThatMerge(t *testing.T) {
w, id := strandedNight(t)
if got := statusOn(w, ctl474, head215); !strings.HasPrefix(got, "error stopped: ") {
t.Fatalf("stopped, its status says %q", got)
}
w.forge.contains[merge215+" in "+merge219] = true
by := deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
d := w.delivery(id)
if d.State != Stopped || !strings.Contains(d.CoveredBy, by) {
t.Fatalf("after #219 it is %s, covered by %q", d.State, d.CoveredBy)
}
got := statusOn(w, ctl474, head215)
if !strings.HasPrefix(got, "success covered") || !strings.Contains(got, by) {
t.Fatalf("covered by %s, its status still says %q", by, got)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
if reads, _ := shown.(map[string]any)["reads"].(string); !strings.HasPrefix(reads, "covered") || !strings.Contains(reads, by) {
t.Fatalf("show reads %q", reads)
}
view := w.forge.views[ctl474+"#215"]
if !strings.Contains(view, "Covered") || !strings.Contains(view, by) {
t.Fatalf("its view does not say it was covered by %s:\n%s", by, view)
}
noted := false
for _, l := range w.forge.notesOn(merge215) {
noted = noted || strings.Contains(l, "covered") && strings.Contains(l, by)
}
if !noted {
t.Fatalf("its merge's note does not say it was covered: %v", w.forge.notesOn(merge215))
}
// Told once: further ticks owe nothing more.
w.later(time.Hour)
w.settleAll()
if d := w.delivery(id); len(d.Owed) != 0 || !d.CoverTold {
t.Fatalf("still owes %+v, told %v", d.Owed, d.CoverTold)
}
}
// A stop covered before this was built (CoveredBy kept, nothing told) is told after a restart, from what was kept.
func TestACoveredStopKeptBeforeIsToldAfterARestart(t *testing.T) {
w, id := strandedNight(t)
w.h.mu.Lock()
d := w.h.deliveries[id]
d.CoveredBy = IDOf(ctl474, head219) + " (merged as " + merge219[:8] + ")"
if err := w.store.PutDelivery(d); err != nil {
t.Fatal(err)
}
w.h.mu.Unlock()
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Hour))
again.settleAll()
if got := statusOn(again, ctl474, head215); !strings.HasPrefix(got, "success covered") ||
!strings.Contains(got, IDOf(ctl474, head219)) {
t.Fatalf("after a restart its status says %q", got)
}
if d := again.delivery(id); d.State != Stopped || len(d.Transitions) != len(w.delivery(id).Transitions) {
t.Fatalf("a restart moved it: %s", d.State)
}
}
// Only a stop for a walk that never built its modules reads covered: a person's stop, or one the cover test has
// not answered, keeps its error.
func TestOnlyACoveredStrandedStopReadsCovered(t *testing.T) {
for _, d := range []*Delivery{
{State: Stopped, CoveredBy: "x"}, // a person's stop, with a cover found while it was held
{State: Stopped, Stranded: "never reached"}, // stranded, not covered
} {
if state, _ := statusOf(d); state != "error" {
t.Fatalf("%+v reads %s", d, state)
}
if d.coveredStop() {
t.Fatalf("%+v reads as a covered stop", d)
}
}
}
func TestAStopStalledListsStaysInDeliveries(t *testing.T) {
w, id := strandedNight(t)
w.later(30 * time.Hour)
w.settleAll()
listed := func() *Line {
for _, l := range w.h.Deliveries("", "", "", false) {
if l.ID == id {
return &l
}
}
return nil
}
if len(w.h.Stalled()) != 1 {
t.Fatalf("stalled lists %+v", w.h.Stalled())
}
l := listed()
if l == nil || !l.Stalled || !strings.Contains(l.Waits, "build-agent") {
t.Fatalf("listed by stalled, deliveries shows %+v", l)
}
// Covered, it ends like any final delivery: a day after its stop.
w.forge.contains[merge215+" in "+merge219] = true
deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
if len(w.h.Stalled()) != 0 || listed() != nil {
t.Fatalf("covered, stalled lists %+v and deliveries %+v", w.h.Stalled(), listed())
}
}
@@ -192,7 +192,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
seat := "mesh-delivery."
return []stdio.Tool{
{Name: seat + "deliveries",
Description: "Every delivery not final, those that ended in the last day, and any stop `stalled` still lists, one line each: its id " +
Description: "Every delivery not final, and those that ended in the last day, one line each: its id " +
"(owner/repository@commit), its state, its group, what it waits for and since when. Narrowed by state, " +
"repository or group.",
Input: map[string]any{"type": "object", "properties": map[string]any{"state": str("one state"),
@@ -347,8 +347,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: "delivery_status",
Description: "Whether this owner works: how many deliveries in each state, the groups, what is owed to " +
"the forge and the bus and not yet done, how many owed effects were tried since it started (done, failed " +
"and tried again, given up) and when the last was, walks with no delivery yet, whether its events arrive, when " +
"the forge and the bus and not yet done, walks with no delivery yet, whether its events arrive, when " +
"the controller's walks were last read or why they could not be, and the last refusals.",
Run: func(map[string]any) (any, error) { return h.Status(l.get()), nil }},
}
@@ -49,17 +49,6 @@ type Delivery struct {
// CheckAsk is this owner's own ask of its check, when the forge's announcement did not bring one: a head
// adopted while it already waited, or one whose announcement the controller missed.
CheckAsk *CheckAsk `json:"check_ask,omitempty"`
// TrunkNow is its trunk's head as this owner last heard it move while it was open: the commit the last merge
// into its trunk landed as (novox/hq ADR 0323 §7). A verdict that names another trunk commit is out of date.
// Empty until a merge into its trunk is heard after it was made.
TrunkNow string `json:"trunk_now,omitempty"`
// BaseSHA is the base branch's head the forge last announced its head with (ADR 0323): sent with a check this
// owner asks, for a verdict no build judges to name its trunk commit.
BaseSHA string `json:"base_sha,omitempty"`
// AskedAgain is when its head was announced again while its verdict was out of date — the merge tool's ask of
// its check against the trunk's head (ADR 0323 §5) — and no verdict has come since: its one-hour bound runs from
// then. Nil while nothing is asked, and cleared by a verdict or another move of its trunk.
AskedAgain *time.Time `json:"asked_again,omitempty"`
// The facts the table's guards read. NewerHead is the head that superseded it; ClosedUnmerged the pull
// request closed; MergedAs the commit it landed on the trunk as; HeldWhy what holds it for a person.
@@ -79,9 +68,6 @@ type Delivery struct {
// Stranded says why a merge whose walk ended done before its word, without moving every module it moves, is
// stopped: nothing later delivered them either (novox/hq issue 474). Set once the cover test says no.
Stranded string `json:"stranded,omitempty"`
// CoverTold is whether the forge was told that such a stop was covered since: its status, its view and its note
// (novox/hq issue 480). Kept, so a restart tells it once, and a stop covered before this was built is told too.
CoverTold bool `json:"cover_told,omitempty"`
// TurnAwaits says, for a group member whose turn waits for a later merge to carry a superseded member before it,
// which merges and until when; TurnAwaitsUntil is that moment, which bounds the wait (ADR 0276 decision 9,
// novox/hq issue 445). `show` shows both, and `stalled` says them past that moment.
@@ -139,38 +125,6 @@ type Verdict struct {
Repo string `json:"repo,omitempty"`
RepoSaid string `json:"repo_summary,omitempty"`
Report string `json:"report,omitempty"`
// Head and Trunk are what it judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, twelve
// characters each. Trunk is empty for a verdict of a build seat that judged the head alone. Beside are the
// commits of the other repositories the check read, by directory; Took how long it ran.
Head string `json:"head,omitempty"`
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
}
// Judged is what a verdict judged, in words: "head 1a2b3c4d5e6f merged into trunk 7a8b9c0d1e2f" (ADR 0323 §3);
// empty for a verdict that names no trunk commit.
func (v *Verdict) Judged() string {
if v == nil || v.Trunk == "" {
return ""
}
return fmt.Sprintf("head %s merged into trunk %s", shortOf(v.Head, 12), shortOf(v.Trunk, 12))
}
// outOfDate is why a delivery's verdict no longer counts: it names a trunk commit that is not its trunk's head
// as this owner heard it (novox/hq ADR 0323 §7). Empty when it counts — or names no trunk commit, as a verdict
// of a build seat from before the rule does, which counts as it did.
func (d *Delivery) outOfDate() string {
if d.Check == nil || d.Check.Trunk == "" || d.TrunkNow == "" || sameCommit(d.Check.Trunk, d.TrunkNow) {
return ""
}
return outOfDateWords(d.Check.Trunk, d.TrunkNow)
}
// outOfDateWords is a verdict out of date, as the forge's holder says it on the pull request (ADR 0323 §4).
func outOfDateWords(judged, now string) string {
return fmt.Sprintf("out of date: judged against trunk %s, the trunk is now %s; judged again on a recheck or when "+
"it is merged", shortOf(judged, 12), shortOf(now, 12))
}
// CheckAsk is a head's own check, asked of the controller by this owner (or re-asked by healer H2).
@@ -208,11 +162,6 @@ type DeliveryPlan struct {
Machines []MachinePlan `json:"machines,omitempty"`
Steps []string `json:"steps,omitempty"`
Summary string `json:"summary"`
// Class is the class of the slowest-budgeted module it moves, as the planner says it: core, leaf, or none when
// it moves no module (novox/hq ADR 0313 §2). Empty in a plan from a controller that did not say it.
Class string `json:"class,omitempty"`
// Trunk is the trunk commit the head was merged into when its verdict judged it (novox/hq ADR 0323 §3).
Trunk string `json:"trunk,omitempty"`
}
// receiving are the machines the plan sends a module to, in the plan's order; none when the plan names no
@@ -299,20 +248,6 @@ func (d *Delivery) FromForgesWord() bool {
// shownOnItsPullRequest is whether its view and status are kept on its pull request: not for history, whose
// pull request the forge may no longer hold — or holds under its number another one.
// coveredStop is whether a delivery stopped because its walk ended done before its word without building its modules
// (novox/hq issue 474) was covered since: a later merge into its trunk that contains it was delivered, carrying every
// module it moves to every machine running it (ADR 0270's cover test, kept as CoveredBy). Its state stays stopped —
// final, as a covered superseded member stays superseded (issue 442) — and it reads covered, by that merge (novox/hq
// issue 480). A stop for any other reason never reads covered.
func (d *Delivery) coveredStop() bool {
return d.State == Stopped && d.Stranded != "" && d.CoveredBy != ""
}
// coveredStopSaid is how a covered stop reads: on the forge's status, in its view and note, and in `show`.
func (d *Delivery) coveredStopSaid() string {
return "covered after its stop: a later merge that contains it was delivered: " + d.CoveredBy
}
func (d *Delivery) shownOnItsPullRequest() bool {
return d.Number > 0 && d.History == "" && d.State != Retired
}
@@ -800,13 +735,6 @@ type GroupCheck struct {
Commits map[string]string `json:"commits"`
Verdict string `json:"verdict,omitempty"`
Summary string `json:"summary,omitempty"`
// Trunks are the trunk commit each member's head was merged into, by member id (novox/hq ADR 0323 §6).
Trunks map[string]string `json:"trunks,omitempty"`
// OutOfDate is why its verdict no longer counts: a merge that is not one of the group's members moved a
// member's trunk, at OutOfDateAt. It counts as outstanding (ADR 0239 §18), and runs again once a member is
// judged again after it.
OutOfDate string `json:"out_of_date,omitempty"`
OutOfDateAt time.Time `json:"out_of_date_at"`
}
// Commits is each member's commit, by id.
@@ -31,16 +31,13 @@ type Member struct {
ModuleDirsSaid bool `json:"module_dirs_said,omitempty"`
CloneURL string `json:"clone_url,omitempty"`
After []string `json:"after,omitempty"`
// BaseSHA is the base branch's head the forge last announced the head with (novox/hq ADR 0323): the trunk commit
// the controller's verdict names when no build judges the head.
BaseSHA string `json:"base_sha,omitempty"`
}
// MemberOf is a delivery as a member.
func MemberOf(d *Delivery) Member {
return Member{ID: d.ID, Repository: d.Repository, Base: d.Base, Head: d.Commit, Number: d.Number, Paths: d.Paths,
PathsTruncated: d.PathsTruncated, Removed: d.Removed, ModuleDirs: d.ModuleDirs, ModuleDirsSaid: d.ModuleDirsSaid,
CloneURL: d.CloneURL, After: d.After, BaseSHA: d.BaseSHA}
CloneURL: d.CloneURL, After: d.After}
}
// Order is a group's order as the controller works it out: each pair with the rules it won over by
@@ -85,7 +85,6 @@ const (
EvCovered Event = "covered" // a later merge into the same trunk delivered every module it moves
EvLifted Event = "lifted" // what held it for a person is no longer so: its group's composed check passed, or a walk now answers its merge
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed, or was stopped for another merge (ADR 0299): it waits for the walk that walks it again
EvTrunkMoved Event = "trunk moved" // another merge moved its trunk, and its verdict names the trunk commit before (novox/hq ADR 0323)
)
// Facts are what a guard reads beyond the delivery itself: the moment, and who asks, for an act.
@@ -168,17 +167,9 @@ var Table = []Row{
Holds: func(d *Delivery, f Facts) error {
return unless(d.MergedAs != "" && (d.Walk != nil || d.MovesNothing()), "not on the trunk yet")
}},
// **A verdict counts only for the trunk's head** (novox/hq ADR 0323 §7): one naming a trunk commit that is no
// longer the trunk's head is kept and shown, and does not count, so the delivery stays proposed and the rows
// cannot loop with `trunk moved`; a recheck or the merge tool's ask judges it again.
{From: []State{Proposed}, Event: EvChecked, To: Checked,
Guard: "a verdict names this commit and the trunk's head as its trunk commit",
Holds: func(d *Delivery, f Facts) error {
if d.Check == nil {
return errors.New("no verdict")
}
return unless(d.outOfDate() == "", "its verdict is "+d.outOfDate())
}},
Guard: "the verdict names this commit",
Holds: func(d *Delivery, f Facts) error { return unless(d.Check != nil, "no verdict") }},
{From: []State{Checked}, Event: EvAccepted, To: Ready,
Guard: "the gate passed or warned, and the repository's own check did not fail",
Holds: func(d *Delivery, f Facts) error { return unless(d.Check.Passes(), "the verdict does not pass") }},
@@ -208,14 +199,6 @@ var Table = []Row{
Says: func(d *Delivery) string {
return "covered: a later merge that contains it was delivered: " + d.CoveredBy
}},
// **A ready delivery whose verdict went out of date goes back to proposed** (novox/hq ADR 0323 §7): another
// merge moved its trunk. A rejected one stays rejected, its status on the forge saying the verdict is out of date.
{From: []State{Ready}, Event: EvTrunkMoved, To: Proposed,
Guard: "its verdict names a trunk commit that is not its trunk's head",
Holds: func(d *Delivery, f Facts) error {
return unless(d.MergedAs == "" && d.outOfDate() != "", "not merged, and its verdict is not out of date")
},
Says: func(d *Delivery) string { return d.outOfDate() }},
{From: []State{Proposed, Checked, Rejected}, Event: EvMergedUnchecked, To: Held,
Guard: "merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on",
Holds: func(d *Delivery, f Facts) error { return unless(d.MergedAs != "", "not merged") }},
@@ -560,9 +543,6 @@ const checkGrace = 15 * time.Minute
// runs from its start plus the grace before it is asked: an ask that cannot be made is still listed.
func boundSince(d *Delivery) time.Time {
if d.State == Proposed {
if d.AskedAgain != nil && d.outOfDate() != "" {
return *d.AskedAgain
}
if d.CheckAsk != nil {
return d.CheckAsk.At
}
@@ -582,17 +562,6 @@ var rewalkBound = Bound{For: unwalked, Says: "the walk that carried a later comm
// boundOf is a delivery's bound: its state's, or none for a delivery held because no walk answers it.
func boundOf(d *Delivery) (Bound, bool) {
if d.State == Proposed && d.outOfDate() != "" {
// Its check is out of date and not asked: nothing is running for it, and it costs nothing (ADR 0323 §7).
// Its bound starts when its check is asked: a recheck puts the verdict away; the merge tool's ask announces the
// head again, and its bound runs from then.
if d.AskedAgain == nil {
return Bound{}, false
}
b := Bounds[Proposed]
b.H2, b.Says = nil, "its verdict is "+d.outOfDate()+"; its check was asked again and not answered: the operator's"
return b, true
}
if d.State == Held && d.Unwalked && d.Walk == nil {
return heldUnwalked, true
}
@@ -82,9 +82,6 @@ func aDeliveryFor(r Row) (*Delivery, Facts) {
case EvRewalk:
d.MergedAs, d.AnsweredBy = "aaaa1111", "bbbb2222"
d.Walk = &WalkSeen{ID: "plan-2", State: walkFailed}
case EvTrunkMoved:
d.Check = &Verdict{Gate: "pass", Head: "c0ffee000000", Trunk: "d28ca95bbbbb"}
d.TrunkNow = "690966dccccccccc"
case EvLifted:
d.MergedAs, d.Lifted = "aaaa1111", "its group's composed check passed"
d.Walk = &WalkSeen{ID: "plan-1", State: walkBuilding, Waits: true, WaitedFor: "mesh-delivery"}
@@ -132,7 +129,7 @@ func TestEveryRowOfTheTableIsTakenWhenItsGuardHoldsAndRefusedWhenNot(t *testing.
func TestEveryPairTheTableDoesNotHoldIsRefusedByName(t *testing.T) {
events := []Event{EvAnnounced, EvAppeared, EvAdopted, EvChecked, EvAccepted, EvRefused, EvRecheck, EvReask, EvNewHead,
EvClosed, EvMerged, EvMergedUnchecked, EvGo, EvHold, EvRelease, EvDone, EvFailed, EvSuperseded, EvStop,
EvHistory, EvRetire, EvCovered, EvLifted, EvRewalk, EvTrunkMoved}
EvHistory, EvRetire, EvCovered, EvLifted, EvRewalk}
for _, from := range append([]State{None}, AllStates...) {
for _, ev := range events {
for _, act := range []bool{false, true} {
@@ -44,10 +44,6 @@ const Unannounced State = "unannounced"
// own for the proposed state's bound: its check was asked and not answered.
const Unanswered State = "unanswered"
// outOfDatePrefix begins the words the forge's holder puts a merge check back to pending with when its trunk moved
// (novox/hq ADR 0323 §4): the gitea module's outOfDateWords.
const outOfDatePrefix = "out of date:"
// UncheckedAfter is how long a head may go without a merge check before it is said: the announcer passes every half
// minute and the check is asked at once, so ten minutes is a fault, never a queue.
const UncheckedAfter = 10 * time.Minute
@@ -123,9 +119,6 @@ func (p OpenPull) unanswered() (waiting []string, pending map[string]string, sta
// since nothing here says whether the one it has answered.
case !ok || got.State == "":
waiting = append(waiting, c+" never set")
case got.State == "pending" && strings.HasPrefix(got.Description, outOfDatePrefix):
// Out of date because its trunk moved, and not asked (novox/hq ADR 0323 §4, §7): nothing runs for it and
// it costs nothing, so it is not a check asked and not answered. Asked again, it is pending anew.
case got.State == "pending":
pending[c] = utcOf(got.UpdatedAt)
waiting = append(waiting, c+" pending since "+orUnknown(pending[c]))
@@ -257,8 +250,8 @@ func (h *Holder) uncheckedLines(now time.Time) []StalledLine {
h2 = "none: its delivery is " + string(u.held) + ", a state the operator moves — the operator's"
}
out = append(out, StalledLine{ID: id, Number: u.pull.Number, State: state, For: now.Sub(u.first).Round(time.Second).String(),
Bound: u.bound().String(), Held: u.held,
H2: h2, Says: says, Waiting: waiting})
Bound: u.bound().String(),
H2: h2, Says: says, Waiting: waiting})
}
return out
}
@@ -71,11 +71,6 @@ func (h *Holder) waitsFor(d *Delivery) string {
if d.Walk != nil {
return "its walk: " + clip(d.Walk.Note, 120)
}
case Stopped:
if strandedUncovered(d) {
return "a later delivered merge to carry " + strings.Join(d.notMovedBy(), ", ") + ": it is stopped, and " +
"listed by stalled until one does (novox/hq issue 480)"
}
}
return ""
}
@@ -91,8 +86,7 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
group != "" && d.Group != group {
continue
}
// A final delivery `stalled` still lists stays, however old: the two never disagree (novox/hq issue 480).
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour && !strandedUncovered(d) {
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour {
continue
}
l := Line{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), For: now.Sub(d.Since).Round(time.Second).String(),
@@ -103,7 +97,7 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
if d.Walk != nil {
l.Walk = d.Walk.ID
}
l.Stalled = pastBound(d, now) || strandedUncovered(d)
l.Stalled = pastBound(d, now)
out = append(out, l)
}
sort.Slice(out, func(i, j int) bool {
@@ -121,9 +115,6 @@ func (h *Holder) Show(id string) (any, error) {
defer h.mu.Unlock()
if d := h.deliveries[id]; d != nil {
out := map[string]any{"delivery": d, "waits": h.waitsFor(d)}
if d.coveredStop() {
out["reads"] = d.coveredStopSaid() // stopped, and its change reached the machines (novox/hq issue 480)
}
if g := h.groups[d.Group]; g != nil {
state, why := GroupState(g, h.membersOf(g))
out["group"] = map[string]any{"group": g, "state": state, "why": why}
@@ -199,14 +190,10 @@ type StalledLine struct {
// the operator's own zone, which it cannot do inside a finished sentence (novox/hq issue 443). Waiting stays
// beside them for a controller that does not read them yet.
Checks []WaitingCheck `json:"checks,omitempty"`
// Held is the state of the delivery a line of the state Unanswered is about, ready or rejected, when one is held
// here (novox/hq issues 441, 451): its check answered once, so the controller advises a recheck of it rather than
// the build queue, reading the state here and never from Says. Empty for a head with no delivery.
Held State `json:"held,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// Stalled is the `stalled` verb: what the controller's self-check raises as `delivery.<id>.stalled`.
@@ -229,8 +216,6 @@ func (h *Holder) Stalled() []StalledLine {
b, _ := boundOf(d)
h2 := "none: the state is the operator's"
switch {
case d.State == Proposed && d.AskedAgain != nil && d.outOfDate() != "":
h2 = "none: its verdict is out of date and its check, asked again, has not answered — the operator's"
case d.State == Proposed && d.CheckAsk == nil:
h2 = "none: its check could not be asked yet — the controller does not answer delivery-check"
case d.State == Proposed && d.CheckAsk.Reasked > 0:
@@ -650,15 +635,6 @@ func (h *Holder) Status(listening string) map[string]any {
out := map[string]any{"deliveries": byState, "groups": len(h.groups), "owed": owed, "unkept": len(h.dirty),
"walks-unmatched": len(h.unmatched), "events": listening, "started": h.started.Format(time.RFC3339),
"seat-on-record": h.held}
// The owed effects tried since this owner started (novox/hq issue 407): done, failed and to be tried again,
// or given up, and when the last was tried. A count since the start, said with that start: two readings in
// one run give a window's tries.
tried := map[string]any{"since": h.started.Format(time.RFC3339), "done": h.tried.done, "failed": h.tried.failed,
"given-up": h.tried.givenUp, "tried": h.tried.done + h.tried.failed + h.tried.givenUp}
if !h.tried.last.IsZero() {
tried["last"] = h.tried.last.Format(time.RFC3339)
}
out["owed-tried"] = tried
// Each held delivery by name, with why and the verb that releases it (novox/hq issue 529): a count alone
// pointed a reader at nothing.
var held []string
+3 -46
View File
@@ -28,7 +28,6 @@ what it refuses.
| `groom` | optionally `pass`, `confirm` | the triage over the backlog: without arguments a dry run in the background, answering its pass id; with `pass` its progress and findings; with `confirm` and `pass` exactly that pass's findings applied, each checked again; see below |
| `tracking` | optionally `conditions` (condition keys) | for every condition key an issue links (link kind `condition`), the issues that link it, those not closed first: status, severity, needs, worker with since, active and stale, and where the fix is (`fix`); the operator's channel says it in a condition's message (hq ADR 0285) |
| `reserve` | `kind` (`decision` or `research`), optionally `for` (one line: what the record is about) | the next number of that kind, spent forever; answers `kind` and `number` (and `prefix`, the number as the record's name starts with it); see below |
| `fix-times` | optionally `days` (1 to 90, default 7) | each class's fix budget, the median and worst fix time of the issues whose last fix was delivered in the window, within and over, and every issue over its budget with its longest phase; see *A fix's time* |
**The cycle's rules, at the write** (hq playbook 03, which `00-META/checks/cycle.py` enforced on the
folders until the import, and hq ADR 0284): a status is one of `open`,
@@ -135,46 +134,11 @@ carries `abandoned` on `get` and its `list` row, beside its `worker`. A move by
`take-over`'s reason, as any stale worker; the answer, the note and the `moved` event say it was abandoned
(`abandoned`).
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`) of an
issue not closed: it is said once — a note on the issue and a warning on the operator's channel
(`seat:operator-channel.notify`, key `issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue
is delivered or the issue is closed. Resolved is the end of the warning as wontfix is: its worker proved the
acceptance live, so a fix whose own walk failed and that a later merge carried is not waited for (hq issue 547). An
issue resolved or wontfix is never said stuck. A module that runs only
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`): it is
said once — a note on the issue and a warning on the operator's channel (`seat:operator-channel.notify`, key
`issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue is delivered. A module that runs only
its bundle cannot raise a controller condition (hq ADR 0240 rule 8, ADR 0256), so the warning is the channel's, not the controller's.
## A fix's time (hq ADR 0313)
An issue's **fix time** runs from its first move to `fixing` (kept as `fixing-from`; a take-over or a later move
back to `fixing` does not restart it) to the moment the last of its fixes is delivered. An issue moved before the
moment was kept has it read from its log: a move's own moment, or, for a value carried into the log after one at
another status, its worker's `active` (the move), never its worker's `since`, which a diagnosis first keeps. Such a
fix time says `from-log`: it is kept and shown, and never warned on, so the first fixes heard of after the tracker
starts measuring raise no warning for work done before it. Once every fix it names is delivered, `get` carries it as `fix-time`:
`from`, `end`, `took`, the issue's `class`, its **fix budget** (`budget`: five minutes for a leaf module, ten for a
core one, counted from the move to `fixing`), `over`, the fix delivered last (`fix`), the `worker`, and four
`phases` read from that fix, which add up to the fix time:
| phase | ends at | read from |
|---|---|---|
| implementing | its pull request opened | mesh-delivery's `opened` (the pull request's first head heard) |
| review | the first approval at its head | the forge's reviews (`gitea_pull_reviews`): the forge's own approval, or a comment review whose first line is `Verdict: approve at <sha>`, the sha the head it was posted at or a prefix of at least seven characters (the forge refuses an approval of one's own pull request, so the mesh's independent reviews are comments) |
| checks | its merge, required checks and any wait for the merge included | mesh-delivery's `merged_at` |
| merge to running | delivered | the delivery's `delivered` (its delivery time) |
A phase whose end came before the end of the phase before it counts zero; a phase whose end is not known (no
approval at the head on the forge, a delivery that did not say when the pull request opened) is `unknown`, and its
time counts in the next phase, which says so. The class is that of the slowest-budgeted module the fixes move, as
mesh-delivery says it from the delivery plan (`class`: `core`, `leaf`, or `none` for a fix that moves no module);
an issue whose fixes move no module, or whose class is not said, is measured and shown with no fix budget.
**Over its budget is loud.** When the last fix is delivered and the fix time exceeds the class's fix budget, the
operator's channel is told (`seat:operator-channel.notify`, key `issue-tracker.fix.<class>.over-budget`), naming the
issue, its worker and its longest phase. Nothing is said while a fix is undelivered: a merged fix not delivered is
loud as a stuck fix. The warning clears when the next issue of that class is delivered within its budget; the
newest fix time of each class decides, and every sweep says or clears what a failed ask left. The delivery budget
(mesh-delivery's `times`, from the merge) stays a measure inside the fix budget.
## The import of hq's issues (slice 3)
`issue-tracker.import` reads every `04-ISSUES/NNN-slug/00-report.md` of hq **through the records module
@@ -397,13 +361,6 @@ reserve spending nothing, the first reserve refused while the knowledge base doe
set, a kind held none of being no answer, hq's folders never read, and a
reservation that cannot be kept leaving its number spent.
A fix's time is tested in `fixtime_test.go`: the start at the first move to `fixing` through a diagnosis, a
take-over and a move away and back; phases that add up to the fix time, one ended early counting zero and one not
known counted in the next; the class of the slowest fix, none and not known with no budget; no warning while
undelivered, the warning once delivered over budget, cleared by the next of its class within budget and not by
another class, and said at the next sweep when the channel did not take it; `fix-times` over a window, an issue
before it not counted; the start read from the log; and the first approval at the head.
The import is tested over `testdata/hq`, copies of seven real reports of hq chosen for their shapes (a
bracketed list with commas inside parentheses, quoted block items, an apostrophe in prose, a `resolved:`
date, a replay, an empty list, a wontfix): the frontmatter read as written, each issue created at its own
@@ -145,7 +145,7 @@ func (t *Tracker) askUnheard() error {
if since.IsZero() {
since = t.now()
}
if _, err := t.recordFix(pr, d.heard(since), n); err != nil {
if _, err := t.recordFix(pr, d.State, d.Why, d.Merged, since, n); err != nil {
return err
}
heard = true
+26 -117
View File
@@ -37,34 +37,6 @@ type Fix struct {
Since time.Time `json:"since"`
// StuckSaid says it was said stuck, on the issue and to the operator's channel.
StuckSaid bool `json:"stuck-said,omitempty"`
// Class is the class of what its delivery moves, as mesh-delivery says it: core, leaf, or none when it moves
// no module; Opened when its pull request was first heard (novox/hq ADR 0313 §2, §4).
Class string `json:"class,omitempty"`
Opened *time.Time `json:"opened,omitempty"`
}
// heardFix is what the delivery said of a pull request at one moment: its state, why, whether it is merged, and
// what a fix's time is read by — the class of what it moves, when its pull request opened and when it merged,
// each when said.
type heardFix struct {
State string
Why string
Merged bool
At time.Time
Class string
Opened *time.Time
MergedAt *time.Time
}
// news says whether what was heard adds anything to what a fix keeps.
func (h heardFix) news(f *Fix, kept string) bool {
if f.State != h.State || f.Why != kept || (h.Merged && !f.Merged) || (h.Class != "" && h.Class != f.Class) {
return true
}
if h.Opened != nil && (f.Opened == nil || !f.Opened.Equal(*h.Opened)) {
return true
}
return h.MergedAt != nil && (f.MergedAt == nil || !f.MergedAt.Equal(*h.MergedAt))
}
// reached says whether a delivery's state says the change reached the machines (mesh-delivery's Reached).
@@ -175,8 +147,7 @@ func DeliveryLine(is *Issue) string {
// recordFix keeps what the delivery said of a pull request on every issue whose fixed-by names it (only, when
// given, on that one issue), and answers the issues it changed.
func (t *Tracker) recordFix(pr string, h heardFix, only uint64) ([]uint64, error) {
state, why, merged, at := h.State, h.Why, h.Merged, h.At
func (t *Tracker) recordFix(pr, state, why string, merged bool, at time.Time, only uint64) ([]uint64, error) {
all, err := t.allIssues()
if err != nil {
return nil, err
@@ -187,7 +158,7 @@ func (t *Tracker) recordFix(pr string, h heardFix, only uint64) ([]uint64, error
continue
}
kept := cut(why, MostReasonKept)
if f := fixOf(is, pr); f != nil && (at.Before(f.Since) || !h.news(f, kept)) {
if f := fixOf(is, pr); f != nil && (at.Before(f.Since) || (f.State == state && f.Why == kept && f.Merged == (merged || f.Merged))) {
// Older than what is kept, or nothing new: transitions can arrive out of order.
continue
}
@@ -208,17 +179,6 @@ func (t *Tracker) recordFix(pr string, h heardFix, only uint64) ([]uint64, error
m := at
f.MergedAt = &m
}
if h.MergedAt != nil && (merged || f.Merged) {
m := h.MergedAt.UTC()
f.MergedAt = &m // the delivery's own moment of the merge, not when it was heard
}
if h.Class != "" {
f.Class = h.Class
}
if h.Opened != nil {
o := h.Opened.UTC()
f.Opened = &o
}
if reached(state) && !contains(i.Delivered, pr) {
i.Delivered = append(i.Delivered, pr)
}
@@ -247,10 +207,6 @@ func (t *Tracker) afterFix(n uint64) error {
t.Logf("[mesh-issues] the warning that %d is delivered and not resolved could not be cleared on the operator's channel (tried again at the next sweep): %v", n, err)
}
if allDelivered(is) {
// Its fix time, final once its last fix is delivered (novox/hq ADR 0313).
if err := t.timeFix(n); err != nil {
return err
}
if unresolved(is) {
prs := liveFixes(is)
sort.Strings(prs)
@@ -310,11 +266,9 @@ func (t *Tracker) afterFix(n uint64) error {
}
// stillStuck says whether an issue has a fix said stuck that is still named, not delivered, and the issue not
// closed. A closed issue is the end of its warning: resolved means its worker proved the acceptance live, so a fix
// the tracker never heard delivered is not waited for any more (a failed walk that a later merge carried stays
// failed, novox/hq issue 547), and wontfix waits for nothing.
// closed as wontfix.
func stillStuck(is *Issue) bool {
if closedStatus(is.Status) {
if is.Status == StatusWontfix {
return false
}
named := liveFixes(is)
@@ -327,30 +281,19 @@ func stillStuck(is *Issue) bool {
}
// clearStuck clears the warning said for an issue on the operator's channel once nothing it was said for is
// stuck — delivered, no longer named, or the issue closed (resolved or wontfix); one that fails is tried again at
// the next sweep.
// stuck — delivered, no longer named, or the issue closed as wontfix; one that fails is tried again at the next
// sweep.
func (t *Tracker) clearStuck(is *Issue) error {
if !is.StuckOpen || stillStuck(is) || t.Notify == nil {
return nil
}
if err := t.Notify(stuckKey(is.Number), "", "", "", "", stuckEnded(is), true); err != nil {
if err := t.Notify(stuckKey(is.Number), "", "", "", "", "the fix of issue "+keyOf(is.Number)+" is delivered or no longer waited for", true); err != nil {
return err
}
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.StuckOpen = false; return nil })
return err
}
// stuckEnded is what the operator's channel says when a stuck warning ends.
func stuckEnded(is *Issue) string {
switch is.Status {
case StatusResolved:
return "resolved: issue " + keyOf(is.Number) + " was proven live by its worker"
case StatusWontfix:
return "issue " + keyOf(is.Number) + " was closed as wontfix: its fix is no longer waited for"
}
return "the fix of issue " + keyOf(is.Number) + " is delivered or no longer waited for"
}
func stuckKey(n uint64) string { return "issue-tracker.fix-stuck." + keyOf(n) }
func prKey(repository string, number int) string {
@@ -365,15 +308,12 @@ func prKey(repository string, number int) string {
// to is kept on each issue whose fixed-by names it.
func (t *Tracker) OnTransition(e stdio.Envelope) error {
var body struct {
Repository string `json:"repository"`
Number int `json:"number"`
To string `json:"to"`
Why string `json:"why"`
MergedAs string `json:"merged_as"`
At time.Time `json:"at"`
Class string `json:"class"`
Opened *time.Time `json:"opened"`
MergedAt *time.Time `json:"merged_at"`
Repository string `json:"repository"`
Number int `json:"number"`
To string `json:"to"`
Why string `json:"why"`
MergedAs string `json:"merged_as"`
At time.Time `json:"at"`
}
if err := json.Unmarshal(e.Body, &body); err != nil || body.To == "" || body.Number <= 0 {
return nil
@@ -383,8 +323,7 @@ func (t *Tracker) OnTransition(e stdio.Envelope) error {
at = t.now()
}
merged := body.MergedAs != ""
changed, err := t.recordFix(prKey(body.Repository, body.Number), heardFix{State: body.To, Why: body.Why, Merged: merged,
At: at, Class: body.Class, Opened: nonZero(body.Opened), MergedAt: nonZero(body.MergedAt)}, 0)
changed, err := t.recordFix(prKey(body.Repository, body.Number), body.To, body.Why, merged, at, 0)
if err != nil {
return err
}
@@ -404,24 +343,6 @@ type Delivery struct {
Since time.Time
// Merged is the pull request's own merged flag, as the forge says it.
Merged bool
// Class, Opened and MergedAt are what a fix's time is read by (novox/hq ADR 0313), when the delivery says them.
Class string
Opened *time.Time
MergedAt *time.Time
}
// heard is what a delivery answered, as a fix keeps it.
func (d Delivery) heard(since time.Time) heardFix {
return heardFix{State: d.State, Why: d.Why, Merged: d.Merged, At: since, Class: d.Class, Opened: d.Opened, MergedAt: d.MergedAt}
}
// nonZero is a moment said, or nil for one not said.
func nonZero(t *time.Time) *time.Time {
if t == nil || t.IsZero() {
return nil
}
u := t.UTC()
return &u
}
// DeliveryState asks mesh-delivery for one pull request's delivery.
@@ -436,12 +357,9 @@ func busDeliveryState(owner string) DeliveryState {
}
var out struct {
Delivery struct {
State string `json:"state"`
Waits string `json:"waits"`
Since time.Time `json:"since"`
Class string `json:"class"`
Opened *time.Time `json:"opened"`
MergedAt *time.Time `json:"merged_at"`
State string `json:"state"`
Waits string `json:"waits"`
Since time.Time `json:"since"`
} `json:"delivery"`
Pull struct {
Merged bool `json:"merged"`
@@ -450,8 +368,7 @@ func busDeliveryState(owner string) DeliveryState {
if err := json.Unmarshal(raw, &out); err != nil {
return Delivery{}, fmt.Errorf("mesh-delivery's checks answered no delivery: %.200s", raw)
}
return Delivery{State: out.Delivery.State, Why: out.Delivery.Waits, Since: out.Delivery.Since.UTC(), Merged: out.Pull.Merged,
Class: out.Delivery.Class, Opened: nonZero(out.Delivery.Opened), MergedAt: nonZero(out.Delivery.MergedAt)}, nil
return Delivery{State: out.Delivery.State, Why: out.Delivery.Waits, Since: out.Delivery.Since.UTC(), Merged: out.Pull.Merged}, nil
}
}
@@ -500,7 +417,7 @@ func (t *Tracker) OnMoved(e stdio.Envelope) error {
if since.IsZero() {
since = t.now()
}
if _, err := t.recordFix(pr, d.heard(since), body.Number); err != nil {
if _, err := t.recordFix(pr, state, d.Why, d.Merged, since, body.Number); err != nil {
return err
}
}
@@ -536,7 +453,7 @@ func (t *Tracker) stuckAfter() time.Duration {
return DefaultStuckAfter
}
// Sweep says every fix of an issue not closed merged and not delivered past the bound: a note on its issue and a warning on the
// Sweep says every fix merged and not delivered past the bound: a note on its issue and a warning on the
// operator's channel, once; and every issue in fixing whose fixes were all delivered past its own bound and is not
// resolved (novox/hq issue 483). First it asks the delivery of a few fixes never heard of. It answers how many
// warnings it said. Run every few minutes.
@@ -551,26 +468,21 @@ func (t *Tracker) Sweep() (int, error) {
bound, now := t.stuckAfter(), t.now()
said := 0
for n, is := range all {
// A fix time missed at its delivery (a failure, or a restart between) is kept now; one of an issue moved to
// fixing before the tracker kept the moment is kept only when its fix is heard of again.
if is.FixingFrom != nil && is.FixTime == nil && allDelivered(is) {
if err := t.timeFix(n); err != nil {
t.Logf("[mesh-issues] the fix time of %d could not be kept (tried again at the next sweep): %v", n, err)
}
}
ok, err := t.sayNotResolved(is, now)
if err != nil {
t.Logf("[mesh-issues] that %d is delivered and not resolved could not be said on the operator's channel (said at the next sweep): %v", n, err)
} else if ok {
said++
}
if is.Status == StatusWontfix {
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
continue
}
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
if closedStatus(is.Status) {
// A resolved or wontfix issue waits for no fix: never said stuck (novox/hq issue 547).
continue
}
named := liveFixes(is)
for _, f := range is.Fixes {
if !contains(named, f.PullRequest) || contains(is.Delivered, f.PullRequest) || !f.Merged || f.MergedAt == nil ||
@@ -614,8 +526,5 @@ func (t *Tracker) Sweep() (int, error) {
said++
}
}
if err := t.judgeFixBudgets(); err != nil {
t.Logf("[mesh-issues] a fix over its budget could not be said or cleared on the operator's channel (tried again at the next sweep): %v", err)
}
return said, nil
}
@@ -11,22 +11,10 @@ import (
)
type notified struct {
mu sync.Mutex
said []string
clear []string
resolved []string
err error
}
// fixedAtFixing names a fix while the issue stays at fixing, as an agent does when the fix merges (ADR 0281 §3,
// ADR 0284): a fix stuck is said only for an issue not closed (issue 547).
func fixedAtFixing(t *testing.T, h *harness, n uint64, fixedBy ...string) {
t.Helper()
h.fixing(t, n)
if _, err := h.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", LocatedIn: []string{"mesh-catalog modules/records"},
FixedBy: fixedBy}); err != nil {
t.Fatal(err)
}
mu sync.Mutex
said []string
clear []string
err error
}
func (n *notified) notify(key, severity, summary, headline, explanation, resolved string, clear bool) error {
@@ -37,7 +25,6 @@ func (n *notified) notify(key, severity, summary, headline, explanation, resolve
}
if clear {
n.clear = append(n.clear, key)
n.resolved = append(n.resolved, resolved)
} else {
n.said = append(n.said, key+" "+severity+" "+summary)
}
@@ -134,7 +121,7 @@ func TestAnIssueKnowsWhetherItsFixIsDelivered(t *testing.T) {
func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
fixedAtFixing(t, h, n, "mesh-catalog#9")
resolve(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
@@ -165,7 +152,7 @@ func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
// Under the bound nothing is said.
h2 := newHarness(t)
m := h2.open(t, "y")
fixedAtFixing(t, h2, m, "mesh-catalog#9")
resolve(t, h2, m, "mesh-catalog#9")
n2 := &notified{}
h2.Notify = n2.notify
transition(t, h2, 9, "delivering", "", "abc", time.Date(2026, 10, 10, 11, 30, 0, 0, time.UTC))
@@ -207,8 +194,7 @@ func TestAFixIsNamedWhileTheIssueStaysLocated(t *testing.T) {
func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
h := newHarness(t)
n := h.open(t, "x")
fixedAtFixing(t, h, n, "mesh-catalog#9")
h.settings.DeliveredNotResolvedAfter = "24h" // only the stuck warning is under test here
resolve(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
@@ -258,78 +244,3 @@ func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
t.Fatalf("%d asks, %q", asks, v.Delivery)
}
}
// A resolved issue is the end of its stuck warning, as wontfix is: resolving clears it on the operator's channel,
// saying so, and the sweep never says a fix stuck for an issue resolved or wontfix, even one whose delivery failed
// and never reads delivered (issue 547).
func TestAResolvedIssueEndsItsStuckWarningAndIsNeverSaidStuck(t *testing.T) {
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
moved := func(t *testing.T, h *harness, n uint64) {
t.Helper()
raw, _ := json.Marshal(map[string]any{"number": n})
if err := h.OnMoved(stdio.Envelope{Body: raw}); err != nil {
t.Fatal(err)
}
}
for _, to := range []string{StatusResolved, StatusWontfix} {
t.Run(to, func(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed on one machine", "abc", merged)
if said, err := h.Sweep(); err != nil || said != 1 || len(note.said) != 1 {
t.Fatalf("not said stuck: %d %v %q", said, err, note.said)
}
ask := MoveAsk{Number: n, Status: to, FixedBy: []string{"mesh-catalog#9"}, ReplayNone: "proven live"}
if to == StatusWontfix {
ask = MoveAsk{Number: n, Status: to, Reason: "not worth it"}
}
if _, err := h.Move(ask); err != nil {
t.Fatal(err)
}
moved(t, h, n)
if len(note.clear) != 1 || note.clear[0] != stuckKey(n) || get(t, h, n).StuckOpen {
t.Fatalf("not cleared at the move: %q", note.clear)
}
if to == StatusResolved && !strings.HasPrefix(note.resolved[0], "resolved: ") {
t.Fatalf("%q", note.resolved)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 1 || len(note.clear) != 1 {
t.Fatalf("said again after %s: %d %q, cleared %q", to, said, note.said, note.clear)
}
})
}
// A warning left open by a tracker that kept it for a resolved issue is cleared at the first sweep, and an
// issue resolved before its fix was ever said stuck is never said stuck.
h := newHarness(t)
n := h.open(t, "x")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{err: errStuck}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed", "abc", merged)
if _, err := h.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
i.Fixes[0].StuckSaid, i.StuckOpen = true, true
return nil
}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: n, Status: StatusResolved, FixedBy: []string{"mesh-catalog#9"}}); err != nil {
t.Fatal(err)
}
moved(t, h, n) // the channel does not answer: kept open, tried at the sweep
note.err = nil
if said, _ := h.Sweep(); said != 0 || len(note.clear) != 1 || get(t, h, n).StuckOpen {
t.Fatalf("said %d, cleared %q", said, note.clear)
}
m := h.open(t, "y")
fixedAtFixing(t, h, m, "mesh-catalog#10")
transition(t, h, 10, "failed", "the gate failed", "abc", merged)
if _, err := h.Move(MoveAsk{Number: m, Status: StatusResolved, FixedBy: []string{"mesh-catalog#10"}}); err != nil {
t.Fatal(err)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 0 {
t.Fatalf("a resolved issue said stuck: %q", note.said)
}
}
@@ -1,647 +0,0 @@
package main
// A fix's time (novox/hq ADR 0313, issue 403). An issue's fix time runs from its first move to fixing to the moment
// the last of its fixes is delivered to every machine it should reach. It is held to the fix budget of its class —
// five minutes for a leaf module, ten for a core one — and split into four phases read from the fix delivered
// last: implementing (to its pull request opened), review (to an approval at its head), checks (to its merge) and
// merge to running (to delivered, its delivery time). A fix over its budget is said on the operator's channel once
// it is final, naming the issue, its worker and its longest phase; the warning clears when the next issue of that
// class is delivered within its budget. `fix-times` answers the last days' median and worst per class.
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// The classes of a fix (ADR 0282 §1, ADR 0313 §2) and the class of a fix that moves no module.
const (
ClassCore = "core"
ClassLeaf = "leaf"
ClassNone = "none"
)
// FixBudgets are the fix budgets by class, counted from the move to fixing (ADR 0313 §2).
var FixBudgets = map[string]time.Duration{ClassLeaf: 5 * time.Minute, ClassCore: 10 * time.Minute}
// fixClasses in the order `fix-times` says them.
var fixClasses = []string{ClassLeaf, ClassCore}
// The phases of a fix time, in order (ADR 0313 §4).
const (
PhaseImplementing = "implementing"
PhaseReview = "review"
PhaseChecks = "checks"
PhaseRunning = "merge to running"
)
// FixTime is an issue's fix time, kept on it once the last of its fixes is delivered.
type FixTime struct {
// From is the first move to fixing; End when the fix delivered last was delivered.
From time.Time `json:"from"`
End time.Time `json:"end"`
TookMS int64 `json:"took_ms"`
Took string `json:"took"`
// Class is core, leaf or none; empty when a fix's class is not known. A fix of class none, or one not known,
// has no budget: its time is shown and never warned on.
Class string `json:"class,omitempty"`
Budget string `json:"budget,omitempty"`
BudgetMS int64 `json:"budget_ms,omitempty"`
Over bool `json:"over,omitempty"`
// Fix is the pull request delivered last, which the phases are read from; Worker who worked on the issue.
Fix string `json:"fix"`
Worker string `json:"worker,omitempty"`
Phases []FixPhase `json:"phases"`
Longest string `json:"longest,omitempty"`
Said string `json:"said"`
// Warned says the over-budget warning was said for it on the operator's channel and not cleared since.
Warned bool `json:"warned,omitempty"`
// FromLog says its start was read back from the log, not kept by the move: an issue moved to fixing before the
// tracker measured fix times. Its fix time is kept and shown, and never warned on, so the first fixes heard of
// after the tracker starts measuring raise no warning for work done before it (novox/hq issue 403).
FromLog bool `json:"from-log,omitempty"`
}
// FixPhase is one phase of a fix time: measured, or unknown when the moment that ends it is not known — its time is
// then counted in the phase after it, which says so.
type FixPhase struct {
Name string `json:"name"`
State string `json:"state"`
TookMS int64 `json:"took_ms"`
Took string `json:"took"`
Said string `json:"said,omitempty"`
}
// Approval is when a pull request was first approved at its head, as the forge says it; a test stubs the asking.
// found is false when no approval at its head is on the forge.
type Approval func(repository string, number int) (at time.Time, found bool, err error)
// classOfFixes is the class of an issue's fixes: that of the slowest-budgeted module they move — core when any
// moves a core module, leaf when any moves a leaf one, none when none moves a module — and empty, with why, when
// a fix's class is not known and none is core.
func classOfFixes(is *Issue, prs []string) (string, string) {
var unknown []string
leaf := false
for _, pr := range prs {
f := fixOf(is, pr)
switch {
case f == nil || f.Class == "":
unknown = append(unknown, pr)
case f.Class == ClassCore:
return ClassCore, ""
case f.Class == ClassLeaf:
leaf = true
}
}
if len(unknown) > 0 {
return "", "the delivery did not say what " + strings.Join(unknown, ", ") + " moves"
}
if leaf {
return ClassLeaf, ""
}
return ClassNone, ""
}
// fixPhases splits the span from start to the delivered moment at the moments that end each phase, in order. A
// phase whose end is not known is unknown, and its time counts in the next phase known; a phase whose end came
// before the end of the phase before it counts zero (ADR 0313 §4). The phases add up to the fix time.
func fixPhases(start time.Time, ends []struct {
name string
at *time.Time
why string
}) ([]FixPhase, string) {
var out []FixPhase
prev := start
var carried []string
for _, e := range ends {
if e.at == nil {
out = append(out, FixPhase{Name: e.name, State: "unknown", Took: "0s", Said: e.why})
carried = append(carried, e.name)
continue
}
took := time.Duration(0)
if e.at.After(prev) {
took = e.at.Sub(prev)
prev = *e.at
}
p := FixPhase{Name: e.name, State: "measured", TookMS: took.Milliseconds(), Took: took.Round(time.Second).String()}
if len(carried) > 0 {
p.Said = "counts " + strings.Join(carried, " and ") + " too: its end is not known"
carried = nil
}
out = append(out, p)
}
best := -1
for i, p := range out {
if p.State == "measured" && (best < 0 || p.TookMS > out[best].TookMS) {
best = i
}
}
longest := ""
if best >= 0 {
longest = out[best].Name + " " + out[best].Took
}
return out, longest
}
// startOfFixing is when an issue was first moved to fixing (ADR 0313 §1), and whether that moment came from its
// log rather than from the move that kept it: the earliest of what it keeps (fixing-from, kept by the move since the
// tracker measures fix times) and what its log says. An issue moved before the moment was kept has it read from its
// log: a move's operation says its own moment. A value carried into the log, or imported, holds the issue whole and
// no moment of its own (it is stamped when it was carried); when the value before it was not at fixing it is the
// move's own write, and its worker's active is the move's moment (a move to fixing marks its worker active), not
// its worker's since, which a move from diagnosing by the same worker keeps (ADR 0284 §6). Only a first value
// already at fixing falls back to its worker's since, the best moment the log holds for it. Zero when it never was.
func (t *Tracker) startOfFixing(is *Issue) (time.Time, bool) {
var kept time.Time
if is.FixingFrom != nil {
kept = is.FixingFrom.UTC()
}
if t.Log == nil {
return kept, false
}
entries, err := t.readKey(is.Number, 0)
if err != nil {
t.Logf("[mesh-issues] the log of %d could not be read for its first move to fixing (its kept moment is used): %v", is.Number, err)
return kept, false
}
var logged time.Time
status, seen := "", false
for _, e := range entries {
var op Operation
if json.Unmarshal(e.Value, &op) != nil {
continue
}
at, became := op.At, ""
switch {
case op.Issue != nil:
became = op.Issue.Status
if w := op.Issue.Worker; w != nil {
switch {
case seen && status != StatusFixing && !w.Active.IsZero():
at = w.Active
case !w.Since.IsZero():
at = w.Since
}
}
case op.Set["status"] != nil:
_ = json.Unmarshal(op.Set["status"], &became)
default:
continue
}
if became == StatusFixing && status != StatusFixing && !at.IsZero() {
logged = at.UTC()
break
}
status, seen = became, true
}
switch {
case kept.IsZero():
return logged, !logged.IsZero()
case !logged.IsZero() && logged.Before(kept):
return logged, true
}
return kept, false
}
// fixTimeOf is an issue's fix time once every fix it names is delivered; nil, with why, when it is not measured.
func (t *Tracker) fixTimeOf(is *Issue) (*FixTime, string) {
if !allDelivered(is) {
return nil, "not every fix is delivered"
}
start, fromLog := t.startOfFixing(is)
if start.IsZero() {
return nil, "it was never moved to fixing"
}
prs := liveFixes(is)
var last *Fix
for _, pr := range prs {
if f := fixOf(is, pr); f != nil && (last == nil || f.Since.After(last.Since)) {
last = f
}
}
if last == nil {
return nil, "no fix is heard of"
}
end := last.Since.UTC()
took := time.Duration(0)
if end.After(start) {
took = end.Sub(start)
}
ft := &FixTime{From: start, End: end, TookMS: took.Milliseconds(), Took: took.Round(time.Second).String(), Fix: last.PullRequest,
FromLog: fromLog}
if is.Worker != nil {
ft.Worker = is.Worker.By
} else if is.FixTime != nil {
ft.Worker = is.FixTime.Worker
}
class, why := classOfFixes(is, prs)
ft.Class = class
if b, ok := FixBudgets[class]; ok {
ft.Budget, ft.BudgetMS, ft.Over = b.String(), b.Milliseconds(), took > b
}
// The phases, from the fix delivered last.
type end_ = struct {
name string
at *time.Time
why string
}
var approved *time.Time
approvalWhy := "no approval at its head is on the forge: review and checks are counted together"
if t.Approval != nil {
repo, number, _ := strings.Cut(last.PullRequest, "#")
var n int
if _, err := fmt.Sscanf(number, "%d", &n); err == nil {
at, found, err := t.Approval(repo, n)
switch {
case err != nil:
approvalWhy = "the forge could not be read for its approval (" + cut(err.Error(), 200) + "): review and checks are counted together"
case found:
a := at.UTC()
approved = &a
}
}
} else {
approvalWhy = "the forge is not asked here: review and checks are counted together"
}
openedWhy := "the delivery did not say when its pull request opened: implementing is counted in review"
mergedWhy := "the delivery did not say when it merged: checks are counted in merge to running"
ft.Phases, ft.Longest = fixPhases(start, []end_{
{PhaseImplementing, last.Opened, openedWhy},
{PhaseReview, approved, approvalWhy},
{PhaseChecks, last.MergedAt, mergedWhy},
{PhaseRunning, &end, ""},
})
var said []string
said = append(said, fmt.Sprintf("fixed in %s, from the first move to fixing (%s) to %s delivered (%s)", ft.Took,
start.Format(time.RFC3339), last.PullRequest, end.Format(time.RFC3339)))
switch {
case class == ClassNone:
said = append(said, "its fixes move no module, so it has no fix budget")
case class == "":
said = append(said, "no fix budget: "+why)
case ft.Over:
said = append(said, fmt.Sprintf("over its %s fix budget of %s", class, ft.Budget))
default:
said = append(said, fmt.Sprintf("within its %s fix budget of %s", class, ft.Budget))
}
if ft.Longest != "" {
said = append(said, "longest: "+ft.Longest)
}
if fromLog {
said = append(said, "its start was read from its log (moved to fixing before fix times were measured), so it is never warned on")
}
ft.Said = strings.Join(said, "; ")
return ft, ""
}
// sameFixTime says whether two fix times say the same, the warning's mark aside.
func sameFixTime(a, b *FixTime) bool {
if a == nil || b == nil {
return a == b
}
x, y := *a, *b
x.Warned, y.Warned = false, false
ra, _ := json.Marshal(x)
rb, _ := json.Marshal(y)
return string(ra) == string(rb)
}
// timeFix keeps an issue's fix time once every fix it names is delivered, and again when a later fix moves its
// end; then the warnings of its class are judged.
func (t *Tracker) timeFix(n uint64) error {
is, _, err := read(t.Store, n)
if err != nil {
return err
}
ft, _ := t.fixTimeOf(is)
if ft == nil || sameFixTime(ft, is.FixTime) {
return nil
}
if _, err := t.change(n, Act{Op: "fix-time", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
ft.Warned = i.FixTime.Warned
}
i.FixTime = ft
return nil
}); err != nil {
return err
}
if err := t.judgeFixBudgets(); err != nil {
t.Logf("[mesh-issues] a fix over its budget could not be said or cleared on the operator's channel (tried again at the next sweep): %v", err)
}
return nil
}
// fixOverKey is the warning a fix over its class's budget raises (ADR 0313 §6).
func fixOverKey(class string) string { return "issue-tracker.fix." + class + ".over-budget" }
// judgeFixBudgets says or clears each class's warning by the newest fix time of the class (ADR 0313 §6): the newest
// over its budget and not yet said is said; the newest within its budget clears what was said for the class. Run
// after every fix time kept and at every sweep, so a warning that failed is said, or cleared, at the next.
func (t *Tracker) judgeFixBudgets() error {
if t.Notify == nil {
return nil
}
all, err := t.allIssues()
if err != nil {
return err
}
var firstErr error
for _, class := range fixClasses {
var newest *Issue
var warned []uint64
for _, is := range all {
ft := is.FixTime
if ft == nil || ft.Class != class {
continue
}
if ft.Warned {
warned = append(warned, is.Number)
}
if newest == nil || ft.End.After(newest.FixTime.End) || (ft.End.Equal(newest.FixTime.End) && is.Number > newest.Number) {
newest = is
}
}
if newest == nil {
continue
}
ft := newest.FixTime
if ft.Over {
if ft.Warned || ft.FromLog {
continue
}
worker := ft.Worker
if worker == "" {
worker = "nobody named"
}
longest := ft.Longest
if longest == "" {
longest = "not known"
}
if err := t.Notify(fixOverKey(class), "warning",
fmt.Sprintf("issue %d: fixed in %s, over its %s fix budget of %s (longest: %s)", newest.Number, ft.Took, class, ft.Budget, longest),
fmt.Sprintf("A fix took longer than its budget (issue %d)", newest.Number),
fmt.Sprintf("Issue %d was fixed in %s, from %s's move to fixing to its last fix delivered, over the %s fix "+
"budget of %s. Its longest phase was %s. Nothing is needed from you: it says where fixes lose time. "+
"It clears when the next %s fix is delivered within its budget.", newest.Number, ft.Took, worker, class,
ft.Budget, longest, class), "", false); err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
if _, err := t.change(newest.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
i.FixTime.Warned = true
}
return nil
}); err != nil && firstErr == nil {
firstErr = err
}
continue
}
if len(warned) == 0 || ft.BudgetMS == 0 {
continue
}
if err := t.Notify(fixOverKey(class), "", "", "", "", fmt.Sprintf("issue %d's fix was delivered within its %s fix budget (%s of %s)",
newest.Number, class, ft.Took, ft.Budget), true); err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
sort.Slice(warned, func(i, j int) bool { return warned[i] < warned[j] })
for _, n := range warned {
if _, err := t.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
if i.FixTime != nil {
i.FixTime.Warned = false
}
return nil
}); err != nil && firstErr == nil {
firstErr = err
}
}
}
return firstErr
}
// FixTimesClass is one class's line of `fix-times`.
type FixTimesClass struct {
Class string `json:"class"`
Budget string `json:"budget"`
BudgetMS int64 `json:"budget_ms"`
Count int `json:"count"`
Within int `json:"within"`
Over int `json:"over"`
Median string `json:"median,omitempty"`
MedianMS int64 `json:"median_ms,omitempty"`
Worst string `json:"worst,omitempty"`
WorstMS int64 `json:"worst_ms,omitempty"`
// WorstIssue is the issue whose fix took the worst time.
WorstIssue uint64 `json:"worst_issue,omitempty"`
}
// FixTimesRow is an issue as `fix-times` names it.
type FixTimesRow struct {
Number uint64 `json:"number"`
Title string `json:"title"`
Class string `json:"class,omitempty"`
Took string `json:"took"`
TookMS int64 `json:"took_ms"`
Budget string `json:"budget,omitempty"`
Longest string `json:"longest,omitempty"`
Worker string `json:"worker,omitempty"`
Delivered string `json:"delivered"`
}
// FixTimes is the answer of `fix-times`.
type FixTimes struct {
Days int `json:"days"`
Since string `json:"since"`
Line string `json:"line"`
Classes []FixTimesClass `json:"classes"`
// Over is every issue over its budget in the window, newest first, with its longest phase.
Over []FixTimesRow `json:"over,omitempty"`
// Unbudgeted are the issues fixed in the window with no fix budget: their fixes move no module, or what they
// move is not known.
Unbudgeted []FixTimesRow `json:"unbudgeted,omitempty"`
}
// MostFixTimesDays is the longest window `fix-times` reads.
const MostFixTimesDays = 90
// FixTimes is the `fix-times` verb: each class's fix budget, the last days' median and worst fix time, and every
// issue over its budget with its longest phase (ADR 0313 §5, §7). An issue counts in the window its last fix was
// delivered in.
func (t *Tracker) FixTimes(days int) (*FixTimes, error) {
if days <= 0 {
days = 7
}
if days > MostFixTimesDays {
return nil, fmt.Errorf("days %d is more than the %d fix-times reads", days, MostFixTimesDays)
}
all, err := t.allIssues()
if err != nil {
return nil, err
}
since := t.now().Add(-time.Duration(days) * 24 * time.Hour)
a := &FixTimes{Days: days, Since: since.Format(time.RFC3339)}
byClass := map[string][]*Issue{}
row := func(is *Issue) FixTimesRow {
ft := is.FixTime
return FixTimesRow{Number: is.Number, Title: is.Title, Class: ft.Class, Took: ft.Took, TookMS: ft.TookMS, Budget: ft.Budget,
Longest: ft.Longest, Worker: ft.Worker, Delivered: ft.End.Format(time.RFC3339)}
}
for _, is := range all {
ft := is.FixTime
if ft == nil || ft.End.Before(since) {
continue
}
if _, ok := FixBudgets[ft.Class]; !ok {
a.Unbudgeted = append(a.Unbudgeted, row(is))
continue
}
byClass[ft.Class] = append(byClass[ft.Class], is)
if ft.Over {
a.Over = append(a.Over, row(is))
}
}
newestFirst := func(rows []FixTimesRow) {
sort.Slice(rows, func(i, j int) bool {
if rows[i].Delivered != rows[j].Delivered {
return rows[i].Delivered > rows[j].Delivered
}
return rows[i].Number > rows[j].Number
})
}
newestFirst(a.Over)
newestFirst(a.Unbudgeted)
var line []string
for _, class := range fixClasses {
b := FixBudgets[class]
c := FixTimesClass{Class: class, Budget: b.String(), BudgetMS: b.Milliseconds()}
is := byClass[class]
sort.Slice(is, func(i, j int) bool {
if is[i].FixTime.TookMS != is[j].FixTime.TookMS {
return is[i].FixTime.TookMS < is[j].FixTime.TookMS
}
return is[i].Number < is[j].Number
})
c.Count = len(is)
for _, i := range is {
if i.FixTime.Over {
c.Over++
} else {
c.Within++
}
}
if n := len(is); n > 0 {
ms := is[n/2].FixTime.TookMS
if n%2 == 0 {
ms = (is[n/2-1].FixTime.TookMS + is[n/2].FixTime.TookMS) / 2
}
med := time.Duration(ms) * time.Millisecond
c.MedianMS, c.Median = ms, med.Round(time.Second).String()
worst := is[n-1]
c.WorstMS, c.Worst, c.WorstIssue = worst.FixTime.TookMS, worst.FixTime.Took, worst.Number
line = append(line, fmt.Sprintf("%s: median %s, worst %s (issue %d), %d of %d within %s", class, c.Median,
c.Worst, c.WorstIssue, c.Within, c.Count, c.Budget))
} else {
line = append(line, fmt.Sprintf("%s: no issue fixed", class))
}
a.Classes = append(a.Classes, c)
}
a.Line = fmt.Sprintf("fix time over the last %d day(s), from the move to fixing to delivered — %s", days, strings.Join(line, "; "))
if len(a.Unbudgeted) > 0 {
a.Line += fmt.Sprintf("; %d fixed with no fix budget", len(a.Unbudgeted))
}
return a, nil
}
// firstApproval is when a pull request was first approved at its head among the forge's reviews: the forge's own
// approval, or — the forge refusing an approval of one's own pull request, so the mesh's independent reviews are
// posted as comments by the account that opened it — a comment review whose first line is "Verdict: approve at
// <sha>", the sha the head it was posted at or a prefix of it of at least seven characters. An approval dismissed,
// stale, or of an older head does not count.
func firstApproval(raw json.RawMessage) (time.Time, bool, error) {
var a struct {
Reviews []struct {
State string `json:"state"`
AtHead bool `json:"at_head"`
Commit string `json:"commit"`
Body string `json:"body"`
Submitted string `json:"submitted"`
Stale bool `json:"stale"`
Dismissed bool `json:"dismissed"`
} `json:"reviews"`
}
var s string
if json.Unmarshal(raw, &s) == nil {
raw = json.RawMessage(s)
}
var wrapped struct {
Content []struct {
Text string `json:"text"`
} `json:"content"`
}
if json.Unmarshal(raw, &wrapped) == nil && len(wrapped.Content) > 0 {
raw = json.RawMessage(wrapped.Content[0].Text)
}
if err := json.Unmarshal(raw, &a); err != nil || a.Reviews == nil {
return time.Time{}, false, fmt.Errorf("the forge's gitea_pull_reviews answered no reviews: %.200s", raw)
}
var first time.Time
for _, r := range a.Reviews {
if !r.AtHead || r.Stale || r.Dismissed {
continue
}
switch r.State {
case "APPROVED":
case "COMMENT":
if !verdictApproves(r.Body, r.Commit) {
continue
}
default:
continue
}
at, err := time.Parse(time.RFC3339, r.Submitted)
if err != nil {
continue
}
if first.IsZero() || at.Before(first) {
first = at
}
}
return first, !first.IsZero(), nil
}
// verdictApproves says whether a review's body opens with the mesh's approving verdict on the commit it was posted
// at: "Verdict: approve at <sha>", the sha that commit or a prefix of it of at least seven characters.
func verdictApproves(body, commit string) bool {
line, _, _ := strings.Cut(strings.TrimLeft(body, " \t\r\n"), "\n")
rest, ok := strings.CutPrefix(strings.TrimSpace(line), "Verdict: approve at ")
if !ok {
return false
}
sha := strings.ToLower(strings.TrimRight(strings.TrimSpace(rest), "."))
commit = strings.ToLower(strings.TrimSpace(commit))
return len(sha) >= 7 && commit != "" && strings.Trim(sha, "0123456789abcdef") == "" && strings.HasPrefix(commit, sha)
}
// busApproval asks the forge's reviews of a pull request over the bus (gitea_pull_reviews).
func busApproval(owner func() string) Approval {
return func(repository string, number int) (time.Time, bool, error) {
raw, err := stdio.Ask("gitea.gitea_pull_reviews", map[string]any{"owner": owner(), "repo": repository, "number": number})
if err != nil {
return time.Time{}, false, err
}
return firstApproval(raw)
}
}
@@ -1,480 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// novox/hq ADR 0313 (issue 403): an issue's fix time runs from its first move to fixing to its last fix
// delivered; it is held to its class's fix budget, split into four phases that add up to it, shown in get and
// fix-times, and a fix over its budget is said on the operator's channel once final, cleared at the next of its
// class delivered within budget.
// fixClock is a harness whose clock a test moves.
type fixClock struct {
*harness
at time.Time
// approvals is when each pull request was approved at its head, as the forge would say it.
approvals map[int]time.Time
}
func newFixClock(t *testing.T) *fixClock {
c := &fixClock{harness: newHarness(t), at: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC), approvals: map[int]time.Time{}}
c.Tracker.Now = func() time.Time { return c.at }
return c
}
func (c *fixClock) approval(repository string, number int) (time.Time, bool, error) {
a, ok := c.approvals[number]
return a, ok, nil
}
func (c *fixClock) later(d time.Duration) time.Time { c.at = c.at.Add(d); return c.at }
// delivery says one transition of a pull request as mesh-delivery says it, with what a fix's time is read by.
func (c *fixClock) delivery(t *testing.T, number int, to, class string, opened, mergedAt *time.Time) {
t.Helper()
body := map[string]any{"repository": "novox/mesh-catalog", "number": number, "to": to, "at": c.at}
if mergedAt != nil {
body["merged_as"] = "m" + strings.Repeat("a", 11)
body["merged_at"] = *mergedAt
}
if class != "" {
body["class"] = class
}
if opened != nil {
body["opened"] = *opened
}
raw, _ := json.Marshal(body)
if err := c.OnTransition(stdio.Envelope{Key: "mesh-delivery.transition", Body: raw}); err != nil {
t.Fatal(err)
}
}
func at(t time.Time) *time.Time { return &t }
func phaseTook(t *testing.T, ft *FixTime) map[string]time.Duration {
t.Helper()
out := map[string]time.Duration{}
var sum int64
for _, p := range ft.Phases {
out[p.Name] = time.Duration(p.TookMS) * time.Millisecond
sum += p.TookMS
}
if sum != ft.TookMS {
t.Fatalf("the phases add up to %dms, the fix time is %dms: %+v", sum, ft.TookMS, ft.Phases)
}
return out
}
// fixOne takes an issue through a fix: opened, moved to fixing, a pull request opened, approved, merged and
// delivered after the given spans; its fix time as kept.
func (c *fixClock) fixOne(t *testing.T, title string, pr int, class string, implementing, review, checks, running time.Duration) uint64 {
t.Helper()
n := c.open(t, title)
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#" + itoa(pr)}}); err != nil {
t.Fatal(err)
}
opened := c.later(implementing)
c.delivery(t, pr, "proposed", "", &opened, nil)
approved := c.later(review)
c.approvals[pr] = approved
merged := c.later(checks)
c.delivery(t, pr, "published", class, &opened, &merged)
c.later(running)
c.delivery(t, pr, "delivered", class, &opened, &merged)
return n
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
func TestAFixTimeRunsFromTheFirstMoveToFixingAndAddsUpItsPhases(t *testing.T) {
c := newFixClock(t)
c.Tracker.Approval = c.approval
note := &notified{}
c.Notify = note.notify
n := c.open(t, "a fix measured")
c.later(time.Minute)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusDiagnosing, By: "fix@g14"}); err != nil {
t.Fatal(err)
}
c.later(10 * time.Minute) // finding the cause is not the fix's time
start := c.at
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#31"}}); err != nil {
t.Fatal(err)
}
c.later(30 * time.Second)
// A take-over, and a move away and back, restart nothing.
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "other@shanks", TakeOver: "the first worker's session ended"}); err != nil {
t.Fatal(err)
}
if _, err := c.Move(MoveAsk{Number: n, Status: StatusOpen}); err != nil {
t.Fatal(err)
}
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "other@shanks"}); err != nil {
t.Fatal(err)
}
opened := c.later(90 * time.Second) // implementing: 2m
c.delivery(t, 31, "proposed", "", &opened, nil)
c.approvals[31] = c.later(time.Minute) // review: 1m
merged := c.later(3 * time.Minute) // checks: 3m
c.delivery(t, 31, "published", "leaf", &opened, &merged)
if is := get(t, c.harness, n); is.FixTime != nil {
t.Fatalf("a fix merged and not delivered has a fix time: %+v", is.FixTime)
}
c.later(2 * time.Minute) // merge to running: 2m
c.delivery(t, 31, "delivered", "leaf", &opened, &merged)
v, err := c.View(n)
if err != nil {
t.Fatal(err)
}
ft := v.FixTime
if ft == nil {
t.Fatalf("no fix time once the fix is delivered: %+v", v.Issue)
}
if !ft.From.Equal(start) || ft.TookMS != (8*time.Minute).Milliseconds() {
t.Fatalf("fix time from %s took %s, want from %s (the first move to fixing) and 8m0s", ft.From, ft.Took, start)
}
got := phaseTook(t, ft)
if got[PhaseImplementing] != 2*time.Minute || got[PhaseReview] != time.Minute || got[PhaseChecks] != 3*time.Minute ||
got[PhaseRunning] != 2*time.Minute {
t.Fatalf("phases %v", got)
}
if ft.Class != ClassLeaf || !ft.Over || ft.Budget != "5m0s" || ft.Longest != "checks 3m0s" || ft.Worker != "other@shanks" {
t.Fatalf("class %q over %v budget %q longest %q worker %q", ft.Class, ft.Over, ft.Budget, ft.Longest, ft.Worker)
}
// Over its budget: said once, naming the issue, its worker and its longest phase.
if len(note.said) != 1 || !strings.HasPrefix(note.said[0], "issue-tracker.fix.leaf.over-budget warning issue "+itoa(int(n))) ||
!strings.Contains(note.said[0], "checks 3m0s") {
t.Fatalf("said %q", note.said)
}
// Heard again: nothing new is kept or said.
c.delivery(t, 31, "delivered", "leaf", &opened, &merged)
if len(note.said) != 1 {
t.Fatalf("said again: %q", note.said)
}
}
// A phase whose end is not known is unknown and counted in the next; one that ended before the phase before it
// counts zero (a pull request opened before the move to fixing).
func TestAPhaseNotKnownIsCountedInTheNextAndOneEndedEarlyCountsZero(t *testing.T) {
c := newFixClock(t)
n := c.open(t, "a fix whose pull request came first")
opened := c.at
c.later(time.Minute)
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#32"}}); err != nil {
t.Fatal(err)
}
merged := c.later(4 * time.Minute)
c.later(time.Minute)
c.delivery(t, 32, "delivered", "leaf", &opened, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil {
t.Fatal("no fix time")
}
got := phaseTook(t, ft)
if got[PhaseImplementing] != 0 || got[PhaseChecks] != 4*time.Minute || got[PhaseRunning] != time.Minute {
t.Fatalf("phases %v", got)
}
for _, p := range ft.Phases {
if p.Name == PhaseReview && (p.State != "unknown" || p.Said == "") {
t.Fatalf("review with no approval asked: %+v", p)
}
if p.Name == PhaseChecks && !strings.Contains(p.Said, "review") {
t.Fatalf("checks does not say it counts review: %+v", p)
}
}
if ft.Over {
t.Fatalf("5m of a leaf's 5m is within it: %+v", ft)
}
}
// The class is the slowest-budgeted of the fixes'; a fix that moves no module has no budget and is never warned
// on; one whose class is not said has none either.
func TestAnIssuesClassIsItsSlowestFixAndOneMovingNothingHasNoBudget(t *testing.T) {
c := newFixClock(t)
note := &notified{}
c.Notify = note.notify
n := c.open(t, "two fixes, one core")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#41", "mesh-catalog#42"}}); err != nil {
t.Fatal(err)
}
merged := c.later(time.Minute)
c.delivery(t, 41, "delivered", "leaf", nil, &merged)
if get(t, c.harness, n).FixTime != nil {
t.Fatal("a fix time before the last fix is delivered")
}
c.later(7 * time.Minute)
c.delivery(t, 42, "delivered", "core", nil, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil || ft.Class != ClassCore || ft.Over || ft.Fix != "mesh-catalog#42" {
t.Fatalf("8m of a core fix: %+v", ft)
}
none := c.fixOne(t, "a fix in hq alone", 43, ClassNone, time.Minute, time.Minute, 10*time.Minute, time.Second)
ft = get(t, c.harness, none).FixTime
if ft == nil || ft.Class != ClassNone || ft.Over || ft.BudgetMS != 0 || !strings.Contains(ft.Said, "no fix budget") {
t.Fatalf("a fix moving nothing: %+v", ft)
}
unknown := c.fixOne(t, "a fix whose class is not said", 44, "", time.Minute, time.Minute, 10*time.Minute, time.Second)
ft = get(t, c.harness, unknown).FixTime
if ft == nil || ft.Class != "" || ft.Over || !strings.Contains(ft.Said, "no fix budget") {
t.Fatalf("a fix of no known class: %+v", ft)
}
if len(note.said) != 0 {
t.Fatalf("warned on a fix with no budget: %q", note.said)
}
}
// Nothing is said while a fix is undelivered, however long; once delivered over budget it is said; the next leaf
// fix delivered within budget clears it.
func TestAFixOverItsBudgetIsSaidOnceFinalAndClearedByTheNextWithin(t *testing.T) {
c := newFixClock(t)
c.Tracker.Approval = c.approval
note := &notified{}
c.Notify = note.notify
n := c.open(t, "a slow fix")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#51"}}); err != nil {
t.Fatal(err)
}
merged := c.later(20 * time.Minute)
c.delivery(t, 51, "published", "leaf", nil, &merged)
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 0 {
t.Fatalf("said while undelivered: %q", note.said)
}
c.later(time.Minute)
c.delivery(t, 51, "delivered", "leaf", nil, &merged)
if len(note.said) != 1 || !strings.HasPrefix(note.said[0], fixOverKey(ClassLeaf)+" warning") {
t.Fatalf("said %q", note.said)
}
if !get(t, c.harness, n).FixTime.Warned {
t.Fatal("not marked said")
}
// A core fix within its budget clears nothing of the leaf's.
c.fixOne(t, "a quick core fix", 52, ClassCore, time.Minute, time.Minute, time.Minute, time.Minute)
if len(note.clear) != 0 {
t.Fatalf("cleared by another class: %q", note.clear)
}
c.fixOne(t, "a quick leaf fix", 53, ClassLeaf, time.Minute, 30*time.Second, time.Minute, time.Minute)
if len(note.clear) != 1 || note.clear[0] != fixOverKey(ClassLeaf) {
t.Fatalf("cleared %q", note.clear)
}
if get(t, c.harness, n).FixTime.Warned {
t.Fatal("still marked said after the clear")
}
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 1 || len(note.clear) != 1 {
t.Fatalf("the sweep said again: %q, cleared %q", note.said, note.clear)
}
}
// A warning the channel did not take is said at the next sweep.
func TestAWarningNotTakenIsSaidAtTheNextSweep(t *testing.T) {
c := newFixClock(t)
note := &notified{err: errFake("the channel is away")}
c.Notify = note.notify
n := c.fixOne(t, "slow", 61, ClassLeaf, 5*time.Minute, time.Minute, time.Minute, time.Minute)
if get(t, c.harness, n).FixTime == nil {
t.Fatal("a fix time is kept although the warning was not taken")
}
note.err = nil
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 1 || !get(t, c.harness, n).FixTime.Warned {
t.Fatalf("said %q", note.said)
}
}
// fix-times answers each class's budget, the window's median and worst, and every issue over its budget with its
// longest phase; an issue delivered before the window does not count, so a lever is judged by the medians before
// and after it.
func TestFixTimesAnswersTheWindowsMedianAndWorst(t *testing.T) {
c := newFixClock(t)
old := c.fixOne(t, "long ago", 70, ClassLeaf, 30*time.Minute, time.Minute, time.Minute, time.Minute)
c.later(8 * 24 * time.Hour)
a := c.fixOne(t, "four minutes", 71, ClassLeaf, time.Minute, time.Minute, time.Minute, time.Minute)
b := c.fixOne(t, "six minutes", 72, ClassLeaf, 3*time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "eight minutes", 73, ClassLeaf, 5*time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "a core fix", 74, ClassCore, time.Minute, time.Minute, time.Minute, time.Minute)
c.fixOne(t, "hq alone", 75, ClassNone, time.Minute, time.Minute, time.Minute, time.Minute)
out, err := c.call(t, "fix-times", map[string]any{})
if err != nil {
t.Fatal(err)
}
ft := out.(*FixTimes)
leaf, core := ft.Classes[0], ft.Classes[1]
if ft.Days != 7 || leaf.Class != ClassLeaf || leaf.Count != 3 || leaf.Median != "6m0s" || leaf.Worst != "8m0s" ||
leaf.Within != 1 || leaf.Over != 2 || leaf.Budget != "5m0s" {
t.Fatalf("leaf %+v (issue %d, delivered before the window, must not count)", leaf, old)
}
if core.Count != 1 || core.Median != "4m0s" || core.Within != 1 {
t.Fatalf("core %+v", core)
}
if len(ft.Over) != 2 || ft.Over[0].Title != "eight minutes" || ft.Over[0].Longest != "implementing 5m0s" || ft.Over[1].Number != b {
t.Fatalf("over %+v", ft.Over)
}
if len(ft.Unbudgeted) != 1 || ft.Unbudgeted[0].Title != "hq alone" {
t.Fatalf("unbudgeted %+v", ft.Unbudgeted)
}
if !strings.Contains(ft.Line, "leaf: median 6m0s, worst 8m0s") {
t.Fatalf("line %q", ft.Line)
}
// Wider, the old one counts and moves the median.
out, _ = c.call(t, "fix-times", map[string]any{"days": 30})
if l := out.(*FixTimes).Classes[0]; l.Count != 4 || l.Median != "7m0s" {
t.Fatalf("over 30 days: %+v", l)
}
if _, err := c.call(t, "fix-times", map[string]any{"days": 91}); err == nil {
t.Fatal("days 91 not refused")
}
_ = a
}
// An issue moved to fixing before the tracker kept the moment has its start read from its log.
func TestTheStartOfAnIssueMovedBeforeItWasKeptIsReadFromItsLog(t *testing.T) {
c := newFixClock(t)
n := c.open(t, "moved to fixing long ago")
c.later(time.Minute)
start := c.at
c.fixing(t, n)
c.later(time.Hour)
is := get(t, c.harness, n)
if is.FixingFrom == nil || !is.FixingFrom.Equal(start) {
t.Fatalf("fixing-from %v, want %s", is.FixingFrom, start)
}
if got, fromLog := c.startOfFixing(is); !got.Equal(start) || fromLog {
t.Fatalf("kept: %s from the log %v, want %s kept by the move", got, fromLog, start)
}
is.FixingFrom = nil
if got, fromLog := c.startOfFixing(is); !got.Equal(start) || !fromLog {
t.Fatalf("from the log: %s (from the log %v), want %s", got, fromLog, start)
}
}
// A value carried into the log holds no moment of its own: after a value at diagnosing, the first value at fixing
// is the move's own write, and its start is its worker's active (the move), not its worker's since (the start of
// the diagnosis, ADR 0313 §1). A first value already at fixing falls back to its worker's since.
func TestACarriedIssueStartsAtTheMoveNotTheDiagnosis(t *testing.T) {
c := newFixClock(t)
diagnosed := time.Date(2026, 10, 1, 8, 0, 0, 0, time.UTC)
moved := diagnosed.Add(40 * time.Minute)
carry := func(n uint64, is Issue) {
t.Helper()
if _, err := c.log.AppendAt(keyOf(n), Operation{Op: OpCarried, By: Seat, At: c.at, Issue: &is}, c.log.lastOf(keyOf(n))); err != nil {
t.Fatal(err)
}
}
carry(900, Issue{Number: 900, Title: "x", Status: StatusDiagnosing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: diagnosed}})
carry(900, Issue{Number: 900, Title: "x", Status: StatusFixing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: moved}})
if got, fromLog := c.startOfFixing(&Issue{Number: 900}); !got.Equal(moved) || !fromLog {
t.Fatalf("start %s (from the log %v), want the move at %s, not the diagnosis at %s", got, fromLog, moved, diagnosed)
}
carry(901, Issue{Number: 901, Title: "y", Status: StatusFixing, Worker: &Worker{By: "fix@g14", Since: diagnosed, Active: moved}})
if got, _ := c.startOfFixing(&Issue{Number: 901}); !got.Equal(diagnosed) {
t.Fatalf("a first value at fixing: %s, want its worker's since %s", got, diagnosed)
}
}
// An issue moved to fixing before fix times were measured gets its fix time when its fix is next heard of, with its
// start from its log; it is kept and shown, never warned on, and leaves the class's warning to a fix measured whole.
func TestAFixWhoseStartCameFromTheLogIsNeverWarnedOn(t *testing.T) {
c := newFixClock(t)
note := &notified{}
c.Notify = note.notify
n := c.open(t, "fixed before the tracker measured")
c.fixing(t, n)
if _, err := c.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", FixedBy: []string{"mesh-catalog#81"}}); err != nil {
t.Fatal(err)
}
// As kept before this change: no fixing-from.
if _, err := c.change(n, Act{Op: "test", By: Seat}, func(i *Issue) error { i.FixingFrom = nil; return nil }); err != nil {
t.Fatal(err)
}
merged := c.later(3 * time.Hour)
c.delivery(t, 81, "delivered", ClassLeaf, nil, &merged)
ft := get(t, c.harness, n).FixTime
if ft == nil || !ft.Over || !ft.FromLog || ft.Warned || ft.TookMS != (3*time.Hour).Milliseconds() {
t.Fatalf("fix time %+v", ft)
}
if _, err := c.Sweep(); err != nil {
t.Fatal(err)
}
if len(note.said) != 0 {
t.Fatalf("warned on a fix whose start came from the log: %q", note.said)
}
// A fix measured whole after it is warned on as before.
c.fixOne(t, "measured whole", 82, ClassLeaf, 6*time.Minute, time.Minute, time.Minute, time.Minute)
if len(note.said) != 1 {
t.Fatalf("said %q", note.said)
}
}
// The forge's reviews: the first approval at the head, never one dismissed, stale or of an older head.
func TestTheReviewEndsAtTheFirstApprovalAtItsHead(t *testing.T) {
raw := json.RawMessage(`{"reviews":[
{"state":"APPROVED","at_head":false,"submitted":"2026-10-12T09:01:00Z"},
{"state":"REQUEST_CHANGES","at_head":true,"submitted":"2026-10-12T09:02:00Z"},
{"state":"APPROVED","at_head":true,"dismissed":true,"submitted":"2026-10-12T09:03:00Z"},
{"state":"APPROVED","at_head":true,"submitted":"2026-10-12T09:05:00Z"},
{"state":"APPROVED","at_head":true,"submitted":"2026-10-12T09:04:00Z"}]}`)
at, found, err := firstApproval(raw)
if err != nil || !found || at.Format(time.RFC3339) != "2026-10-12T09:04:00Z" {
t.Fatalf("%s %v %v", at, found, err)
}
// The mesh's own reviews: comments by the account that opened the pull request, whose first line is the verdict.
head := "43df7ebc5aa21c0bceacd1580f01f67d83d3495d"
comments := json.RawMessage(`{"reviews":[
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: request changes at ` + head + `\n\nBlocking: …","submitted":"2026-10-12T09:01:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Looks fine. Verdict: approve at ` + head + `","submitted":"2026-10-12T09:02:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: approve at 43df7eb\n\nAll clauses hold.","submitted":"2026-10-12T09:07:00Z"},
{"state":"COMMENT","at_head":true,"commit":"` + head + `","body":"Verdict: approve at ` + head + `\n\nAgain.","submitted":"2026-10-12T09:08:00Z"}]}`)
at, found, err = firstApproval(comments)
if err != nil || !found || at.Format(time.RFC3339) != "2026-10-12T09:07:00Z" {
t.Fatalf("a comment's verdict: %s %v %v", at, found, err)
}
for _, c := range []struct {
body, commit string
approves bool
}{
{"Verdict: approve at " + head, head, true},
{"Verdict: approve at 43df7eb", head, true},
{"Verdict: approve at 43df7e", head, false}, // shorter than seven characters
{"Verdict: approve at 0123456789", head, false}, // another commit
{"Verdict: request changes at " + head, head, false}, // not an approval
{"Not yet.\nVerdict: approve at " + head, head, false}, // not the first line
{"Verdict: approve at 43df7ebc", "", false}, // no commit to hold it to
} {
if got := verdictApproves(c.body, c.commit); got != c.approves {
t.Errorf("%q at %q approves %v, want %v", c.body, c.commit, got, c.approves)
}
}
if _, found, err := firstApproval(json.RawMessage(`{"reviews":[]}`)); err != nil || found {
t.Fatalf("no reviews: %v %v", found, err)
}
if _, _, err := firstApproval(json.RawMessage(`"nothing"`)); err == nil {
t.Fatal("an answer with no reviews read as none")
}
}
type errFake string
func (e errFake) Error() string { return string(e) }
@@ -112,12 +112,6 @@ type Issue struct {
Triage *Verdict `json:"triage,omitempty"`
// Worker is who works on it, while it is diagnosing or fixing (ADR 0284); nil at every other status.
Worker *Worker `json:"worker,omitempty"`
// FixingFrom is when it was first moved to fixing: where its fix time starts (novox/hq ADR 0313 §1). A
// take-over or a later move back to fixing keeps it.
FixingFrom *time.Time `json:"fixing-from,omitempty"`
// FixTime is its fix time, phase by phase, once the last of its fixes is delivered (ADR 0313 §4): measured,
// never set by a verb.
FixTime *FixTime `json:"fix-time,omitempty"`
// Opening is the id of the call that opened it, so a write whose answer was lost is known as its own.
Opening string `json:"opening"`
// Imported says that the issue, or its status, was taken from its folder in hq by the import of
+8 -34
View File
@@ -22,7 +22,7 @@ import (
const Seat = "issue-tracker"
// Verbs are the seat's verbs, as the manifest declares and claims them.
var Verbs = []string{"open", "get", "list", "move", "note", "link", "import", "groom", "set", "tracking", "reserve", "fix-times"}
var Verbs = []string{"open", "get", "list", "move", "note", "link", "import", "groom", "set", "tracking", "reserve"}
func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) }
@@ -95,16 +95,13 @@ func main() {
CallerOf: stdio.CallerOf,
}
t.Notify = busNotifier()
forgeOwner := func() string {
if s, err := t.Settings(); err == nil && strings.TrimSpace(s.ForgeOwner) != "" {
return strings.TrimSpace(s.ForgeOwner)
}
return "novox"
}
t.Deliveries = func(repository, number string) (Delivery, error) {
return busDeliveryState(forgeOwner())(repository, number)
owner := "novox"
if s, err := t.Settings(); err == nil && strings.TrimSpace(s.ForgeOwner) != "" {
owner = strings.TrimSpace(s.ForgeOwner)
}
return busDeliveryState(owner)(repository, number)
}
t.Approval = busApproval(forgeOwner)
if len(os.Args) > 1 && os.Args[1] == "dry-run" {
os.Exit(dryRunAtTheTerminal(t, os.Args[2:]))
}
@@ -384,10 +381,8 @@ func Tools(t *Tracker) []stdio.Tool {
Description: "One issue whole, by its number: title, status, severity, what it needs, report, who opened it " +
"and when, the owners it is located in, who works on it and since when (worker, and worker-stale when " +
"nobody did anything on it for a day), what fixed it and whether that is delivered, its notes and its " +
"links; the issues it waits for, those that depend on it (needed-by), and whether it is ready; " +
"whether its fix is delivered, as the delivery last said it (merged, held and why, delivered); and, once " +
"the last of its fixes is delivered, its fix time (fix-time) from its first move to fixing, its class and " +
"fix budget, and its phases: implementing, review, checks, merge to running.",
"links; the issues it waits for, those that depend on it (needed-by), and whether it is ready; and " +
"whether its fix is delivered, as the delivery last said it (merged, held and why, delivered).",
Input: schema([]string{"number"}, map[string]any{"number": number("the issue's number")}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("get", m, "number")
@@ -715,26 +710,5 @@ func Tools(t *Tracker) []stdio.Tool {
}
return t.reserve(t.callerOf(m), ask)
}},
{Name: Seat + ".fix-times",
Description: "How long fixes take, from the move to fixing to the last fix delivered (the fix time): each class's fix budget (leaf 5m, core 10m), the median and worst fix time of the issues whose last " +
"fix was delivered in the last days (7 unless days says otherwise), how many were within and over, and every " +
"issue over its budget with its longest phase (implementing, review, checks, merge to running), newest first; " +
"issues whose fixes move no module, or whose class is not known, are listed apart with no budget. One issue's " +
"fix time, phase by phase, is in its get. A lever meant to shorten fixes is judged by comparing this median " +
"over the seven days before it was delivered with the seven days after. Refused: days above 90.",
Input: schema(nil, map[string]any{
"days": map[string]any{"type": "integer", "minimum": 1, "maximum": MostFixTimesDays, "description": "the window, in days back from now (default 7)"},
}),
Run: func(m map[string]any) (any, error) {
a, err := readArgs("fix-times", m, "days")
if err != nil {
return nil, err
}
days, err := a.number("days", false)
if err != nil {
return nil, err
}
return t.FixTimes(int(days))
}},
}
}
@@ -106,7 +106,7 @@ func TestItDeclaresAndHoldsTheIssueTrackerSeatOnceForTheMesh(t *testing.T) {
// the import and a reservation read the record through the records module (issue 424).
if !reflect.DeepEqual(m.Consumes, []string{"mesh-issues." + EventOpened, "mesh-issues." + EventMoved, "mesh-delivery.transition"}) ||
!reflect.DeepEqual(m.Invokes, []string{"claude-code.claude_code_judge", "seat:mesh-delivery.checks", "seat:operator-channel.notify",
"gitea.gitea_pull_reviews", "records.records_status", "records.records_list", "records.records_read", "mesh-knowledge.knowledge_highest"}) {
"records.records_status", "records.records_list", "records.records_read", "mesh-knowledge.knowledge_highest"}) {
t.Fatalf("consumes %v, invokes %v", m.Consumes, m.Invokes)
}
if len(m.Build.Artifacts) != 1 || m.Build.Artifacts[0].From != "cmd/mesh-issues" ||
+1 -10
View File
@@ -36,12 +36,8 @@ type Tracker struct {
Judge Judgement
// Deliveries answers a pull request's delivery (ADR 0279 §3, ADR 0281); nil asks nothing.
Deliveries DeliveryState
// Notify tells the operator's channel of a stuck fix (ADR 0281) and of a fix over its budget (ADR 0313); nil
// says nothing there.
// Notify tells the operator's channel of a stuck fix (ADR 0281); nil says nothing there.
Notify Notifier
// Approval answers when a pull request was first approved at its head (ADR 0313 §4); nil asks nothing, and a
// fix's review is counted with its checks.
Approval Approval
// CallerOf is who called, for the arguments one call was handed (stdio.CallerOf); nil proves nobody.
CallerOf func(map[string]any) string
// Log is the tracker's record, the log `changes` (novox/hq ADR 0300); the bucket `issues` is its fold.
@@ -460,11 +456,6 @@ func (t *Tracker) move(by string, a MoveAsk) (*Moved, error) {
return err
}
is.Status = to
if to == StatusFixing && from != StatusFixing && is.FixingFrom == nil {
// The first move to fixing starts the fix time; none after it restarts it (ADR 0313 §1).
start := at
is.FixingFrom = &start
}
if (to == StatusResolved || to == StatusWontfix) && from != to {
is.Closed = &at
} else if to != StatusResolved && to != StatusWontfix {
+2 -5
View File
@@ -17,8 +17,7 @@
"groom",
"set",
"tracking",
"reserve",
"fix-times"
"reserve"
]
}
],
@@ -37,8 +36,7 @@
"groom",
"set",
"tracking",
"reserve",
"fix-times"
"reserve"
]
}
],
@@ -63,7 +61,6 @@
"claude-code.claude_code_judge",
"seat:mesh-delivery.checks",
"seat:operator-channel.notify",
"gitea.gitea_pull_reviews",
"records.records_status",
"records.records_list",
"records.records_read",
+6 -20
View File
@@ -23,7 +23,7 @@ operator, it is accepted only on the operator's Approve, given on a channel that
| `reserve` | `kind`, `for` | the next decision or research number, asked of the issue tracker (`issue-tracker.reserve`); spent whether or not its record is accepted |
| `propose` | the record whole | proposed, or refused naming every rule it breaks; an accepted record changes only by a progressive insight (`insight`). Each version proposed is then reviewed by the knowledge base itself (below) |
| `import` | `dry-run`, `commit` | hq's decision records and research efforts at the commit the records module stands at, each whole at its own number |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again; refused for a record whose clauses another record changed or links by number |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again |
`ask`, `get`, `holds` and `history` stay within `bound` bytes (4096 unless asked for, at most 65536) and say what they cut
and how to read the rest. The module's own tool `knowledge_health` answers its health (below).
@@ -78,7 +78,7 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- 25 on the structure: kind, reserved number, title, topic, date, deciders, research status, at least one
clause, clause numbering, bold lead, clause size, the Context, Consequences and "How it is checked" sections,
a research effort's overview, link kind, link target, link fit, no link to itself, what it extends, what it
supersedes or changes is accepted, a changed clause is in force or changed (a drafted one too), a superseded record cited beside its
supersedes or changes is accepted, a changed clause is in force, a superseded record cited beside its
superseder, a graduated effort says what it became, and the links in its text resolve;
- 4 on the prose: no word the glossary retired, no daemon's line, a deferral names its tracking issue, and a
progressive insight in the dated, marked form.
@@ -132,16 +132,8 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- `partial`: both ends of a change agree;
- `rests-on-live`: no accepted record extends a superseded one.
A failure is kept as a finding and told to the operator once, as a warning naming the rule and the records
(`operator-channel.notify`, keyed `knowledge.finding.<rule>.<records>`, sent again at the next step when it
could not be sent, and cleared once the rule holds for them again). It is a fault of the records, which their
authors repair, not of the module: the module's health names it and stays healthy, so it never holds the
module's own rollouts (the operator's decision of 2026-10-12 on hq issue 504). The acceptance stands.
`cited` is also judged ahead, in the review: a decision that names no home and that no accepted (or
superseded) record links is not judged and not accepted. Its review, by the knowledge base itself, asks changes
naming the rule (hq issue 504). It is not refused at propose, because no record can link a number only
reserved: a record citing a new one is proposed after it and accepted, and the new one is proposed again.
A failure is kept as a finding and makes the module unhealthy, so the mesh raises a condition naming the rule
and the records. The acceptance stands.
## The import
@@ -178,12 +170,6 @@ the older clause marked `changed`. When every prose record of a topic has been t
operator is told once (`operator-channel.notify`, a warning) with the counts. `return` sends a record back to be
drafted again.
A drafted clause may be changed by a later record (`changes 0281 §9`, novox/hq issue 494, ADR 0321): on its acceptance the
clause is marked `changed` with the change named, keeps its number and stays `drafted`, so it is still never served
as in force, and `get` says both. A redraft numbers a record's clauses anew, so `return` refuses a record any of
whose clauses another record changed or links by number, served or proposed, naming each link: a link never comes
to point at another clause.
## Where it keeps its records
In a database in the store: the module requires a `postgres-database` provision named `knowledge`, as
@@ -195,14 +181,14 @@ the consolidated schema for a fresh database is `schema.sql`. A test holds the t
- **the bundle**: the verbs, `knowledge_health`, the operator's answers heard, and once a minute (and at once when
a record is proposed) the router's record read, reservations never proposed withdrawn, the reviews of proposals,
and the drafting worker; and it tells the whole-set findings. It never changes the schema, and
and the drafting worker. It never changes the schema, and
refuses to serve while the schema is behind.
- **`prepare`** (run once): brings the schema to this version.
- **`keep`** (judged by its unit): judges the whole record set every five minutes and keeps the findings.
- **`watch`** (judged by `knowledge_health`, run as the machine's account, the one the runtime runs the bundle
as, so the tool can read what it writes): every minute, reads whether the database serves and what the
findings say, and leaves its verdict in `${dir:state}/health.json`. That is the file the health tool answers
from. A verdict older than three minutes is unhealthy; a whole-set finding is named in it but leaves it healthy.
from. A verdict older than three minutes is unhealthy.
## Settings
@@ -70,13 +70,6 @@ func held(r *Record, c Clause) string {
if c.Unconfirmed != "" {
s += " (the second judgement did not confirm it: " + c.Unconfirmed + ")"
}
if len(c.ChangedBy) > 0 {
if c.Status == ClauseSuperseded {
s += "; superseded by " + strings.Join(c.ChangedBy, ", ")
} else {
s += "; changed by " + strings.Join(c.ChangedBy, ", ") + ": read both"
}
}
return s
case r.Status == StatusProposed || c.Status == ClauseProposed:
return "proposed: not in force"
@@ -503,15 +503,6 @@ func (k *KB) Return(caller string, key Key, reason string) (map[string]any, erro
if r.Drafting == nil {
return nil, fmt.Errorf("%s's clauses were written by its author, not drafted: a written clause is changed by a new record", key)
}
all, err := k.Store.All(ctx)
if err != nil {
return nil, err
}
if held := clauseHolds(r, all); len(held) > 0 {
return nil, fmt.Errorf("%s is not returned: drafting it again would number its clauses anew, and these hold them to their numbers: %s. "+
"A clause keeps its number once another record links it (novox/hq issue 494). To say a drafted clause is wrong "+
"or no longer holds, propose a record that changes it (changes %s §N)", key, strings.Join(held, "; "), strings.TrimPrefix(key.String(), "ADR "))
}
r.Clauses = nil
r.Drafting = &Drafting{State: DraftWaiting, Returned: reason, At: k.now()}
why := "returned to be drafted again: " + reason
@@ -521,36 +512,3 @@ func (k *KB) Return(caller string, key Key, reason string) (map[string]any, erro
k.emit(EventReturned, r, caller, why, e.Versions+1)
return map[string]any{"record": key.String(), "says": fmt.Sprintf("%s's clauses are taken back and will be drafted again, with your reason; until then it serves none.", key)}, nil
}
// clauseHolds is what holds a record's clauses to their numbers: each clause another record changed or superseded,
// and each link of another record, served or proposed, that names one of its clauses. A redraft numbers the clauses
// anew, so a record any of these name is never returned: its links would point at other clauses.
func clauseHolds(r *Record, all []*Entry) []string {
var out []string
for _, c := range r.Clauses {
if len(c.ChangedBy) > 0 && (c.Status == ClauseChanged || c.Status == ClauseSuperseded) {
out = append(out, fmt.Sprintf("§%d is %s by %s", c.N, c.Status, strings.Join(c.ChangedBy, ", ")))
}
}
key := r.Key()
seen := map[string]bool{}
for _, e := range all {
for _, v := range []*Record{e.Current, e.Pending} {
if v == nil || v.Key() == key || v.Status == StatusWithdrawn {
continue
}
for _, l := range v.Links {
to, ok := l.To.Record()
if !ok || to != key || l.To.Clause == 0 {
continue
}
s := fmt.Sprintf("%s (%s) %s §%d", v.Key(), v.Status, l.Kind, l.To.Clause)
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
}
}
return out
}
@@ -1,108 +0,0 @@
package main
import (
"context"
"strings"
"testing"
)
// novox/hq issue 494: a record may change a clause the knowledge base holds only as drafted, and a record whose
// clauses another record links by number is never drafted again, so no link ever points at another clause.
// draftedNotConfirmed drafts 0240 and has the second judgement reject its first clause: it stays drafted.
func draftedNotConfirmed(t *testing.T, w *world) Key {
t.Helper()
k := Key{KindDecision, 240}
draftUntil(t, w, k, DraftDrafted)
e := mustGet(t, w.k.Store, k)
r := e.Current.Clone()
r.Clauses[0].Text += " NOT EXACT"
if err := w.k.Store.Save(context.Background(), Version{By: "test", Why: "a draft the checker will reject", Record: r}, true, false, e.Revision); err != nil {
t.Fatal(err)
}
r = draftUntil(t, w, k, DraftDone)
if r.Clauses[0].Source != SourceDrafted || !r.Accepted() {
t.Fatalf("0240 §1 is not a drafted clause of an accepted record: %s %+v", r.Status, r.Clauses[0])
}
return k
}
func TestARecordMayChangeAClauseHeldOnlyAsDrafted(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
k := draftedNotConfirmed(t, w)
n := w.reserved(t, "author@test")
p := valid(n)
p.Links = append(p.Links, LinkAsk{Kind: LinkChanges, To: "0240 §1", From: 2})
if _, err := w.k.Propose("author@test", p); err != nil {
t.Fatalf("a change of a drafted clause was refused: %v", err)
}
w.reviewAll(t)
if e := mustGet(t, s, Key{KindDecision, n}); !e.Current.Accepted() {
t.Fatalf("not accepted: %s", e.Current.Status)
}
// The drafted clause is changed by it, keeps its number and stays drafted: still never served as in force.
r := mustGet(t, s, k).Current
c := r.Clause(1)
want := itoa4(n) + " §2"
if c == nil || c.Status != ClauseChanged || c.Source != SourceDrafted || len(c.ChangedBy) != 1 || !strings.HasSuffix(c.ChangedBy[0], want) {
t.Fatalf("0240 §1 after the change: %+v", c)
}
if Served(r, *c) {
t.Fatal("a drafted clause that was changed is served as holding")
}
g, err := w.k.Get(GetAsk{Kind: KindDecision, Number: 240})
if err != nil {
t.Fatal(err)
}
if h := g.Clauses[0].Holds; !strings.HasPrefix(h, "drafted, not confirmed") || !strings.Contains(h, "changed by ADR "+want) {
t.Fatalf("get says of 0240 §1: %q", h)
}
// The whole-set rules find nothing wrong with the change.
f, _ := s.Findings(context.Background())
for _, x := range f {
if x.Rule == "partial" {
t.Fatalf("a whole-set finding: %+v", x)
}
}
})
}
func TestARecordWhoseClausesAreLinkedByNumberIsNeverReturned(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
k := draftedNotConfirmed(t, w)
// A proposed record cites 0240 §2: returning 0240 would point it at whatever is drafted second next time.
n := w.reserved(t, "author@test")
p := valid(n)
p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: "0240 §2", From: 1})
w.judge.review = `{"verdict":"changes","findings":["not yet"]}`
if _, err := w.k.Propose("author@test", p); err != nil {
t.Fatal(err)
}
w.reviewAll(t)
before := mustGet(t, s, k)
_, err := w.k.Return("operator@shanks", k, "clause 1 says more than the section")
if err == nil {
t.Fatal("a record a proposed record links by clause was returned")
}
if !strings.Contains(err.Error(), "ADR "+itoa4(n)+" (proposed) cites §2") || !strings.Contains(err.Error(), "changes 0240 §N") {
t.Fatalf("the refusal does not say what holds the clauses or the way forward: %v", err)
}
after := mustGet(t, s, k)
if after.Versions != before.Versions || len(after.Current.Clauses) != len(before.Current.Clauses) || len(w.said.of(EventReturned)) != 0 {
t.Fatalf("a refused return changed 0240: %d versions (was %d), %d clauses", after.Versions, before.Versions, len(after.Current.Clauses))
}
// A clause another record changed holds the record to its numbers too, with no link left to scan.
w2 := imported(t, newMemStore())
k2 := draftedNotConfirmed(t, w2)
n2 := w2.reserved(t, "author@test")
p2 := valid(n2)
p2.Links = append(p2.Links, LinkAsk{Kind: LinkChanges, To: "0240 §1", From: 2})
w2.accepted(t, "author@test", p2)
if _, err := w2.k.Return("operator@shanks", k2, "again"); err == nil || !strings.Contains(err.Error(), "§1 is changed by ADR "+itoa4(n2)+" §2") {
t.Fatalf("a record with a changed clause was returned, or the refusal does not name the change: %v", err)
}
})
}
@@ -514,73 +514,6 @@ func (k *KB) judgeSet(ctx context.Context) []Finding {
return findings
}
// findingPrefix keys the notice of one whole-set finding: the prefix, the rule, and the records it names.
const findingPrefix = "knowledge.finding."
func findingKey(f Finding) string {
names := make([]string, len(f.Records))
for i, r := range f.Records {
names[i] = strings.ReplaceAll(strings.ToLower(r), " ", "-")
}
return findingPrefix + f.Rule + "." + strings.Join(names, ".")
}
// TellFindings, called by the background work and never under the write lock, tells the operator, once, each finding of the rules that judge the whole record set, as a warning
// naming the rule and the records, and clears the notice once the rule holds for them again. A finding is a fault
// of the records, which their authors repair; it is not the module failing, so it leaves the module's health as it
// is and never holds the module's own rollouts (the operator's decision of 2026-10-12 on novox/hq issue 504, as
// issue 508 decided for a proposal never reviewed). A notice that cannot be sent is not kept as told, and is sent
// again at the next step.
func (k *KB) TellFindings() {
if k.Notify == nil || k.Clear == nil {
return
}
ctx := context.Background()
findings, err := k.Store.Findings(ctx)
if err != nil {
k.logf("[mesh-knowledge] the whole-set findings cannot be read to tell them: %v", err)
return
}
now := map[string]bool{}
for _, f := range findings {
key := findingKey(f)
now[key] = true
first, err := k.Store.Told(ctx, key)
if err != nil || !first {
continue
}
records := strings.Join(f.Records, ", ")
summary := fmt.Sprintf("The rule %s, which judges the whole record set, fails: %s.", f.Rule, f.Says)
explanation := "The knowledge base works; its records break a rule. The acceptance stands. The author of " + records +
" repairs it: a decision cited by nothing names its home in a progressive insight (propose with homes and insight), " +
"or a record that cites it is accepted. get and links show the record."
resolved := fmt.Sprintf("The rule %s holds again for %s", f.Rule, records)
if err := k.Notify(key, summary, "A rule of the record set fails: "+records, explanation, resolved); err != nil {
k.logf("[mesh-knowledge] the operator could not be told that %s fails for %s: %v; told again at the next step", f.Rule, records, err)
if uerr := k.Store.Untold(ctx, key); uerr != nil {
k.logf("[mesh-knowledge] and that it was not told could not be kept: %v", uerr)
}
}
}
told, err := k.Store.ToldOf(ctx, findingPrefix)
if err != nil {
k.logf("[mesh-knowledge] the notices of whole-set findings cannot be read: %v", err)
return
}
for _, key := range told {
if now[key] {
continue
}
if err := k.Clear(key, ""); err != nil {
k.logf("[mesh-knowledge] the notice %s could not be cleared: %v; it is cleared again at the next step", key, err)
continue
}
if err := k.Store.Untold(ctx, key); err != nil {
k.logf("[mesh-knowledge] the notice %s was cleared, and that could not be kept: %v", key, err)
}
}
}
// ---- reservations that were never proposed -------------------------------------------------------------------
// Withdraw withdraws every number reserved longer ago than the settings allow and never proposed: it stays spent,
@@ -40,9 +40,6 @@ func TestMain(m *testing.M) {
serve(k, os.Getenv("MESH_KNOWLEDGE_STATUS"))
os.Exit(0)
}
// Every test runs in a zone that is not UTC, so a time said locally is proven to be, wherever the tests run. It
// is set once, before any goroutine starts: a test that set it while a bus test's server read it raced.
time.Local = testZone
code := m.Run()
stopPostgres()
os.Exit(code)
@@ -439,13 +436,9 @@ func valid(n int) ProposeAsk {
{Name: "Consequences", Text: "Merges land sooner."},
{Name: "How it is checked", Text: "The controller's merge-window tests."},
},
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}},
Homes: []string{testHome}}
Links: []LinkAsk{{Kind: LinkExtends, To: "0276"}, {Kind: LinkChanges, To: "0276 §1", From: 1}}}
}
// testHome is the hq design valid names as its home: a decision is accepted only cited (novox/hq issue 504).
const testHome = "03-DESIGN/01-to-be/55-the-knowledge-base.md"
// imported is a world whose knowledge base holds the fixture's records.
func imported(t *testing.T, s Store) *world {
t.Helper()
@@ -164,8 +164,6 @@ func background(k *KB) {
logf("[mesh-knowledge] withdrew %d reservation(s) never proposed", n)
}
review()
// The keeper judges the record set in a process of its own, which tells nobody: its findings are told here.
k.TellFindings()
for {
said, err := k.DraftStep()
if err != nil {
@@ -554,11 +552,10 @@ type HealthAnswer struct {
At time.Time `json:"at,omitempty"`
}
// Health is whether the knowledge base works: unhealthy when its database cannot serve. A rule that judges the whole
// record set and fails is a fault of the records, which their authors repair, not of the module: it is told to the
// operator as a warning naming the rule and the records (TellFindings), and named here, but leaves the module
// healthy (the operator's decision of 2026-10-12 on novox/hq issue 504). So does a proposal that ran out of review
// tries (novox/hq issue 508). Neither holds the module's own rollouts.
// Health is whether the knowledge base works: unhealthy when its database cannot serve, or when a rule that judges
// the whole record set fails — the mesh raises a condition naming the rule and the records. A proposal that ran out
// of review tries waits on its author, not on the module: it is told to its author and the operator, and named
// here, but leaves the module healthy, so it never holds the module's own rollouts (novox/hq issue 508).
func (k *KB) Health(ready Healther) HealthAnswer {
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
@@ -589,14 +586,17 @@ func (k *KB) Health(ready Healther) HealthAnswer {
waiting = fmt.Sprintf("%d proposed record(s) could not be reviewed in %d tries and wait for their authors to propose them again: %s",
n, MostReviewTries, strings.Join(unreviewed, "; "))
}
why := "the database serves and every rule that judges the whole record set holds"
if len(says) > 0 {
if len(says) > 8 {
says = append(says[:8], fmt.Sprintf("and %d more", len(says)-8))
}
why = fmt.Sprintf("the database serves; %d rule(s) that judge the whole record set fail, told to the operator as a warning "+
"for the records' authors to repair: %s", len(findings), strings.Join(says, "; "))
why := fmt.Sprintf("%d rule(s) that judge the whole record set fail: %s", len(findings), strings.Join(says, "; "))
if waiting != "" {
why += ". " + waiting
}
return HealthAnswer{State: "unhealthy", Why: why}
}
why := "the database serves and every rule that judges the whole record set holds"
if waiting != "" {
why += "; " + waiting
}
@@ -605,11 +605,10 @@ func (k *KB) Health(ready Healther) HealthAnswer {
func healthTool(status string, now func() time.Time) stdio.Tool {
return stdio.Tool{Name: HealthTool,
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version, unhealthy " +
"otherwise. A rule that judges the whole record set and fails — a decision accepted that nothing cites, a supersession " +
"or a change said on one end only, a record extending a superseded one — is named but leaves it healthy: it is told " +
"to the operator as a warning for the records' authors (novox/hq issue 504). So is a proposed record that ran out of " +
"review tries: it waits on its author (novox/hq issue 508).",
Description: "How the knowledge base is, as its watch last saw it: healthy when its database serves this version and every " +
"rule that judges the whole record set holds; unhealthy otherwise, naming the rule and the records — a decision accepted " +
"that nothing cites, a supersession or a change said on one end only, a record extending a superseded one. A " +
"proposed record that ran out of review tries is named but leaves it healthy: it waits on its author (novox/hq issue 508).",
Run: func(map[string]any) (any, error) { return ReadStatus(status, now()), nil }}
}
@@ -919,8 +918,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
"link to nothing, a clause without a bold lead, a superseded record cited alone, an unknown topic, a retired word. A " +
"proposal is revised by proposing it again; an accepted record changes only by a progressive insight (insight: what " +
"stood before). The knowledge base then reviews each version itself, in a judgement that sees only the record, the " +
"records it links and the rules. A decision that names no home and that no accepted record links is not judged: " +
"its review asks changes for the whole-set rule cited, which its acceptance would break. On approve it is accepted (a record that needs the operator waits for the operator's " +
"records it links and the rules: on approve it is accepted (a record that needs the operator waits for the operator's " +
"Approve), on changes it stays proposed with the findings for its author to revise, and when no judgement can be had " +
"it stays proposed and says so. Nothing proposed is in force.",
Input: schema([]string{"number", "title"}, map[string]any{
@@ -935,7 +933,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
"links": map[string]any{"type": "array", "description": "its links: kind, to (\"0276\", \"0276 §4\", \"research 045\", \"issue 365\", a design's path), from (the clause it goes out from)",
"items": map[string]any{"type": "object", "properties": map[string]any{"kind": map[string]any{"type": "string", "enum": LinkKinds},
"to": text("the target"), "from": integer("the clause of this record it goes out from")}, "required": []string{"kind", "to"}}},
"homes": texts("hq documents that cite it, such as a design's path; a decision needs one unless an accepted record links it"),
"homes": texts("hq documents that cite it, such as a design's path"),
"needs-operator": flag("it holds only once the operator approves it"),
"status": map[string]any{"type": "string", "enum": ResearchStatuses, "description": "a research effort's status"},
"insight": text("for an accepted record: a progressive insight, saying what stood before"),
@@ -1010,8 +1008,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
{Name: Seat + ".return",
Description: "Send the clauses drafted from a prose decision back to be drafted again, with your reason, which the next " +
"drafting reads. Until they are drafted and confirmed again the record serves none. Refused: a record whose clauses its " +
"author wrote, an empty reason, and a record whose clauses another record changed or links by number (a redraft " +
"numbers them anew; a record that changes the drafted clause says it is wrong instead).",
"author wrote, an empty reason.",
Input: schema([]string{"number", "reason"}, map[string]any{"kind": kindArg(), "number": integer("the record's number"),
"reason": text("what is wrong with the drafted clauses")}),
Run: func(m map[string]any) (any, error) {
@@ -408,10 +408,7 @@ func (k *KB) reviewOne(key Key) (string, error) {
k.mu.Lock()
e, err := k.Store.Get(ctx, key)
var linked []*Record
var all []*Entry
var allErr error
if err == nil && e != nil {
all, allErr = k.Store.All(ctx)
seen := map[Key]bool{key: true}
for _, l := range e.Working().Links {
to, ok := l.To.Record()
@@ -433,20 +430,9 @@ func (k *KB) reviewOne(key Key) (string, error) {
var verdict string
var findings []string
by := ReviewBy
uncited := ""
if allErr == nil {
uncited = UncitedOnAcceptance(w, all)
}
switch {
case allErr != nil:
err = fmt.Errorf("the record set cannot be read to judge whether a record cites it: %v", allErr)
case uncited != "":
// The whole-set rule cited, judged ahead: changes, asked by the knowledge base itself without a judgement,
// so its acceptance never turns the module's health red for it (novox/hq issue 504).
verdict, findings = VerdictChanges, []string{"the rule cited, which judges the whole record set: " + uncited}
case len(question) > MostReviewQuestion:
if len(question) > MostReviewQuestion {
err = fmt.Errorf("the record is %d bytes as a question, and a judgement takes at most %d: shorten it", len(question), MostReviewQuestion)
default:
} else {
var answer json.RawMessage
var judge string
answer, judge, err = k.Judge(question, reviewSchema, k.judgeModel())
@@ -155,16 +155,13 @@ func TestProposedAgainAfterTheLastTryClearsTheNotice(t *testing.T) {
})
}
// testZone is the local zone every test runs in, set once by TestMain: not UTC.
var testZone = time.FixedZone("CEST", 2*60*60)
// inZone says the test needs the local zone not to be UTC; TestMain set it before any goroutine started. Setting it
// here, under a running bus server's goroutines, was a data race the race detector failed.
// inZone sets the local zone to one that is not UTC for one test, so a time said locally is proven to be, wherever
// the test runs.
func inZone(t *testing.T) {
t.Helper()
if time.Local != testZone {
t.Fatal("the local zone is not the tests' own: TestMain sets it")
}
was := time.Local
time.Local = time.FixedZone("CEST", 2*60*60)
t.Cleanup(func() { time.Local = was })
}
// novox/hq issue 497: a clear the operator's channel refused keeps its notice as told, and is sent again at the next
@@ -48,7 +48,7 @@ var PerRecordRules = []Rule{
{"self-link", "no link points at the record itself", "records.py supersession"},
{"rests-on", "a record it extends is accepted or proposed, never superseded or withdrawn", "records.py rests-on"},
{"supersedes-accepted", "a record it supersedes or changes is accepted", "records.py supersession, partial-supersession"},
{"changes-in-force", "a clause it changes is in force or changed, never superseded or proposed; a drafted one may be changed", "records.py partial-supersession"},
{"changes-in-force", "a clause it changes is in force or changed, never superseded, proposed or drafted", "records.py partial-supersession"},
{"live-citation", "a superseded record it cites is cited beside the record that replaced it", "records.py live-citation"},
{"became", "a graduated research effort names what it became", "cycle.py research"},
{"text-links", "every link its text writes to a decision record or research effort names one the knowledge base holds", "records.py links"},
@@ -261,10 +261,8 @@ func JudgeRecord(r *Record, held *Entry, lookup Lookup, ref *Reference) []Refusa
if !target.Accepted() || target.Status == StatusSuperseded {
refuse("supersedes-accepted", "%s changes %s, which is %s, not accepted", self, to, target.Status)
} else if c := target.Clause(l.To.Clause); c != nil {
// A drafted clause may be changed (novox/hq issue 494, ADR 0321): the change is recorded against the clause the
// knowledge base holds, by its number, which nothing renumbers once another record links it (Return).
if c.Status != ClauseInForce && c.Status != ClauseChanged {
refuse("changes-in-force", "%s changes %s §%d, which is %s", self, to, l.To.Clause, c.Status)
if (c.Status != ClauseInForce && c.Status != ClauseChanged) || c.Source == SourceDrafted {
refuse("changes-in-force", "%s changes %s §%d, which is %s (%s)", self, to, l.To.Clause, c.Status, c.Source)
}
}
case LinkCites:
@@ -548,36 +546,6 @@ func JudgeSet(records []*Record) []Finding {
return out
}
// UncitedOnAcceptance is the whole-set rule `cited` judged ahead, on a decision's working version as it would
// stand once accepted, over the records as JudgeSet reads them: "" when it names a home or a record accepted (or
// superseded) links it, else what the rule would say. hq held this rule at the merge of the record's own pull
// request, which carried the design citing it; the knowledge base holds it in the review, before the judgement, so
// an acceptance never leaves a decision cited by nothing (novox/hq issue 504). It is not refused at propose: no
// record can link a number only reserved, so a record that will cite a new one is proposed after it, accepted, and
// the new one proposed again. A citing record changed by a correction after the acceptance is what the rule, run
// after every acceptance, still says. An imported record was held to hq's checks, and its correction is not judged
// here, as JudgeSet does not judge it.
func UncitedOnAcceptance(r *Record, all []*Entry) string {
if r.Kind != KindDecision || len(r.Homes) > 0 || (r.Pending != nil && r.AcceptedAt.IsZero()) {
return ""
}
k := r.Key()
for _, e := range all {
from := e.Current
if from.Key() == k || !(from.Accepted() || from.Status == StatusSuperseded) {
continue
}
for _, l := range from.Links {
if to, ok := l.To.Record(); ok && to == k {
return ""
}
}
}
return fmt.Sprintf("%s would be accepted with nothing citing it: it names no home and no accepted record links it. Name "+
"its home (homes: the hq design or playbook that cites it), or have the record that links it accepted first; then "+
"propose this one again", k)
}
// RuleFrom is the day each dated rule began to hold a record, as hq's checks hold it: a record dated earlier was
// never held to it, and a dated correction of it (a progressive insight, which keeps the record's date) is not.
// words.py held the retired words from 2026-10-07 and the daemons' lines from 2026-10-09; cycle.py the deferrals
@@ -1,119 +1,38 @@
package main
import (
"context"
"errors"
"strings"
"testing"
)
// A decision nothing would cite is not accepted: its review asks changes itself, without a judgement, naming the
// rule cited, and the module stays healthy (novox/hq issue 504). Before, the review approved it, the acceptance
// broke the rule, and the module turned unhealthy for a record its author could only repair after the fact.
func TestADecisionNothingWouldCiteIsNotAccepted(t *testing.T) {
// to-be 55 §10: "A whole-set rule that fails is loud — the module's test: an acceptance that leaves a decision
// cited by nothing raises the condition." The condition is the mesh's, raised when the module's health answers
// unhealthy; the test reads the health answer the node-engine reads.
func TestAnAcceptanceThatLeavesADecisionCitedByNothingMakesTheModuleUnhealthy(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy after the import of real records: %s", h.Why)
}
n := w.reserved(t, "author@g14")
p := valid(n)
p.Homes = nil
if _, err := w.k.Propose("author@g14", p); err != nil {
t.Fatal(err)
}
judged := len(w.judge.reviews)
w.reviewAll(t)
e := mustGet(t, s, Key{KindDecision, n})
if e.Current.Status != StatusProposed {
t.Fatalf("a decision nothing cites was %s by its review", e.Current.Status)
}
rv := e.Current.Reviews
if len(rv) == 0 || rv[len(rv)-1].Verdict != VerdictChanges || !strings.Contains(rv[len(rv)-1].Findings, "the rule cited") ||
rv[len(rv)-1].By != ReviewBy {
t.Fatalf("its review asked no changes for the rule cited: %+v", rv)
}
if len(w.judge.reviews) != judged {
t.Fatal("a judgement was spent on a record the rule cited refuses")
}
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy for a record not accepted: %s", h.Why)
}
// Named with its home, it is accepted, and the module stays healthy.
p.Homes = []string{testHome}
w.accepted(t, "author@g14", p)
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy after a decision with its home was accepted: %s", h.Why)
}
})
}
// A new decision is cited by a record proposed after it: no record can link a number only reserved. Once the
// citing record is accepted, the new one proposed again is accepted. A correction of the citing record that drops
// the link still leaves the decision cited by nothing, and that is loud: told to the operator once, as a warning
// naming the rule and the record, while the module stays healthy and its rollouts are not held (the operator's
// decision of 2026-10-12 on novox/hq issue 504; to-be 55 §10). A record that cites it clears the notice.
func TestARecordAcceptedCitingANewDecisionLetsItBeAcceptedAndDroppingTheLinkIsLoud(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
n := w.reserved(t, "author@g14")
a := valid(n)
a.Homes = nil
if _, err := w.k.Propose("author@g14", a); err != nil {
t.Fatal(err)
}
w.reviewAll(t)
m := w.reserved(t, "author@g14")
b := valid(m)
b.Links = append(b.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", b)
w.accepted(t, "author@g14", a) // its revision: an accepted record links it now
if h := w.k.Health(nil); !h.Healthy {
t.Fatalf("unhealthy with both accepted, one citing the other: %s", h.Why)
}
w.k.TellFindings()
if len(w.told) != 0 {
t.Fatalf("told with every rule holding: %v", w.told)
}
// A correction of the citing record drops its link: the rule, run after the acceptance, says so.
b.Links = b.Links[:len(b.Links)-1]
b.Insight = "it cited " + itoaKey(n)
w.accepted(t, "author@g14", b)
w.accepted(t, "author@g14", valid(n))
h := w.k.Health(nil)
if !h.Healthy || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
t.Fatalf("a decision left cited by nothing: the health should stay healthy and name it, not %+v", h)
if h.Healthy || h.State != "unhealthy" || !strings.Contains(h.Why, "the rule cited") || !strings.Contains(h.Why, itoaKey(n)) {
t.Fatalf("an accepted decision nothing cites left the module %+v", h)
}
w.k.TellFindings()
w.k.TellFindings()
if len(w.told) != 1 || !strings.Contains(w.told[0], "rule cited") || !strings.Contains(w.told[0], itoaKey(n)) ||
!strings.HasPrefix(w.toldKeys[0], findingPrefix) {
t.Fatalf("the operator is not told once, naming the rule and the record: %v %v", w.told, w.toldKeys)
}
// A restart tells nothing again: what was told is kept in the store.
told := w.toldKeys[0]
restarted := newWorld(t, s, fixture())
restarted.clock = w.clock
restarted.k.TellFindings()
if len(restarted.told) != 0 {
t.Fatalf("told again after a restart: %v", restarted.told)
}
w.told = nil
// The acceptance stands: the condition says it, and undoes nothing.
if mustGet(t, s, Key{KindDecision, n}).Current.Status != StatusAccepted {
t.Fatal("the acceptance was undone")
}
// A record that cites it heals it.
k := w.reserved(t, "author@g14")
c := valid(k)
c.Links = append(c.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", c)
if h := w.k.Health(nil); strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
m := w.reserved(t, "author@g14")
p := valid(m)
p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: itoa4(n)})
w.accepted(t, "author@g14", p)
h = w.k.Health(nil)
if strings.Contains(h.Why, itoaKey(n)+" is accepted and nothing cites it") {
t.Fatalf("still said after a record cited it: %s", h.Why)
}
w.k.TellFindings()
if len(w.cleared) != 1 || w.cleared[0][0] != told {
t.Fatalf("the notice is not cleared once the rule holds again: %v", w.cleared)
}
w.k.TellFindings()
if len(w.cleared) != 1 || len(w.told) != 0 {
t.Fatalf("after the clear: cleared %v, told %v", w.cleared, w.told)
}
})
}
@@ -164,29 +83,3 @@ func TestEachWholeSetRuleFails(t *testing.T) {
t.Errorf("a consistent set fails: %+v", f)
}
}
// A finding's notice the operator's channel refused is not kept as told: it is sent again at the next step.
func TestAFindingsNoticeThatFailsIsSentAgain(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := newWorld(t, s, fixture())
f := Finding{Rule: "cited", Records: []string{"ADR 0300"}, Says: "ADR 0300 is accepted and nothing cites it"}
if err := s.SetFindings(context.Background(), []Finding{f}); err != nil {
t.Fatal(err)
}
tries := 0
w.k.Notify = func(key, summary, headline, explanation, resolved string) error {
tries++
if tries == 1 {
return errors.New("nothing answers operator-channel.notify")
}
w.told = append(w.told, summary)
return nil
}
w.k.TellFindings()
w.k.TellFindings()
w.k.TellFindings()
if tries != 2 || len(w.told) != 1 {
t.Fatalf("a refused notice: %d tries, told %v", tries, w.told)
}
})
}
-63
View File
@@ -1,63 +0,0 @@
# mesh-reads
A node's files and accounts, read through the mesh (novox/hq ADR 0310, extending ADR 0268 §5; hq to-be 41
WP8; issue 418). It holds the node seat `node-reads` on every node, the control-node included, so reading a
file, searching one, listing a directory, the owner and mode of each component of a path and looking up an
account or a group never go through the login shell's `execute`, and an approval is asked for an act, never
for a read.
## Who it reads as
The account `mesh-reads`, which this module makes on every node (`runs-as`, ADR 0259 §8): no login shell, a
home of its own under `/var/lib`, and no group beside its own. The tool runner serves it in a runtime of its
own, on its own bus credential, never the machine's runtime, which runs as the operator's account. So the
node's file modes bound what it sees: a file only the operator, root or a module's own account may read is
refused by its mode, and the answer says so. It refuses every read when it finds itself running as root.
## What it refuses by name
Before any read, whatever the file modes would allow (ADR 0310 §5):
- **The places the mesh keeps a secret on this node**, which the controller composes from the manifests
and gives it as `MESH_SECRET_PLACES`: the directory of every module's own secret (its bus credential
among them), every credential file and grant directory, the vault's directories, and where the
operator-sealed secrets are kept. Without that list the module does not start.
- **The known credential files of a machine's programs** (`refuse.go`): an account's `.ssh`, `.gnupg`,
`.aws`, `.kube`; the agent's `.credentials.json`, `.netrc`, `.pgpass`, `.git-credentials`; private key
names (`id_ed25519`, `*.key`, `ssh_host_*_key`); `/etc/shadow`; `/etc/ssl/private`, `/etc/wireguard`
and the network manager's connections.
- **Histories and profiles**: shell and program histories (`.zsh_history`, `.bash_history`, `.psql_history`,
`.viminfo` …), browser and mail profiles (`.mozilla`, `.config/google-chrome`, `.config/chromium` …), a
process's `environ`, `mem`, `fd`, `map_files`, `root` and `cwd` under `/proc`, and `/etc/rancher` and
`/etc/kubernetes/admin.conf`.
**What was opened is judged, not the name.** The type is checked before the open, so a device or a fifo is
never opened; the file is opened non-blocking (in a search without following a link), checked again on its
descriptor, and the path the kernel gives for that descriptor is judged by name like any path asked. A link
flipped between the check and the open therefore reaches nothing a name refuses, and a deleted file is refused.
**Limit:** a hard link to a secret file under another name is not seen by the name rule; the file's own mode,
and the kernel's protected hard links, are what bound it. `stat` and `readlink` say a secret place's owner
and mode, and mark it, because those are not its content; `list`, `read` and `search` refuse it.
## The content rule
Every answer passes the content rule's secret class, from `mesh-sdk`'s `outward` package (ADR 0310 §4): a
line of a file or a matching line shaped like a secret is withheld and its number said, as is a link target
or an account's words. Every line of a `BEGIN … END` block — a private key's or a certificate's — is withheld whole, because one
base64 line of a key can pass the rule alone. A path and an entry's name are not judged: they are what the caller named or what the
file system calls a thing, and the rule would take a temporary directory's or a container layer's name for a
token.
## Its verbs and their bounds
| verb | replaces | bounds |
|---|---|---|
| `stat` | `stat`, `ls -ld`, `namei -l`, `findmnt -T` | 200 paths, 64 components each; links followed only with `follow` |
| `readlink` | `readlink`, `readlink -f`, `realpath` | one path |
| `list` | `ls`, `ls -la` | 2000 entries, cut and said |
| `read` | `cat`, `head`, `tail`, `less` | 64 KiB, or up to 256 KiB when asked; cut at a line and said; text only |
| `search` | `grep`, `grep -r`, `grep -F` | a fixed string; 100 matches (500 asked), 4 deep (8 asked), 5000 files, 4 MiB a file, 20 seconds (checked inside a file too); the named path's file system only, the others named; the bound met is said |
| `account` | `getent passwd`, `getent group`, `id`, `groups` | 2000 accounts and groups; never a password field |
No answer is larger than 1 MiB. Nothing here writes a file.
-204
View File
@@ -1,204 +0,0 @@
// mesh-reads (novox/hq ADR 0310, to-be 41 WP8): the holder of node-reads. A node's files and accounts read
// through the mesh — the owner and mode of each component of a path, a link, a directory, a text file, a search
// by fixed string, an account or a group — so these reads never go through the login shell's execute.
//
// It runs as an account of its own, mesh-reads, served by the tool runner in a runtime of its own (ADR 0259
// §8): in no group that holds a secret, with no login shell and no home of the operator's, so the node's file
// modes bound what it sees. On top, it refuses by name the places a secret is kept (refuse.go) and withholds
// any piece of an answer shaped like a secret (mesh-sdk's outward). Read-only: nothing here writes a file.
//
// stdout is the MCP channel; what this module says, it says on stderr.
package main
import (
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Seat is the role this module holds.
const Seat = "node-reads"
// PlacesWord is where the controller names this node's secret places (novox/hq ADR 0310 §5).
const PlacesWord = "MESH_SECRET_PLACES"
func main() {
places, err := placesFrom(os.Getenv(PlacesWord))
if err != nil {
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
os.Exit(1)
}
if err := stdio.Serve("", tools(newReader(places))); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-reads] %v\n", err)
os.Exit(1)
}
}
// placesFrom reads the controller's list. Absent is refused: without it, a module's own-secret directory
// would be read wherever the account's modes allow, and a refusal the decision promises would be silent.
func placesFrom(raw string) ([]string, error) {
if strings.TrimSpace(raw) == "" {
return nil, fmt.Errorf("%s is not given, so this node's secret places are unknown and nothing would be "+
"refused by them (novox/hq ADR 0310 §5): a controller that knows node-reads composes it", PlacesWord)
}
var places []string
if err := json.Unmarshal([]byte(raw), &places); err != nil {
return nil, fmt.Errorf("%s is not a list of paths: %v", PlacesWord, err)
}
return places, nil
}
func str(description string) map[string]any {
return map[string]any{"type": "string", "description": description}
}
func arg(a map[string]any, k string) string {
switch v := a[k].(type) {
case string:
return strings.TrimSpace(v)
case float64:
return strconv.FormatFloat(v, 'f', -1, 64)
case bool:
return strconv.FormatBool(v)
}
return ""
}
func number(a map[string]any, k string) (int, error) {
s := arg(a, k)
if s == "" {
return 0, nil
}
n, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("%s is a whole number, and %q was given", k, s)
}
return n, nil
}
// pathsArg is the paths given, one per line — or, from a caller that sends a list, the list.
func pathsArg(a map[string]any) []string {
var raw []string
switch v := a["paths"].(type) {
case string:
raw = strings.Split(v, "\n")
case []any:
for _, p := range v {
if s, ok := p.(string); ok {
raw = append(raw, s)
}
}
}
var out []string
for _, p := range raw {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
// bounded is the last bound (novox/hq ADR 0310): whatever a verb made, no answer is larger than MaxAnswer.
func bounded(answer any, err error) (any, error) {
if err != nil {
return nil, err
}
body, err := json.Marshal(answer)
if err != nil {
return nil, err
}
if len(body) > MaxAnswer {
return nil, fmt.Errorf("the answer is %d bytes, and an answer of node-reads is at most %d: ask for less", len(body), MaxAnswer)
}
return answer, nil
}
// verb is one of the seat's verbs: listed as `<seat>.<verb>`, so the runtime serves it on the seat's subject.
// The descriptions are the controller's (internal/catalogue/reads.go), said again here for a caller of the
// module's own address.
func verb(name, description string, input map[string]any, run func(a map[string]any) (any, error)) stdio.Tool {
return stdio.Tool{Name: Seat + "." + name, Description: description, Input: input,
Run: func(a map[string]any) (any, error) { return bounded(run(a)) }}
}
func object(props map[string]any, required ...string) map[string]any {
if required == nil {
required = []string{}
}
return map[string]any{"type": "object", "properties": props, "required": required}
}
func tools(r Reader) []stdio.Tool {
return []stdio.Tool{
verb("stat", "Each component of one path or several, from / down: its type, owner and group (name and "+
"number), mode, whether it is a link and to what, and whether it is a mount point and of which file "+
"system. A link is not followed unless follow is \"true\". Read as the account mesh-reads, which holds no "+
"secret: a component it may not reach says so. At most 200 paths in one call.",
object(map[string]any{"paths": str("one absolute path, or several separated by line breaks"),
"follow": str("\"true\": also say where the path leads once every link is followed")}, "paths"),
func(a map[string]any) (any, error) { return r.Stat(pathsArg(a), arg(a, "follow") == "true") }),
verb("readlink", "What one link points to, as written, and where the path leads once every link is "+
"followed, and whether that exists.",
object(map[string]any{"path": str("an absolute path")}, "path"),
func(a map[string]any) (any, error) { return r.Readlink(arg(a, "path")) }),
verb("list", "Every entry of one directory with its type, owner, group, mode, size and, for a link, its "+
"target. At most 2000 entries; a longer directory is cut and says so. A place a secret is kept is "+
"refused by name.",
object(map[string]any{"path": str("an absolute path to a directory")}, "path"),
func(a map[string]any) (any, error) { return r.List(arg(a, "path")) }),
verb("read", "One text file, or a part of it by line: at most 256 KiB in one answer (64 KiB unless more "+
"is asked), cut at a line and said so. A negative from counts from the end, as tail does. A binary file "+
"is refused; a place a secret is kept, or a known credential file, is refused by name; a line shaped "+
"like a secret is withheld and its number said. /proc and /sys are read as far as mesh-reads may.",
object(map[string]any{"path": str("an absolute path to a file"),
"from": str("the first line to answer, counted from 1; negative: that many lines from the end (optional)"),
"lines": str("how many lines to answer (optional)"),
"bytes": str("the most bytes to answer, up to 262144 (optional, 65536 unless given)")}, "path"),
func(a map[string]any) (any, error) {
var ask ReadAsk
var err error
if ask.From, err = number(a, "from"); err != nil {
return nil, err
}
if ask.Lines, err = number(a, "lines"); err != nil {
return nil, err
}
if ask.Bytes, err = number(a, "bytes"); err != nil {
return nil, err
}
return r.Read(arg(a, "path"), ask)
}),
verb("search", "Find a fixed string (never a pattern) in one file or under one directory: every matching "+
"line with its file and number. Bounded: at most 500 matches (100 unless more is asked), 8 directories "+
"deep (4 unless more is asked), 5000 files, 4 MiB per file and 20 seconds, on the file system of the path "+
"named; links are not followed and binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like "+
"a secret is withheld. The answer says which bound it met.",
object(map[string]any{"text": str("the fixed string to find"),
"path": str("an absolute path to a file or a directory"),
"depth": str("how many directories deep, up to 8 (optional)"),
"matches": str("the most matches to answer, up to 500 (optional)"),
"case": str("\"false\": match whatever the case (optional)")}, "text", "path"),
func(a map[string]any) (any, error) {
text, _ := a["text"].(string) // as given: a space at either end is part of what is found
ask := SearchAsk{Text: text, Path: arg(a, "path"), CaseInsensitive: arg(a, "case") == "false"}
var err error
if ask.Depth, err = number(a, "depth"); err != nil {
return nil, err
}
if ask.Matches, err = number(a, "matches"); err != nil {
return nil, err
}
return r.Search(ask)
}),
verb("account", "One account or group by name or number, as this node's account files say: an account's "+
"number, group, home, shell and every group it is in; a group's number and members. With neither, every "+
"account and group. Never a password field.",
object(map[string]any{"name": str("an account's name or number (optional)"),
"group": str("a group's name or number (optional)")}),
func(a map[string]any) (any, error) { return r.Account(arg(a, "name"), arg(a, "group")) }),
}
}
@@ -1,59 +0,0 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
)
// ADR 0310 §3: the account the reads run as is the module's own, made by the mesh, with no login shell, no
// home of the operator's and no group beside its own, and the module runs as it.
func TestTheAccountHoldsNoSecretsGroupAndNoLogin(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
RunsAs string `json:"runs-as"`
SecretsOwner string `json:"secrets-owner"`
Claims []map[string]any `json:"claims"`
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
if m.RunsAs != "mesh-reads" || m.SecretsOwner != "mesh-reads" {
t.Errorf("runs as %q, secrets owned by %q", m.RunsAs, m.SecretsOwner)
}
if len(m.Claims) != 1 || m.Claims[0]["name"] != Seat {
t.Errorf("claims %v", m.Claims)
}
found := false
for _, r := range m.Resources {
if r["type"] != "user" {
continue
}
found = true
if r["name"] != m.RunsAs {
t.Errorf("a user other than its own: %v", r)
}
if r["shell"] != "/usr/bin/nologin" {
t.Errorf("the account has a login shell: %v", r["shell"])
}
if home, _ := r["home"].(string); !strings.HasPrefix(home, "/var/lib/") {
t.Errorf("the account's home is %q, not one of the mesh's", home)
}
if _, grouped := r["groups"]; grouped {
t.Errorf("the account is put in groups: %v", r["groups"])
}
}
if !found {
t.Error("the module makes no account of its own")
}
for _, r := range m.Resources {
if r["type"] != "user" && r["type"] != "directory" {
t.Errorf("a read-only holder declares a %v: %v", r["type"], r)
}
}
}
File diff suppressed because it is too large Load Diff
@@ -1,562 +0,0 @@
package main
import (
crand "crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/json"
"encoding/pem"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"testing"
"time"
)
// A token shaped as a secret, written into files the tests read.
const aToken = "ghp_abcdefghijklmnopqrstuvwxyz0123456789"
func testReader(t *testing.T, places ...string) (Reader, string) {
t.Helper()
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
group := filepath.Join(dir, "group")
mountinfo := filepath.Join(dir, "mountinfo")
write(t, passwd, "root:x:0:0::/root:/usr/bin/bash\nops:x:1000:1000:The Operator:/home/ops:/usr/bin/zsh\n"+
"mesh-reads:x:961:961::/var/lib/mesh-reads:/usr/bin/nologin\n")
write(t, group, "root:x:0:\nwheel:x:998:ops\nops:x:1000:\nmesh-reads:x:961:\ndocker:x:970:ops,builder\n")
data := filepath.Join(dir, "data")
write(t, mountinfo, "22 1 0:21 / / rw,relatime - btrfs /dev/nvme0n1p2 rw\n"+
"40 22 0:40 / "+strings.ReplaceAll(data, " ", `\040`)+" rw - nfs4 home-server:/tank/data rw\n")
r := Reader{Refuser: Refuser{Places: places}, Passwd: passwd, Group: group, MountInfo: mountinfo,
Euid: func() int { return 961 }, Now: time.Now}
return r, dir
}
func write(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// ADR 0310 §3: run as root, every read is refused, loudly — its bound is an account of its own.
func TestEveryReadIsRefusedAsRoot(t *testing.T) {
r, dir := testReader(t)
r.Euid = func() int { return 0 }
write(t, filepath.Join(dir, "f"), "x\n")
if _, err := r.Read(filepath.Join(dir, "f"), ReadAsk{}); err == nil || !strings.Contains(err.Error(), "root") {
t.Errorf("a read as root: %v", err)
}
if _, err := r.Stat([]string{dir}, false); err == nil {
t.Error("a stat as root was answered")
}
if _, err := r.Account("ops", ""); err == nil {
t.Error("an account lookup as root was answered")
}
}
// ADR 0310 §5: without the controller's list of secret places it does not start: a refusal it promises would be
// silent.
func TestItDoesNotStartWithoutTheNodesSecretPlaces(t *testing.T) {
if _, err := placesFrom(""); err == nil || !strings.Contains(err.Error(), PlacesWord) {
t.Errorf("no places: %v", err)
}
if p, err := placesFrom(`["/var/lib/mesh-vault"]`); err != nil || len(p) != 1 {
t.Errorf("a list: %v %v", p, err)
}
if p, err := placesFrom(`[]`); err != nil || len(p) != 0 {
t.Errorf("an empty list is a node with no secret place: %v %v", p, err)
}
}
// ADR 0310 §5: a path under the vault's directory, a module's own-secret directory and a known credential file
// are each refused by name before any read — though the account's modes would allow it — and a link to one is
// refused as the place is.
func TestSecretPlacesAreRefusedByNameBeforeAnyRead(t *testing.T) {
r0, dir := testReader(t)
vault := filepath.Join(dir, "var/lib/mesh-vault")
own := filepath.Join(dir, "var/lib/desk-channel")
r, _ := r0, dir
r.Places = []string{vault, own}
for _, p := range []string{
filepath.Join(vault, "ledger/entries.json"),
filepath.Join(own, "broker"),
filepath.Join(own, "settings.json"),
filepath.Join(dir, "home/ops/.claude/.credentials.json"),
filepath.Join(dir, "home/ops/.ssh/config"),
filepath.Join(dir, "home/ops/.pgpass"),
filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key"),
filepath.Join(dir, "etc/shadow"),
filepath.Join(dir, "srv/tls/server.key"),
} {
write(t, p, "harmless words\n")
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "refused by name") {
t.Errorf("%s was read: %v", p, err)
}
}
if _, err := r.List(vault); err == nil || !strings.Contains(err.Error(), "refused by name") {
t.Errorf("the vault's directory was listed: %v", err)
}
link := filepath.Join(dir, "tmp/innocent")
write(t, filepath.Join(dir, "tmp/x"), "")
if err := os.Symlink(filepath.Join(own, "broker"), link); err != nil {
t.Fatal(err)
}
if _, err := r.Read(link, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "leads to") {
t.Errorf("a link to a secret place was read: %v", err)
}
// A search under a directory holding a secret place skips it and names it.
write(t, filepath.Join(dir, "home/ops/notes.txt"), "harmless words\n")
a, err := r.Search(SearchAsk{Text: "harmless", Path: filepath.Join(dir, "home"), Depth: 8})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "notes.txt") || len(a.Skipped) != 3 {
t.Errorf("search: matches %v, skipped %v", a.Matches, a.Skipped)
}
// What a public key or a certificate is, is not refused.
for _, p := range []string{filepath.Join(dir, "etc/ssh/ssh_host_ed25519_key.pub"), filepath.Join(dir, "etc/ssl/certs/ca.pem")} {
write(t, p, "harmless words\n")
if _, err := r.Read(p, ReadAsk{}); err != nil {
t.Errorf("%s was refused: %v", p, err)
}
}
}
// ADR 0310 §3: what the account's file modes refuse is refused — the first bound, below the names.
func TestAFileTheAccountMayNotReadIsRefusedByItsMode(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("run as root, a file's mode refuses nothing")
}
r, dir := testReader(t)
p := filepath.Join(dir, "operators-only")
write(t, p, "the operator's words\n")
if err := os.Chmod(p, 0o000); err != nil {
t.Fatal(err)
}
if _, err := r.Read(p, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "may not reach it") {
t.Errorf("a file of mode 0000: %v", err)
}
closed := filepath.Join(dir, "closed")
write(t, filepath.Join(closed, "inside"), "x")
if err := os.Chmod(closed, 0o700&^0o100); err != nil {
t.Fatal(err)
}
defer os.Chmod(closed, 0o755)
a, err := r.Stat([]string{filepath.Join(closed, "inside")}, false)
if err != nil {
t.Fatal(err)
}
if got := a.Paths[0]; !strings.Contains(got.Stopped, "may not reach") {
t.Errorf("a path through a directory it may not search: %+v", got)
}
}
// ADR 0310 §2 and the bounds: a read over the size bound is cut at a line and says so; more than the most is
// refused; lines and a tail are answered as asked.
func TestAReadIsBounded(t *testing.T) {
r, dir := testReader(t)
p := filepath.Join(dir, "big")
var b strings.Builder
for i := 1; i <= 20000; i++ {
fmt.Fprintf(&b, "line %d of the file\n", i)
}
write(t, p, b.String())
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
if len(a.Text) > DefaultReadSize || !strings.Contains(a.Cut, fmt.Sprint(DefaultReadSize)) || a.From != 1 {
t.Errorf("a default read: %d bytes, cut %q", len(a.Text), a.Cut)
}
if _, err := r.Read(p, ReadAsk{Bytes: MaxReadSize + 1}); err == nil {
t.Error("more than the most a read answers was answered")
}
a, err = r.Read(p, ReadAsk{From: 10, Lines: 2})
if err != nil || a.Text != "line 10 of the file\nline 11 of the file\n" || a.From != 10 || a.Cut != "" {
t.Errorf("lines 10 and 11: %+v %v", a, err)
}
a, err = r.Read(p, ReadAsk{From: -2})
if err != nil || a.Text != "line 19999 of the file\nline 20000 of the file\n" || a.From != 19999 {
t.Errorf("the last two lines: %q from %d, %v", a.Text, a.From, err)
}
if _, err := r.Read(p, ReadAsk{From: 30000}); err == nil || !strings.Contains(err.Error(), "20000 lines") {
t.Errorf("a line past the end: %v", err)
}
}
// A binary file, a directory and a fifo are refused, and the fifo without waiting for a writer.
func TestOnlyATextFileIsRead(t *testing.T) {
r, dir := testReader(t)
bin := filepath.Join(dir, "bin")
write(t, bin, "ELF\x00\x01\x02")
if _, err := r.Read(bin, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "binary") {
t.Errorf("a binary file: %v", err)
}
if _, err := r.Read(dir, ReadAsk{}); err == nil || !strings.Contains(err.Error(), "directory") {
t.Errorf("a directory: %v", err)
}
fifo := filepath.Join(dir, "fifo")
if err := syscall.Mkfifo(fifo, 0o644); err != nil {
t.Fatal(err)
}
done := make(chan error, 1)
go func() { _, err := r.Read(fifo, ReadAsk{}); done <- err }()
select {
case err := <-done:
if err == nil || !strings.Contains(err.Error(), "fifo") {
t.Errorf("a fifo: %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("a read of a fifo waits for a writer")
}
}
// ADR 0310 §4: a line shaped like a secret is withheld, its number said, and the rest is answered.
func TestALineShapedLikeASecretIsWithheld(t *testing.T) {
r, dir := testReader(t)
p := filepath.Join(dir, "app.conf")
write(t, p, "listen = /run/app.sock\ntoken = "+aToken+"\nuser = app\n")
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
if strings.Contains(a.Text, aToken) || !strings.Contains(a.Text, "user = app") ||
!strings.Contains(a.Text, "listen = /run/app.sock") || fmt.Sprint(a.Withheld) != "[2]" {
t.Errorf("read: %q withheld %v", a.Text, a.Withheld)
}
m, err := r.Search(SearchAsk{Text: "token", Path: dir})
if err != nil {
t.Fatal(err)
}
body, _ := json.Marshal(m)
if strings.Contains(string(body), aToken) || len(m.Matches) != 1 || len(m.Withheld) != 1 {
t.Errorf("search: %s", body)
}
}
// The bounds of a search: matches, depth, the most asked, and time — each met is said.
func TestASearchIsBounded(t *testing.T) {
r, dir := testReader(t)
for i := 0; i < 30; i++ {
write(t, filepath.Join(dir, fmt.Sprintf("f%02d", i)), "needle\nneedle\n")
}
a, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: 7})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 7 || !strings.HasPrefix(a.Bound, "matches") {
t.Errorf("seven matches asked: %d, bound %q", len(a.Matches), a.Bound)
}
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Matches: MaxMatches + 1}); err == nil {
t.Error("more matches than the most were asked and answered")
}
if _, err := r.Search(SearchAsk{Text: "needle", Path: dir, Depth: MaxDepth + 1}); err == nil {
t.Error("deeper than the most was asked and answered")
}
deep := t.TempDir()
write(t, filepath.Join(deep, "a/b/c/shallow"), "needle\n")
write(t, filepath.Join(deep, "a/b/c/d/e/deep"), "needle\n")
a, err = r.Search(SearchAsk{Text: "needle", Path: deep, Depth: 3})
if err != nil {
t.Fatal(err)
}
if len(a.Matches) != 1 || !strings.HasSuffix(a.Matches[0].File, "shallow") {
t.Errorf("three deep: %+v", a.Matches)
}
// Case: as asked.
write(t, filepath.Join(deep, "case"), "NEEDLE\n")
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case")}); len(a.Matches) != 0 {
t.Error("a match whatever the case, unasked")
}
if a, _ := r.Search(SearchAsk{Text: "needle", Path: filepath.Join(deep, "case"), CaseInsensitive: true}); len(a.Matches) != 1 {
t.Error("no match whatever the case, asked")
}
// Time: a clock past the bound stops the walk and says so.
start := time.Now()
calls := 0
r.Now = func() time.Time { calls++; return start.Add(time.Duration(calls) * SearchTime / 3) }
a, err = r.Search(SearchAsk{Text: "needle", Path: dir})
if err != nil || !strings.HasPrefix(a.Bound, "time") {
t.Errorf("the time bound: %q %v", a.Bound, err)
}
// A fixed string: a pattern's characters are themselves.
write(t, filepath.Join(deep, "pattern"), "a.c\nabc\n")
if a, _ := r.Search(SearchAsk{Text: "a.c", Path: filepath.Join(deep, "pattern")}); len(a.Matches) != 1 || a.Matches[0].Line != 1 {
t.Errorf("a.c as a fixed string: %+v", a.Matches)
}
}
// The survey issue 418 was asked for: each component of a path with its owner, group, mode, whether it is a
// link and to what, and whether it is a mount point — the link not followed unless asked.
func TestStatSaysEachComponent(t *testing.T) {
r, dir := testReader(t)
data := filepath.Join(dir, "data")
write(t, filepath.Join(data, "real/file"), "x")
if err := os.Chmod(filepath.Join(data, "real"), os.ModeSetgid|0o750); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(data, "link")); err != nil {
t.Fatal(err)
}
a, err := r.Stat([]string{filepath.Join(data, "link"), filepath.Join(data, "real/file")}, false)
if err != nil {
t.Fatal(err)
}
link := a.Paths[0]
last := link.Components[len(link.Components)-1]
if link.Components[0].Path != "/" || last.Type != "link" || last.Link != "real" || link.LeadsTo != "" {
t.Errorf("the link, not followed: %+v", link)
}
var dataC, realC Component
for _, c := range a.Paths[1].Components {
switch c.Path {
case data:
dataC = c
case filepath.Join(data, "real"):
realC = c
}
}
if dataC.Mount == nil || dataC.Mount.FS != "nfs4" {
t.Errorf("the mount point is not said: %+v", dataC)
}
if realC.Mode != "2750" || realC.Type != "directory" || realC.UID == nil || realC.Perms == "" {
t.Errorf("a component's mode: %+v", realC)
}
if uid := *realC.UID; uid == 1000 && realC.Owner != "ops" {
t.Errorf("owner %d named %q", uid, realC.Owner)
}
a, err = r.Stat([]string{filepath.Join(data, "link")}, true)
if err != nil {
t.Fatal(err)
}
if a.Paths[0].LeadsTo != filepath.Join(data, "real") || a.Paths[0].Leads == nil || a.Paths[0].Leads.Type != "directory" {
t.Errorf("followed: %+v", a.Paths[0])
}
a, err = r.Stat([]string{filepath.Join(dir, "missing/deeper")}, false)
if err != nil || !strings.Contains(a.Paths[0].Stopped, "does not exist") {
t.Errorf("a missing path: %+v %v", a, err)
}
if _, err := r.Stat([]string{"relative/path"}, false); err == nil {
t.Error("a relative path was answered")
}
many := make([]string, MaxPaths+1)
for i := range many {
many[i] = "/"
}
if _, err := r.Stat(many, false); err == nil {
t.Error("more paths than the most were answered")
}
}
func TestReadlinkSaysTheTargetAndWhereItLeads(t *testing.T) {
r, dir := testReader(t)
write(t, filepath.Join(dir, "target"), "")
if err := os.Symlink("target", filepath.Join(dir, "l1")); err != nil {
t.Fatal(err)
}
if err := os.Symlink("l1", filepath.Join(dir, "l2")); err != nil {
t.Fatal(err)
}
if err := os.Symlink("nowhere", filepath.Join(dir, "dangling")); err != nil {
t.Fatal(err)
}
a, err := r.Readlink(filepath.Join(dir, "l2"))
if err != nil || !a.IsLink || a.Link != "l1" || a.LeadsTo != filepath.Join(dir, "target") || !a.Exists {
t.Errorf("l2: %+v %v", a, err)
}
a, err = r.Readlink(filepath.Join(dir, "dangling"))
if err != nil || !a.IsLink || a.Exists || !strings.Contains(a.Stopped, "does not exist") {
t.Errorf("dangling: %+v %v", a, err)
}
}
func TestListSaysEveryEntry(t *testing.T) {
r, dir := testReader(t)
write(t, filepath.Join(dir, "d/a"), "abc")
if err := os.Symlink("a", filepath.Join(dir, "d/b")); err != nil {
t.Fatal(err)
}
a, err := r.List(filepath.Join(dir, "d"))
if err != nil {
t.Fatal(err)
}
if a.Count != 2 || a.Entries[0].Name != "a" || *a.Entries[0].Size != 3 || a.Entries[1].Type != "link" || a.Entries[1].Link != "a" {
t.Errorf("list: %+v", a)
}
for i := 0; i < MaxEntries+5; i++ {
write(t, filepath.Join(dir, "many", fmt.Sprintf("e%05d", i)), "")
}
a, err = r.List(filepath.Join(dir, "many"))
if err != nil || len(a.Entries) != MaxEntries || a.Count != MaxEntries+5 || a.Cut == "" {
t.Errorf("a long directory: %d of %d, cut %q, %v", len(a.Entries), a.Count, a.Cut, err)
}
}
// getent passwd, getent group and id: an account with every group it is in, a group with its members, never a
// password field.
func TestAccountAndGroupLookup(t *testing.T) {
r, _ := testReader(t)
a, err := r.Account("ops", "")
if err != nil {
t.Fatal(err)
}
u := a.Accounts[0]
sort.Strings(u.Groups)
if u.UID != 1000 || u.Group != "ops" || u.Shell != "/usr/bin/zsh" || fmt.Sprint(u.Groups) != "[docker ops wheel]" {
t.Errorf("ops: %+v", u)
}
if a, err := r.Account("961", ""); err != nil || a.Accounts[0].Name != "mesh-reads" {
t.Errorf("by number: %+v %v", a, err)
}
g, err := r.Account("", "docker")
if err != nil || g.Groups[0].GID != 970 || fmt.Sprint(g.Groups[0].Members) != "[ops builder]" {
t.Errorf("docker: %+v %v", g, err)
}
if _, err := r.Account("nobody-here", ""); err == nil {
t.Error("a missing account was answered")
}
all, err := r.Account("", "")
if err != nil || len(all.Accounts) != 3 || len(all.Groups) != 5 {
t.Errorf("all: %+v %v", all, err)
}
body, _ := json.Marshal(all)
if strings.Contains(string(body), `"x"`) || strings.Contains(string(body), "password") {
t.Errorf("a password field is answered: %s", body)
}
}
// No answer is larger than the bound, whatever a verb made.
func TestNoAnswerIsLargerThanTheBound(t *testing.T) {
if _, err := bounded(strings.Repeat("x", MaxAnswer+1), nil); err == nil {
t.Error("an answer over the bound was given")
}
if _, err := bounded("small", nil); err != nil {
t.Error(err)
}
}
// The seat's six verbs, as the controller defines them (mesh-controller internal/catalogue/reads.go).
func TestItServesTheSeatsVerbs(t *testing.T) {
r, _ := testReader(t)
var names []string
for _, tool := range tools(r) {
names = append(names, tool.Name)
}
if fmt.Sprint(names) != "[node-reads.stat node-reads.readlink node-reads.list node-reads.read node-reads.search node-reads.account]" {
t.Errorf("verbs: %v", names)
}
}
// The review of #244 (required 1): a link flipped between a harmless file and a secret place, while it is read
// and searched, never answers the secret place's content — what was opened is judged, not the name.
func TestALinkFlippedDuringAReadNeverReadsASecretPlace(t *testing.T) {
r, dir := testReader(t)
secret := filepath.Join(dir, "var/lib/mesh-reads")
r.Places = []string{secret}
write(t, filepath.Join(secret, "broker"), "THE-BROKER-CREDENTIAL\n")
write(t, filepath.Join(dir, "plain"), "THE-PLAIN-WORDS\n")
link := filepath.Join(dir, "shared/link")
if err := os.MkdirAll(filepath.Dir(link), 0o755); err != nil {
t.Fatal(err)
}
stop := make(chan struct{})
done := make(chan struct{})
go func() {
defer close(done)
targets := []string{filepath.Join(dir, "plain"), filepath.Join(secret, "broker")}
for i := 0; ; i++ {
select {
case <-stop:
return
default:
}
tmp := link + ".new"
os.Remove(tmp)
if os.Symlink(targets[i%2], tmp) == nil {
os.Rename(tmp, link)
}
}
}()
leaked := 0
for i := 0; i < 5000; i++ {
if a, err := r.Read(link, ReadAsk{}); err == nil && strings.Contains(a.Text, "BROKER") {
leaked++
}
if a, err := r.Search(SearchAsk{Text: "BROKER", Path: filepath.Dir(link)}); err == nil && len(a.Matches) > 0 {
leaked++
}
}
close(stop)
<-done
if leaked > 0 {
t.Errorf("a secret place was answered %d times through a flipped link", leaked)
}
}
// The review of #244 (required 2): every line of a private key's block is withheld, in a read and in a search,
// though the file's name is not refused.
func TestEveryLineOfAKeyBlockIsWithheld(t *testing.T) {
r, dir := testReader(t)
key, err := rsa.GenerateKey(crand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
der, err := x509.MarshalPKCS8PrivateKey(key)
if err != nil {
t.Fatal(err)
}
body := string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}))
p := filepath.Join(dir, "server.pem")
write(t, p, "# the server's key\n"+body+"after = the block\n")
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
lines := strings.Count(body, "\n")
for _, l := range strings.Split(strings.TrimSpace(body), "\n")[1:] {
if strings.Contains(a.Text, l) && !strings.HasPrefix(l, "-----END") {
t.Fatalf("a line of the key was answered: %q", l)
}
}
if len(a.Withheld) != lines || !strings.Contains(a.Text, "after = the block") || !strings.Contains(a.Text, "the server's key") {
t.Errorf("withheld %d of the block's %d lines; text:\n%s", len(a.Withheld), lines, a.Text)
}
// A search for a character every base64 line holds answers none of the key.
m, err := r.Search(SearchAsk{Text: "A", Path: p, Matches: MaxMatches})
if err != nil {
t.Fatal(err)
}
for _, x := range m.Matches {
if !strings.HasPrefix(x.Text, "‹withheld") {
t.Errorf("a search answered a line of the key: %q", x.Text)
}
}
}
// The review of #244 (required 3 and should): histories, browser profiles, a process's private entries and the
// cluster credentials are refused by name.
func TestHistoriesProfilesAndProcessesAreRefusedByName(t *testing.T) {
var r Refuser
for _, p := range []string{"/home/ops/.zsh_history", "/home/ops/.bash_history", "/root/.psql_history",
"/home/ops/.viminfo", "/home/ops/.mozilla/firefox/x.default/logins.json",
"/home/ops/.config/google-chrome/Default/Cookies", "/home/ops/.config/chromium/Default/Login Data",
"/proc/self/environ", "/proc/1234/mem", "/proc/self/fd/3", "/proc/thread-self/root/etc/shadow",
"/etc/rancher/k3s/k3s.yaml", "/etc/kubernetes/admin.conf"} {
if r.Refused(p) == "" {
t.Errorf("%s is not refused", p)
}
}
for _, p := range []string{"/proc/mdstat", "/proc/self/status", "/home/ops/.config/i3/config", "/etc/hosts"} {
if why := r.Refused(p); why != "" {
t.Errorf("%s is refused: %s", p, why)
}
}
}
-129
View File
@@ -1,129 +0,0 @@
package main
import (
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/outward"
)
// What is refused by name, before any read (novox/hq ADR 0310 §5), and what an answer withholds (§4).
//
// **Two lists of places.** The places the mesh knows a secret is kept on this node — every module's own-secret
// directory, every credential file and grant directory, the vault's directories — come from the controller,
// which composes them from the manifests (MESH_SECRET_PLACES). The known credential files of the programs
// on a machine are this module's list, below: the mesh does not write them, so nothing else names them. Both
// hold whatever the account's file modes would allow: the file modes are the first bound, the names the second.
// knownDirs are directories that hold nothing but credentials, wherever they are: an account's keys.
var knownDirs = map[string]bool{
".ssh": true, ".gnupg": true, ".password-store": true, ".aws": true, ".kube": true, ".azure": true,
".mozilla": true, ".thunderbird": true, // browser and mail profiles: saved logins and cookies
}
// knownNames are file names that are a credential whatever directory they are in.
var knownNames = map[string]bool{
".credentials.json": true, // the agent's credentials (Claude Code)
".netrc": true, ".pgpass": true, ".git-credentials": true, ".npmrc": true, ".pypirc": true,
".my.cnf": true, ".vault-token": true, "credentials": true, "credentials.json": true,
"id_rsa": true, "id_dsa": true, "id_ecdsa": true, "id_ed25519": true, "id_ecdsa_sk": true, "id_ed25519_sk": true,
"shadow": true, "gshadow": true, "shadow-": true, "gshadow-": true, "opasswd": true,
"broker": true, // the mesh's name for a module's bus credential
// Shell and program histories: where a token is typed.
".bash_history": true, ".zsh_history": true, ".histfile": true, ".python_history": true, ".psql_history": true,
".mysql_history": true, ".lesshst": true, ".viminfo": true, ".node_repl_history": true, ".sqlite_history": true,
".rediscli_history": true,
}
// knownSuffixes are file name endings a credential has.
var knownSuffixes = []string{".key", ".p12", ".pfx", ".jks", ".keystore", ".creds", ".nk", ".kdbx", "_key"}
// knownPaths are the places on a machine that hold credentials and are not a module's.
var knownPaths = []string{
"/etc/ssl/private", "/etc/wireguard", "/etc/NetworkManager/system-connections", "/etc/ipsec.d/private",
"/var/lib/iwd", "/etc/docker/key.json", "/root/.docker/config.json", "/var/lib/sss/secrets",
"/etc/rancher", "/etc/kubernetes/admin.conf",
}
// knownProfiles are directories under an account's .config that hold a browser's saved logins and cookies.
var knownProfiles = map[string]bool{"google-chrome": true, "chromium": true, "BraveSoftware": true, "vivaldi": true,
"microsoft-edge": true}
// procPrivate are the entries of a process under /proc that reach its memory, its environment or its files.
var procPrivate = map[string]bool{"environ": true, "mem": true, "fd": true, "map_files": true, "root": true,
"cwd": true}
// Refuser says whether a path is a place a secret is kept.
type Refuser struct {
// Places are the mesh's, from the controller: refused with everything beneath them.
Places []string
}
// Refused says why a path may not be read, or "" when it may. The path is clean and absolute.
func (r Refuser) Refused(path string) string {
path = filepath.Clean(path)
for _, p := range r.Places {
if within(path, p) {
return "it is a place the mesh keeps a secret (" + p + ")"
}
}
for _, p := range knownPaths {
if within(path, p) {
return "it is a place a machine keeps credentials (" + p + ")"
}
}
parts := strings.Split(strings.TrimPrefix(path, "/"), "/")
if len(parts) >= 3 && parts[0] == "proc" && procPrivate[parts[2]] {
return "it reaches a process's " + parts[2]
}
for i, part := range parts {
if i > 0 && parts[i-1] == ".config" && knownProfiles[part] {
return "it is " + part + "'s profile, which keeps saved logins"
}
if knownDirs[part] {
return "it is in " + part + ", which holds an account's credentials"
}
if i == len(parts)-1 {
if knownNames[part] {
return part + " is a known credential file"
}
for _, s := range knownSuffixes {
if strings.HasSuffix(part, s) && len(part) > len(s) {
return part + " is named as a key or a credential (" + s + ")"
}
}
if strings.HasSuffix(part, ".pem") && (strings.Contains(part, "key") || strings.Contains(part, "private")) {
return part + " is named as a private key"
}
if part == "config.json" && i > 0 && parts[i-1] == ".docker" {
return "a container registry login is kept there"
}
}
}
return ""
}
func within(path, place string) bool {
return path == place || strings.HasPrefix(path, strings.TrimSuffix(place, "/")+"/")
}
// Withheld is one piece of an answer the content rule kept back: where it was, and the class it carried.
type Withheld struct {
At string `json:"at"`
Class string `json:"why"`
}
// clean is the content rule's secret class (mesh-sdk outward, novox/hq ADR 0310 §4) over one piece of an
// answer: the piece, or a mark in its place when it is shaped like a secret.
//
// **What a file holds, not what the file system names.** It judges a file's lines, a matching line, a link's
// target, a mount's source and an account's words. A path and an entry's name are not judged: they are what
// the caller named or the file system calls a thing, and the rule's random-run test takes a temporary
// directory's or a container layer's name for a token, which would withhold the very survey asked for. A
// place where a secret sits by name is refused before any read instead (Refuser).
func clean(s string) (string, string) {
if refusal, ok := outward.CheckSecret(s); !ok {
return "‹withheld: " + refusal.What + "›", refusal.What
}
return s, ""
}
-5
View File
@@ -1,5 +0,0 @@
module mesh-reads
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2
-2
View File
@@ -1,2 +0,0 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2 h1:1E5iapAO1o4vkwV/8CUBQvNfL3OjXRtiQGBaKHXJVW8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
-54
View File
@@ -1,54 +0,0 @@
{
"module": "mesh-reads",
"version": "1",
"runs-as": "mesh-reads",
"claims": [
{
"name": "node-reads",
"scope": "node",
"serves": [
"stat",
"readlink",
"list",
"read",
"search",
"account"
]
}
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-reads",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-reads",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-reads"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-reads",
"place": "."
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-reads",
"binary": "mesh-reads",
"loads": [
"mesh-reads"
]
}
]
}
}
+7 -42
View File
@@ -16,20 +16,15 @@ channel's client and no channel's secret.
its sender) and `desk-channel` (kind `desktop`, the machines' notifiers, proves nothing). Which kinds exist
and what each promises is the controller's word, in this module's membership; whether each can send now
is the channel's own, said on `intake.standing`.
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages, and the restart passes made),
`asks` (every ask, closed ones for 30 days), `identities` (the operator's linked accounts, one per kind, and a
link asked for), `walks` (the delivery walks open now, each with what it holds).
- **Keeps its state on the bus** (ADR 0201): `open` and `sent` (the messages), `asks` (every ask, closed ones
for 30 days), `identities` (the operator's linked accounts, one per kind, and a link asked for).
- **Lost answers are said again from here, never from a channel's memory** (issue 384,
`cmd/messenger/restart.go`): a channel whose messages' answers were lost — the desk's notifier started
again — says when in its standing (`answers-lost-at`). The router says again on that kind, directly and not
through the outbox, each message that is still open, asked there, never answered and last said before that
time: one per problem, under its handle, with the words and answers of now, so it replaces the message in
place. **A restart says nothing it has said** (novox/hq ADR 0308 §6, issue 531): a silenced message, one
withheld for its content, one answered and one a walk holds are left as they are, and what ended never comes
back. The standings of one restart (a notifier on each machine, seconds apart) are one pass: a loss is acted
on once it stood 10 s, and the pass made is kept in the state (`sent`, key `passes`), so a router that starts
again does not make it again. A kind that cannot send waits; it never goes to another channel. Telegram's
answers are never lost and it says no time.
through the outbox, each message that is still open, asked there and last said before that time: one per
problem, under its handle, with the words and answers of now, so it replaces the message in place. What is
silenced is edited to say so, with no answers and no urgency; what ended never comes back. A kind that cannot
send waits; it never goes to another channel. Telegram's answers are never lost and it says no time.
## Asks, answers and warrants (novox/hq ADR 0259)
@@ -148,10 +143,7 @@ new message is shown, so it rings once and no second message stands beside the f
notification by its id; Telegram deletes the earlier message). A message said again goes alone, never as a
line of a digest beside the message it follows; one first said inside a digest gets a message of its own the
next time, and later words replace or edit that one. An ask's buttons live on that one message, so an ask that
ends takes them away everywhere; its expiry is said with its day when it is not today. An ask replaced by a
newer one about the same thing ends its own message too, which then says "Replaced by a newer question"; the
newer ask shows a message of its own (issue 548). A message left open by an ask that already ended is ended on
the next minute's sweep.
ends takes them away everywhere; its expiry is said with its day when it is not today.
- **Routing:** urgent to every kind that can send; a warning to the desk where it can send, otherwise to
the channel that reaches the operator away; an ask to every kind that carries one of its answers.
@@ -173,37 +165,10 @@ the next minute's sweep.
history. An event that cannot be read
is refused by name, counted, and told to the operator once an hour.
## While a delivery walks (novox/hq ADR 0308)
The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`started`, `ended`), `touches`
(`[{machine, modules}]`: what it moves and what a moved tool runner or node-engine carries), `outcome`
(`delivered`, `failed`, `stopped`), `bound_seconds` and `budget_class` (`cmd/messenger/walks.go`).
- **Held:** while a walk is open and within its bound, a message whose subject is a machine it touches, or a
module it touches (on a machine it touches, when the subject names one), is held: kept on the walk with when
and what, and the history says `held`. A message several open walks touch is told when the last ends or passes
its bound. At a walk's end, a later open walk that touches a message takes it over only as a first hold would:
still open, not silenced, not said since, and not one that is never held.
- **Never held:** an ask waiting for the operator (and a condition an open ask is about), a secret given or
replaced, a subject no open walk touches, and a walk's own failure (a plan, walk, delivery or build).
- **The end is one message:** the outcome, the modules and machines reached, how long; each held message still
open, one line each; the ones that cleared counted. Delivered with nothing held and nothing open: one line —
and nothing on a kind listed in the setting `only-what-needs-me`. What it carries is marked told on a kind
only once the message was sent there. Until then each is kept owed in the state (`owed`): a message lost with a
restart before it was sent leaves them untold and owed, and the restarted router says each on its own. A
record that clears owes nothing more.
- **The bound:** at its start plus `bound_seconds` (900 when not given), what is held and still open is said,
with "The walk … is still running, since HH:MM", and the cleared counted; the walk holds nothing more. A walk
whose end never comes is forgotten a day after it started. A start heard after its end opens nothing: the
controller's moments are compared on its own clock, never with one capped at the router's now.
- **Kept in the state** (`walks`): a router that starts again mid-walk holds and tells as before.
## What the operator gives
1. **The time zone**, as a setting: `{"time-zone": "<an IANA zone name>"}`. Every time a message says is
said in it; not given, the zone of the machine this runs on.
**The kinds that say only what needs the operator**, as a setting: `{"only-what-needs-me": ["<kind>"]}`. A
walk that delivered with nothing held and nothing open is not told on them (ADR 0308 §4); empty by default.
2. **The channels**: assign `telegram` and `desk-channel` (each says in its README what it needs). Linking
an account as the operator is done on the channel, never by a setting.
+3 -32
View File
@@ -691,58 +691,30 @@ func (h *Holder) showClosed(rec AskRecord) {
}
told := h.toldLine(rec) // read before the lock: it reads the ask's record
h.mu.Lock()
ownMessage := r.Origin == "ask"
// The first line is how it ended, which an ask's own message shows as its title (sayOn); the rest follows.
switch {
case rec.Warrant != nil && rec.Warrant.Outcome == asks.OutcomeChosen:
r.Answered = chosenLine(rec.Warrant, rec.Closed.In(h.zone())) + "\n" + told
case rec.State == string(asks.OutcomeExpired):
r.Answered = "No longer answerable: nobody answered in time.\n" + strings.TrimSuffix(capital(rec.Ask.OnExpiry), ".") + "."
case rec.State == string(asks.OutcomeReplaced) && !ownMessage:
// On a condition's message, which the newer ask takes over and shows in place.
r.Answered = ""
case rec.State == string(asks.OutcomeReplaced):
// Its own message: the newer ask looks for its message under its own key and opens one of its own, so
// this one ends here, on every channel that showed it (issue 548).
r.Answered = "Replaced by a newer question.\n" + rec.Asker + " asked again about the same thing; that question is the one to answer."
r.Answered = ""
default:
r.Answered = "No longer needed."
}
ownMessage := r.Origin == "ask"
h.mu.Unlock()
if r.told() {
h.edit(r, "answered", false)
}
if ownMessage {
if ownMessage && rec.State != string(asks.OutcomeReplaced) {
h.mu.Lock()
r.Cleared = h.Now()
r.Owed = "" // an ended question is owed to nobody (the review of PR 238)
h.mu.Unlock()
}
h.persist(r)
}
// endLeftovers ends the own message of an ask that already ended and whose message stayed open: one left by a
// router before issue 548 was fixed, whose replaced asks kept their messages for good. Its copies say how the
// ask ended, as its end would have.
func (h *Holder) endLeftovers() {
h.mu.Lock()
var ended []AskRecord
for _, r := range h.open {
if r.Origin != "ask" || r.Ask == "" || !r.Cleared.IsZero() {
continue
}
if e := h.asks[r.Ask]; e != nil && e.rec.State != StateOpen && e.rec.Record == r.Key {
ended = append(ended, e.rec)
}
}
h.mu.Unlock()
sort.Slice(ended, func(i, j int) bool { return ended[i].Record < ended[j].Record })
for _, rec := range ended {
h.Logf("[messenger] the message of %s stayed open after its ask ended (%s): ended now", rec.Record, rec.State)
h.showClosed(rec)
}
}
// chosenLine is how a chosen ask ended, in the operator's words: "You chose Approve at 13:30 on Telegram."
func chosenLine(w *asks.Warrant, at time.Time) string {
who, where := "Somebody", ""
@@ -803,7 +775,6 @@ func (h *Holder) sweepAsks(now time.Time) {
}
}
h.mu.Unlock()
h.endLeftovers()
sort.Strings(expired)
for _, k := range expired {
e := h.askOf(k)
+12 -55
View File
@@ -154,11 +154,6 @@ type Record struct {
Subject string `json:"subject"`
Severity string `json:"severity"`
Summary string `json:"summary"`
// Scope, SubjectID and Machine are the condition's subject as the controller gave it, which a walk's touches
// are matched against (ADR 0308 §2); empty for a record kept before, whose key names them.
Scope string `json:"scope,omitempty"`
SubjectID string `json:"subject_id,omitempty"`
Machine string `json:"machine,omitempty"`
// Headline, Explanation and Resolved are the plain words the operator reads (novox/hq ADR 0253):
// the controller's, or a module's through notify; empty, the message is said from the summary.
Headline string `json:"headline,omitempty"`
@@ -200,17 +195,14 @@ type Record struct {
// ClearHeld is what was waiting to be said of it when it cleared (a reminder, an escalation): said after
// all if it comes back before its clearing was said. Blips counts those comebacks, never said, so a
// condition that flaps is known as one and its message's "again, Nth time" counts only what was said.
ClearHeld string `json:"clear_held,omitempty"`
Blips int `json:"blips,omitempty"`
Reminded bool `json:"reminded,omitempty"`
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
// Owed is what a walk held of it and has not told yet (ADR 0308 §4, §5): kept in the state until a message
// carrying it was sent, so a router that restarts before says it on its own (issue 538), never history.
Owed string `json:"owed,omitempty"`
Folded bool `json:"folded,omitempty"` // held by the cap
Cleared time.Time `json:"cleared,omitempty"`
Count int `json:"count"`
Refused string `json:"refused,omitempty"`
ClearHeld string `json:"clear_held,omitempty"`
Blips int `json:"blips,omitempty"`
Reminded bool `json:"reminded,omitempty"`
Pending string `json:"pending,omitempty"` // what could not be said yet: raised, reminder, …
Folded bool `json:"folded,omitempty"` // held by the cap
Cleared time.Time `json:"cleared,omitempty"`
Count int `json:"count"`
Refused string `json:"refused,omitempty"`
// History: known from the state or from an old event, and never said — not news.
History bool `json:"history,omitempty"`
// Seen is when this holder last took word of it; the open set read after it is authoritative.
@@ -219,8 +211,7 @@ type Record struct {
// recordOf is a condition as this holder keeps it.
func recordOf(c Condition) Record {
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Scope: c.Scope, SubjectID: c.ID, Machine: c.Machine,
Severity: c.Severity, Summary: c.Summary,
return Record{Key: c.Key, Kind: c.Kind, Subject: c.SubjectWords(), Severity: c.Severity, Summary: c.Summary,
Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved, Needs: c.Needs, Actions: c.Actions,
Origin: "condition",
More: "conditions show " + c.Key, Raised: c.Raised, SilencedTill: c.SilencedTill, SilencedWhy: c.SilencedWhy}
@@ -241,13 +232,6 @@ func (r *Record) takeWords(from Record) {
}
}
// takeSubject keeps the subject as the controller says it now, when it says one.
func (r *Record) takeSubject(from Record) {
if from.Scope != "" || from.SubjectID != "" || from.Machine != "" {
r.Scope, r.SubjectID, r.Machine = from.Scope, from.SubjectID, from.Machine
}
}
func (r *Record) silenced(now time.Time) bool {
return !r.SilencedTill.IsZero() && now.Before(r.SilencedTill)
}
@@ -308,9 +292,6 @@ type Holder struct {
Zone func() *time.Location
// Tracking asks the issue tracker which issues track which conditions; nil asks it through Ask (tracking.go).
Tracking func() (map[string][]TrackedIssue, error)
// NeedsOnly says whether a kind is set to say only what needs the operator (the setting
// `only-what-needs-me`, ADR 0308 §4); nil, none is.
NeedsOnly func(kind string) bool
work sync.Mutex // one event, call or tick at a time
mu sync.Mutex // what the status reads
@@ -348,14 +329,8 @@ type Holder struct {
lastUnlink map[string]Unlinked
// rootWord is the controller's last word on whether an agent can become root, by the machines asked.
rootWord map[string]rootWord
// passes is, by kind, the restart pass this router last made in full, kept in its state (restart.go); lost
// the loss of answers it has yet to act on, by kind.
passes map[string]Pass
lost map[string]time.Time
// walks are the walks open now, by id, each with what it holds (walks.go, ADR 0308).
walks map[string]*Walk
// ended are the walks whose end was heard, by id: what a start heard after its end is matched against.
ended map[string]*Walk
// shows is, by kind, the AnswersLostAt this router last acted on in full (restart.go).
shows map[string]time.Time
// track is the issue tracker's last word on which issues track which conditions (tracking.go).
track trackWord
}
@@ -373,10 +348,7 @@ func (h *Holder) init() {
h.chanOK = map[string]time.Time{}
h.saidOnce = map[string]time.Time{}
h.heard = map[string]int{}
h.passes = map[string]Pass{}
h.lost = map[string]time.Time{}
h.walks = map[string]*Walk{}
h.ended = map[string]*Walk{}
h.shows = map[string]time.Time{}
}
if h.Now == nil {
h.Now = time.Now
@@ -410,20 +382,11 @@ func (h *Holder) Load() error {
r.History = true
}
}
if r.Owed != "" && r.Cleared.IsZero() {
// Held from the operator on purpose and not told yet: said on its own, as the next tick tries what is
// unsent (issue 538). What the operator could have heard is history; this they could not.
r.Pending, r.History = r.Owed, false
}
h.open[r.Key] = &r
}
if sent, err := h.Store.Recent(); err == nil {
h.recent = sent
}
if err := h.loadWalks(); err != nil {
h.storeErr = err.Error()
return err
}
h.mu.Unlock()
err = h.loadAsks()
h.mu.Lock()
@@ -522,7 +485,6 @@ func (h *Holder) raised(rec Record, quiet bool) {
wasQuiet := needsNothing(old)
wasSilenced := old.silenced(now)
old.Summary, old.Subject, old.Kind = rec.Summary, rec.Subject, rec.Kind
old.takeSubject(rec)
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
old.takeWords(rec)
if rec.Severity != "" {
@@ -622,7 +584,6 @@ func (h *Holder) changed(rec Record, quiet bool) {
wasQuiet := needsNothing(old)
wasSilenced := old.silenced(h.Now())
old.Summary, old.Subject = rec.Summary, rec.Subject
old.takeSubject(rec)
old.SilencedTill, old.SilencedWhy = rec.SilencedTill, rec.SilencedWhy
old.takeWords(rec)
if rec.Severity != "" {
@@ -687,7 +648,6 @@ func (h *Holder) cleared(key string, quiet bool) {
}
old.Cleared = now
old.Pending = ""
old.Owed = "" // what a walk held of it is owed no more: it ended (the review of PR 238)
old.Folded = false
old.Seen = now
told := old.told()
@@ -750,8 +710,6 @@ func (h *Holder) Tick() {
sort.Slice(recs, func(i, j int) bool { return recs[i].Raised.Before(recs[j].Raised) })
h.sweepAsks(now)
h.breakGlass()
h.boundWalks(now)
h.duePasses(now)
for _, r := range recs {
switch {
case !r.Cleared.IsZero():
@@ -817,7 +775,6 @@ func (h *Holder) Flush() {
h.init()
now := h.Now()
h.mu.Unlock()
h.duePasses(now)
h.flush(now, false)
}
+1 -21
View File
@@ -66,9 +66,6 @@ func (f *fakeChannel) Edit(id string, m Message) error {
type memStore struct {
recs map[string]Record
recent []Sent
// walks and passes are what ADR 0308 keeps: the open walks and the restart passes.
walks map[string]Walk
passes map[string]Pass
// puts counts the writes, so a write of bytes identical to what is stored is still seen (issue 422's review).
puts int
}
@@ -90,24 +87,7 @@ func (m *memStore) All() ([]Record, error) {
return out, nil
}
func (m *memStore) PutRecent(s []Sent) error { m.recent = s; return nil }
func (m *memStore) PutWalk(w Walk) error {
if m.walks == nil {
m.walks = map[string]Walk{}
}
m.walks[w.ID] = w
return nil
}
func (m *memStore) DeleteWalk(id string) error { delete(m.walks, id); return nil }
func (m *memStore) Walks() ([]Walk, error) {
var out []Walk
for _, w := range m.walks {
out = append(out, w)
}
return out, nil
}
func (m *memStore) PutPasses(p map[string]Pass) error { m.passes = p; return nil }
func (m *memStore) Passes() (map[string]Pass, error) { return m.passes, nil }
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
func (m *memStore) Recent() ([]Sent, error) { return m.recent, nil }
type clock struct{ t time.Time }
-80
View File
@@ -29,9 +29,6 @@ type Settings struct {
// TimeZone is the operator's, an IANA name such as Europe/Brussels: every time a message says is said
// in it. Empty is this machine's own zone.
TimeZone string `json:"time-zone"`
// OnlyWhatNeedsMe are the channel kinds set to say only what needs the operator: a walk that delivered
// with nothing held and nothing open is not told there at all (novox/hq ADR 0308 §4).
OnlyWhatNeedsMe []string `json:"only-what-needs-me"`
}
// zoneOf is the location a time-zone setting names, or this machine's own when it names none or one
@@ -63,19 +60,6 @@ func readSettings(path string) (Settings, error) {
if tz, ok := loose["time-zone"].(string); ok {
s.TimeZone = strings.TrimSpace(tz)
}
switch v := loose["only-what-needs-me"].(type) {
case nil:
case []any:
for _, k := range v {
kind, ok := k.(string)
if !ok || strings.TrimSpace(kind) == "" {
return s, fmt.Errorf("only-what-needs-me lists something that is not a channel kind: %v", k)
}
s.OnlyWhatNeedsMe = append(s.OnlyWhatNeedsMe, strings.TrimSpace(kind))
}
default:
return s, fmt.Errorf("only-what-needs-me is not a list of channel kinds")
}
return s, nil
}
@@ -125,50 +109,6 @@ func (stateStore) All() ([]Record, error) {
}
return out, nil
}
// The walks open now, one key each in the state `walks`, and the restart passes under one key of `sent`
// (novox/hq ADR 0308): a router that starts again holds, tells and passes as if it had not.
func (stateStore) PutWalk(w Walk) error {
_, err := stdio.State("walks").Put(kvKey(w.ID), w)
return err
}
func (stateStore) DeleteWalk(id string) error { return stdio.State("walks").Delete(kvKey(id)) }
func (stateStore) Walks() ([]Walk, error) {
keys, err := stdio.State("walks").Keys()
if err != nil {
return nil, err
}
var out []Walk
for _, k := range keys {
e, err := stdio.State("walks").Get(k)
if err != nil {
return nil, err
}
if e == nil {
continue
}
var w Walk
if err := json.Unmarshal(e.Value, &w); err != nil {
logf("[messenger] the walk kept as %s cannot be read (%v); left as it is", k, err)
continue
}
out = append(out, w)
}
return out, nil
}
func (stateStore) PutPasses(p map[string]Pass) error {
_, err := stdio.State("sent").Put("passes", p)
return err
}
func (stateStore) Passes() (map[string]Pass, error) {
e, err := stdio.State("sent").Get("passes")
if err != nil || e == nil {
return nil, err
}
var p map[string]Pass
return p, json.Unmarshal(e.Value, &p)
}
func (stateStore) PutRecent(s []Sent) error {
_, err := stdio.State("sent").Put("recent", s)
return err
@@ -331,14 +271,6 @@ func main() {
return loc
},
Emit: func(event string, body any) error { return stdio.Emit(event, body) },
NeedsOnly: func(kind string) bool {
for _, k := range settings().OnlyWhatNeedsMe {
if k == kind {
return true
}
}
return false
},
}
bench.To = func(kind string) string {
if ids := h.identitiesOf(kind); len(ids) > 0 {
@@ -394,18 +326,6 @@ func run(h *Holder, l *listening) {
}
}()
handle := func(e stdio.Envelope) error {
if e.Key == ControllerSeat+"."+EventPlanMoved || e.Key == ControllerSeat+"."+EventWalk {
// A walk's start or end (novox/hq ADR 0308 §1); any other move of a plan is not read here.
ev, isWalk, err := DecodeWalk(e.Body)
if err != nil {
h.Unreadable(e.Key, err)
return nil
}
if isWalk {
h.Walked(ev)
}
return nil
}
event, ok := eventOf(e.Key)
if !ok {
return nil
@@ -81,7 +81,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
}
if !reflect.DeepEqual(m.Consumes, []string{
ControllerSeat + "." + EventRaised, ControllerSeat + "." + EventChanged, ControllerSeat + "." + EventCleared,
ControllerSeat + "." + EventPlanMoved, "intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
"intake.choice", "intake.link", "intake.failed", "intake.standing"}) {
t.Fatalf("consumes %v", m.Consumes)
}
// It performs nothing: the verbs it calls read the open conditions, whether root is free, and which issues
@@ -100,7 +100,7 @@ func TestItDeclaresTheSeatsOfAnAskAndHoldsTheOperatorChannel(t *testing.T) {
}
state = append(state, name)
}
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities", "walks"}) {
if !reflect.DeepEqual(state, []string{"open", "sent", "asks", "identities"}) {
t.Fatalf("state %v", state)
}
// It holds no channel's secret: only its own bus account, read by its own account alone (ADR 0259 §8).

Some files were not shown because too many files have changed in this diff Show More