Author SHA1 Message Date
jochen 05c0396f6b mesh-reads: drop the binary a local build left in the tree (issue 418)
mesh/merge-gate fail: builds new: modules/mesh-reads, sent nowhere; no bus step; a manifest the change touches fails the module check: modules/mesh-reads/m…
mesh/repo-check fail: its merge-check.sh failed: files asking for a setting without saying whether it is trusted: 25
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/delivery stopped: the pull request closed unmerged
2026-10-12 14:04:40 +02:00
jochen c36677b9e5 mesh-reads: read a node's files and accounts as an account that holds no secret (hq ADR 0310, issue 418)
Reading a file, searching one, listing a directory, the owner and mode of each
component of a path and looking up an account went through the login shell's
execute as the operator's account, which sees the operator's keys and the
agent's credentials, and execute is not served on the control-node. This module
holds node-reads on every node, runs as mesh-reads (no login, no group beside
its own), refuses the node's secret places by name before any read (the
controller's list and the known credential files), withholds any line shaped
like a secret through mesh-sdk's outward rule, and bounds every answer.
2026-10-12 14:04:12 +02:00
74 changed files with 422 additions and 6625 deletions
-2
View File
@@ -19,5 +19,3 @@ modules/mesh-issues/cmd/mesh-issues/mesh-issues
modules/mounts/cmd/mounts/mounts
modules/mesh-knowledge/cmd/mesh-knowledge/mesh-knowledge
modules/mesh-knowledge/mesh-knowledge
modules/mesh-reads/mesh-reads
modules/mesh-reads/cmd/mesh-reads/mesh-reads
+4 -22
View File
@@ -1,8 +1,7 @@
# artifact-store-tools
Tools that say what the mesh's artifact store holds, and what the controller's records say of it
(novox/hq ADR 0251 §1–3, to-be 51). It changes nothing on its machine; its only resources are its state
directory and the file the mesh writes there from its setting `older-store-names`.
(novox/hq ADR 0251 §1–3, to-be 51). It declares no resources and changes nothing on its machine.
## Why a module beside the store
@@ -19,7 +18,7 @@ on that machine.
|---|---|---|
| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size |
| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) |
| `artifact_store_references` | r | what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_references` | r | what the controller's records say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` |
### How the store is read
@@ -44,7 +43,6 @@ it lists. The store's collector removes every blob no manifest marks, so these n
| state | what | removable |
|---|---|---|
| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no |
| `named-by-a-definition` | no record names it, and a definition in the catalogue pins it by digest, under the store's current name or an older one; `why` names the module (hq issue 323) | no |
| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no |
| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` |
| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive |
@@ -58,20 +56,6 @@ its own right; the controller lets an index go with the platform manifests only
counted as the index's. Two indexes naming one platform give it the more keeping state. An index no
record names leaves its platform manifests unrecorded with it.
**What the definitions pin** (hq issue 323). A definition may pin an image of the store the mesh did not
make: pushed before the mesh built it, and pinned under the store's current name or a name it had before.
No record of the controller names it, yet a machine runs it, so `artifact_store_references` reads every
definition the catalogue holds (`mesh-catalog.catalog_modules`, then `catalog_module` for each, eight at
a time and within twenty seconds) and finds every `<name>/<repository>@sha256:…` in any of its strings,
under each of the store's names: `artifact-store://`, the store's address as the controller answers it,
and the names it had before. Those are the mesh's setting `older-store-names` (space-separated, `none` by
default, set with the controller's `settings`), which the mesh writes to the module's `store-names` file: a
catalogue names no installation. When they cannot be read, the answer's first line says so. A manifest no record
names that a definition pins is `named-by-a-definition`; one the records name keeps the records' state. A
definition read at its commit is kept, so the next call asks only for the list. Definitions that could
not be read are listed under `definitions.unread`, and the answer's first line says that what they pin
reads as unrecorded until they are.
**Unrecorded copies of upstream images** are listed in `unrecorded_mirrors`: each manifest at the top of
a repository only the build's mirror writes (`upstream/<host>/<path>`, or a module's `<module>/on-<argument>`
from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the
@@ -104,8 +88,6 @@ with slashes, a cut listing refused), manifests read with one absent or unreadab
digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks),
every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept
manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked,
the fall-back when the references let go of cannot be carried, a manifest a definition pins under the
store's current or an older name, a definition not read said first, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs and the
catalogue's two tools.
the fall-back when the references let go of cannot be carried, escalation through `sudo -n`, and that the
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs.
Any command other than the two read-only scripts fails the test.
@@ -7,15 +7,12 @@ import (
// The states a manifest the store holds can be in, as the records see it (novox/hq to-be 51).
const (
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
// StateNamedByDefinition is a manifest no record names that a definition pins by digest, under the
// store's current name or an older one (novox/hq issue 323): a machine may run it, so it is kept.
StateNamedByDefinition = "named-by-a-definition"
StateKept = "kept"
StateHolderKept = "holder-of-kept-archive"
StateEligible = "eligible"
StateHolderEligible = "holder-of-eligible-archive"
StateCollectedPresent = "let-go-yet-present"
StateNamedDocument = "named-document"
StateUnrecorded = "unrecorded"
collectionRemovesThese = "eligible and holder-of-eligible-archive, through the controller's collect; nothing else"
)
@@ -48,8 +45,8 @@ var mirrorRepository = regexp.MustCompile(`^(upstream/[a-z0-9][a-z0-9._/-]*|[a-z
// keeping orders the states a platform manifest may take from the indexes naming it: the most keeping
// wins, so a platform two indexes name is as kept as the more kept of them.
var keeping = map[string]int{
StateKept: 0, StateNamedByDefinition: 1, StateNamedDocument: 2, StateHolderKept: 3, StateEligible: 4,
StateHolderEligible: 5, StateCollectedPresent: 6,
StateKept: 0, StateNamedDocument: 1, StateHolderKept: 2, StateEligible: 3, StateHolderEligible: 4,
StateCollectedPresent: 5,
}
// recordIndex finds a record by repository and digest.
@@ -67,9 +64,8 @@ func indexRecords(r *Records) recordIndex {
return idx
}
// Classify gives every manifest the store holds one state. A manifest no record names that a definition
// pins is named by a definition; defs may be nil, when the definitions were not read.
func Classify(v *View, recs *Records, defs *Definitions) []Entry {
// Classify gives every manifest the store holds one state.
func Classify(v *View, recs *Records) []Entry {
idx := indexRecords(recs)
var out []Entry
for _, name := range v.L.RepoNames() {
@@ -103,12 +99,6 @@ func Classify(v *View, recs *Records, defs *Definitions) []Entry {
e.State = StateNamedDocument
}
}
if e.State == StateUnrecorded && defs != nil {
for _, p := range defs.Pins[Key(name, d)] {
e.State = StateNamedByDefinition
e.Why = append(e.Why, p.why())
}
}
out = append(out, e)
}
}
@@ -1,343 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"regexp"
"sort"
"strings"
"sync"
"time"
)
// What the definitions pin in the store (novox/hq issue 323). A definition may pin an image of the store by
// digest that the mesh did not make: one pushed before the mesh built it, pinned under the store's current
// name or under a name the store had before. No record of the controller names such a manifest, and yet a
// machine runs it, so it is said to be named by a definition and never unrecorded.
// NamesFileEnv names the file the mesh writes from the setting older-store-names: a line `older=` with the
// names the store had before, as a definition may still write them — host names, or host:port, separated by
// spaces — or `none`. A catalogue names no installation, so the names are the mesh's setting, not the
// manifest's.
const NamesFileEnv = "MESH_ARTIFACT_STORE_NAMES_FILE"
// CatalogueModules and CatalogueModule are the catalogue's tools these definitions are read through.
const (
CatalogueModules = "mesh-catalog.catalog_modules"
CatalogueModule = "mesh-catalog.catalog_module"
)
// DefinitionsTimeout bounds reading every definition, below the runtime's thirty-second call limit; what
// was read is kept, so a call that runs out reads only the rest the next time.
const DefinitionsTimeout = 20 * time.Second
// definitionReaders is how many definitions are asked at once.
const definitionReaders = 8
// StoreNames are the names a definition may pin an image of the store under: the scheme the mesh writes
// (artifact-store://), the store's address now, and the names it had before.
type StoreNames struct {
Current []string
Older []string
}
// NamesOf is the store's names: its address as the controller answers it, and the older ones given.
func NamesOf(address, older string) StoreNames {
n := StoreNames{Current: []string{"artifact-store://"}}
if a := strings.Trim(strings.TrimSpace(address), "/"); a != "" {
n.Current = append(n.Current, a+"/")
}
for _, o := range strings.FieldsFunc(older, func(r rune) bool { return r == ',' || r == ' ' || r == '\n' || r == '\t' }) {
if o = strings.Trim(o, "/"); o != "" && o != "none" {
n.Older = append(n.Older, o+"/")
}
}
return n
}
// OlderNames reads the older names from the file the mesh writes, and says why when it cannot.
func OlderNames() (string, string) {
path := os.Getenv(NamesFileEnv)
if path == "" {
return "", NamesFileEnv + " is not set, so no older name of the store was read"
}
raw, err := os.ReadFile(path)
if err != nil {
return "", "the store's older names could not be read: " + err.Error()
}
for _, line := range strings.Split(string(raw), "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "older="); ok {
return v, ""
}
}
return "", path + " holds no line older=, so no older name of the store was read"
}
// Pin is one definition pinning a manifest of the store by digest.
type Pin struct {
Module string `json:"module"`
// Name is the store's name it is pinned under, as written: "artifact-store://", an address, or an older name.
Name string `json:"name"`
Older bool `json:"older,omitempty"`
}
// pinned is one pin found in a manifest: where in the store, and under which name.
type pinned struct {
Repository, Digest, Name string
Older bool
}
var pinShapes sync.Map // name -> *regexp.Regexp
// pinShape is `<name><repository>@sha256:<hex>`, the name standing on its own: not the tail of a longer host.
func pinShape(name string) *regexp.Regexp {
if re, ok := pinShapes.Load(name); ok {
return re.(*regexp.Regexp)
}
re := regexp.MustCompile(`(?:^|[^A-Za-z0-9.-])` + regexp.QuoteMeta(name) +
`([a-z0-9][a-z0-9._/-]*)@(sha256:[0-9a-f]{64})`)
pinShapes.Store(name, re)
return re
}
// PinsIn finds every image of the store a manifest pins by digest, in any string it holds: a resource's
// image, a build's base, a file's content.
func PinsIn(manifest json.RawMessage, names StoreNames) []pinned {
var doc any
if json.Unmarshal(manifest, &doc) != nil {
return nil
}
seen := map[string]bool{}
var out []pinned
var walk func(any)
walk = func(x any) {
switch v := x.(type) {
case map[string]any:
for _, e := range v {
walk(e)
}
case []any:
for _, e := range v {
walk(e)
}
case string:
for _, group := range []struct {
names []string
older bool
}{{names.Current, false}, {names.Older, true}} {
for _, name := range group.names {
for _, m := range pinShape(name).FindAllStringSubmatch(v, -1) {
p := pinned{Repository: strings.Trim(m[1], "/"), Digest: m[2], Name: name, Older: group.older}
if k := p.Repository + "@" + p.Digest + " " + p.Name; !seen[k] {
seen[k] = true
out = append(out, p)
}
}
}
}
}
}
walk(doc)
return out
}
// Definitions is what the catalogue's definitions pin in the store, and what of them could not be read.
type Definitions struct {
// Pins are by Key(repository, digest).
Pins map[string][]Pin
// Read is how many definitions were read; Unread says, per module, why its definition was not.
Read int
Unread map[string]string
// NotAsked says why the catalogue could not be asked at all, when it could not.
NotAsked string
// NamesNotRead says why the store's older names could not be read, when they could not.
NamesNotRead string
Names StoreNames
}
// Catalogue reads the definitions through the catalogue's tools, and keeps each definition it read by its
// module and commit, which never change together.
type Catalogue struct {
Ask Asking
mu sync.Mutex
kept map[string][]pinned // "<module>@<commit>|<names>" -> its pins
}
type catalogueEntry struct {
Module string `json:"module"`
Commit string `json:"commit"`
}
// Read asks the catalogue for every module and reads each definition, bounded by DefinitionsTimeout.
func (c *Catalogue) Read(ctx context.Context, names StoreNames) *Definitions {
defs := &Definitions{Pins: map[string][]Pin{}, Unread: map[string]string{}, Names: names}
if c == nil || c.Ask == nil {
defs.NotAsked = "this bundle has no way to ask the catalogue"
return defs
}
raw, err := c.Ask(CatalogueModules, map[string]any{})
if err != nil {
defs.NotAsked = "the catalogue's modules could not be asked: " + err.Error()
return defs
}
answer, output, ok := answerOf(raw)
if !ok {
defs.NotAsked = "the catalogue refused its modules: " + lastLine(output)
return defs
}
var list struct {
Modules []catalogueEntry `json:"modules"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &list); err != nil || list.Error != "" || list.Modules == nil {
defs.NotAsked = "the catalogue's modules answer is not readable: " + firstOf(list.Error, errText(err), lastLine(output))
return defs
}
c.mu.Lock()
if c.kept == nil {
c.kept = map[string][]pinned{}
}
c.mu.Unlock()
nameKey := strings.Join(names.Current, ",") + ";" + strings.Join(names.Older, ",")
ctx, cancel := context.WithTimeout(ctx, DefinitionsTimeout)
defer cancel()
type result struct {
module string
pins []pinned
why string
}
results := make(chan result, len(list.Modules))
work := make(chan catalogueEntry)
var wg sync.WaitGroup
for i := 0; i < definitionReaders; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for m := range work {
pins, why := c.definition(m, names, nameKey)
results <- result{m.Module, pins, why}
}
}()
}
go func() {
defer close(work)
for _, m := range list.Modules {
select {
case work <- m:
case <-ctx.Done():
return
}
}
}()
go func() { wg.Wait(); close(results) }()
got := map[string]bool{}
collect:
for {
select {
case r, open := <-results:
if !open {
break collect
}
got[r.module] = true
if r.why != "" {
defs.Unread[r.module] = r.why
continue
}
defs.Read++
for _, p := range r.pins {
k := Key(p.Repository, p.Digest)
defs.Pins[k] = append(defs.Pins[k], Pin{Module: r.module, Name: p.Name, Older: p.Older})
}
case <-ctx.Done():
break collect
}
}
for _, m := range list.Modules {
if !got[m.Module] {
defs.Unread[m.Module] = fmt.Sprintf("not read within %d s", int(DefinitionsTimeout/time.Second))
}
}
for k := range defs.Pins {
sort.Slice(defs.Pins[k], func(i, j int) bool { return defs.Pins[k][i].Module < defs.Pins[k][j].Module })
}
return defs
}
// definition reads one module's definition, or answers it from what was read before.
func (c *Catalogue) definition(m catalogueEntry, names StoreNames, nameKey string) ([]pinned, string) {
key := m.Module + "@" + m.Commit + "|" + nameKey
if m.Commit != "" {
c.mu.Lock()
pins, had := c.kept[key]
c.mu.Unlock()
if had {
return pins, ""
}
}
args := map[string]any{"module": m.Module}
if m.Commit != "" {
args["commit"] = m.Commit
}
raw, err := c.Ask(CatalogueModule, args)
if err != nil {
return nil, "could not be asked: " + err.Error()
}
answer, output, ok := answerOf(raw)
if !ok {
return nil, "refused: " + lastLine(output)
}
var one struct {
Module *struct {
Manifest json.RawMessage `json:"manifest"`
} `json:"module"`
Error string `json:"error"`
}
if err := json.Unmarshal(answer, &one); err != nil || one.Error != "" || one.Module == nil || len(one.Module.Manifest) == 0 {
return nil, "not readable: " + firstOf(one.Error, errText(err), "no manifest in the answer")
}
pins := PinsIn(one.Module.Manifest, names)
if m.Commit != "" {
c.mu.Lock()
c.kept[key] = pins
c.mu.Unlock()
}
return pins, ""
}
// why says which definitions name a manifest, and under which of the store's names.
func (p Pin) why() string {
if p.Older {
return fmt.Sprintf("a definition names it: %s, under %s, an older name of the store", p.Module, shownName(p.Name))
}
return "a definition names it: " + p.Module
}
func firstOf(s ...string) string {
for _, x := range s {
if strings.TrimSpace(x) != "" {
return x
}
}
return ""
}
func errText(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// shownName is a name of the store as a person reads it: a host without the slash a pin puts after it, the
// scheme as written.
func shownName(name string) string {
if strings.HasSuffix(name, "://") {
return name
}
return strings.TrimSuffix(name, "/")
}
@@ -1,259 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// A definition may pin an image of the store the mesh did not make, under a name the store had before
// (novox/hq issue 323): three apps the anchor runs are pinned so. No record names them; a definition does.
const oldName = "registry-api.example.org"
// container is a definition whose one container runs image.
func container(image string) string {
return fmt.Sprintf(`{"module":"x","resources":[{"id":"server","type":"container","image":%q}]}`, image)
}
func TestAnImageADefinitionPinsUnderAnOlderNameOfTheStoreIsNamedByADefinition(t *testing.T) {
v := view(t, world())
names := NamesOf("store:5000", oldName)
defs := &Definitions{Pins: map[string][]Pin{}}
for module, manifest := range map[string]string{
// m-stranger: no record names it; the invoicing app pins it under the store's older name.
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
// m-old: the records say eligible; a definition pinning it does not change what the records say.
"website": container("store:5000/app/server@" + d("m-old")),
} {
for _, p := range PinsIn(json.RawMessage(manifest), names) {
defs.Pins[Key(p.Repository, p.Digest)] = append(defs.Pins[Key(p.Repository, p.Digest)], Pin{Module: module, Name: p.Name, Older: p.Older})
}
}
got := stateByName(Classify(v, withFill(records()), defs))
stranger := got["app/server "+d("m-stranger")]
if stranger.State != StateNamedByDefinition {
t.Fatalf("a manifest a definition pins under an older name reads %q, want %q", stranger.State, StateNamedByDefinition)
}
if want := []string{"a definition names it: invoicing, under " + oldName + ", an older name of the store"}; !reflect.DeepEqual(stranger.Why, want) {
t.Errorf("why %q, want %q", stranger.Why, want)
}
if s := got["app/server "+d("m-old")].State; s != StateEligible {
t.Errorf("a recorded manifest a definition pins reads %q: the records decide what the mesh made", s)
}
// Without the definitions it is what it was: unrecorded.
if s := stateByName(Classify(v, withFill(records()), nil))["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("without definitions %q", s)
}
}
func TestPinsAreFoundInEveryStringAndOnlyUnderTheStoresNames(t *testing.T) {
names := NamesOf("store:5000", oldName+", older.example/")
manifest := `{"resources":[
{"type":"container","image":"` + oldName + `/team/www@` + d("www") + `"},
{"type":"file","content":"IMAGE=older.example/team/tool@` + d("tool") + `\n"},
{"type":"container","image":"artifact-store://mod/art@` + d("built") + `"},
{"type":"container","image":"store:5000/mod/art@` + d("addressed") + `"},
{"type":"container","image":"not` + oldName + `/team/www@` + d("lookalike") + `"},
{"type":"container","image":"docker.io/library/nginx@` + d("upstream") + `"},
{"type":"container","image":"` + oldName + `/team/www:latest"}],
"build":{"on":{"base":"` + oldName + `/team/www@` + d("www") + `"}}}`
var got []string
for _, p := range PinsIn(json.RawMessage(manifest), names) {
got = append(got, fmt.Sprintf("%s@%s %s %v", p.Repository, p.Digest, p.Name, p.Older))
}
want := map[string]bool{
"team/www@" + d("www") + " " + oldName + "/ true": true,
"team/tool@" + d("tool") + " older.example/ true": true,
"mod/art@" + d("built") + " artifact-store:// false": true,
"mod/art@" + d("addressed") + " store:5000/ false": true,
}
if len(got) != len(want) {
t.Fatalf("found %d pins, want %d (one per image and name; not a longer host's tail, not another registry, not a tag): %v", len(got), len(want), got)
}
for _, g := range got {
if !want[g] {
t.Errorf("found %s", g)
}
}
}
func TestAPlatformTakesTheStateOfAnIndexADefinitionNames(t *testing.T) {
v := mirrorWorld()
defs := &Definitions{Pins: map[string][]Pin{
Key("route-proxy/on-go_base", d("i-pre")): {{Module: "route-proxy", Name: oldName + "/", Older: true}},
}}
got := stateByName(Classify(v, withFill(mirrorRecords()), defs))
for _, name := range []string{"i-pre", "p-pre"} {
if s := got["route-proxy/on-go_base "+d(name)].State; s != StateNamedByDefinition {
t.Errorf("%s reads %q, want %q", name, s, StateNamedByDefinition)
}
}
if e := got["route-proxy/on-go_base "+d("p-pre")]; e.Index != d("i-pre") {
t.Errorf("p-pre takes its state from %q", e.Index)
}
if s := got["route-proxy/on-go_base "+d("p-lost")].State; s != StateUnrecorded {
t.Errorf("a platform nothing names reads %q", s)
}
for _, um := range UnrecordedMirrors(v, Classify(v, withFill(mirrorRecords()), defs)) {
if strings.Contains(um.Reference, hexOf(d("i-pre"))) {
t.Errorf("a copy a definition names is offered to the mirrors verb: %v", um)
}
}
}
// olderNames writes the file the mesh writes from the setting older-store-names.
func olderNames(t *testing.T, names string) {
t.Helper()
path := filepath.Join(t.TempDir(), "store-names")
if err := os.WriteFile(path, []byte("# written by the mesh\nolder="+names+"\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(NamesFileEnv, path)
}
func TestTheOlderNamesAreReadFromTheFileTheMeshWrites(t *testing.T) {
olderNames(t, oldName+" older.example")
got, why := OlderNames()
if why != "" || !reflect.DeepEqual(NamesOf("", got).Older, []string{oldName + "/", "older.example/"}) {
t.Errorf("read %q (%s)", got, why)
}
olderNames(t, "none")
if got, why := OlderNames(); why != "" || len(NamesOf("", got).Older) != 0 {
t.Errorf("none read as %q (%s)", got, why)
}
t.Setenv(NamesFileEnv, filepath.Join(t.TempDir(), "absent"))
if _, why := OlderNames(); why == "" {
t.Error("an absent file was not said")
}
}
func TestOlderNamesNotReadAreSaidFirst(t *testing.T) {
t.Setenv(NamesFileEnv, "")
a := &asked{definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger"))}}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, NamesFileEnv) {
t.Errorf("first said %q", w)
}
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q without the older names", s)
}
}
// references is one artifact_store_references tool, kept across calls as the runtime keeps the bundle.
func references(t *testing.T, a *asked) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
if tl.Name == "artifact_store_references" {
return tl.Run
}
}
t.Fatal("no artifact_store_references")
return nil
}
func manifestsOf(t *testing.T, out any) map[string]Entry {
t.Helper()
entries, ok := out.(map[string]any)["manifests"].([]Entry)
if !ok {
t.Fatalf("no manifests in %v", out)
}
return stateByName(entries)
}
func TestReferencesReadTheDefinitionsUnderTheStoresOlderNames(t *testing.T) {
olderNames(t, oldName)
a := &asked{definitions: map[string]string{
"invoicing": container(oldName + "/app/server@" + d("m-stranger")),
"grafana": container("grafana/grafana@" + d("upstream")),
}}
run := references(t, a)
out, err := run(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
e := manifestsOf(t, out)["app/server "+d("m-stranger")]
if e.State != StateNamedByDefinition || len(e.Why) != 1 || !strings.Contains(e.Why[0], "invoicing") || !strings.Contains(e.Why[0], oldName) {
t.Errorf("m-stranger: %+v", e)
}
said := out.(map[string]any)["definitions"].(map[string]any)
if said["read"] != 2 || !reflect.DeepEqual(said["older_store_names"], []string{oldName}) ||
!reflect.DeepEqual(said["store_names"], []string{"artifact-store://", "store:5000"}) {
t.Errorf("definitions said %v", said)
}
if w := out.(map[string]any)["said"].([]string)[0]; strings.Contains(w, "could not be read") {
t.Errorf("a warning with every definition read: %s", w)
}
// A definition read at its commit is kept: the next call asks only for the list.
before := 0
for _, c := range a.calls {
if c == CatalogueModule {
before++
}
}
if _, err := run(map[string]any{"repository": "app/server"}); err != nil {
t.Fatal(err)
}
after := 0
for _, c := range a.calls {
if c == CatalogueModule {
after++
}
}
if before != 2 || after != 2 {
t.Errorf("asked catalog_module %d times, then %d: each definition at its commit is read once", before, after)
}
}
func TestADefinitionNotReadIsSaidFirstAndLeavesWhatItPinsUnrecorded(t *testing.T) {
olderNames(t, oldName)
a := &asked{
definitions: map[string]string{"invoicing": container(oldName + "/app/server@" + d("m-stranger")), "grafana": container("x")},
failing: map[string]bool{"invoicing": true},
}
out, err := references(t, a)(map[string]any{"repository": "app/server"})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if s := manifestsOf(t, out)["app/server "+d("m-stranger")].State; s != StateUnrecorded {
t.Errorf("m-stranger reads %q with its definition unread", s)
}
if w := m["said"].([]string)[0]; !strings.Contains(w, "1 definitions could not be read") {
t.Errorf("first said %q", w)
}
if u := m["definitions"].(map[string]any)["unread"].(map[string]string); u["invoicing"] == "" || len(u) != 1 {
t.Errorf("unread %v", u)
}
// The catalogue not answering at all: the records are still answered, and the warning stands first.
a = &asked{noCatalogue: true}
out, err = references(t, a)(map[string]any{})
if err != nil {
t.Fatal(err)
}
m = out.(map[string]any)
if w := m["said"].([]string)[0]; !strings.Contains(w, "the definitions could not be read") || !strings.Contains(w, "no responders") {
t.Errorf("first said %q", w)
}
if m["definitions"].(map[string]any)["not_asked"] == nil {
t.Errorf("definitions %v", m["definitions"])
}
}
func TestNoCatalogueIsSaidAndNothingIsAsked(t *testing.T) {
defs := (&Catalogue{}).Read(context.Background(), NamesOf("", ""))
if defs.NotAsked == "" || defs.Read != 0 {
t.Errorf("%+v", defs)
}
}
@@ -24,7 +24,7 @@ func main() {
UID: os.Getuid(),
}
// An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools.
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask}, &Catalogue{Ask: stdio.Ask})); err != nil {
if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask})); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
@@ -65,7 +65,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
for i := range recs.References {
fill(&recs.References[i])
}
got := stateByName(Classify(v, recs, nil))
got := stateByName(Classify(v, recs))
up := "upstream/docker.io/library/golang "
for name, want := range map[string]string{
up + d("p-amd"): StateEligible, // only the eligible index names it
@@ -92,7 +92,7 @@ func TestAPlatformManifestTakesTheStateOfTheIndexNamingIt(t *testing.T) {
if len(mirrors) != 2 || !got["route-proxy/on-go_base "+d("i-pre")].Mirror || !got["route-proxy/on-go_base "+d("p-lost")].Mirror {
t.Errorf("offered as unrecorded copies: %v", mirrors)
}
um := UnrecordedMirrors(v, Classify(v, recs, nil))
um := UnrecordedMirrors(v, Classify(v, recs))
if len(um) != 2 || um[0].Reference != "artifact-store://route-proxy/on-go_base@"+d("i-pre") || um[0].OnlyTheir != 1+1+1000 {
t.Errorf("unrecorded copies: %+v", um)
}
@@ -104,7 +104,7 @@ func TestLettingAnIndexGoTakesItsOwnPlatformsAndSaysWhatStaysAndNamesOne(t *test
for i := range recs.References {
fill(&recs.References[i])
}
entries := Classify(v, recs, nil)
entries := Classify(v, recs)
old := "artifact-store://upstream/docker.io/library/golang@" + d("i-old")
gone := LetGoKeys(v, []string{old}, Spared(v, entries))
up := "upstream/docker.io/library/golang"
@@ -215,7 +215,7 @@ func TestTheMarkIsTheCollectorsIncludingAnIndexAndSharedBlobs(t *testing.T) {
func TestEveryStateIsGiven(t *testing.T) {
v := view(t, world())
got := map[string]string{}
for _, e := range Classify(v, withFill(records()), nil) {
for _, e := range Classify(v, withFill(records())) {
got[e.Repository+" "+e.Digest] = e.State
}
want := map[string]string{
@@ -239,7 +239,7 @@ func TestALetGoReferenceStillPresentIsSaidSo(t *testing.T) {
v := view(t, world())
r := records()
r.References = append(r.References, Record{Reference: "artifact-store://app/server@" + d("m-stranger"), State: "collected"})
for _, e := range Classify(v, withFill(r), nil) {
for _, e := range Classify(v, withFill(r)) {
if e.Digest == d("m-stranger") && e.State != StateCollectedPresent {
t.Errorf("state %s", e.State)
}
@@ -270,7 +270,7 @@ func TestBytesFreedCountASharedBlobAsKept(t *testing.T) {
func TestOnlyTheirBytesLeaveOutWhatAnotherStateMarks(t *testing.T) {
v := view(t, world())
entries := Classify(v, withFill(records()), nil)
entries := Classify(v, withFill(records()))
sums := Summarise(v, entries, entries)
el := sums[StateEligible]
if el.OnlyTheirBytes != 500+v.L.Blobs[d("m-old")] {
@@ -284,11 +284,6 @@ type asked struct {
calls []string
args []map[string]any
fail map[string]bool
// definitions are the catalogue's modules and their manifests; failing, the modules whose
// definition cannot be read; noCatalogue, the catalogue not answering at all.
definitions map[string]string
failing map[string]bool
noCatalogue bool
}
func (a *asked) ask(key string, body any) (json.RawMessage, error) {
@@ -304,31 +299,6 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
switch key {
case "seat:mesh-controller.artifacts":
answer = records()
case CatalogueModules:
if a.noCatalogue {
return nil, fmt.Errorf("no responders")
}
ms := []map[string]any{}
for name := range a.definitions {
ms = append(ms, map[string]any{"module": name, "commit": "c-" + name, "path": "modules/" + name})
}
answer = map[string]any{"modules": ms}
case CatalogueModule:
name, _ := args["module"].(string)
if a.failing[name] {
return nil, fmt.Errorf("no answer")
}
if args["commit"] != "c-"+name {
return nil, fmt.Errorf("asked %s at %v", name, args["commit"])
}
manifest, has := a.definitions[name]
if !has {
answer = map[string]any{"error": "the catalogue holds no " + name}
break
}
// The catalogue's tool answers through the runtime as the protocol's content list.
raw, _ := json.Marshal(map[string]any{"module": map[string]any{"module": name, "manifest": json.RawMessage(manifest)}})
return json.Marshal(map[string]any{"content": []map[string]any{{"type": "text", "text": string(raw)}}})
case "seat:mesh-controller.collect":
if args["confirm"] == "true" {
answer = CollectAnswer{LetGo: []string{"artifact-store://app/server@" + d("m-old")}}
@@ -345,7 +315,7 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) {
func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) {
f := world()
s := &Store{Container: "mesh-registry", Run: f.run(t)}
for _, tl := range Tools(s, Controller{Ask: a.ask}, &Catalogue{Ask: a.ask}) {
for _, tl := range Tools(s, Controller{Ask: a.ask}) {
if tl.Name == name {
return tl.Run
}
@@ -416,7 +386,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
t.Fatal(err)
}
var served []string
for _, tl := range Tools(&Store{}, Controller{}, &Catalogue{}) {
for _, tl := range Tools(&Store{}, Controller{}) {
if !strings.HasPrefix(tl.Name, "artifact_store_") || tl.Description == "" || tl.Run == nil {
t.Errorf("tool %q", tl.Name)
}
@@ -428,7 +398,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) {
if !reflect.DeepEqual(served, listed) {
t.Fatalf("served %v, manifest %v", served, listed)
}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect", CatalogueModules, CatalogueModule}
want := []string{"seat:mesh-controller.artifacts", "seat:mesh-controller.collect"}
if !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes %v", m.Invokes)
}
@@ -13,7 +13,7 @@ import (
// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect
// only reads too, unless it is a real run, and then it asks the controller, which decides and deletes.
func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
func Tools(s *Store, c Controller) []stdio.Tool {
ctx := context.Background()
repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (<module>/<artifact>)"}
return []stdio.Tool{
@@ -53,9 +53,7 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
{
Name: "artifact_store_references",
Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " +
"names it, or one of the five most recent builds of a module the mesh holds), named by a definition (no " +
"record names it, and a definition in the catalogue pins it by digest, under the store's current name or " +
"an older one: the module is said), the holder of a kept archive, " +
"names it, or one of the five most recent builds of a module the mesh holds), the holder of a kept archive, " +
"eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " +
"a named document the controller keeps under a tag, or unrecorded — no record names it. A platform manifest " +
"takes the state of the recorded index naming it. Counted and sized by " +
@@ -69,19 +67,11 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
if err != nil {
return nil, err
}
// The definitions are read while the store is: each bounded below the runtime's call limit.
read := make(chan *Definitions, 1)
go func() {
older, why := OlderNames()
defs := cat.Read(ctx, NamesOf(recs.Store, older))
defs.NamesNotRead = why
read <- defs
}()
v, err := s.View(ctx)
if err != nil {
return nil, err
}
return References(v, recs, <-read, repo)
return References(v, recs, repo)
},
},
{
@@ -126,7 +116,7 @@ func Tools(s *Store, c Controller, cat *Catalogue) []stdio.Tool {
if err != nil {
return map[string]any{"controller": answer, "bytes_not_worked_out": "the records could not be read: " + err.Error()}, nil
}
return Collected(v, answer, dry, Classify(v, recs, nil)), nil
return Collected(v, answer, dry, Classify(v, recs)), nil
},
},
}
@@ -235,11 +225,11 @@ func Usage(v *View, top int) map[string]any {
}
// References answers artifact_store_references.
func References(v *View, recs *Records, defs *Definitions, only string) (map[string]any, error) {
func References(v *View, recs *Records, only string) (map[string]any, error) {
if only != "" && v.L.Repos[only] == nil {
return nil, fmt.Errorf("the store holds no repository %q", only)
}
entries := Classify(v, recs, defs)
entries := Classify(v, recs)
byRepo := map[string][]Entry{}
for _, e := range entries {
byRepo[e.Repository] = append(byRepo[e.Repository], e)
@@ -253,7 +243,7 @@ func References(v *View, recs *Records, defs *Definitions, only string) (map[str
}
whole := Summarise(v, entries, entries)
said := []string{}
for _, state := range []string{StateKept, StateNamedByDefinition, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
for _, state := range []string{StateKept, StateHolderKept, StateEligible, StateHolderEligible, StateCollectedPresent, StateNamedDocument, StateUnrecorded} {
if s := whole[state]; s != nil {
said = append(said, fmt.Sprintf("%s: %d manifests marking %s, %s of it marked by nothing else", state, s.Manifests, human(s.Bytes), human(s.OnlyTheirBytes)))
}
@@ -286,57 +276,9 @@ func References(v *View, recs *Records, defs *Definitions, only string) (map[str
if u := unreadNote(v); u != nil {
out["unread"] = u
}
out["definitions"] = definitionsSaid(defs)
if warn := definitionsWarning(defs); warn != "" {
// Said first: what it leaves out is a manifest a machine may run, read as unrecorded.
out["said"] = append([]string{warn}, said...)
}
return out, nil
}
// definitionsSaid is how the definitions were read: how many, under which names, and what was not read.
func definitionsSaid(defs *Definitions) map[string]any {
if defs == nil {
return map[string]any{"read": 0, "not_asked": "the definitions were not read"}
}
var current, older []string
for _, n := range defs.Names.Current {
current = append(current, shownName(n))
}
for _, n := range defs.Names.Older {
older = append(older, shownName(n))
}
out := map[string]any{"read": defs.Read, "store_names": nonNil(current), "older_store_names": nonNil(older)}
if defs.NotAsked != "" {
out["not_asked"] = defs.NotAsked
}
if defs.NamesNotRead != "" {
out["older_store_names_not_read"] = defs.NamesNotRead
}
if len(defs.Unread) > 0 {
out["unread"] = defs.Unread
}
return out
}
// definitionsWarning says, when any definition was not read, that a manifest it pins reads as unrecorded.
func definitionsWarning(defs *Definitions) string {
switch {
case defs == nil:
return ""
case defs.NotAsked != "":
return "the definitions could not be read (" + defs.NotAsked + "), so a manifest a definition pins " +
"and no record names reads as unrecorded here: ask again before reading unrecorded as unused"
case defs.NamesNotRead != "":
return defs.NamesNotRead + ", so a manifest a definition pins under an older name of the store and no " +
"record names reads as unrecorded here"
case len(defs.Unread) > 0:
return fmt.Sprintf("%d definitions could not be read (listed in definitions.unread), so a manifest one "+
"of them pins and no record names reads as unrecorded here: ask again — what was read is kept", len(defs.Unread))
}
return ""
}
// Collected answers artifact_store_collect: the controller's answer, and what the store's collector frees.
func Collected(v *View, a *CollectAnswer, dry bool, entries []Entry) map[string]any {
refs := a.LetGo
+2 -28
View File
@@ -3,9 +3,7 @@
"version": "1",
"invokes": [
"seat:mesh-controller.artifacts",
"seat:mesh-controller.collect",
"mesh-catalog.catalog_modules",
"mesh-catalog.catalog_module"
"seat:mesh-controller.collect"
],
"tools": [
"artifact_store_repositories",
@@ -13,29 +11,6 @@
"artifact_store_references",
"artifact_store_collect"
],
"settings": {
"older-store-names": {
"kind": "preference",
"default": "none",
"why": "the names the artifact store had before, as a definition may still pin an image under them: host names or host:port, separated by spaces; none when it had no other. artifact_store_references reads every definition under these as well as under the store's current name, so an image a definition pins under one is said to be named by a definition (novox/hq issue 323)"
}
},
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": "."
},
{
"id": "store-names",
"type": "file",
"path": "${dir:state}/store-names",
"mode": "0600",
"trusted": false,
"content": "# The names the artifact store had before, from the setting older-store-names, written by the mesh.\n# artifact_store_references reads every definition under these too (novox/hq issue 323).\n# Replaced on every push; change the setting, never this file.\nolder=${setting:older-store-names}\n"
}
],
"build": {
"artifacts": [
{
@@ -49,8 +24,7 @@
"artifact-store-tools"
],
"env": {
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry",
"MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names"
"MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry"
}
}
]
+32 -155
View File
@@ -1,10 +1,8 @@
# betterbird
Betterbird, the mail and calendar client of g14 and shanks (a Thunderbird fork, with the operator's
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208, ADR 0316, ADR 0317,
to-be 56). It requires `x11-display`, so it is assigned only where a display server is held on the same
machine. It installs Betterbird from a pinned upstream release, ships Owl with it through Betterbird's
enterprise policy, and declares the operator's mail accounts without their credentials.
Exchange calendar through the Owl add-on), as a module (novox/hq issue 345, ADR 0208). It requires
`x11-display`, so it is assigned only where a display server is held on the same machine.
It exists because a symptom had no tools: on a workstation, a calendar reminder appeared as a
transparent window with nothing in it (2026-10-09). Nothing in the mesh could say which window that
@@ -13,65 +11,24 @@ issue 345.
## Owns
| what | where | class |
|---|---|---|
| Betterbird, the upstream release (`archive`, pinned by URL and sha256) | `/opt/betterbird` | owned, whole |
| the enterprise policy (`install/distribution/policies.json`) | `/opt/betterbird/distribution` | owned, whole |
| Owl's xpi (`owl`, pinned by the vendor's sha256) | `/opt/betterbird/owl` | owned, whole |
| the spelling dictionaries' default (`install/defaults-pref/vendor-prefs.js`) | `/opt/betterbird/defaults/pref` | owned: its one file |
| the tools' program, for the start's preparation | `/usr/local/lib/mesh-betterbird` | owned, whole |
| the start | `/usr/local/bin/betterbird` | owned, whole |
| the launcher entry | `/usr/local/share/applications/eu.betterbird.Betterbird.desktop` | owned, whole |
| the module's directory, with its `user.js` (setting `prefs`), `accounts.json` (setting `accounts`) and the record of adopted and created accounts | `~/.config/betterbird/` | owned, whole |
| the profile's `user.js`, placed by the tools | the profile | the module's, or never touched |
| the account lists in the profile's `prefs.js` | the profile | written into: the module's keys only |
| the profile: mail, address books, calendars, logins | `~/.thunderbird/` | found (ADR 0182) |
| what | where |
|---|---|
| the module's directory | `~/.config/betterbird/` |
| the module's `user.js`, filled from the setting `prefs` | `~/.config/betterbird/user.js` |
**The community package `betterbird-bin` is declared absent** (ADR 0180), before anything is placed in
`/opt/betterbird`: the node-engine removes it with `pacman -R`, then unpacks the release where the package
had it. Betterbird chooses its profile by its installation's path, so the same profile opens after the
switch. A package left in place (its removal failed) holds the path, and the release is refused there by
the node-engine rather than mixed in. The module creates no link: the start, a launcher entry naming the
icon in the installation, and the dictionaries as a default preference replace the package's links.
**Every new build waits for the operator** (`upgrade: record`): the first push to a node removes the
package and replaces what a running Betterbird has open, so the operator quits Betterbird, then pushes.
An upgrade is a change to the pins, reviewed and delivered like any other (ADR 0316 §5).
**The pins.** The release is `betterbird-153.4.0esr-bb10.en-US.linux-x86_64.tar.xz` (81,620,684 bytes) and
Owl is 1.5.6.1 (430,201 bytes), each pinned by sha256 in `module.json`. The build seat fetches each once,
refuses it when the digest differs, keeps it under its sha256, and packs it into the artifact store; a node
receives the mesh's archive by digest and never asks the vendor. **Owl is never in this repository**: its
licence forbids copying it to third parties, and its vendor serves only its newest version, at one
address. Keeping the pinned copy in the mesh's private artifact store for the operator's own machines was
the operator's call (2026-10-12, ADR 0316 §6).
**Owl's range.** Owl declares which Betterbird versions it supports (`strict_min_version` to
`strict_max_version`: 115 to 156 for 1.5.6.1). `owlSupports` in `manifest_test.go` holds the range read
from each pinned xpi, and the module's tests refuse a Betterbird pin outside it: moving Owl's pin means
reading the new range there (ADR 0316 §8).
**The policy** turns Betterbird's own updater off (`DisableAppUpdate`, which Betterbird's update check
honours) and installs Owl into every profile from `file:///opt/betterbird/owl/owl.xpi` as
`force_installed`, with `updates_disabled`. A profile that already runs another Owl keeps it: the policy
installs the pinned one where Owl is missing. A newer Owl in the profile (g14 runs 1.5.7) is said as a
fact by `betterbird_status`; Owl installed by policy cannot be removed in Betterbird, so the remedy is
moving the pin. An older one is a finding of `betterbird_check`, with the same remedy.
**Betterbird never asks to be the default client.** The policy locks `mail.shell.checkDefaultClient` off
(`Preferences`, `Status: locked`): Betterbird's own default (`all-thunderbird.js`) is to ask at every start,
with a window to make it the default for mail, news, feeds and calendar files, while that preference is true
(`messenger.js`). Betterbird 153 has no policy of its own for it (its `policies-schema.json`), and its
`Preferences` policy accepts the `mail.` prefix. A lock, not a default: the window's own "always check" box
writes the preference into the profile, which wins over a default; a lock wins over the profile and greys the
box out in the settings. `betterbird_check` says when Betterbird would ask (novox/hq issue 345). **Owl's licence stays the person's**: the mesh holds no licence, key or payment (ADR 0316 §9).
**No start of its own.** Betterbird is started by the operator from the launcher. The start
(`/usr/local/bin/betterbird`) first runs `betterbird-tools prepare`, which, with no Betterbird of the
account running, places the user.js and writes the created accounts into the account lists; then it
starts Betterbird either way. `betterbird_restart` starts it as the transient user unit
`mesh-betterbird`, so its output then goes to the journal.
Nothing else. It holds no seat and adds no start.
- **No package.** g14 and shanks both run `betterbird-bin` 153.1.0esr-1, installed explicitly from the
AUR: `pacman` counts it as foreign. The node-engine installs packages from the official
repositories only, so the module does not declare it, and **installing and upgrading Betterbird stay
the operator's** (an AUR helper). `betterbird_status` says the version and that it is from the AUR;
`betterbird_check` says when it is missing.
- **No start.** Betterbird is started by the operator from the launcher (on the workstation measured,
it runs in the login session's scope, started by hand, with no autostart entry and no line in i3's
configuration). `betterbird_restart` starts it as the transient user unit `mesh-betterbird`, so its
output then goes to the journal.
- **The profile is Betterbird's** (ADR 0182): `~/.thunderbird/` holds the mail, the address books,
the calendars' caches, the logins and the keys. The module declares none of it.
## The user.js: how a preference is declared
@@ -95,104 +52,33 @@ default, the one in use, is the other.
what to do. The tools refuse a declared file that holds anything but comments and `user_pref`
statements, or that sets a preference whose name can hold a credential.
The file is **trusted** (`"trusted": true`): a preference can weaken how the mail client checks a server,
so the setting is changed only at the controller's terminal or proposed and set on the operator's warrant
(novox/hq issue 339, ADR 0277). An account's preferences (`mail.account.`, `mail.identity.`, `mail.server.`,
`mail.smtpserver.` and the account lists) are refused in `prefs`: an account is declared in `accounts`, so
it has one owner (ADR 0316 §11).
## The accounts: declared without credentials (ADR 0316 §10–§16, ADR 0317)
The setting `accounts` reaches the machine as `~/.config/betterbird/accounts.json`, a trusted mergeable
file: a map of accounts by name, each with `address`, `from-name` and `label` (optional), `incoming` (`kind`
imap or pop3, `host`, `port`, `security` tls, starttls or none, `sign-in` password, encrypted-password,
oauth, kerberos, ntlm or tls-certificate, `user`) and `outgoing` (the same, without a kind; its sign-in may
be none), and, only in a node's own layer, `key`. **Never a password or a token**: the whole file is
refused when any key in it is shaped like one, and an account signs in at its first use, in Betterbird's
own window, where Betterbird keeps the result in its login store. The work account through Owl is never
declared: the person creates it in Owl's setup (ADR 0316 §13).
At every apply (the start from the launcher, `betterbird_user_js` with apply, `betterbird_restart`), each
declared account, in its place in the setting, is (ADR 0317 §1):
1. **kept by its recorded keys**, when the module recorded it on this node for this profile;
2. **its creation**, when it is under the module's own keys (`account_mesh_<name>`, `id_mesh_<name>`,
`server_mesh_<name>`, `smtp_mesh_<name>`);
3. **adopted by the key a node's layer names** (`key`), when that account is in the list, its default
identity has the address and it is not Owl's;
4. **adopted by its address**, when exactly one account's default identity has it: its keys (account,
default identity, incoming server, and the outgoing server that identity names) become the module's, and
the account lists are left as they are;
5. **created** under the module's own keys when no account has it: its keys are added at the end of
`mail.accountmanager.accounts` and `mail.smtpservers` in `prefs.js`, every other key kept in its place,
only while Betterbird is not running.
**Refused, for that account alone and with the remedy** (§7): two accounts with the address (remove the
duplicate in Betterbird, or name the `key`), Owl's account at the address, a named key that does not
qualify, an account of another kind than declared (never converted), an identity that names no outgoing
server, and an outgoing server that another identity, neither adopted nor the module's, also uses
(Betterbird lets identities share one, and the declared values would change that other account's sending;
ADR 0317 §2 leaves the identity's server and the lists as they are, so the module refuses rather than give
it a server of its own). A recorded adoption follows the identity's current outgoing server at every apply,
under the same rule. The other accounts go ahead; the start and `betterbird_check` say it.
`betterbird_user_js` answers an account's preferences by name with their values withheld.
The declared fields are written in the user.js under the account's keys, so they win at every start; only
the fields the setting names, never the account's directory, folders, filters, caches, mail or kept
sign-in, so an adopted account keeps its sign-in (§5). The apply names, by preference name only, each one
whose value it changed (§4). What was adopted or created is recorded in
`~/.config/betterbird/accounts-record.json`. **Leaving the setting** releases an adopted account (its
lines and its record go, the account stays the person's) and takes a created account's keys out of the
lists (its mail stays in the profile).
**The value is derived, not retyped** (§9): `betterbird_accounts` with `propose` builds `accounts` from the
profile (every account but Owl's, no credential) and proposes it through the controller's settings verb;
the operator reads it whole at their desk (`settings proposals <id>`) and approves on a channel that proves
who answers. Its answer to its caller holds counts and kinds only.
**A node with no profile yet.** The preparation needs the profile Betterbird makes at its first start: on a
node set up from nothing, the declared accounts are written at the second start from the launcher (or at
`betterbird_restart`).
The file is **trusted** (it carries no `"trusted": false`): a preference can weaken how the mail client
checks a server, so the setting is changed only at the controller's terminal, never through a verb
(novox/hq issue 339).
## Tools
They are served by the node's runtime as the operator account (ADR 0175). **No answer carries mail, a
calendar entry, an address, a host, an account's name or a credential**: accounts are said by count, place
and key.
calendar entry, an account, an address or a credential.**
| tool | does |
|---|---|
| `betterbird_status` (r) | whether it runs (pid, since, scope, children); what is installed and from where (the mesh's release, or the community package before the switch), its version and build beside the newest published build (Betterbird's own `versions.txt`); the policy; Owl's version against the pin and whether it is active; where its output goes; the profile and how it was found; the user.js (declared, placed, in effect); the declared accounts by count and, per place, adopted, created or refused with their keys; the add-ons in the profile (ids, versions); the preferences that decide how its windows and notifications look |
| `betterbird_status` (r) | whether it runs (pid, since, scope, children), the package and version and that it is from the AUR, where its output goes, the profile and how it was found, the user.js (declared, placed, in effect), the add-ons the operator installed (ids), and the preferences that decide how its windows and notifications look |
| `betterbird_prefs` (r) | named preferences or prefixes (ending in `.` or `*`, at most 200; an empty name, `*` or `.` is refused): value and source (user.js, prefs.js or the default). Withheld without its value: a name that can hold a credential (`password`, `token`, `oauth`, `signon`, …); one that names an account, server, calendar, recent folder or person (`mail.server.`, `mail.identity.`, `mail.last_msg…`, `messenger.account.`, `mail.cloud_files.`, `calendar.registry.`, …); and every `extensions.*` but the theme's id (`extensions.activeThemeID`). In every value answered, the home is `~`, and a mail address, a `user%40domain@host` URI and a host name are masked |
| `betterbird_user_js` (r; a with `apply`) | the declared user.js against the profile's; `apply` places it |
| `betterbird_log` (r) | the last lines of the journal of `mesh-betterbird` (or of the reminder test), addresses and credentials masked (the value after `password=`, `token:` and the like, and the token after `Bearer`/`Basic`). A Betterbird the module did not start prints elsewhere, and the answer says where |
| `betterbird_restart` (a) | ends Betterbird (asked to quit, forced after 8 s), places the user.js where allowed, starts it as `mesh-betterbird`. **Refused while a composer window is open** (X instance `Msgcompose`, whatever its class), because the unsent message would be lost, unless `discard_unsent` is true, which only the operator's yes justifies. It answers within the runtime's 30 s: a start not yet seen after 4 s is said in `note`, and `betterbird_status` shows it |
| `betterbird_check` (r) | installed from the mesh's release and not behind the newest published build; the policy placed; no ask at the start to be the default mail client; Owl at its pin and active; one running; profile found; the user.js on the machine, valid, placed, in effect; the declared accounts in the list, none refused, each signed in once; its output read |
| `betterbird_accounts` (r; a with `propose`) | the profile's accounts by count and kind (declarable, Owl's, others); with `propose`, the setting `accounts` built from them and proposed to the operator (ADR 0317 §9) |
| `betterbird_check` (r) | installed; one running; profile found; the user.js on the machine, valid, placed, in effect; its output read |
| `betterbird_reminder_test` (a) | a reminder on demand without the operator's calendar (below) |
### A reminder on demand
**It opens windows on the operator's screen**, and its description says so first, so a caller tells the
operator before calling it. On 2026-10-12 at 12:05 a test Betterbird with an empty profile carrying the
operator's theme looked like the operator's configuration lost; so the test is **visibly a test**: its
profile carries a `userChrome.css` that frames every window of the test Betterbird in red and runs a red
band reading MESH TEST across its top, and its event is called "MESH TEST reminder (a throw-away
Betterbird of the mesh, not your calendar)". The reminder window itself is left unstyled, because how
Betterbird draws it is what the test looks at; its title is the event's. A window title cannot be set by a
preference, so the marking is the band, the frame and the event's name.
`betterbird_reminder_test` starts a **second** Betterbird with `--no-remote` in a throw-away profile
under `/run/user/<uid>/mesh-betterbird-reminder-test/`, as the user unit
`mesh-betterbird-reminder-test` (ended after 10 minutes at most). The profile holds the module's
user.js (the setting `prefs`; never a declared account) and one calendar: a local `.ics` file with one
event, "MESH TEST reminder", whose reminder is
user.js and one calendar: a local `.ics` file with one event, "Mesh reminder test", whose reminder is
due `after` seconds from the call (default 40). `carry` copies named preferences of the operator's
profile into it (a theme, a colour switch), under the same limits as `betterbird_prefs`. The test's own
preferences come last in its user.js, so nothing declared or carried overrides them; one of them turns
Betterbird's update check off (`betterbird.update.checkIntervalDays` 0), because a fresh profile checks
at its start and, when a newer build is published, opens the About window on the operator's screen
(seen 2026-10-12). Then:
profile into it (a theme, a colour switch), under the same limits as `betterbird_prefs`. Then:
- `xorg_windows` with class `Calendar` answers the reminder window: role `AlarmWindow`, type `Dialog`,
its depth (32: a visual with alpha, which the compositor blends), its map state;
@@ -220,19 +106,13 @@ A window of depth 32 that its program has not drawn is transparent under a compo
that is why the symptom looks the way it does. Why Betterbird did not draw it is hq issue 345's
question.
## What changes when it is pushed
## What changes when it is assigned
The module is assigned to shanks today (look-only, from issue 345's first part); g14 runs Betterbird
outside the mesh. With this version, a push to a node (each waits for the operator: `upgrade: record`):
| | on a node with the community package (g14, shanks) |
|---|---|
| the package | `betterbird-bin` removed (quit Betterbird first) |
| `/opt/betterbird` | the mesh's 153.4.0esr-bb10, in the same place, so the same profile opens |
| the policy, Owl | placed; Owl installed into the profile by Betterbird at its next start where it is missing |
| `/usr/local` | the start, the launcher entry, the tools' program |
| `~/.config/betterbird/` | `user.js`, `accounts.json` (no account until the setting `accounts` is approved) |
| the profile | untouched until the next start from the launcher, `betterbird_user_js` with apply, or `betterbird_restart` |
| | g14 | shanks |
|---|---|---|
| files | `~/.config/betterbird/` and its `user.js` (no preference) | the same |
| the profile | untouched until `betterbird_user_js` with `apply` or `betterbird_restart` | the same; the profile has no user.js today, so `apply` may place one |
| Betterbird | keeps running as started | keeps running as started (since 2026-10-04, its output on the session's pipe nobody reads: `betterbird_check` names it until `betterbird_restart`) |
## Relies on
@@ -240,6 +120,3 @@ outside the mesh. With this version, a push to a node (each waits for the operat
- **`xorg`** for `xorg_windows` and `xorg_screenshot`, which look at Betterbird's windows. This module
carries no window code of its own.
- `systemd-run` and `journalctl` of the account's user manager.
- The build seat's archive pinned by URL and sha256 (mesh-controller, ADR 0316 §2): a controller that does
not know it refuses this manifest.
- The controller's `settings` verb, for `betterbird_accounts` with `propose` (`invokes`).
@@ -1,811 +0,0 @@
package main
// The operator's mail accounts, declared without credentials (novox/hq ADR 0316 §10–§14, to-be 56 §4).
//
// The setting `accounts` reaches the machine as ~/.config/betterbird/accounts.json, a file the node-engine
// writes from the module's settings layers. Per account it names the address, the incoming server and the
// outgoing server; never a password or a token, and the whole file is refused when any key in it is shaped
// like one. Betterbird's accounts are preferences, so a declared account becomes:
//
// - its declared fields as preferences in the user.js the module places, where they win at every start
// (§11): under the keys of the account the profile already holds at that address, which the module adopts
// (novox/hq ADR 0317), or, where the profile holds none, under keys of the module's own that end in
// mesh_<name> (account_mesh_<name>, id_mesh_<name>, server_mesh_<name>, smtp_mesh_<name>);
// - for an account the module created, its key in the account list (mail.accountmanager.accounts) and in the
// list of outgoing servers (mail.smtpservers) of the profile's prefs.js: written into, never owned, and
// only while Betterbird is not running, because it rewrites prefs.js when it quits (§12). Every other key
// keeps its place, and an adopted account's keys are in the lists already.
//
// Which account is adopted, created or refused is adoption.go's.
//
// Nobody's password passes through here: an account signs in at its first use, in Betterbird's own window,
// and Betterbird keeps the result in its login store (§14). Of that store the module reads only the server
// names each entry is kept for, which Betterbird writes in the clear, to count the declared accounts that
// have not signed in yet (§16); never a user name or a password, which are encrypted there anyway.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// accountsFile is the setting accounts on the machine, written by the node-engine.
const accountsFile = ".config/betterbird/accounts.json"
// Account is one declared mail account, in the shape of the setting accounts: keyed by its name.
type Account struct {
Address string `json:"address"`
// FromName is the name the account's mail is sent under. Absent: Betterbird's, none.
FromName string `json:"from-name,omitempty"`
// Label is what Betterbird shows the account as. Absent: its address.
Label string `json:"label,omitempty"`
Incoming Server `json:"incoming"`
Outgoing Server `json:"outgoing"`
// Key is the profile's account key to adopt, given only in a node's own layer to settle an address the
// profile holds more than once (novox/hq ADR 0317 §8).
Key string `json:"key,omitempty"`
}
// Server is an account's incoming or outgoing server: never a password or a token.
type Server struct {
// Kind is imap or pop3, for the incoming server; the outgoing one is smtp and says nothing.
Kind string `json:"kind,omitempty"`
Host string `json:"host"`
Port int `json:"port,omitempty"`
Security string `json:"security"`
SignIn string `json:"sign-in"`
User string `json:"user,omitempty"`
}
// The words a server's connection security and sign-in method are said in, and Betterbird's numbers for
// them (socketType / try_ssl, authMethod).
var (
securities = map[string]int{"none": 0, "starttls": 2, "tls": 3}
signIns = map[string]int{"none": 1, "password": 3, "encrypted-password": 4, "kerberos": 5, "ntlm": 6,
"tls-certificate": 7, "oauth": 10}
)
func wordFor(words map[string]int, n int) string {
for w, v := range words {
if v == n {
return w
}
}
return ""
}
var (
accountName = regexp.MustCompile(`^[a-z][a-z0-9]{0,30}$`)
addressShape = regexp.MustCompile(`^[^\s@"\\]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$`)
hostShape = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9.\-]*[A-Za-z0-9])?$`)
profileKey = regexp.MustCompile(`^[A-Za-z0-9_]{1,64}$`)
// plainText is a value written into a preference: no quote, backslash or line break.
plainText = regexp.MustCompile(`^[^"\\\n\r]*$`)
// secretKey is a key the accounts file may never carry, at any depth: the setting holds no credential.
secretKey = regexp.MustCompile(`(?i)(pass|secret|token|oauth.?refresh|refresh|credential|api.?key|private.?key|cookie|bearer)`)
)
// keysOf are the module's own keys for a declared account: each ends in mesh_<name>.
func keysOf(name string) (account, identity, server, smtp string) {
return "account_mesh_" + name, "id_mesh_" + name, "server_mesh_" + name, "smtp_mesh_" + name
}
// ownKey is a key of an account list the module wrote, and the name it was declared under.
var ownKey = regexp.MustCompile(`^(?:account|smtp)_mesh_([a-z][a-z0-9]{0,30})$`)
// readAccounts reads the declared accounts. A missing file is no account (the module is older here); a file
// that carries a key shaped like a credential, an unknown field or a value that cannot be written is refused
// whole, naming the key and never a value.
func (m *Machine) readAccounts() (map[string]Account, bool, error) {
raw, err := readBounded(m.home(accountsFile))
if errors.Is(err, os.ErrNotExist) {
return map[string]Account{}, false, nil
}
if err != nil {
return nil, false, err
}
accounts, err := parseAccounts(raw)
return accounts, true, err
}
func parseAccounts(raw []byte) (map[string]Account, error) {
var tree any
if err := json.Unmarshal(raw, &tree); err != nil {
return nil, fmt.Errorf("~/%s is not JSON: %v", accountsFile, err)
}
if where := secretKeyIn(tree, ""); where != "" {
// An account's name is said by nothing: its segment of the path is withheld.
if parts := strings.SplitN(where, ".", 3); len(parts) == 3 && parts[0] == "accounts" {
where = "accounts.<an account>." + parts[2]
}
return nil, fmt.Errorf("the setting accounts carries %s, a key shaped like a credential: an account is declared "+
"without its password or token, which it signs in with at its first use (novox/hq ADR 0316 §10, §14). "+
"Take it out at the controller's terminal", where)
}
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
return nil, fmt.Errorf("~/%s is not an object: %v", accountsFile, err)
}
accounts := map[string]Account{}
if body, ok := top["accounts"]; ok && string(body) != "null" {
dec := json.NewDecoder(bytes.NewReader(body))
dec.DisallowUnknownFields()
if err := dec.Decode(&accounts); err != nil {
return nil, fmt.Errorf("the setting accounts is not a map of accounts by name: %v", err)
}
}
var problems []string
names := sortedNames(accounts)
for i, name := range names {
problems = append(problems, accountProblems(fmt.Sprintf("declared account %d of %d", i+1, len(names)),
name, accounts[name])...)
}
if len(problems) > 0 {
return nil, fmt.Errorf("the setting accounts is refused: %s", strings.Join(problems, "; "))
}
return accounts, nil
}
// secretKeyIn is the path of the first key shaped like a credential in a JSON tree, or "".
func secretKeyIn(node any, at string) string {
switch v := node.(type) {
case map[string]any:
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
here := k
if at != "" {
here = at + "." + k
}
if secretKey.MatchString(k) {
return here
}
if w := secretKeyIn(v[k], here); w != "" {
return w
}
}
case []any:
for i, x := range v {
if w := secretKeyIn(x, fmt.Sprintf("%s[%d]", at, i)); w != "" {
return w
}
}
}
return ""
}
func sortedNames(accounts map[string]Account) []string {
names := make([]string, 0, len(accounts))
for n := range accounts {
names = append(names, n)
}
sort.Strings(names)
return names
}
// accountProblems is what is wrong with one declared account, said of it by its place (label) and never by a
// value of it, its name included (novox/hq ADR 0316 §16).
func accountProblems(label, name string, a Account) []string {
var out []string
say := func(format string, args ...any) { out = append(out, label+": "+fmt.Sprintf(format, args...)) }
if !accountName.MatchString(name) {
say("an account's name is lower-case letters and digits, starting with a letter (it names the module's keys, mesh_<name>)")
}
if !addressShape.MatchString(a.Address) {
say("the address is not a mail address")
}
if a.Key != "" && !profileKey.MatchString(a.Key) {
say("key is an account key of the profile, such as account2")
}
for field, v := range map[string]string{"from-name": a.FromName, "label": a.Label} {
if !plainText.MatchString(v) {
say("%s holds a quote, a backslash or a line break", field)
}
}
for side, s := range map[string]Server{"incoming": a.Incoming, "outgoing": a.Outgoing} {
switch {
case side == "incoming" && s.Kind != "imap" && s.Kind != "pop3":
say("the incoming server's kind is imap or pop3")
case side == "outgoing" && s.Kind != "" && s.Kind != "smtp":
say("the outgoing server is smtp")
}
if !hostShape.MatchString(s.Host) {
say("the %s server's host is not a host name", side)
}
if s.Port < 0 || s.Port > 65535 {
say("the %s server's port is not a port", side)
}
if _, ok := securities[s.Security]; !ok {
say("the %s server's security is tls, starttls or none", side)
}
if _, ok := signIns[s.SignIn]; !ok || (side == "incoming" && s.SignIn == "none") {
say("the %s server's sign-in is password, encrypted-password, oauth, kerberos, ntlm or tls-certificate%s",
side, map[bool]string{true: ", or none", false: ""}[side == "outgoing"])
}
if !plainText.MatchString(s.User) {
say("the %s server's user holds a quote, a backslash or a line break", side)
}
}
return out
}
// quoted is a preference's string value as user.js writes it.
func quoted(s string) string { return strconv.Quote(s) }
// declaredPrefs are a declared account's fields as preferences under the keys it resolved to, name to value
// as user.js writes it, in a fixed order. Only the fields the setting names (novox/hq ADR 0317 §5): never
// the account's directory, folders, filters or caches. An adopted account keeps its kind, the outgoing server
// its identity names, and its own name and sender's name where the declaration says none; a created one
// gets every field.
func declaredPrefs(k keySet, a Account, adopted bool) [][2]string {
label := a.Label
if label == "" && !adopted {
label = a.Address
}
inPort := a.Incoming.Port
if inPort == 0 {
inPort = defaultPort(a.Incoming.Kind, a.Incoming.Security)
}
outPort := a.Outgoing.Port
if outPort == 0 {
outPort = defaultPort("smtp", a.Outgoing.Security)
}
var out [][2]string
add := func(key, value string) { out = append(out, [2]string{key, value}) }
if !adopted {
add("mail.account."+k.Account+".identities", quoted(k.Identity))
add("mail.account."+k.Account+".server", quoted(k.Server))
}
if a.FromName != "" || !adopted {
add("mail.identity."+k.Identity+".fullName", quoted(a.FromName))
}
add("mail.identity."+k.Identity+".useremail", quoted(a.Address))
if !adopted {
add("mail.identity."+k.Identity+".smtpServer", quoted(k.SMTP))
add("mail.identity."+k.Identity+".valid", "true")
add("mail.server."+k.Server+".type", quoted(a.Incoming.Kind))
}
add("mail.server."+k.Server+".hostname", quoted(a.Incoming.Host))
add("mail.server."+k.Server+".port", strconv.Itoa(inPort))
add("mail.server."+k.Server+".socketType", strconv.Itoa(securities[a.Incoming.Security]))
add("mail.server."+k.Server+".authMethod", strconv.Itoa(signIns[a.Incoming.SignIn]))
add("mail.server."+k.Server+".userName", quoted(a.Incoming.User))
if label != "" {
add("mail.server."+k.Server+".name", quoted(label))
}
add("mail.smtpserver."+k.SMTP+".hostname", quoted(a.Outgoing.Host))
add("mail.smtpserver."+k.SMTP+".port", strconv.Itoa(outPort))
add("mail.smtpserver."+k.SMTP+".try_ssl", strconv.Itoa(securities[a.Outgoing.Security]))
add("mail.smtpserver."+k.SMTP+".authMethod", strconv.Itoa(signIns[a.Outgoing.SignIn]))
add("mail.smtpserver."+k.SMTP+".username", quoted(a.Outgoing.User))
if label != "" {
add("mail.smtpserver."+k.SMTP+".description", quoted(label))
}
return out
}
// defaultPort is the port a server listens on when the declaration says none.
func defaultPort(kind, security string) int {
switch {
case kind == "imap" && security == "tls":
return 993
case kind == "imap":
return 143
case kind == "pop3" && security == "tls":
return 995
case kind == "pop3":
return 110
case security == "tls":
return 465
case security == "starttls":
return 587
}
return 25
}
// composeUserJS is the user.js the module places in the profile: the declared file (the setting prefs), then
// each adopted or created account's declared fields under its keys. A refused account has none.
func composeUserJS(declaredText string, accounts map[string]Account, resolved []resolvedAccount) string {
var lines []string
for _, r := range resolved {
if r.How == refused {
continue
}
lines = append(lines, fmt.Sprintf("// declared account %d: %s under %s", r.Place, r.How, r.Keys.Account))
for _, p := range declaredPrefs(r.Keys, accounts[r.name], r.How == adopted) {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(p[0]), p[1]))
}
}
if len(lines) == 0 {
return declaredText
}
return strings.TrimRight(declaredText, "\n") + "\n\n" +
"// The accounts the setting accounts declares (novox/hq ADR 0316 §11, ADR 0317): the fields it names, under\n" +
"// the keys of the account adopted or created. No password or token: an account signs in at its first use.\n" +
strings.Join(lines, "\n") + "\n"
}
// profileAccount is one account the profile holds, as its preferences say it.
type profileAccount struct {
Key, Server, Kind string
// Identities are its identities, the default first, and Addresses their addresses in that order.
Identities []string
Addresses []string
Owl, Ours bool
}
// profileAccounts reads the accounts the profile's list names, from prefs.js and user.js (user.js wins).
func profileAccounts(prefs map[string]string) []profileAccount {
var out []profileAccount
for _, key := range listOf(prefs["mail.accountmanager.accounts"]) {
a := profileAccount{Key: key, Server: unquote(prefs["mail.account."+key+".server"])}
a.Kind = unquote(prefs["mail.server."+a.Server+".type"])
a.Owl = strings.HasPrefix(a.Kind, "owl")
a.Ours = ownKey.MatchString(key)
for _, id := range listOf(prefs["mail.account."+key+".identities"]) {
a.Identities = append(a.Identities, id)
a.Addresses = append(a.Addresses, unquote(prefs["mail.identity."+id+".useremail"]))
}
out = append(out, a)
}
return out
}
// mergedPrefs are a profile's preferences as Betterbird starts with them: prefs.js, then user.js over it.
func mergedPrefs(profile string) map[string]string {
prefsJS, _ := readBounded(filepath.Join(profile, "prefs.js"))
userJS, _ := readBounded(filepath.Join(profile, "user.js"))
out := readPrefs(prefsJS)
for k, v := range readPrefs(userJS) {
out[k] = v
}
return out
}
// listOf reads a comma-separated list preference's value.
func listOf(v string) []string {
var out []string
for _, k := range strings.Split(unquote(v), ",") {
if k = strings.TrimSpace(k); k != "" {
out = append(out, k)
}
}
return out
}
// unquote is a preference's string value without its quotes; any other value as written.
func unquote(v string) string {
if s, err := strconv.Unquote(v); err == nil {
return s
}
return v
}
// writeAccountLists writes the module's keys into the profile's account list and list of outgoing servers
// (§12): the declared accounts' keys added at the end, the module's keys of accounts no longer declared taken
// out, every other key and line kept as it was. Only while Betterbird is not running. Says what changed.
func (m *Machine) writeAccountLists(profile string, resolved []resolvedAccount) (string, error) {
if ps := m.mainProcs(); len(ps) > 0 {
return "", fmt.Errorf("Betterbird is running (pid %d), and it rewrites prefs.js when it quits: the account "+
"list is written at its next start from the launcher, or by betterbird_restart", ps[0].PID)
}
path := filepath.Join(profile, "prefs.js")
raw, err := readBounded(path)
if err != nil && !errors.Is(err, os.ErrNotExist) {
return "", err
}
var wantAccounts, wantSMTP []string
for _, r := range resolved {
if r.How == created {
wantAccounts, wantSMTP = append(wantAccounts, r.Keys.Account), append(wantSMTP, r.Keys.SMTP)
}
}
out, changed := withListed(string(raw), map[string][]string{
"mail.accountmanager.accounts": wantAccounts, "mail.smtpservers": wantSMTP})
if !changed {
return "unchanged", nil
}
mode := os.FileMode(0o600)
if st, err := os.Stat(path); err == nil {
mode = st.Mode().Perm()
}
tmp := path + ".mesh-new"
if err := os.WriteFile(tmp, []byte(out), mode); err != nil {
return "", err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return "", err
}
return fmt.Sprintf("written: %d account(s) the module created in the account list", len(wantAccounts)), nil
}
// withListed is prefs.js with each list preference holding the module's wanted keys: its own keys that are
// not wanted taken out, the wanted ones not there added at its end, everything else as it was. A list the
// file does not hold yet is added as a line, when anything is wanted in it.
func withListed(text string, want map[string][]string) (string, bool) {
changed := false
done := map[string]bool{}
var lines []string
s := bufio.NewScanner(strings.NewReader(text))
s.Buffer(make([]byte, 64<<10), 4<<20)
for s.Scan() {
line := s.Text()
if mm := prefLine.FindStringSubmatch(line); mm != nil {
if keys, ok := want[mm[1]]; ok && !done[mm[1]] {
done[mm[1]] = true
next := listedWith(listOf(mm[2]), keys)
if next != strings.Join(listOf(mm[2]), ",") {
line = fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), quoted(next))
changed = true
}
}
}
lines = append(lines, line)
}
names := make([]string, 0, len(want))
for n := range want {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
if !done[n] && len(want[n]) > 0 {
lines = append(lines, fmt.Sprintf("user_pref(%s, %s);", quoted(n), quoted(strings.Join(want[n], ","))))
changed = true
}
}
out := strings.Join(lines, "\n")
if text == "" || strings.HasSuffix(text, "\n") {
out += "\n"
}
return out, changed
}
// listedWith is a list with the module's own keys made the wanted ones, every other key in its place.
func listedWith(have, wanted []string) string {
want := map[string]bool{}
for _, k := range wanted {
want[k] = true
}
var out []string
there := map[string]bool{}
for _, k := range have {
if ownKey.MatchString(k) && !want[k] {
continue
}
out = append(out, k)
there[k] = true
}
for _, k := range wanted {
if !there[k] {
out = append(out, k)
}
}
return strings.Join(out, ",")
}
// AccountCount is what status and check say of the accounts: numbers, places and keys, never an address, a
// host or a name (novox/hq ADR 0316 §16, ADR 0317 §3).
type AccountCount struct {
Declared int `json:"declared"`
// InList are the declared accounts the profile's account list names, adopted or created.
InList int `json:"declared_in_the_list"`
// Others are the accounts the profile holds that no declaration names (the work account in Owl among them).
Others int `json:"other_accounts"`
// WithoutSignIn are the declared accounts with no sign-in kept in Betterbird's login store yet.
WithoutSignIn int `json:"declared_without_a_sign_in,omitempty"`
// Each is every declared account by its place in the setting: adopted, created or refused, and its keys.
Each []resolvedAccount `json:"each,omitempty"`
Problem string `json:"problem,omitempty"`
}
// countAccounts counts the declared accounts against the profile's.
func (m *Machine) countAccounts(profile, profileDir string) AccountCount {
var c AccountCount
accounts, _, err := m.readAccounts()
if err != nil {
c.Problem = err.Error()
return c
}
c.Declared = len(accounts)
prefs := mergedPrefs(profile)
have := profileAccounts(prefs)
c.Each = resolveAccounts(accounts, prefs, m.readRecord(profileDir))
mine := map[string]bool{}
refusedN := 0
for _, r := range c.Each {
if r.How == refused {
refusedN++
continue
}
mine[r.Keys.Account] = true
}
for _, a := range have {
switch {
case mine[a.Key]:
c.InList++
case a.Kind != "none":
c.Others++
}
}
if refusedN > 0 {
c.Problem = fmt.Sprintf("%d declared account(s) are refused: each says why and what to do", refusedN)
}
c.WithoutSignIn = len(withoutSignIn(accounts, keptSignIns(profile)))
return c
}
// keptSignIns are the server names Betterbird's login store keeps a sign-in for: the `hostname` of each
// entry of logins.json, which Betterbird writes in the clear. Nothing else of the file is read.
func keptSignIns(profile string) map[string]bool {
raw, err := readBounded(filepath.Join(profile, "logins.json"))
out := map[string]bool{}
if err != nil {
return out
}
var store struct {
Logins []struct {
Hostname string `json:"hostname"`
} `json:"logins"`
}
if json.Unmarshal(raw, &store) != nil {
return out
}
for _, l := range store.Logins {
out[strings.ToLower(l.Hostname)] = true
}
return out
}
// oauthIssuers are the sign-in services of the providers whose servers sign in through OAuth, by a suffix of
// the server's host: their kept sign-in is under the service's name, not the server's.
var oauthIssuers = map[string]string{"gmail.com": "oauth://accounts.google.com",
"googlemail.com": "oauth://accounts.google.com", "office365.com": "oauth://login.microsoftonline.com",
"outlook.com": "oauth://login.microsoftonline.com"}
// withoutSignIn are the declared accounts whose incoming server has no sign-in kept yet. A server signed in
// to without a password (none, kerberos, ntlm, a certificate) keeps nothing and is never counted.
func withoutSignIn(accounts map[string]Account, kept map[string]bool) []string {
var out []string
for _, name := range sortedNames(accounts) {
in := accounts[name].Incoming
switch in.SignIn {
case "password", "encrypted-password":
if !kept[strings.ToLower(in.Kind+"://"+in.Host)] {
out = append(out, name)
}
case "oauth":
issuer := ""
for suffix, i := range oauthIssuers {
if strings.HasSuffix(strings.ToLower(in.Host), suffix) {
issuer = i
}
}
found := false
for k := range kept {
if (issuer != "" && k == issuer) || (issuer == "" && strings.HasPrefix(k, "oauth://")) {
found = true
}
}
if !found {
out = append(out, name)
}
}
}
return out
}
// AccountsAnswer is what betterbird_accounts answers its caller: counts and kinds, never an address, a host or
// a name (novox/hq ADR 0317 §9). The value itself goes only to the operator, in a proposal they read at
// their own desk.
type AccountsAnswer struct {
Profile string `json:"profile"`
// Declarable are the profile's mail accounts a declaration can hold (IMAP and POP3), by kind.
Declarable map[string]int `json:"declarable"`
// KeptByOwl are the accounts Owl keeps (the work account through Exchange): never declared (ADR 0316 §13).
KeptByOwl int `json:"kept_by_owl"`
// NotDeclarable are accounts of another kind (local folders, feeds, news, chat), by kind.
NotDeclarable map[string]int `json:"not_declarable,omitempty"`
// AlreadyDeclared are the declarable accounts under the module's own keys.
AlreadyDeclared int `json:"under_the_modules_keys"`
// Proposed is the controller's answer to the proposal, with every address and host masked.
Proposed string `json:"proposed,omitempty"`
Note string `json:"note"`
}
// profileDeclarable reads the profile's accounts: the value a declaration of them would be, by name, and
// what is answered of them.
func (m *Machine) profileDeclarable() (map[string]Account, AccountsAnswer, error) {
p, err := m.profile()
if err != nil {
return nil, AccountsAnswer{}, err
}
prefs := mergedPrefs(p.path)
a := AccountsAnswer{Profile: p.Dir, Declarable: map[string]int{}, NotDeclarable: map[string]int{}}
value := map[string]Account{}
used := map[string]bool{}
for _, pa := range profileAccounts(prefs) {
switch {
case pa.Owl:
a.KeptByOwl++
continue
case pa.Kind != "imap" && pa.Kind != "pop3":
kind := pa.Kind
if kind == "none" {
kind = "local folders"
}
a.NotDeclarable[kind]++
continue
}
a.Declarable[pa.Kind]++
if pa.Ours {
a.AlreadyDeclared++
}
address := ""
if len(pa.Addresses) > 0 {
address = pa.Addresses[0]
}
name := nameFor(pa, address, used)
used[name] = true
value[name] = readAccount(prefs, pa)
}
if len(a.NotDeclarable) == 0 {
a.NotDeclarable = nil
}
return value, a, nil
}
// asker calls a verb of the mesh as this module, through the runtime: a variable so a test answers.
var asker = stdio.Ask
// ProfileAccounts says, by count and kind, which accounts the profile in use holds; with propose it builds
// the setting accounts from them (every account but Owl's, without any credential) and proposes it through
// the controller's settings verb for the operator's Approve (novox/hq ADR 0317 §9, ADR 0277). The value
// never comes back to the caller: the operator reads it whole at their desk (`settings proposals <id>`) and
// checks the fingerprint the ask shows.
func (m *Machine) ProfileAccounts(propose bool) (AccountsAnswer, error) {
value, a, err := m.profileDeclarable()
if err != nil {
return a, err
}
a.Note = "counts and kinds only: no address, host or name leaves through this answer. With propose, the " +
"setting accounts is built from these accounts (never Owl's, never a password or token) and proposed to " +
"the operator, who reads it whole at their desk before approving (novox/hq ADR 0317 §9)"
if !propose {
return a, nil
}
if len(value) == 0 {
return a, errors.New("the profile holds no IMAP or POP3 account to declare: nothing was proposed")
}
var problems []string
for i, name := range sortedNames(value) {
if p := accountProblems("", name, value[name]); len(p) > 0 {
problems = append(problems, fmt.Sprintf("account %d of %d: %d field(s) cannot be declared as the profile "+
"holds them", i+1, len(value), len(p)))
}
}
if len(problems) > 0 {
return a, fmt.Errorf("nothing was proposed: %s; betterbird_prefs shows the account's preferences by name",
strings.Join(problems, "; "))
}
body, err := json.Marshal(map[string]any{"accounts": value})
if err != nil {
return a, err
}
raw, err := asker("seat:mesh-controller.settings", map[string]any{"module": "betterbird", "values": string(body),
"propose": "true"})
if err != nil {
return a, fmt.Errorf("the controller's settings verb did not answer: %v", err)
}
said, ok := verbSaid(raw)
a.Proposed = m.maskValue(said)
if !ok {
return a, fmt.Errorf("the controller did not take the proposal: %s", a.Proposed)
}
return a, nil
}
// verbSaid is a verb's words and whether it went ahead, whatever wraps them: the controller's {ok, output},
// or the protocol's content list.
func verbSaid(raw json.RawMessage) (string, bool) {
var s string
if json.Unmarshal(raw, &s) == nil {
return verbSaid(json.RawMessage(s))
}
var m map[string]json.RawMessage
if json.Unmarshal(raw, &m) != nil {
return string(raw), true
}
if content, has := m["content"]; has {
var items []struct {
Text string `json:"text"`
}
var e struct {
IsError bool `json:"isError"`
}
_ = json.Unmarshal(raw, &e)
if json.Unmarshal(content, &items) == nil && len(items) > 0 {
out, ok := verbSaid(json.RawMessage(items[0].Text))
return out, ok && !e.IsError
}
}
if okRaw, has := m["ok"]; has {
var ok bool
_ = json.Unmarshal(okRaw, &ok)
var output string
_ = json.Unmarshal(m["output"], &output)
return output, ok
}
return string(raw), true
}
// readAccount reads one profile account's preferences into the shape of the setting.
func readAccount(prefs map[string]string, pa profileAccount) Account {
num := func(key string, def int) int {
if n, err := strconv.Atoi(prefs[key]); err == nil {
return n
}
return def
}
text := func(key string) string { return unquote(prefs[key]) }
srv := "mail.server." + pa.Server + "."
acc := Account{Incoming: Server{Kind: pa.Kind, Host: text(srv + "hostname"), User: text(srv + "userName"),
Security: wordFor(securities, num(srv+"socketType", 0)), SignIn: wordFor(signIns, num(srv+"authMethod", 3))}}
acc.Incoming.Port = num(srv+"port", defaultPort(pa.Kind, acc.Incoming.Security))
ids := listOf(prefs["mail.account."+pa.Key+".identities"])
if len(ids) > 0 {
id := "mail.identity." + ids[0] + "."
acc.Address, acc.FromName = text(id+"useremail"), text(id+"fullName")
smtp := text(id + "smtpServer")
if smtp == "" {
smtp = text("mail.smtp.defaultserver")
}
if smtp != "" {
s := "mail.smtpserver." + smtp + "."
acc.Outgoing = Server{Host: text(s + "hostname"), User: text(s + "username"),
Security: wordFor(securities, num(s+"try_ssl", 0)), SignIn: wordFor(signIns, num(s+"authMethod", 3))}
acc.Outgoing.Port = num(s+"port", defaultPort("smtp", acc.Outgoing.Security))
}
}
if label := text(srv + "name"); label != "" && label != acc.Address {
acc.Label = label
}
return acc
}
// nameFor is the name a profile account is answered under: its own name when the module declared it, else
// the first label of its address's domain, made unique.
func nameFor(pa profileAccount, address string, used map[string]bool) string {
if mm := ownKey.FindStringSubmatch(pa.Key); mm != nil {
return mm[1]
}
base := "account"
if _, domain, ok := strings.Cut(address, "@"); ok {
label, _, _ := strings.Cut(strings.ToLower(domain), ".")
label = regexp.MustCompile(`[^a-z0-9]`).ReplaceAllString(label, "")
if label != "" && label[0] >= 'a' && label[0] <= 'z' {
base = label
}
}
if len(base) > 28 {
base = base[:28]
}
name := base
for i := 2; used[name]; i++ {
name = base + strconv.Itoa(i)
}
return name
}
@@ -1,518 +0,0 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// The setting accounts (novox/hq ADR 0316 §10–§16, ADR 0317 §9 for betterbird_accounts): declared without
// credentials, written under the module's own keys, the account list written into and never owned, refused
// rather than duplicated, and said by count.
const personal = `{"accounts": {
"personal": {"address": "me@mail.example.org", "from-name": "Me",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password", "user": "me"},
"outgoing": {"host": "smtp.example.org", "port": 587, "security": "starttls", "sign-in": "password", "user": "me"}}
}}`
// profileWithAccounts is a profile holding an account of the person's at another address, Owl's work account,
// local folders, and an account list in that order.
func (f *fake) profileWithAccounts() {
f.betterbird()
f.write(profileDir+"/prefs.js", "// Mozilla User Preferences\n"+
`user_pref("mail.account.account1.server", "server1");`+"\n"+
`user_pref("mail.account.account1.identities", "id1");`+"\n"+
`user_pref("mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account3.server", "server3");`+"\n"+
`user_pref("mail.account.account3.identities", "id3");`+"\n"+
`user_pref("mail.accountmanager.accounts", "account1,account3,account2");`+"\n"+
`user_pref("mail.identity.id1.useremail", "other@mail.example.org");`+"\n"+
`user_pref("mail.identity.id1.fullName", "Other");`+"\n"+
`user_pref("mail.identity.id1.smtpServer", "smtp1");`+"\n"+
`user_pref("mail.identity.id3.useremail", "work@corp.example.com");`+"\n"+
`user_pref("mail.server.server1.type", "imap");`+"\n"+
`user_pref("mail.server.server1.hostname", "imap.example.org");`+"\n"+
`user_pref("mail.server.server1.socketType", 3);`+"\n"+
`user_pref("mail.server.server1.authMethod", 10);`+"\n"+
`user_pref("mail.server.server1.userName", "other");`+"\n"+
`user_pref("mail.server.server1.directory", "/home/operator/.thunderbird/x/ImapMail/imap.example.org");`+"\n"+
`user_pref("mail.server.server2.type", "none");`+"\n"+
`user_pref("mail.server.server3.type", "owl-ews");`+"\n"+
`user_pref("mail.smtpserver.smtp1.hostname", "smtp.example.org");`+"\n"+
`user_pref("mail.smtpserver.smtp1.try_ssl", 3);`+"\n"+
`user_pref("mail.smtpserver.smtp1.authMethod", 10);`+"\n"+
`user_pref("mail.smtpserver.smtp1.username", "other");`+"\n"+
`user_pref("mail.smtpservers", "smtp1");`+"\n"+
`user_pref("mailnews.tags.version", 2);`+"\n")
}
func (f *fake) declareAccounts(body string) {
f.write(testHome+"/"+accountsFile, body)
}
func (f *fake) read(path string) string {
f.t.Helper()
raw, err := os.ReadFile(filepath.Join(f.Root, path))
if err != nil {
f.t.Fatal(err)
}
return string(raw)
}
func TestAnAccountsValueCarryingAPasswordLikeFieldIsRefusedWithoutItsValue(t *testing.T) {
for _, body := range []string{
`{"accounts": {"personal": {"address": "me@mail.example.org", "password": "hunter2"}}}`,
`{"accounts": {"personal": {"incoming": {"oauth_refresh": "1//hunter2"}}}}`,
`{"token": "hunter2", "accounts": {}}`,
} {
_, err := parseAccounts([]byte(body))
if err == nil || !strings.Contains(err.Error(), "shaped like a credential") || strings.Contains(err.Error(), "hunter2") {
t.Errorf("%s: %v", body, err)
}
}
// An unknown field is refused too, and a value that cannot be declared, said by the account's place.
if _, err := parseAccounts([]byte(`{"accounts": {"personal": {"address": "me@mail.example.org", "keychain": 1}}}`)); err == nil {
t.Error("an unknown field was taken")
}
_, err := parseAccounts([]byte(strings.Replace(personal, `"security": "tls"`, `"security": "maybe"`, 1)))
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 1") || strings.Contains(err.Error(), "personal") ||
strings.Contains(err.Error(), "example.org") {
t.Errorf("a bad value is said by the account's place, never its name or a value: %v", err)
}
}
func TestTheSettingPrefsRefusesAnAccountsPreferences(t *testing.T) {
for _, name := range []string{"mail.account.account1.server", "mail.identity.id1.useremail", "mail.server.server1.hostname",
"mail.smtpserver.smtp1.hostname", "mail.accountmanager.accounts", "mail.smtpservers"} {
err := validUserJS(mark + ")\nuser_pref(\"" + name + "\", \"x\");\n")
if err == nil || !strings.Contains(err.Error(), "setting accounts") {
t.Errorf("%s was taken in prefs: %v", name, err)
}
}
if err := validUserJS(mark + ")\nuser_pref(\"calendar.alarms.show\", true);\n"); err != nil {
t.Fatal(err)
}
}
// noPackage answers pacman as a machine without the community package.
func (f *fake) noPackage() {
f.answer = func(name string, args []string) Output {
if name == "pacman" {
return Output{Code: 1}
}
return Output{}
}
}
// userPrefLines are the user_pref lines of a user.js, by preference name.
func userPrefLines(t *testing.T, text string) map[string]string {
t.Helper()
out := map[string]string{}
for _, line := range strings.Split(text, "\n") {
if !strings.HasPrefix(line, "user_pref(") {
continue
}
if !statement.MatchString(line) {
t.Errorf("not a statement Betterbird reads as one: %s", line)
}
out[line[len(`user_pref("`):strings.Index(line, `",`)]] = line
}
return out
}
func TestADeclaredAccountTheProfileDoesNotHoldIsCreatedUnderTheModulesKeysAndListed(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
// Betterbird is running: the user.js is placed, the list waits for its next start.
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(u.Lists, "not written") || !strings.Contains(u.Lists, "running") {
t.Fatalf("the list is never written while Betterbird runs: %+v", u)
}
if len(u.Each) != 1 || u.Each[0].How != created || u.Each[0].Keys != (keySet{"account_mesh_personal", "id_mesh_personal",
"server_mesh_personal", "smtp_mesh_personal"}) {
t.Fatalf("%+v", u.Each)
}
placed := f.read(profileDir + "/user.js")
for name := range userPrefLines(t, placed) {
if key := strings.Split(name, ".")[2]; !strings.HasSuffix(key, "mesh_personal") {
t.Errorf("a created account's preference outside the module's keys: %s", name)
}
if strings.Contains(strings.ToLower(name), "password") || strings.Contains(name, "directory") {
t.Errorf("the module writes no credential and none of Betterbird's own preferences: %s", name)
}
}
for _, want := range []string{`user_pref("mail.server.server_mesh_personal.port", 993);`,
`user_pref("mail.server.server_mesh_personal.socketType", 3);`,
`user_pref("mail.smtpserver.smtp_mesh_personal.try_ssl", 2);`,
`user_pref("mail.identity.id_mesh_personal.useremail", "me@mail.example.org");`} {
if !strings.Contains(placed, want) {
t.Errorf("the user.js lacks %s:\n%s", want, placed)
}
}
// Quit: the list gains the module's keys at its end, every other line as it was.
before := f.read(profileDir + "/prefs.js")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
after := f.read(profileDir + "/prefs.js")
if !strings.Contains(after, `user_pref("mail.accountmanager.accounts", "account1,account3,account2,account_mesh_personal");`) ||
!strings.Contains(after, `user_pref("mail.smtpservers", "smtp1,smtp_mesh_personal");`) {
t.Fatalf("the lists:\n%s", after)
}
bl, al := strings.Split(before, "\n"), strings.Split(after, "\n")
if len(bl) != len(al) {
t.Fatalf("lines were added or lost: %d then %d", len(bl), len(al))
}
for i := range bl {
if bl[i] != al[i] && !strings.Contains(bl[i], "mail.accountmanager.accounts") && !strings.Contains(bl[i], "mail.smtpservers") {
t.Errorf("line %d changed: %q to %q", i+1, bl[i], al[i])
}
}
// The record lost: an account under the module's own keys is still its creation, never adopted.
os.Remove(filepath.Join(f.Root, testHome, recordFile))
u, err = f.userJS(true)
if err != nil || u.Each[0].How != created {
t.Fatalf("without the record, the module's keys still say created: %+v %v", u.Each, err)
}
// The account leaves the setting: only the module's keys go, the mail stays in the profile.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if got := f.read(profileDir + "/prefs.js"); got != before {
t.Fatalf("removing the module's keys left the file otherwise as it was:\n%s", got)
}
}
// adoptable is a declaration of the profile's account1 (other@mail.example.org): equal but for its outgoing
// port, said explicitly, and its incoming user.
const adoptable = `{"accounts": {
"mail": {"address": "Other@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "oauth", "user": "other2"},
"outgoing": {"host": "smtp.example.org", "port": 465, "security": "tls", "sign-in": "oauth", "user": "other"}}}}`
func TestADeclaredAccountTheProfileHoldsIsAdoptedUnderItsOwnKeysAndTheListIsLeftAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
f.declareAccounts(adoptable)
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
before := f.read(profileDir + "/prefs.js")
u, err := f.userJS(true)
if err != nil {
t.Fatal(err)
}
if len(u.Each) != 1 || u.Each[0].How != adopted || u.Each[0].Keys != (keySet{"account1", "id1", "server1", "smtp1"}) {
t.Fatalf("adopted under the profile's keys: %+v", u.Each)
}
// The address's case and the user name differ; the default port declared explicitly is no change. Names
// only, no value.
if strings.Join(u.Each[0].Changed, ",") != "mail.identity.id1.useremail,mail.server.server1.userName" {
t.Fatalf("the changed preferences, by name: %v", u.Each[0].Changed)
}
if f.read(profileDir+"/prefs.js") != before || u.Lists != "unchanged" {
t.Fatalf("an adopted account's keys are in the lists already: nothing written to prefs.js (%s)", u.Lists)
}
lines := userPrefLines(t, f.read(profileDir+"/user.js"))
for name := range lines {
key := strings.Split(name, ".")[2]
if key != "account1" && key != "id1" && key != "server1" && key != "smtp1" {
t.Errorf("an adopted account's preference outside its keys: %s", name)
}
for _, never := range []string{"directory", ".type", "smtpServer", "identities", ".server"} {
if strings.HasSuffix(name, never) {
t.Errorf("the module wrote %s, which is Betterbird's or the account's own", name)
}
}
}
if !strings.Contains(lines["mail.server.server1.userName"], `"other2"`) {
t.Fatalf("the declared value wins at every start: %v", lines)
}
// The status says it by place and keys, never by address, host or name.
s, err := f.Status()
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(s.Accounts)
if s.Accounts.InList != 1 || s.Accounts.Others != 1 || strings.Contains(string(raw), "example") ||
strings.Contains(string(raw), `"mail"`) || !strings.Contains(string(raw), `"account1"`) {
t.Fatalf("%s", raw)
}
// Leaving the setting releases it: its lines and its record go, the account and the lists stay.
f.declareAccounts(`{"accounts": {}}`)
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
if f.read(profileDir+"/prefs.js") != before || strings.Contains(f.read(profileDir+"/user.js"), "server1") ||
strings.Contains(f.read(testHome+"/"+recordFile), "account1") {
t.Fatal("a released account stays the person's, with no line or record of the module's")
}
}
func TestAnAmbiguousOwlsOrOtherKindOfMatchIsRefusedForThatAccountAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account with account1's address.
prefs := f.read(profileDir + "/prefs.js")
prefs = strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account4"`, 1) +
`user_pref("mail.account.account4.server", "server4");` + "\n" +
`user_pref("mail.account.account4.identities", "id4");` + "\n" +
`user_pref("mail.identity.id4.useremail", "other@mail.example.org");` + "\n" +
`user_pref("mail.identity.id4.smtpServer", "smtp4");` + "\n" +
`user_pref("mail.smtpserver.smtp4.hostname", "smtp.example.org");` + "\n" +
`user_pref("mail.server.server4.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
both := strings.Replace(adoptable, `{"accounts": {`, `{"accounts": {
"zz": {"address": "new@mail.example.org",
"incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},`, 1)
f.declareAccounts(both)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "declared account 1 of 2: 2 accounts in the profile have its address (account1, account4)") ||
!strings.Contains(err.Error(), "key") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused by place, with the matches' keys and the remedy, never the address: %v", err)
}
u, _ := f.userJS(false)
if u.Each[0].How != refused || u.Each[1].How != created {
t.Fatalf("the other declared account goes ahead: %+v", u.Each)
}
if strings.Contains(f.read(profileDir+"/user.js"), "server1") || strings.Contains(f.read(profileDir+"/user.js"), "server4") {
t.Fatal("a refused account gets no line")
}
// The node's layer names the key: that one is adopted.
f.declareAccounts(strings.Replace(both, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err != nil {
t.Fatalf("%v", err)
}
if u, _ := f.userJS(false); u.Each[0].How != adopted || u.Each[0].Keys.Account != "account4" {
t.Fatalf("%+v", u.Each)
}
os.Remove(filepath.Join(f.Root, testHome, recordFile))
for key, want := range map[string]string{"account9": "not in the profile's account list", "account3": "Owl's",
"account2": "another address"} {
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "`+key+`",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("key %s: %v", key, err)
}
}
// Owl's account at the address, and an account of another kind than declared.
f.declareAccounts(`{"accounts": {"work": {"address": "work@corp.example.com",
"incoming": {"kind": "imap", "host": "outlook.example.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.example.com", "security": "tls", "sign-in": "oauth"}}}}`)
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "Owl's") {
t.Errorf("Owl's account is never adopted: %v", err)
}
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.server.server4.type", "imap"`, `"mail.server.server4.type", "pop3"`, 1))
f.declareAccounts(strings.Replace(adoptable, `"address": "Other@mail.example.org",`, `"address": "Other@mail.example.org", "key": "account4",`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "never converts") {
t.Errorf("an account of another kind is refused, not converted: %v", err)
}
}
func TestDeclaredAccountsWithoutASignInAreCountedFromTheServerNamesAlone(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
links(t, nil)
f.declareAccounts(`{"accounts": {
"personal": {"address": "me@mail.example.org", "incoming": {"kind": "imap", "host": "imap.example.org", "security": "tls", "sign-in": "password"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "password"}},
"google": {"address": "me@gmail.com", "incoming": {"kind": "imap", "host": "imap.gmail.com", "security": "tls", "sign-in": "oauth"},
"outgoing": {"host": "smtp.gmail.com", "security": "tls", "sign-in": "oauth"}},
"lan": {"address": "me@lan.example.org", "incoming": {"kind": "pop3", "host": "pop.example.org", "security": "tls", "sign-in": "kerberos"},
"outgoing": {"host": "smtp.example.org", "security": "tls", "sign-in": "none"}}}}`)
// The username and password fields are encrypted and never read; a fixture holding junk there proves it.
f.write(profileDir+"/logins.json", `{"logins": [
{"hostname": "imap://imap.example.org", "httpRealm": "imap://imap.example.org", "encryptedUsername": "MEIE...", "encryptedPassword": "MDoE..."},
{"hostname": "smtp://smtp.example.org", "encryptedUsername": "x", "encryptedPassword": "y"}]}`)
c := f.countAccounts(filepath.Join(f.Root, profileDir), "~/.thunderbird/60016jsp.default-default")
if c.Declared != 3 || c.WithoutSignIn != 1 || c.Others != 2 || c.InList != 0 {
t.Fatalf("three declared, the OAuth one not signed in, the person's and Owl's other (local folders not counted): %+v", c)
}
raw, _ := json.Marshal(c)
if strings.Contains(string(raw), "example") || strings.Contains(string(raw), "gmail") {
t.Fatalf("a count names nothing: %s", raw)
}
}
func TestBetterbirdAccountsAnswersCountsAndProposesTheValueWithoutOwlOrACredential(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
links(t, nil)
a, err := f.ProfileAccounts(false)
if err != nil {
t.Fatal(err)
}
if a.Declarable["imap"] != 1 || a.KeptByOwl != 1 || a.NotDeclarable["local folders"] != 1 || a.Proposed != "" {
t.Fatalf("%+v", a)
}
raw, _ := json.Marshal(a)
for _, never := range []string{"example", "other", "Other", "work@", "smtp1"} {
if strings.Contains(string(raw), never) {
t.Errorf("the answer to the caller names %q: %s", never, raw)
}
}
var asked map[string]any
asker = func(key string, body any) (json.RawMessage, error) {
if key != "seat:mesh-controller.settings" {
t.Fatalf("asked %s", key)
}
asked = body.(map[string]any)
return json.RawMessage(`{"ok": true, "output": "proposed p-17 for betterbird on the whole mesh: accounts (other@mail.example.org); fingerprint ab12"}`), nil
}
t.Cleanup(func() { asker = nil })
a, err = f.ProfileAccounts(true)
if err != nil {
t.Fatal(err)
}
if asked["module"] != "betterbird" || asked["propose"] != "true" || asked["node"] != nil {
t.Fatalf("proposed for the whole mesh, never set: %v", asked)
}
value, err := parseAccounts([]byte(asked["values"].(string)))
if err != nil {
t.Fatalf("the proposed value is one the module takes: %v", err)
}
other, ok := value["mail"]
if len(value) != 1 || !ok || other.Address != "other@mail.example.org" || other.Incoming.SignIn != "oauth" ||
other.Incoming.Port != 993 || other.Outgoing.Security != "tls" || other.Outgoing.Port != 465 {
t.Fatalf("the value holds the person's account and not Owl's: %+v", value)
}
if strings.Contains(a.Proposed, "other@") || !strings.Contains(a.Proposed, "p-17") {
t.Fatalf("the controller's words come back with every address masked: %s", a.Proposed)
}
}
func TestAnAdoptedAccountNeverWritesIntoAnOutgoingServerAnotherIdentityUses(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// A second account of the person's, not declared, sends through smtp1 too.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"account1,account3,account2"`, `"account1,account3,account2,account5"`, 1) +
`user_pref("mail.account.account5.server", "server5");` + "\n" +
`user_pref("mail.account.account5.identities", "id5");` + "\n" +
`user_pref("mail.identity.id5.useremail", "fifth@mail.example.org");` + "\n" +
`user_pref("mail.identity.id5.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.server.server5.type", "imap");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
err := f.Prepare()
if err == nil || !strings.Contains(err.Error(), "also used by identity id5") || strings.Contains(err.Error(), "example.org") {
t.Fatalf("refused with the remedy, by keys: %v", err)
}
if user, _ := os.ReadFile(filepath.Join(f.Root, profileDir, "user.js")); strings.Contains(string(user), "smtpserver.smtp1") {
t.Fatal("the shared outgoing server was written")
}
// Recorded adoption, then the person moves the identity to a server of its own: the module follows it.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"mail.identity.id5.smtpServer", "smtp1"`, `"mail.identity.id5.smtpServer", "smtp5"`, 1))
if err := f.Prepare(); err != nil {
t.Fatal(err)
}
moved := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.identity.id1.smtpServer", "smtp1"`, `"mail.identity.id1.smtpServer", "smtp9"`, 1) +
`user_pref("mail.smtpserver.smtp9.hostname", "smtp.example.org");` + "\n"
f.write(profileDir+"/prefs.js", moved)
u, err := f.userJS(true)
if err != nil || u.Each[0].How != adopted || u.Each[0].Keys.SMTP != "smtp9" {
t.Fatalf("the identity's current outgoing server is followed: %+v %v", u.Each, err)
}
user := f.read(profileDir + "/user.js")
if strings.Contains(user, "smtpserver.smtp1.") || !strings.Contains(user, "smtpserver.smtp9.hostname") {
t.Fatalf("never the old one:\n%s", user)
}
// And the move back onto a shared server is refused again.
f.write(profileDir+"/prefs.js", strings.Replace(moved, `"mail.identity.id5.smtpServer", "smtp5"`, `"mail.identity.id5.smtpServer", "smtp9"`, 1))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id5") {
t.Fatalf("%v", err)
}
}
func TestTheUserJSAnswerWithholdsTheAccountsValues(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare(`user_pref("mail.compose.default_to", "x");`)
f.declareAccounts(personal)
u, err := f.userJS(false)
if err != nil {
t.Fatal(err)
}
for _, never := range []string{"example.org", `"me"`, `"Me"`} {
if strings.Contains(u.Content, never) {
t.Errorf("the answer carries %s:\n%s", never, u.Content)
}
}
if !strings.Contains(u.Content, `user_pref("mail.server.server_mesh_personal.hostname", "<withheld>");`) {
t.Fatalf("an account's preference is named and its value withheld:\n%s", u.Content)
}
// What is placed in the profile keeps the values: only the answer withholds them.
if f.read(testHome+"/"+accountsFile) == "" {
t.Fatal("no declaration")
}
}
func TestACredentialRefusalNamesNoAccount(t *testing.T) {
_, err := parseAccounts([]byte(`{"accounts": {"personal": {"incoming": {"password": "hunter2"}}}}`))
if err == nil || strings.Contains(err.Error(), "personal") || !strings.Contains(err.Error(), "accounts.<an account>.incoming.password") {
t.Fatalf("%v", err)
}
}
func TestAnAccountsOwnAliasesAndNonMailIdentitiesDoNotShareItsOutgoingServer(t *testing.T) {
f := newFake(t)
f.profileWithAccounts()
f.noPackage()
links(t, nil)
f.declare("// none")
os.RemoveAll(filepath.Join(f.Root, "proc", "4000"))
// account1 has an alias identity on smtp1; smtp1 is the default server; Owl's account and local folders
// have identities that name no outgoing server.
prefs := strings.Replace(f.read(profileDir+"/prefs.js"), `"mail.account.account1.identities", "id1"`, `"mail.account.account1.identities", "id1,id1b"`, 1)
prefs = strings.Replace(prefs, `"mail.account.account2.server", "server2");`, `"mail.account.account2.server", "server2");`+"\n"+
`user_pref("mail.account.account2.identities", "id2");`, 1) +
`user_pref("mail.identity.id1b.useremail", "alias@mail.example.org");` + "\n" +
`user_pref("mail.identity.id1b.smtpServer", "smtp1");` + "\n" +
`user_pref("mail.identity.id2.useremail", "local@mail.example.org");` + "\n" +
`user_pref("mail.smtp.defaultserver", "smtp1");` + "\n"
f.write(profileDir+"/prefs.js", prefs)
f.declareAccounts(adoptable)
if err := f.Prepare(); err != nil {
t.Fatalf("an account's own alias, and identities that do not send by SMTP, are no sharing: %v", err)
}
// An identity of another mail account naming no server does use the default one.
f.write(profileDir+"/prefs.js", strings.Replace(prefs, `"account1,account3,account2"`, `"account1,account3,account2,account6"`, 1)+
`user_pref("mail.account.account6.server", "server6");`+"\n"+
`user_pref("mail.account.account6.identities", "id6");`+"\n"+
`user_pref("mail.identity.id6.useremail", "six@mail.example.org");`+"\n"+
`user_pref("mail.server.server6.type", "pop3");`+"\n")
os.Remove(filepath.Join(f.Root, testHome, recordFile))
if err := f.Prepare(); err == nil || !strings.Contains(err.Error(), "also used by identity id6") {
t.Fatalf("%v", err)
}
}
@@ -1,330 +0,0 @@
package main
// Which account of the profile a declared account is (novox/hq ADR 0317): adopted, created or refused.
//
// At every apply, per declared account in its place in the setting (§1):
//
// 1. an account the module recorded on this node for this profile, adopted or created, is kept by its
// recorded keys without matching again, while its account key is still in the profile's account list;
// 2. otherwise an account under the module's own mesh_<name> keys is the module's creation;
// 3. otherwise a node's layer may name the key to adopt (§8): adopted only when it is in the account list,
// its default identity has the declared address and it is not Owl's;
// 4. otherwise the profile's accounts whose default identity has the declared address: one is adopted, none
// means the account is created under mesh_<name> keys (ADR 0316 §11).
//
// Refused, with the remedy, for that account alone (§7): two or more accounts with the address, the one that
// has it being Owl's, a named key that does not qualify, an account of another kind than the declaration says
// (the module never converts one: the review of ADR 0317), and an identity that names no outgoing server. The
// other declared accounts go ahead.
//
// Adopting takes the account's keys as the module's (§2): its account key, its default identity, its incoming
// server and the outgoing server that identity names. The account lists are left as they are, because the keys
// are in them already; the declared fields are written under those keys (§4, §5). What was adopted or created
// is recorded in the module's directory in the home, by the tools and never by a setting (§3); leaving the
// setting releases an adopted account (its lines and its record go, the account stays the person's, §6).
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// recordFile is where the module records, per profile, what it adopted or created (ADR 0317 §3).
const recordFile = ".config/betterbird/accounts-record.json"
// How a declared account resolved.
const (
adopted = "adopted"
created = "created"
refused = "refused"
)
// keySet are an account's four keys.
type keySet struct {
Account string `json:"account"`
Identity string `json:"identity"`
Server string `json:"server"`
SMTP string `json:"smtp"`
}
// resolvedAccount is one declared account as it resolved, said by its place and keys, never its name, address
// or host (ADR 0316 §16, ADR 0317 §3).
type resolvedAccount struct {
Place int `json:"place"`
How string `json:"how"`
Keys keySet `json:"keys,omitempty"`
// Why is a refusal's reason and its remedy.
Why string `json:"why,omitempty"`
// Changed are the preferences an apply changed for it, by name only (ADR 0317 §4).
Changed []string `json:"changed,omitempty"`
name string
}
// record is what the module adopted or created, per profile (its directory as the tools say it), per
// declared account's name.
type record map[string]map[string]recorded
type recorded struct {
How string `json:"how"`
Keys keySet `json:"keys"`
}
func (m *Machine) readRecord(profileDir string) map[string]recorded {
raw, err := readBounded(m.home(recordFile))
if err != nil {
return map[string]recorded{}
}
var r record
if json.Unmarshal(raw, &r) != nil || r[profileDir] == nil {
return map[string]recorded{}
}
return r[profileDir]
}
// writeRecord keeps what this apply resolved for the profile: a refused account is not recorded, and an
// account no longer declared is dropped (§6).
func (m *Machine) writeRecord(profileDir string, resolved []resolvedAccount) error {
r := record{}
if raw, err := readBounded(m.home(recordFile)); err == nil {
_ = json.Unmarshal(raw, &r)
}
mine := map[string]recorded{}
for _, a := range resolved {
if a.How != refused {
mine[a.name] = recorded{How: a.How, Keys: a.Keys}
}
}
if len(mine) == 0 {
delete(r, profileDir)
} else {
r[profileDir] = mine
}
body, err := json.MarshalIndent(r, "", " ")
if err != nil {
return err
}
path := m.home(recordFile)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return err
}
tmp := path + ".mesh-new"
if err := os.WriteFile(tmp, append(body, '\n'), 0o600); err != nil {
return err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return err
}
return nil
}
// resolveAccounts resolves every declared account, in its place in the setting (the file's order, by name).
func resolveAccounts(accounts map[string]Account, prefs map[string]string, rec map[string]recorded) []resolvedAccount {
have := profileAccounts(prefs)
inList := map[string]profileAccount{}
for _, a := range have {
inList[a.Key] = a
}
names := sortedNames(accounts)
taken := map[string]bool{}
var out []resolvedAccount
for i, name := range names {
r := resolveOne(i+1, len(names), name, accounts[name], prefs, have, inList, rec, taken)
r.name = name
if r.How != refused {
taken[r.Keys.Account] = true
}
out = append(out, r)
}
return refuseSharedOutgoing(out, prefs, have)
}
// refuseSharedOutgoing refuses an adopted account whose outgoing server another identity uses that is neither
// adopted nor the module's (review 34 of mesh-catalog #232): Betterbird lets identities share an outgoing
// server, and the declared values written into it would change that other account's sending at every start.
// ADR 0317 §2 adopts the outgoing server the identity names and leaves the lists as they are, so the module
// does not give the identity a server of its own; it refuses, with the remedy.
func refuseSharedOutgoing(resolved []resolvedAccount, prefs map[string]string, have []profileAccount) []resolvedAccount {
// Every identity of an adopted or created account is its own, its aliases too (review 36).
accountOf := map[string]profileAccount{}
kindOf := map[string]string{}
for _, a := range have {
accountOf[a.Key] = a
for _, id := range a.Identities {
kindOf[id] = a.Kind
}
}
ours := map[string]bool{}
for _, r := range resolved {
if r.How == refused {
continue
}
ours[r.Keys.Identity] = true
for _, id := range accountOf[r.Keys.Account].Identities {
ours[id] = true
}
}
defaultServer := unquote(prefs["mail.smtp.defaultserver"])
users := map[string][]string{}
seen := map[string]bool{}
use := func(id string) {
if seen[id] || ours[id] {
return
}
seen[id] = true
server := unquote(prefs["mail.identity."+id+".smtpServer"])
// Only a mail account's identity sends through the default server when it names none: Owl's, local
// folders and feeds do not send by SMTP (review 36).
if server == "" && (kindOf[id] == "imap" || kindOf[id] == "pop3") {
server = defaultServer
}
if server != "" {
users[server] = append(users[server], id)
}
}
for _, a := range have {
for _, id := range a.Identities {
use(id)
}
}
for k := range prefs {
if strings.HasPrefix(k, "mail.identity.") && strings.HasSuffix(k, ".smtpServer") {
use(strings.TrimSuffix(strings.TrimPrefix(k, "mail.identity."), ".smtpServer"))
}
}
for i, r := range resolved {
if r.How != adopted {
continue
}
if others := users[r.Keys.SMTP]; len(others) > 0 {
sort.Strings(others)
resolved[i] = resolvedAccount{Place: r.Place, How: refused, name: r.name, Why: fmt.Sprintf(
"declared account %d of %d: the outgoing server %s of the profile's account %s is also used by "+
"identity %s, whose sending the declared values would change: give the account an outgoing "+
"server of its own in Betterbird's account settings, then start Betterbird again",
r.Place, len(resolved), r.Keys.SMTP, r.Keys.Account, strings.Join(others, ", "))}
}
}
return resolved
}
func resolveOne(place, of int, name string, a Account, prefs map[string]string, have []profileAccount,
inList map[string]profileAccount, rec map[string]recorded, taken map[string]bool) resolvedAccount {
r := resolvedAccount{Place: place}
refuse := func(format string, args ...any) resolvedAccount {
r.How, r.Keys = refused, keySet{}
r.Why = fmt.Sprintf("declared account %d of %d: ", place, of) + fmt.Sprintf(format, args...)
return r
}
// 1. Recorded, while its account is still in the list.
if was, ok := rec[name]; ok {
if _, there := inList[was.Keys.Account]; there || was.How == created {
r.How, r.Keys = was.How, was.Keys
if was.How == adopted {
// The identity's outgoing server as it is now: a person may have moved it since.
smtpKey := unquote(prefs["mail.identity."+was.Keys.Identity+".smtpServer"])
if smtpKey == "" {
return refuse("the profile's account %s has an identity that names no outgoing server: choose one "+
"for it in Betterbird's account settings, then start Betterbird again", was.Keys.Account)
}
r.Keys.SMTP = smtpKey
}
return r
}
}
// 2. Under the module's own keys: its creation.
account, identity, server, smtp := keysOf(name)
own := keySet{account, identity, server, smtp}
if _, there := inList[account]; there {
r.How, r.Keys = created, own
return r
}
adopt := func(pa profileAccount) resolvedAccount {
if pa.Kind != a.Incoming.Kind {
return refuse("the profile's account %s is of kind %s and the declaration says %s: the module never "+
"converts an account. Declare the kind the account has, or remove the account in Betterbird",
pa.Key, pa.Kind, a.Incoming.Kind)
}
smtpKey := unquote(prefs["mail.identity."+pa.Identities[0]+".smtpServer"])
if smtpKey == "" {
return refuse("the profile's account %s has an identity that names no outgoing server: choose one for it "+
"in Betterbird's account settings, then start Betterbird again", pa.Key)
}
r.How, r.Keys = adopted, keySet{pa.Key, pa.Identities[0], pa.Server, smtpKey}
return r
}
matchesDefault := func(pa profileAccount) bool {
return len(pa.Addresses) > 0 && strings.EqualFold(pa.Addresses[0], a.Address)
}
// 3. The key a node's layer names.
if a.Key != "" {
pa, there := inList[a.Key]
switch {
case !there:
return refuse("the key %s this node's layer names is not in the profile's account list", a.Key)
case pa.Owl:
return refuse("the key %s this node's layer names is Owl's account, which is never declared "+
"(novox/hq ADR 0316 §13)", a.Key)
case !matchesDefault(pa):
return refuse("the key %s this node's layer names is an account whose default identity has another "+
"address", a.Key)
case taken[pa.Key]:
return refuse("the key %s is already another declared account's", a.Key)
}
return adopt(pa)
}
// 4. By the default identity's address.
var matches []profileAccount
for _, pa := range have {
if !pa.Ours && !taken[pa.Key] && matchesDefault(pa) {
matches = append(matches, pa)
}
}
switch {
case len(matches) == 0:
r.How, r.Keys = created, own
return r
case len(matches) > 1:
var keys []string
for _, pa := range matches {
keys = append(keys, pa.Key)
}
return refuse("%d accounts in the profile have its address (%s): remove the duplicate in Betterbird, or "+
"give the one to adopt as key in this node's layer of the setting accounts (novox/hq ADR 0317 §7, §8)",
len(matches), strings.Join(keys, ", "))
case matches[0].Owl:
return refuse("the profile's account with its address, %s, is Owl's, which is never declared (novox/hq "+
"ADR 0316 §13): take it out of the setting accounts", matches[0].Key)
}
return adopt(matches[0])
}
// changedPrefs are the preferences of a resolved account whose declared value differs from the profile's
// saved one (prefs.js), by name only (ADR 0317 §4). A preference prefs.js does not hold is changed unless the
// declared value is what Betterbird reads in its place.
func changedPrefs(r resolvedAccount, a Account, saved map[string]string) []string {
defaults := map[string]string{}
if r.How == adopted {
// What Betterbird reads when prefs.js holds no value: the same defaults betterbird_accounts reads with.
pa := profileAccount{Key: r.Keys.Account, Server: r.Keys.Server, Kind: a.Incoming.Kind,
Identities: []string{r.Keys.Identity}}
view := readAccount(saved, pa)
for _, p := range declaredPrefs(r.Keys, view, true) {
defaults[p[0]] = p[1]
}
}
var out []string
for _, p := range declaredPrefs(r.Keys, a, r.How == adopted) {
have, ok := saved[p[0]]
if !ok {
have, ok = defaults[p[0]]
}
if !ok || have != p[1] {
out = append(out, p[0])
}
}
return out
}
@@ -10,23 +10,19 @@ package main
// (passwordSecret, accountPrefix);
// - the profile's extensions.json, for the add-ons' ids and whether each is on;
// - the module's declared user.js (~/.config/betterbird/user.js), which the node-engine writes from
// the module's setting `prefs`, and its declared accounts (~/.config/betterbird/accounts.json, the
// setting `accounts`, accounts.go);
// - of logins.json, only the server name each kept sign-in is for, which Betterbird keeps in the clear,
// to count the declared accounts that have not signed in yet (novox/hq ADR 0316 §16).
// the module's setting `prefs`.
//
// Never read: the mail store, the address books, a user name or password in logins.json, key4.db,
// cert9.db, cookies, the calendar's caches. Nothing of the operator's mail or calendar is answered, and of
// their accounts only counts and kinds.
// Never read: the mail store, the address books, logins.json, key4.db, cert9.db, cookies, the
// calendar's caches. Nothing of the operator's mail, calendar or accounts is answered.
//
// The files a tool writes in the profile are user.js, and only the module's own (the declared file and
// the declared accounts' preferences, betterbird_user_js with apply, betterbird_restart, and the start
// from the launcher), and the module's own keys in the account lists of prefs.js, only while Betterbird is
// not running (accounts.go). A user.js that is not the module's is never overwritten.
// The one file a tool writes in the profile is user.js, and only the module's own: the declared file,
// copied whole (betterbird_user_js with apply, betterbird_restart). A user.js that is not the module's
// is never overwritten.
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
@@ -39,12 +35,8 @@ import (
)
const (
mainComm = "betterbird-bin"
// startBin is Betterbird itself, where the mesh's release and the community package both put it. The
// module's start (/usr/local/bin/betterbird) runs the preparation first; a tool that starts Betterbird
// prepares itself and starts this.
startBin = installDir + "/betterbird"
// packageFor is the community package the mesh's release replaces (novox/hq ADR 0316 §4).
mainComm = "betterbird-bin"
startBin = "/usr/bin/betterbird"
packageFor = "betterbird-bin"
appIni = "/opt/betterbird/application.ini"
restartAs = "mesh-betterbird"
@@ -270,11 +262,6 @@ var (
allowedExtensions = map[string]bool{"extensions.activeThemeID": true}
)
// accountOwned are the preferences of an account and of the account lists: the setting accounts' alone, so
// the setting prefs refuses them and an account has one owner (novox/hq ADR 0316 §11).
var accountOwned = []string{"mail.account.", "mail.identity.", "mail.server.", "mail.smtpserver.",
"mail.accountmanager.", "mail.smtpservers", "mail.smtp.defaultserver"}
// withheld says why a preference's value is never answered, or "" when it may be.
func withheld(name string) string {
if strings.HasPrefix(name, "extensions.") && !allowedExtensions[name] {
@@ -405,12 +392,6 @@ func validUserJS(text string) error {
if passwordSecret.MatchString(name) {
return fmt.Errorf("line %d sets %s, a preference that can hold a credential: not through the mesh", i+1, name)
}
for _, p := range accountOwned {
if strings.HasPrefix(name, p) {
return fmt.Errorf("line %d sets %s, an account's preference: an account is declared in the setting "+
"accounts, never in prefs (novox/hq ADR 0316 §11)", i+1, name)
}
}
}
}
return nil
@@ -452,35 +433,13 @@ type UserJS struct {
Effective string `json:"effective,omitempty"`
Content string `json:"content,omitempty"`
Applied bool `json:"applied,omitempty"`
// Accounts are how many accounts the setting accounts declares, and what is wrong with them.
Accounts int `json:"declared_accounts"`
AccountsProblem string `json:"accounts_problem,omitempty"`
// Each is every declared account by its place: adopted, created or refused (with why), its keys, and the
// preferences an apply changed for it, by name (novox/hq ADR 0317 §3, §4).
Each []resolvedAccount `json:"accounts,omitempty"`
// Lists says whether the keys of the accounts the module created were written into the account lists.
Lists string `json:"account_lists,omitempty"`
}
// refusedAccounts are the reasons of the declared accounts that were refused.
func (u UserJS) refusedAccounts() []string {
var out []string
for _, r := range u.Each {
if r.How == refused {
out = append(out, r.Why)
}
}
return out
}
// userJS compares the user.js the module places — the declared file, then the declared accounts' fields under
// the keys each was adopted or created under (adoption.go) — with the profile's; apply places it when the
// profile has none or the module's, records what was adopted or created, and writes the keys of the accounts
// it created into the profile's account lists when Betterbird is not running. A refused account gets nothing;
// the others go ahead (novox/hq ADR 0317 §7).
// userJS compares the declared user.js with the profile's; apply places the declared one when the
// profile has none or the module's.
func (m *Machine) userJS(apply bool) (UserJS, error) {
a := UserJS{Declared: "~/" + declared}
declaredText, err := readBounded(m.home(declared))
want, err := readBounded(m.home(declared))
a.Present = err == nil
p, perr := m.profile()
if perr != nil {
@@ -488,21 +447,11 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
}
target := filepath.Join(p.path, "user.js")
a.Profile = m.tilde(strings.TrimPrefix(target, m.Root))
accounts, _, aerr := m.readAccounts()
if aerr != nil {
a.AccountsProblem = aerr.Error()
accounts = nil
} else {
a.Accounts = len(accounts)
a.Each = resolveAccounts(accounts, mergedPrefs(p.path), m.readRecord(p.Dir))
}
var want []byte
if a.Present {
if verr := validUserJS(string(declaredText)); verr != nil {
if verr := validUserJS(string(want)); verr != nil {
a.Valid = verr.Error()
}
want = []byte(composeUserJS(string(declaredText), accounts, a.Each))
a.Content = m.answeredUserJS(string(want))
a.Content = string(want)
for _, line := range strings.Split(string(want), "\n") {
for _, st := range splitStatements(strings.TrimSpace(line)) {
if statement.MatchString(st) {
@@ -528,17 +477,6 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
return a, fmt.Errorf("the module's user.js is not on this machine (~/%s): the module is not pushed here yet", declared)
case a.Valid != "":
return a, fmt.Errorf("the module's user.js is refused: %s; change the setting prefs", a.Valid)
case a.AccountsProblem != "":
return a, fmt.Errorf("nothing was placed: %s", a.AccountsProblem)
}
saved, _ := readBounded(filepath.Join(p.path, "prefs.js"))
savedPrefs := readPrefs(saved)
for i, r := range a.Each {
if r.How != refused {
a.Each[i].Changed = changedPrefs(r, accounts[r.name], savedPrefs)
}
}
switch {
case herr == nil && !bytes.HasPrefix(have, []byte(mark)):
return a, fmt.Errorf("%s is not the module's: move it aside (its preferences belong in the setting prefs), then apply again", a.Profile)
}
@@ -553,14 +491,6 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
}
}
a.Applied, a.Placed = true, "the module's, as declared"
if err := m.writeRecord(p.Dir, a.Each); err != nil {
return a, fmt.Errorf("the user.js is placed, and what was adopted or created could not be recorded: %v", err)
}
if lists, err := m.writeAccountLists(p.path, a.Each); err != nil {
a.Lists = "not written: " + err.Error()
} else {
a.Lists = lists
}
}
if ps := m.mainProcs(); len(ps) > 0 {
if st, err := os.Stat(target); err == nil {
@@ -574,74 +504,27 @@ func (m *Machine) userJS(apply bool) (UserJS, error) {
return a, nil
}
// answeredUserJS is the user.js as a tool answers it (novox/hq ADR 0316 §16, ADR 0317 §9): an account's
// preferences with their values withheld, every other value masked as betterbird_prefs masks it.
func (m *Machine) answeredUserJS(text string) string {
lines := strings.Split(text, "\n")
for i, line := range lines {
for _, st := range splitStatements(strings.TrimSpace(line)) {
mm := prefLine.FindStringSubmatch(st)
if mm == nil {
continue
}
value := m.maskValue(mm[2])
if why := withheld(mm[1]); why != "" {
value = `"<withheld>"`
}
lines[i] = fmt.Sprintf("user_pref(%s, %s);", quoted(mm[1]), value)
}
}
return strings.Join(lines, "\n")
}
// Prepare is what the module's start runs before Betterbird (`betterbird-tools prepare`): with no Betterbird
// of this account running, the user.js placed and the declared accounts written into the account lists. With
// one running it does nothing, since the start then only hands over to it. Its error is said by the start,
// which starts Betterbird either way.
func (m *Machine) Prepare() error {
if len(m.mainProcs()) > 0 {
return nil
}
u, err := m.userJS(true)
if err != nil {
return err
}
var problems []string
if strings.HasPrefix(u.Lists, "not written") {
problems = append(problems, "the account lists were "+u.Lists)
}
problems = append(problems, u.refusedAccounts()...)
if len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
return nil
}
// Status is what betterbird_status answers.
type Status struct {
Running []Proc `json:"running"`
Children int `json:"children"`
// Installation is what is installed and from where, the policy and Owl (novox/hq ADR 0316 §16).
Installation Installation `json:"installation"`
// Installed is the community package's version, while it is still there.
Installed string `json:"community_package,omitempty"`
Running []Proc `json:"running"`
Children int `json:"children"`
Installed string `json:"installed,omitempty"`
Foreign bool `json:"from_the_aur,omitempty"`
Version string `json:"version,omitempty"`
Build string `json:"build,omitempty"`
Profile *Profile `json:"profile,omitempty"`
Problem string `json:"profile_problem,omitempty"`
Output []Stream `json:"output,omitempty"`
UserJS *UserJS `json:"user_js,omitempty"`
// Accounts are counts: the declared, those in the profile's list, the others, those without a sign-in.
Accounts *AccountCount `json:"accounts,omitempty"`
AddOns []AddOn `json:"add_ons,omitempty"`
AddOns []AddOn `json:"add_ons,omitempty"`
// Display are the preferences that decide how its windows and notifications look.
Display []Pref `json:"display_prefs,omitempty"`
}
// AddOn is one add-on the operator installed: its id, and whether it is on.
type AddOn struct {
ID string `json:"id"`
Version string `json:"version,omitempty"`
Active bool `json:"active"`
ID string `json:"id"`
Active bool `json:"active"`
}
// displayPrefs are the preferences betterbird_status answers: the reminder dialog, the theme, the
@@ -667,24 +550,39 @@ func (m *Machine) Status() (Status, error) {
if v, err := m.installed(packageFor); err == nil && v != "" {
s.Installed, s.Foreign = v, m.foreign(packageFor)
}
if raw, err := readBounded(m.path(appIni)); err == nil {
_, sec := iniSections(raw)
s.Version, s.Build = sec["App"]["Version"], sec["App"]["BuildID"]
}
if len(s.Running) > 0 {
s.Output = m.streams(s.Running[0].PID)
}
p, err := m.profile()
if err != nil {
s.Installation = m.installation("")
s.Problem = err.Error()
return s, nil
}
s.Profile = &p
s.Installation = m.installation(p.path)
if u, err := m.userJS(false); err == nil {
u.Content = ""
s.UserJS = &u
}
c := m.countAccounts(p.path, p.Dir)
s.Accounts = &c
s.AddOns = addOns(p.path)
if raw, err := readBounded(filepath.Join(p.path, "extensions.json")); err == nil {
var ext struct {
Addons []struct {
ID string `json:"id"`
Active bool `json:"active"`
Location string `json:"location"`
} `json:"addons"`
}
if json.Unmarshal(raw, &ext) == nil {
for _, a := range ext.Addons {
if a.Location == "app-profile" {
s.AddOns = append(s.AddOns, AddOn{a.ID, a.Active})
}
}
}
}
if got, err := m.Prefs(displayPrefs); err == nil {
s.Display = got["prefs"].([]Pref)
}
@@ -813,8 +711,7 @@ func (m *Machine) Restart(discard bool) (RestartAnswer, error) {
if u, err := m.userJS(true); err != nil {
a.UserJS = "not placed: " + err.Error()
} else {
a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s, %d declared account(s); account lists %s",
u.Statements, declared, u.Accounts, u.Lists)
a.UserJS = fmt.Sprintf("placed: %d preference(s) from ~/%s", u.Statements, declared)
}
// As detach does, with shorter bounds: the worst case of the whole call (3 s for the window tree,
// 8 s to quit, 3 + 5 s to start, 4 s to see it) stays under the runtime's 30 s.
@@ -849,65 +746,16 @@ type CheckAnswer struct {
}
// Check verifies what the module promises: Betterbird installed, one running in the session, its
// profile found, the module's user.js declared, valid, placed and in effect, its output read, and no ask at its
// start to be the default mail client (novox/hq issue 345).
// profile found, the module's user.js declared, valid, placed and in effect, and its output read.
func (m *Machine) Check() (CheckAnswer, error) {
a := CheckAnswer{Findings: []Finding{}}
add := func(what, do string) { a.Findings = append(a.Findings, Finding{what, do}) }
if _, err := m.installed(packageFor); err != nil {
v, err := m.installed(packageFor)
if err != nil {
return a, err
}
p, perr := m.profile()
profilePath := ""
if perr == nil {
profilePath = p.path
}
in := m.installation(profilePath)
switch {
case in.From == "not installed":
add("Betterbird is not installed in "+installDir, "the module's push places the mesh's release; it waits for the operator's push (upgrade policy record)")
case in.From != "the mesh":
add("Betterbird is "+in.From, "quit Betterbird, then push the module to this node: the push removes the community package and places the mesh's release")
}
if in.Behind {
add("Betterbird "+in.Version+" (build "+in.Build+") is behind the newest published build "+in.Newest,
"move the module's pin in a reviewed change (novox/hq ADR 0316 §5)")
}
if in.From == "the mesh" && !strings.HasPrefix(in.Policy, "placed:") {
add("the enterprise policy is "+in.Policy, "push the module to this node")
}
if why := m.defaultClientAsk(profilePath); why != "" && in.From != "not installed" {
add("Betterbird asks at its start to be the default mail client: "+why,
"push the module to this node: its policy locks "+defaultClientPref+" off (novox/hq issue 345)")
}
switch {
case in.From != "the mesh":
case in.Owl == nil || in.Owl.Pinned == "":
add("Owl's xpi is not placed ("+owlFile+")", "push the module to this node")
case in.Owl.InProfile == "":
add("Owl is not installed in the profile yet", "Betterbird installs it from the policy at its next start")
case !in.Owl.AtThePin && !newerVersion(in.Owl.InProfile, in.Owl.Pinned):
add("Owl "+in.Owl.InProfile+" runs in the profile, older than the mesh's pin "+in.Owl.Pinned,
"Owl is installed by policy and cannot be removed in Betterbird: move the module's Owl pin in a reviewed change (novox/hq ADR 0316 §6)")
case !in.Owl.Active:
add("Owl is installed and not active", "turn it on in Betterbird's add-ons")
}
if perr == nil {
c := m.countAccounts(p.path, p.Dir)
for _, r := range c.Each {
if r.How == refused {
add("refused: "+r.Why, "as it says; the other declared accounts go ahead")
}
}
switch {
case c.InList < c.Declared:
add(fmt.Sprintf("%d of %d declared account(s) are not in the profile's account list yet", c.Declared-c.InList, c.Declared),
"quit Betterbird and start it from the launcher, or betterbird_restart")
}
if c.WithoutSignIn > 0 {
add(fmt.Sprintf("%d declared account(s) have not signed in yet", c.WithoutSignIn),
"a person signs in once, in Betterbird's own window, when it asks (novox/hq ADR 0316 §14)")
}
if v == "" {
add("Betterbird ("+packageFor+") is not installed", "install it from the AUR: the node-engine installs packages from the official repositories only, so the module does not")
}
ps := m.mainProcs()
switch {
@@ -924,8 +772,6 @@ func (m *Machine) Check() (CheckAnswer, error) {
add("the module's user.js is not on this machine (~/"+declared+")", "the module is not pushed here: the controller's push")
case u.Valid != "":
add("the module's user.js is refused: "+u.Valid, "change the module's setting prefs")
case u.AccountsProblem != "":
add("the module's user.js is not placed: "+u.AccountsProblem, "change the module's setting accounts at the controller's terminal, or propose it (betterbird_accounts)")
case !strings.HasPrefix(u.Placed, "the module's, as declared"):
add("the profile's user.js is "+u.Placed, "betterbird_user_js with apply, then betterbird_restart")
case strings.HasPrefix(u.Effective, "not yet"):
@@ -161,7 +161,7 @@ func TestStatusSaysWhatRunsTheProfileTheAddOnsAndTheDisplayPrefs(t *testing.T) {
t.Fatal(err)
}
if len(s.Running) != 1 || s.Running[0].PID != 4000 || s.Children != 1 || s.Installed != "153.1.0esr-1" || !s.Foreign ||
s.Installation.Version != "153.1.0" || s.Profile == nil || len(s.AddOns) != 1 || s.AddOns[0].ID != "owl@beonex.com" {
s.Version != "153.1.0" || s.Profile == nil || len(s.AddOns) != 1 || s.AddOns[0].ID != "owl@beonex.com" {
t.Fatalf("%+v", s)
}
if !s.Output[0].Dead {
@@ -238,13 +238,6 @@ func TestTheReminderTestTouchesNothingOfTheOperatorsProfile(t *testing.T) {
!strings.Contains(string(user), "file:///run/user/1000/"+testUnit+"/test.ics") || strings.Contains(string(user), "someone@") {
t.Fatalf("%s", user)
}
// Betterbird's update check stays off, so no About window opens on the operator's screen, and the
// test's own preferences come last, so nothing carried or declared overrides them.
if !strings.Contains(string(user), `"betterbird.update.checkIntervalDays", 0`) ||
strings.Index(string(user), "extensions.activeThemeID") > strings.Index(string(user), "calendar.registry.meshtest.uri") ||
strings.Index(string(user), `"calendar.alarms.show"`) > strings.Index(string(user), "calendar.registry.meshtest.uri") {
t.Fatalf("%s", user)
}
// The event starts 10 minutes after the reminder, which is due 40 s from now.
if !strings.Contains(string(cal), "DTSTART:20270115T081040Z") || !strings.Contains(string(cal), "TRIGGER:-PT10M") ||
a.Due != "2027-01-15T08:00:40Z" {
@@ -1,290 +0,0 @@
package main
// What the mesh installs (novox/hq ADR 0316 §1–§9, to-be 56 §1–§3): Betterbird from its pinned upstream release
// in /opt/betterbird, where the community package betterbird-bin put it, so Betterbird keeps choosing the same
// profile; the enterprise policy in its distribution directory, which turns Betterbird's own updater off and
// installs Owl into every profile from the xpi the mesh placed beside it; and a defaults file naming the
// system's spelling dictionaries. The tools read these files and say what they find: the version against the
// pin and the newest published build, whether the policy is placed, Owl's version against its pin.
//
// **The pin on the machine is what the mesh placed.** The manifest pins the release and the xpi by sha256; the
// node-engine places exactly those bytes, so the version in /opt/betterbird/application.ini (when the mesh
// placed it) and the version in the placed xpi are the pins. The community package's own copy is said as
// such, and the switch from it waits for the operator (ADR 0316 §4).
import (
"archive/zip"
"encoding/json"
"fmt"
"io"
"net/http"
"path/filepath"
"strings"
"time"
)
const (
installDir = "/opt/betterbird"
// policyFile is the enterprise policy the module places (ADR 0316 §7).
policyFile = installDir + "/distribution/policies.json"
// owlFile is Owl's xpi as the mesh placed it, and the URL the policy installs it from.
owlFile = installDir + "/owl/owl.xpi"
owlID = "owl@beonex.com"
// defaultClientPref is Betterbird's ask at its start to be the default mail, news, feed and calendar
// client (Thunderbird's all-thunderbird.js: pref("mail.shell.checkDefaultClient", true); messenger.js opens
// systemIntegrationDialog.xhtml while it is true). The module's policy locks it off (novox/hq issue 345).
defaultClientPref = "mail.shell.checkDefaultClient"
// preparation is the tools' program as the module places it for its start (betterbird-tools prepare).
preparation = "/usr/local/lib/mesh-betterbird/betterbird-tools"
// versionsURL is where Betterbird publishes its newest build per major version: what its own updater
// reads (MailGlue's BetterbirdUpdateCheck), and what the status compares the installed build with.
versionsURL = "https://www.betterbird.eu/start/versions.txt"
)
// publishedBuilds fetches Betterbird's list of newest builds: a variable so a test answers.
var publishedBuilds = func() (string, error) {
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Get(versionsURL)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("%s answered %s", versionsURL, resp.Status)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
return string(body), err
}
// Installation is what is installed, and from where.
type Installation struct {
// From is the mesh, the community package (betterbird-bin, before the switch), or nothing.
From string `json:"from"`
Version string `json:"version,omitempty"`
Build string `json:"build,omitempty"`
// Newest is the newest build Betterbird published for this major version, and whether this one is behind.
Newest string `json:"newest_published_build,omitempty"`
Behind bool `json:"behind,omitempty"`
// Policy says whether the module's enterprise policy is placed, and what it does.
Policy string `json:"policy"`
// Owl is the add-on's version as the mesh placed it (the pin), and as the profile runs it.
Owl *OwlState `json:"owl,omitempty"`
}
// OwlState is Owl against its pin.
type OwlState struct {
Pinned string `json:"pinned,omitempty"`
InProfile string `json:"in_profile,omitempty"`
Active bool `json:"active"`
AtThePin bool `json:"at_the_pin"`
// Note says, as a fact, a profile running an Owl newer than the pin: the policy's install keeps a newer
// version, and Owl installed by policy cannot be removed in Betterbird.
Note string `json:"note,omitempty"`
}
// newerVersion says whether version a is newer than b, comparing dotted numbers.
func newerVersion(a, b string) bool {
if a == "" || b == "" {
return false
}
pa, pb := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; i < len(pa) || i < len(pb); i++ {
var x, y int
if i < len(pa) {
fmt.Sscanf(pa[i], "%d", &x)
}
if i < len(pb) {
fmt.Sscanf(pb[i], "%d", &y)
}
if x != y {
return x > y
}
}
return false
}
// installation reads what is installed. profile may be empty when none is found yet.
func (m *Machine) installation(profile string) Installation {
var in Installation
community, _ := m.installed(packageFor)
raw, err := readBounded(m.path(appIni))
switch {
case community != "":
in.From = "the community package " + packageFor + " " + community + ": the switch to the mesh's release waits for the operator's push, with Betterbird quit (novox/hq ADR 0316 §4)"
case err == nil:
in.From = "the mesh"
default:
in.From = "not installed"
}
if err == nil {
_, sec := iniSections(raw)
in.Version, in.Build = sec["App"]["Version"], sec["App"]["BuildID"]
in.Newest, in.Behind = newestBuild(in.Version, in.Build)
}
in.Policy = m.policyState()
pinned := xpiVersion(m.path(owlFile))
owl := &OwlState{Pinned: pinned}
if profile != "" {
for _, a := range addOns(profile) {
if a.ID == owlID {
owl.InProfile, owl.Active = a.Version, a.Active
}
}
}
owl.AtThePin = owl.Pinned != "" && owl.Pinned == owl.InProfile
if newerVersion(owl.InProfile, owl.Pinned) {
owl.Note = "the profile runs Owl " + owl.InProfile + ", newer than the mesh's pin " + owl.Pinned + ": the policy " +
"keeps a newer version and installs the pin only where Owl is missing. To run the same Owl everywhere, " +
"move the module's pin in a reviewed change"
}
if owl.Pinned != "" || owl.InProfile != "" {
in.Owl = owl
}
return in
}
// newestBuild is the newest build Betterbird published for a version's major, and whether build is older.
func newestBuild(version, build string) (string, bool) {
major, _, _ := strings.Cut(version, ".")
if major == "" || build == "" {
return "", false
}
list, err := publishedBuilds()
if err != nil {
return "unknown: " + err.Error(), false
}
for _, l := range strings.Split(list, "\n") {
f := strings.Split(strings.TrimSpace(l), ",")
if len(f) == 3 && f[0] == "Linux" && f[1] == major {
return f[2], f[2] > build
}
}
return "unknown: " + versionsURL + " names no build for " + major, false
}
// policyState says whether the module's policy is placed and does what ADR 0316 §5 and §7 say.
func (m *Machine) policyState() string {
raw, err := readBounded(m.path(policyFile))
if err != nil {
return "not placed: " + policyFile + " is missing"
}
if why := policyProblem(raw); why != "" {
return "placed, and " + why
}
return "placed: Betterbird's updater off, Owl installed into every profile from the mesh's xpi, its updates off, " +
"and the ask to be the default mail client locked off"
}
// policyProblem is what the policy fails to say of what ADR 0316 asks, or "".
func policyProblem(raw []byte) string {
var p struct {
Policies struct {
DisableAppUpdate bool `json:"DisableAppUpdate"`
ExtensionSettings map[string]struct {
Mode string `json:"installation_mode"`
URL string `json:"install_url"`
Disabled bool `json:"updates_disabled"`
} `json:"ExtensionSettings"`
Preferences map[string]struct {
Value any `json:"Value"`
Status string `json:"Status"`
} `json:"Preferences"`
} `json:"policies"`
}
if err := json.Unmarshal(raw, &p); err != nil {
return "it is not JSON: " + err.Error()
}
owl, ok := p.Policies.ExtensionSettings[owlID]
switch {
case !p.Policies.DisableAppUpdate:
return "it does not turn Betterbird's updater off (DisableAppUpdate)"
case !ok || owl.Mode != "force_installed":
return "it does not install Owl into every profile (force_installed)"
case owl.URL != "file://"+owlFile:
return "it installs Owl from " + owl.URL + ", not from the mesh's xpi"
case !owl.Disabled:
return "it leaves Owl's updates on"
}
if pr, ok := p.Policies.Preferences[defaultClientPref]; !ok || pr.Value != false || pr.Status != "locked" {
return "it does not lock Betterbird's ask to be the default mail client off (Preferences " + defaultClientPref + ": false, locked)"
}
return ""
}
// defaultClientAsk says why Betterbird asks at its start to be the default mail client, or "" when it does
// not: the module's policy locks the ask off, else the profile's user.js (which wins at the start) or prefs.js
// turns it off. Betterbird's own default is to ask.
func (m *Machine) defaultClientAsk(profile string) string {
raw, err := readBounded(m.path(policyFile))
if err == nil && !strings.Contains(policyProblem(raw), defaultClientPref) {
return ""
}
if profile != "" {
for _, f := range []string{"user.js", "prefs.js"} {
b, err := readBounded(filepath.Join(profile, f))
if err != nil {
continue
}
if v, ok := readPrefs(b)[defaultClientPref]; ok {
if v == "false" {
return ""
}
return "the profile's " + f + " sets " + defaultClientPref + " to " + v + " and no policy locks it off"
}
}
}
return "no policy locks " + defaultClientPref + " off and Betterbird's default is to ask"
}
// xpiVersion is the version an add-on's manifest.json says, or "" when it cannot be read.
func xpiVersion(path string) string {
r, err := zip.OpenReader(path)
if err != nil {
return ""
}
defer r.Close()
for _, f := range r.File {
if f.Name != "manifest.json" {
continue
}
rc, err := f.Open()
if err != nil {
return ""
}
defer rc.Close()
var mf struct {
Version string `json:"version"`
}
if json.NewDecoder(io.LimitReader(rc, 1<<20)).Decode(&mf) != nil {
return ""
}
return mf.Version
}
return ""
}
// addOns reads a profile's extensions.json: each add-on's id, version, whether it is on, and where from.
func addOns(profile string) []AddOn {
raw, err := readBounded(filepath.Join(profile, "extensions.json"))
if err != nil {
return nil
}
var ext struct {
Addons []struct {
ID string `json:"id"`
Version string `json:"version"`
Active bool `json:"active"`
Location string `json:"location"`
} `json:"addons"`
}
if json.Unmarshal(raw, &ext) != nil {
return nil
}
var out []AddOn
for _, a := range ext.Addons {
if a.Location == "app-profile" {
out = append(out, AddOn{ID: a.ID, Version: a.Version, Active: a.Active})
}
}
return out
}
@@ -1,231 +0,0 @@
package main
import (
"archive/zip"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// What the mesh installs, as the tools say it (novox/hq ADR 0316 §16): the version against the pin and the
// newest published build, the policy, Owl against its pin; and the reminder test, visibly a test.
func (f *fake) xpi(path, version string) {
f.t.Helper()
full := filepath.Join(f.Root, path)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
f.t.Fatal(err)
}
out, err := os.Create(full)
if err != nil {
f.t.Fatal(err)
}
w := zip.NewWriter(out)
m, _ := w.Create("manifest.json")
m.Write([]byte(`{"version": "` + version + `", "browser_specific_settings": {"gecko": {"id": "owl@beonex.com"}}}`))
w.Close()
out.Close()
}
// placedByTheMesh is a node after the switch: the release, the policy and Owl's xpi in /opt/betterbird, and
// the profile running Owl at the pin.
func (f *fake) placedByTheMesh() {
f.noPackage()
f.write(appIni, "[App]\nVendor=Betterbird Project\nVersion=153.4.0\nBuildID=20260929222610\n")
raw, err := os.ReadFile(filepath.Join("..", "..", "install", "distribution", "policies.json"))
if err != nil {
f.t.Fatal(err)
}
f.write(policyFile, string(raw))
f.xpi(owlFile, "1.5.6.1")
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.6.1","active":true,"location":"app-profile"}]}`)
}
func TestTheStatusSaysTheInstallationAgainstThePinTheNewestBuildThePolicyAndOwl(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
publishedBuilds = func() (string, error) {
return "blocklist,{}\nLinux,153,20261101000000\nLinux,140,20260829111240\n", nil
}
t.Cleanup(func() { publishedBuilds = func() (string, error) { return "", os.ErrDeadlineExceeded } })
s, err := f.Status()
if err != nil {
t.Fatal(err)
}
in := s.Installation
if in.From != "the mesh" || in.Version != "153.4.0" || in.Newest != "20261101000000" || !in.Behind ||
!strings.HasPrefix(in.Policy, "placed:") || in.Owl == nil || in.Owl.Pinned != "1.5.6.1" || !in.Owl.AtThePin || !in.Owl.Active {
t.Fatalf("%+v %+v", in, in.Owl)
}
c, err := f.Check()
if err != nil {
t.Fatal(err)
}
found := false
for _, x := range c.Findings {
found = found || strings.Contains(x.What, "behind the newest published build 20261101000000")
}
if !found {
t.Fatalf("a release left behind is said: %+v", c.Findings)
}
}
func TestTheCommunityPackageAPolicyThatIsNotTheModulesAndAnOwlOffItsPinAreSaid(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
f.write(policyFile, `{"policies": {"ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "https://vendor.example/owl.xpi", "updates_disabled": true}}}}`)
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.7","active":true,"location":"app-profile"}]}`)
in := f.installation(filepath.Join(f.Root, profileDir))
if !strings.Contains(in.Policy, "DisableAppUpdate") || in.Owl.AtThePin || in.Owl.InProfile != "1.5.7" {
t.Fatalf("%+v %+v", in, in.Owl)
}
// A newer Owl than the pin is a fact with the real remedy, never a finding to remove it.
if !strings.Contains(in.Owl.Note, "move the module's pin") {
t.Fatalf("%+v", in.Owl)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`)
c, _ := f.Check()
for _, x := range c.Findings {
if strings.Contains(x.What, "Owl") {
t.Errorf("a newer Owl is not a finding: %+v", x)
}
}
f.write(profileDir+"/extensions.json", `{"addons":[{"id":"owl@beonex.com","version":"1.5.5","active":true,"location":"app-profile"}]}`)
c, _ = f.Check()
found := false
for _, x := range c.Findings {
found = found || (strings.Contains(x.What, "older than the mesh's pin") && strings.Contains(x.Do, "move the module's Owl pin"))
}
if !found {
t.Fatalf("an older Owl is a finding whose remedy is the pin: %+v", c.Findings)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "https://vendor.example/owl.xpi", "updates_disabled": true}}}}`)
if in := f.installation(""); !strings.Contains(in.Policy, "not from the mesh's xpi") {
t.Fatalf("%s", in.Policy)
}
f.answer = func(name string, args []string) Output {
if name == "pacman" && args[0] == "-Q" {
return Output{Stdout: "betterbird-bin 153.1.0esr-1\n"}
}
return Output{}
}
in = f.installation("")
if !strings.HasPrefix(in.From, "the community package betterbird-bin 153.1.0esr-1") || !strings.Contains(in.From, "operator") {
t.Fatalf("%s", in.From)
}
raw, _ := json.Marshal(in)
if strings.Contains(string(raw), "example.org") {
t.Fatalf("%s", raw)
}
}
func TestTheReminderTestIsVisiblyATestAndSaysItOpensAWindow(t *testing.T) {
f := newFake(t)
f.desktopSession()
f.betterbird()
links(t, nil)
f.declare("// none")
f.declareAccounts(personal)
a, err := f.ReminderTest(40*time.Second, nil, false)
if err != nil {
t.Fatal(err)
}
dir := filepath.Join("/run/user/1000", testUnit)
css := f.read(dir + "/profile/chrome/userChrome.css")
user := f.read(dir + "/profile/user.js")
cal := f.read(dir + "/test.ics")
if !strings.Contains(css, "MESH TEST") || !strings.Contains(css, `:root:not([windowtype="Calendar:AlarmWindow"])`) {
t.Fatalf("the test's windows but the reminder carry a MESH TEST band: %s", css)
}
if !strings.Contains(user, `user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);`) ||
strings.Contains(user, "mesh_personal") || strings.Contains(user, "example.org") {
t.Fatalf("the test profile reads its userChrome.css and holds no declared account: %s", user)
}
if !strings.Contains(cal, "SUMMARY:MESH TEST reminder") || !strings.Contains(a.OnScreen, "operator's screen") {
t.Fatalf("the event and the answer say it is a test on the operator's screen: %s %+v", cal, a)
}
if !f.called("systemd-run --user --collect --quiet --unit="+testUnit) || !strings.Contains(strings.Join(f.calls, "\n"), startBin+" --no-remote") {
t.Fatalf("the test starts Betterbird itself, never the module's start: %v", f.calls)
}
for _, tool := range tools() {
if tool.Name == "betterbird_reminder_test" && !strings.HasPrefix(tool.Description, "OPENS WINDOWS ON THE OPERATOR'S SCREEN") {
t.Fatalf("the description says first that a window opens on the operator's screen: %s", tool.Description)
}
}
}
func TestTheUserJSDescriptionSaysAdoption(t *testing.T) {
for _, tool := range tools() {
if tool.Name == "betterbird_user_js" && (!strings.Contains(tool.Description, "adopted under its own keys") ||
strings.Contains(tool.Description, "refuses the apply")) {
t.Fatalf("%s", tool.Description)
}
}
}
// Betterbird never asks at its start to be the default mail client: the module's policy locks
// mail.shell.checkDefaultClient off, and the check says so when it is on (novox/hq issue 345).
func TestTheDefaultClientAskIsLockedOffAndTheCheckSaysWhenItIsOn(t *testing.T) {
f := newFake(t)
f.betterbird()
f.placedByTheMesh()
links(t, nil)
asks := func() string {
t.Helper()
c, err := f.Check()
if err != nil {
t.Fatal(err)
}
for _, x := range c.Findings {
if strings.Contains(x.What, "asks at its start") {
return x.What + " | " + x.Do
}
}
return ""
}
if got := asks(); got != "" {
t.Fatalf("the module's policy locks the ask off, yet: %s", got)
}
unlocked := `{"policies": {"DisableAppUpdate": true, "ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`
if why := policyProblem([]byte(unlocked)); !strings.Contains(why, defaultClientPref) {
t.Fatalf("a policy that leaves the ask on is said: %q", why)
}
f.write(policyFile, unlocked)
if got := asks(); !strings.Contains(got, "default mail client") || !strings.Contains(got, "push the module") {
t.Fatalf("the ask on is a finding with its remedy: %q", got)
}
f.write(policyFile, `{"policies": {"DisableAppUpdate": true, "Preferences": {"mail.shell.checkDefaultClient": {"Value": false, "Status": "default"}},
"ExtensionSettings": {"owl@beonex.com": {"installation_mode": "force_installed", "install_url": "file:///opt/betterbird/owl/owl.xpi", "updates_disabled": true}}}}`)
if why := policyProblem(mustRead(t, f, policyFile)); !strings.Contains(why, "locked") {
t.Fatalf("a default the profile can override is not a lock: %q", why)
}
f.write(profileDir+"/prefs.js", `user_pref("mail.shell.checkDefaultClient", true);`+"\n")
if got := asks(); got == "" {
t.Fatal("a profile that turned the ask back on over an unlocked default is a finding")
}
f.write(policyFile, unlocked)
f.write(profileDir+"/prefs.js", `user_pref("mail.shell.checkDefaultClient", false);`+"\n")
if got := asks(); got != "" {
t.Fatalf("a profile with the ask off does not ask: %s", got)
}
}
func mustRead(t *testing.T, f *fake, p string) []byte {
t.Helper()
raw, err := os.ReadFile(filepath.Join(f.Root, p))
if err != nil {
t.Fatal(err)
}
return raw
}
+18 -58
View File
@@ -16,15 +16,6 @@ import (
)
func main() {
// The module's start runs this before Betterbird (novox/hq ADR 0316 §11, §12): the user.js placed and the
// declared accounts written into the account lists, while no Betterbird of this account runs.
if len(os.Args) == 2 && os.Args[1] == "prepare" {
if err := machine.Prepare(); err != nil {
fmt.Fprintln(os.Stderr, "betterbird: "+err.Error())
os.Exit(1)
}
return
}
if err := stdio.Serve("", tools()); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
@@ -38,13 +29,10 @@ func tools() []stdio.Tool {
{
Name: "betterbird_status",
Description: "Betterbird: whether it runs (the main process's pid, since, and the scope it runs in; how " +
"many child processes); what is installed in /opt/betterbird and from where (the mesh's pinned release, " +
"or the community package before the switch), its version and build beside the newest published build; " +
"whether the enterprise policy is placed; Owl's version against the mesh's pin and whether it is active; " +
"where its output goes; the profile it uses and how that was found; the module's user.js (declared, " +
"placed, in effect); the declared accounts by count (in the profile's list, other accounts, not signed " +
"in yet), never an address, host or name; the add-ons in the profile; and the preferences that decide " +
"how its windows and notifications look. (r)",
"many child processes), the installed package and version and whether it is from the AUR, where its " +
"output goes, the profile it uses and how that was found, the module's user.js (declared, placed, in " +
"effect), the add-ons the operator installed, and the preferences that decide how its windows and " +
"notifications look (reminder dialog, theme, renderer, system notifications). (r)",
Run: func(map[string]any) (any, error) { return machine.Status() },
},
{
@@ -67,15 +55,11 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_user_js",
Description: "The module's user.js — the preferences the module declares (setting prefs), then the declared " +
"accounts' preferences under the module's own keys (setting accounts) — against the profile's user.js: " +
"whether it is placed, and whether the running Betterbird started after it was. With apply, place it in " +
"the profile (only where the profile has no user.js or the module's own; an operator's file is never " +
"overwritten). A declared account the profile already holds at its address is adopted under its own keys; " +
"one it does not hold is created under the module's keys, which are added to the account lists when " +
"Betterbird is not running; an ambiguous one is refused alone, with the remedy (novox/hq ADR 0317). " +
"Accounts' values are withheld in the answer. Betterbird reads it at its next start: betterbird_restart. " +
"(r; a with apply)",
Description: "The module's user.js — the preferences the module declares, from its setting prefs, written by " +
"the mesh to ~/.config/betterbird/user.js — against the profile's user.js: whether it is placed, and " +
"whether the running Betterbird started after it was. With apply, place it in the profile (only where " +
"the profile has no user.js or the module's own; an operator's file is never overwritten). Betterbird " +
"reads it at its next start: betterbird_restart. (r; a with apply)",
Input: map[string]any{
"apply": map[string]any{"type": "boolean", "description": "place the declared user.js in the profile (default false)"},
},
@@ -134,43 +118,19 @@ func tools() []stdio.Tool {
},
{
Name: "betterbird_check",
Description: "Check what the module promises: Betterbird installed from the mesh's pinned release and not " +
"behind the newest published build; the enterprise policy placed; Owl at its pin and active; one " +
"Betterbird runs; its profile is found; the module's user.js on the machine, valid, placed and in " +
"effect; the declared accounts in the profile's list, none refused, each signed in once; its output " +
"read. Answers ok and each finding with what to do, never an address, host or name. (r)",
Description: "Check what the module promises: Betterbird is installed; one runs; its profile is found; the " +
"module's user.js is on the machine, valid, placed in the profile and in effect; its output is read " +
"(a pipe nobody reads loses what it prints). Answers ok and each finding with what to do. (r)",
Run: func(map[string]any) (any, error) { return machine.Check() },
},
{
Name: "betterbird_accounts",
Description: "The accounts of the profile Betterbird uses, by count and kind: the IMAP and POP3 accounts a " +
"declaration can hold, the accounts Owl keeps (never declared), and the rest. Never an address, a host " +
"or a name, and never anything of the login store. With propose, it builds the module's setting " +
"accounts from them (every account but Owl's, without any password or token) and proposes it through " +
"the controller's settings verb for the operator's Approve on a channel that proves who answers " +
"(novox/hq ADR 0317, ADR 0277): nothing changes until then, and the operator reads the value whole at " +
"their desk. (r; a with propose)",
Input: map[string]any{
"propose": map[string]any{"type": "boolean", "description": "build the setting accounts from the profile and propose it to the operator (default false)"},
},
Run: func(args map[string]any) (any, error) {
propose, err := flag(args, "propose", false)
if err != nil {
return nil, err
}
return machine.ProfileAccounts(propose)
},
},
{
Name: "betterbird_reminder_test",
Description: "OPENS WINDOWS ON THE OPERATOR'S SCREEN: tell the operator before calling it. Shows a calendar " +
"reminder without touching the operator's calendar: a second Betterbird in a throw-away profile under " +
"the runtime directory (--no-remote, unit mesh-betterbird-reminder-test, at most 10 minutes), its main " +
"window marked MESH TEST in red, with the module's user.js (no accounts) and one local calendar file " +
"holding one event, \"MESH TEST reminder\", whose reminder is due after `after` seconds. carry copies " +
"named preferences of the operator's profile (a theme, a colour switch) into it, never a credential or " +
"an account. Then xorg_windows (class Calendar) and xorg_screenshot show the reminder window. stop ends " +
"it and removes the profile. (a)",
Description: "Show a calendar reminder without touching the operator's calendar: a second Betterbird in a " +
"throw-away profile under the runtime directory (--no-remote, unit mesh-betterbird-reminder-test, at most " +
"10 minutes), with the module's user.js and one local calendar file holding one event whose reminder is " +
"due after `after` seconds. carry copies named preferences of the operator's profile (a theme, a " +
"colour switch) into it, never a credential or an account. Then xorg_windows (class Calendar) and " +
"xorg_screenshot show the reminder window. stop ends it and removes the profile. (a)",
Input: map[string]any{
"after": map[string]any{"type": "integer", "description": "seconds until the reminder is due (default 40, 15 to 300)"},
"carry": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
@@ -2,32 +2,25 @@ package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"regexp"
"strconv"
"strings"
"testing"
)
// betterbird's shape (novox/hq issue 345, ADR 0208, ADR 0316): the X display on its own machine; its directory
// in the home with the user.js the setting prefs fills and the accounts file the setting accounts fills, both
// trusted; the community package declared absent before anything is placed in /opt/betterbird; Betterbird from
// its pinned upstream release, Owl pinned by its vendor's digest, the policy and the dictionaries' default
// unpacked into the installation; the tools' program for the start's preparation, the start and the launcher;
// every new build waiting for the operator's push; and the Go bundle serving exactly the listed tools.
// betterbird's shape (novox/hq issue 345, ADR 0208): no package (Betterbird comes from the AUR, and the
// node-engine installs from the official repositories only), no seat, no start of its own (the operator
// starts it from the launcher); the X display on its own machine; one directory and one file of its own,
// the user.js the setting prefs fills; and the Go bundle serving exactly the listed betterbird_ tools.
type manifest struct {
Module string `json:"module"`
Version string `json:"version"`
Upgrade map[string]string `json:"upgrade"`
Requires []string `json:"requires"`
Invokes []string `json:"invokes"`
Settings map[string]any `json:"settings"`
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
Module string `json:"module"`
Version string `json:"version"`
Requires []string `json:"requires"`
Settings map[string]any `json:"settings"`
Tools []string `json:"tools"`
Resources []map[string]any `json:"resources"`
Build struct {
Artifacts []map[string]any `json:"artifacts"`
} `json:"build"`
@@ -48,216 +41,43 @@ func readManifest(t *testing.T) (manifest, string) {
return m, string(raw)
}
func resource(t *testing.T, m manifest, id string) map[string]any {
t.Helper()
for _, r := range m.Resources {
if r["id"] == id {
return r
}
}
t.Fatalf("module.json has no resource %s", id)
return nil
}
func artifact(t *testing.T, m manifest, name string) map[string]any {
t.Helper()
for _, a := range m.Build.Artifacts {
if a["name"] == name {
return a
}
}
t.Fatalf("module.json builds no artifact %s", name)
return nil
}
func TestItPlacesBetterbirdWhereTheCommunityPackageDidAndOnlyAfterItIsGone(t *testing.T) {
func TestItOwnsTheUserJSAndNothingOfTheProfile(t *testing.T) {
m, raw := readManifest(t)
if m.Module != "betterbird" || !reflect.DeepEqual(m.Requires, []string{"x11-display"}) {
t.Fatalf("%+v", m)
}
// Every new build waits for a person's push: the switch removes a package and replaces what a running
// Betterbird has open (ADR 0316 §4).
if m.Upgrade["policy"] != "record" || strings.TrimSpace(m.Upgrade["why"]) == "" {
t.Fatalf("the module is marked for a person's approval, with why: %v", m.Upgrade)
}
var ids []string
for _, r := range m.Resources {
ids = append(ids, r["id"].(string))
}
want := "configuration-dir,user-js,accounts,community-package,installation,owl,policies,spelling,preparation,start,launcher"
if strings.Join(ids, ",") != want {
t.Fatalf("the resources, in the order the node-engine applies them: %v, want %s", ids, want)
}
pkg := resource(t, m, "community-package")
if pkg["type"] != "package" || pkg["package"] != packageFor || pkg["absent"] != true {
t.Fatalf("the community package is declared absent (ADR 0180): %v", pkg)
}
places := map[string]string{"installation": installDir, "owl": filepath.Dir(owlFile),
"policies": filepath.Dir(policyFile), "spelling": installDir + "/defaults/pref", "preparation": filepath.Dir(preparation)}
arts := map[string]string{"installation": "archive", "owl": "owl", "policies": "distribution", "spelling": "defaults", "preparation": "tools"}
for id, path := range places {
r := resource(t, m, id)
if r["type"] != "archive" || r["path"] != path || r["artifact"] != arts[id] || r["owner"] != nil {
t.Errorf("%s is the %s archive unpacked at %s, root's: %v", id, arts[id], path, r)
if r["type"] == "package" {
t.Errorf("betterbird-bin is the AUR's: a package resource asks the node-engine for what it cannot install")
}
if p, _ := r["path"].(string); !strings.HasPrefix(p, "${machine:account-home}/.config/betterbird") {
t.Errorf("a path outside the module's directory: %s", p)
}
}
if strings.Join(ids, ",") != "configuration-dir,user-js" {
t.Fatalf("%v", ids)
}
for _, never := range []string{".thunderbird", "prefs.js", "logins.json", "key4.db"} {
if strings.Contains(raw, never) {
t.Errorf("module.json names %q: the profile is Betterbird's", never)
}
}
s := strings.ToLower(raw)
for _, never := range []string{"/home/", "jochen", "g14", "shanks", "novox.be", "http://", "mediahuis"} {
if strings.Contains(s, never) {
t.Errorf("module.json names %q", never)
}
}
}
func TestTheUserJSAndTheAccountsAreTrustedFilesOfTheModulesOwn(t *testing.T) {
m, _ := readManifest(t)
file := resource(t, m, "user-js")
file := m.Resources[1]
content := file["content"].(string)
if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != true {
t.Fatalf("the user.js begins with the mark, holds the setting, and is trusted (only the terminal or the operator's warrant sets it): %v", file)
if !strings.HasPrefix(content, mark) || !strings.Contains(content, "${setting:prefs}") || file["trusted"] != nil {
t.Fatalf("the user.js begins with the mark, holds the setting, and stays trusted (only the terminal sets it): %v", file)
}
// The default is a comment, so a machine without the setting places a valid, empty user.js.
def := m.Settings["prefs"].(map[string]any)["default"].(string)
if err := validUserJS(strings.ReplaceAll(content, "${setting:prefs}", def)); err != nil {
t.Fatal(err)
}
acc := resource(t, m, "accounts")
if acc["path"] != "${machine:account-home}/"+accountsFile || acc["merge"] != "json" || acc["trusted"] != true ||
acc["mode"] != "0600" || acc["owner"] != "${machine:account}" {
t.Fatalf("the setting accounts reaches the machine as a trusted mergeable file of the account's (ADR 0316 §10, §15): %v", acc)
}
got, err := parseAccounts([]byte(acc["content"].(string)))
if err != nil || len(got) != 0 {
t.Fatalf("its default declares no account: %v %v", got, err)
}
if _, declared := m.Settings["accounts"]; declared {
t.Fatal("accounts is the operator's value and has no default in the definition (ADR 0262)")
}
}
func TestBothPinsAreAURLAVersionAndASha256AndOwlIsNeverInTheRepository(t *testing.T) {
m, _ := readManifest(t)
sha := regexp.MustCompile(`^[0-9a-f]{64}$`)
for _, name := range []string{"archive", "owl"} {
a := artifact(t, m, name)
u, _ := a["url"].(string)
if a["kind"] != "archive" || !strings.HasPrefix(u, "https://") || !sha.MatchString(fmt.Sprint(a["sha256"])) ||
strings.TrimSpace(fmt.Sprint(a["version"])) == "" || a["from"] != nil {
t.Errorf("%s is pinned by URL, version and sha256, and copied from nowhere in the repository: %v", name, a)
}
}
if a := artifact(t, m, "archive"); a["within"] != "betterbird" {
t.Errorf("the release unpacks into betterbird/, whose content is the installation: %v", a)
}
if a := artifact(t, m, "owl"); a["file"] != filepath.Base(owlFile) {
t.Errorf("Owl's xpi is kept whole as %s: %v", filepath.Base(owlFile), a)
}
// Owl's licence forbids copying it to third parties: no xpi anywhere in the module's tree.
filepath.Walk(filepath.Join("..", ".."), func(p string, info os.FileInfo, err error) error {
if err == nil && strings.HasSuffix(strings.ToLower(p), ".xpi") {
t.Errorf("%s is in the repository", p)
}
return nil
})
}
// owlSupports is the range of Betterbird versions each pinned Owl declares in its manifest.json
// (browser_specific_settings.gecko strict_min_version, strict_max_version), read from the vendor's xpi
// when its pin was taken. Moving Owl's pin means reading the new xpi's range into here.
var owlSupports = map[string][2]string{
"1.5.6.1": {"115.0", "156.*"},
}
// owlAccepts says why a Betterbird version is outside the range an Owl version declares, or nil.
func owlAccepts(owl, betterbird string) error {
r, known := owlSupports[owl]
if !known {
return fmt.Errorf("Owl %s's supported range is not recorded in owlSupports: read it from the xpi's manifest.json", owl)
}
major := func(v string) int {
head, _, _ := strings.Cut(v, ".")
n, err := strconv.Atoi(head)
if err != nil {
return -1
}
return n
}
b, lo, hi := major(betterbird), major(r[0]), major(r[1])
if b < 0 || b < lo || b > hi {
return fmt.Errorf("Betterbird %s is outside the range Owl %s supports (%s to %s): a Betterbird Owl does not support "+
"waits for Owl's pin to move (novox/hq ADR 0316 §8)", betterbird, owl, r[0], r[1])
}
return nil
}
func TestABetterbirdOwlDoesNotSupportIsRefused(t *testing.T) {
m, _ := readManifest(t)
owl, bb := fmt.Sprint(artifact(t, m, "owl")["version"]), fmt.Sprint(artifact(t, m, "archive")["version"])
if err := owlAccepts(owl, bb); err != nil {
t.Fatal(err)
}
for _, outside := range []string{"157.0esr-bb1", "102.15.1esr-bb4"} {
if err := owlAccepts(owl, outside); err == nil {
t.Errorf("Betterbird %s was accepted with Owl %s", outside, owl)
}
}
if err := owlAccepts("9.9", bb); err == nil {
t.Error("an Owl whose range nobody read was accepted")
}
}
func TestThePolicyTurnsTheUpdaterOffAndInstallsTheMeshsOwlEverywhere(t *testing.T) {
m, _ := readManifest(t)
if a := artifact(t, m, "distribution"); a["kind"] != "archive" || a["from"] != "install/distribution" {
t.Fatalf("%v", a)
}
raw, err := os.ReadFile(filepath.Join("..", "..", "install", "distribution", "policies.json"))
if err != nil {
t.Fatal(err)
}
if why := policyProblem(raw); why != "" {
t.Fatalf("the module's policies.json: %s", why)
}
entries, _ := os.ReadDir(filepath.Join("..", "..", "install", "distribution"))
if len(entries) != 1 {
t.Fatalf("the distribution directory holds the policy alone: %v", entries)
}
if a := artifact(t, m, "defaults"); a["kind"] != "archive" || a["from"] != "install/defaults-pref" {
t.Fatalf("%v", a)
}
prefs, err := os.ReadFile(filepath.Join("..", "..", "install", "defaults-pref", "vendor-prefs.js"))
if err != nil || !strings.Contains(string(prefs), `pref("spellchecker.dictionary_path", "/usr/share/hunspell");`) {
t.Fatalf("the dictionaries are named as a default, in place of a link: %s %v", prefs, err)
}
}
func TestTheStartPreparesThenStartsBetterbirdAndTheLauncherUsesIt(t *testing.T) {
m, _ := readManifest(t)
start := resource(t, m, "start")
content := start["content"].(string)
if start["path"] != "/usr/local/bin/betterbird" || start["mode"] != "0755" ||
!strings.Contains(content, "\n"+preparation+" prepare ||") || !strings.HasSuffix(content, "\nexec "+startBin+" \"$@\"\n") {
t.Fatalf("the start runs the preparation, says when it did not complete, and starts Betterbird either way: %v", start)
}
launcher := resource(t, m, "launcher")
lc := launcher["content"].(string)
if launcher["path"] != "/usr/local/share/applications/eu.betterbird.Betterbird.desktop" ||
!strings.Contains(lc, "\nExec=/usr/local/bin/betterbird %u\n") || !strings.Contains(lc, "\nIcon="+installDir+"/") ||
!strings.Contains(lc, "\nStartupWMClass=eu.betterbird.Betterbird\n") {
t.Fatalf("the launcher starts the module's start and names the icon in the installation: %v", launcher)
}
if b := artifact(t, m, "tools"); filepath.Base(preparation) != b["binary"] {
t.Fatalf("the preparation runs the tools' binary: %v", b)
}
}
func TestTheToolsAgreeWithTheManifest(t *testing.T) {
m, _ := readManifest(t)
m, raw := readManifest(t)
served := map[string]bool{}
for _, tool := range tools() {
served[tool.Name] = true
@@ -274,12 +94,17 @@ func TestTheToolsAgreeWithTheManifest(t *testing.T) {
for name := range served {
t.Errorf("the bundle serves %s, which module.json does not list", name)
}
b := artifact(t, m, "tools")
if len(m.Build.Artifacts) != 1 {
t.Fatalf("%v", m.Build.Artifacts)
}
b := m.Build.Artifacts[0]
if b["kind"] != "bundle" || b["language"] != "go" || b["from"] != "cmd/betterbird-tools" || b["binary"] != "betterbird-tools" {
t.Errorf("the Go tools bundle: %v", b)
}
// betterbird_accounts proposes the setting accounts through the controller's settings verb (ADR 0317 §9).
if !reflect.DeepEqual(m.Invokes, []string{"seat:mesh-controller.settings"}) {
t.Errorf("the module invokes the controller's settings verb and nothing else: %v", m.Invokes)
s := strings.ToLower(raw)
for _, never := range []string{"/home/", "jochen", "g14", "shanks", "novox.be", "http"} {
if strings.Contains(s, never) {
t.Errorf("module.json names %q", never)
}
}
}
@@ -10,13 +10,6 @@ package main
// module declares. A theme or another preference of the operator's profile can be carried over by
// name (carry), within the same limits as betterbird_prefs: never a credential, never an account.
//
// **It is visibly a test** (the operator, 2026-10-12): on 2026-10-12 at 12:05 a test Betterbird with an empty
// profile carrying the operator's theme looked like the operator's configuration lost. So the test profile
// carries a userChrome.css that marks its main window with a red band reading MESH TEST, and the event is
// called "MESH TEST reminder". The reminder window itself is left as Betterbird draws it, because how it is
// drawn is what the test looks at. The tool's description says a window opens on the operator's screen, so
// a caller tells the operator first.
//
// It lives for at most ten minutes (the unit's RuntimeMaxSec), and stop ends it early and removes the
// profile. The reminder window is then found with xorg_windows (class Calendar, role AlarmWindow) and
// looked at with xorg_screenshot.
@@ -48,7 +41,6 @@ type ReminderAnswer struct {
Carried []string `json:"carried,omitempty"`
Withheld []string `json:"withheld,omitempty"`
Look string `json:"look"`
OnScreen string `json:"on_screen,omitempty"`
Stopped bool `json:"stopped,omitempty"`
}
@@ -58,38 +50,11 @@ func ics(now time.Time, after time.Duration, stamp string) string {
start := now.Add(after + 10*time.Minute)
return strings.Join([]string{"BEGIN:VCALENDAR", "VERSION:2.0", "PRODID:-//novox mesh//betterbird reminder test//EN",
"BEGIN:VEVENT", "UID:mesh-reminder-test-" + stamp + "@mesh", "DTSTAMP:" + f(now), "DTSTART:" + f(start),
"DTEND:" + f(start.Add(30*time.Minute)), "SUMMARY:" + testTitle, "BEGIN:VALARM", "ACTION:DISPLAY",
"DESCRIPTION:" + testTitle, "TRIGGER:-PT10M", "END:VALARM", "END:VEVENT", "END:VCALENDAR", ""}, "\r\n")
"DTEND:" + f(start.Add(30*time.Minute)), "SUMMARY:Mesh reminder test", "BEGIN:VALARM", "ACTION:DISPLAY",
"DESCRIPTION:Mesh reminder test", "TRIGGER:-PT10M", "END:VALARM", "END:VEVENT", "END:VCALENDAR", ""}, "\r\n")
}
// testTitle is the test event's title: what the reminder window shows.
const testTitle = "MESH TEST reminder (a throw-away Betterbird of the mesh, not your calendar)"
// testChrome is the test profile's userChrome.css: a red band reading MESH TEST across the top of every
// Betterbird window but the reminder (windowtype Calendar:AlarmWindow), and a red frame around it, so the test
// Betterbird is never taken for the operator's.
const testChrome = `/* The mesh's reminder test (module betterbird, novox/hq issue 345): a throw-away Betterbird, not yours. */
:root:not([windowtype="Calendar:AlarmWindow"]) {
outline: 6px solid #d00000 !important;
outline-offset: -6px !important;
}
:root:not([windowtype="Calendar:AlarmWindow"]) > body::before {
content: "MESH TEST \2014 a throw-away Betterbird started by the mesh for a reminder test; your mail and calendar are not here. It ends by itself within 10 minutes.";
display: block !important;
background: #d00000 !important;
color: #ffffff !important;
font: bold 15px sans-serif !important;
padding: 6px 12px !important;
text-align: center !important;
}
`
// testPrefs are the test profile's own preferences: its one calendar, and no first-run questions.
//
// betterbird.update.checkIntervalDays 0 is Betterbird's own switch for its update check (MailGlue): a
// fresh profile has never checked, so Betterbird checks at its start and, when betterbird.eu publishes a
// newer build than the installed one, opens the About window on the operator's screen and gives it the
// focus (seen on the workstation 2026-10-12, novox/hq issue 345). The test is about the reminder only.
func testPrefs(calendar string) string {
return strings.Join([]string{
`user_pref("calendar.registry.meshtest.type", "ics");`,
@@ -103,9 +68,7 @@ func testPrefs(calendar string) string {
`user_pref("datareporting.policy.dataSubmissionEnabled", false);`,
`user_pref("toolkit.telemetry.reportingpolicy.firstRun", false);`,
`user_pref("app.update.auto", false);`,
`user_pref("betterbird.update.checkIntervalDays", 0);`,
`user_pref("mail.rights.version", 1);`,
`user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);`,
"",
}, "\n")
}
@@ -162,22 +125,14 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) (
if err := os.WriteFile(calendar, []byte(ics(now, after, strconv.FormatInt(now.Unix(), 10))), 0o600); err != nil {
return a, err
}
// The module's user.js first, then what is carried, then the test's own preferences: the last
// statement wins, so neither can take the test's calendar or its quiet start away.
user := ""
user := testPrefs(strings.TrimPrefix(calendar, m.Root))
if mine, err := readBounded(m.home(declared)); err == nil && validUserJS(string(mine)) == nil {
user = string(mine) + "\n"
user = string(mine) + "\n" + user
}
user += strings.Join(carried, "\n") + "\n" + testPrefs(strings.TrimPrefix(calendar, m.Root))
user += strings.Join(carried, "\n") + "\n"
if err := os.WriteFile(filepath.Join(dir, "profile", "user.js"), []byte(user), 0o600); err != nil {
return a, err
}
if err := os.MkdirAll(filepath.Join(dir, "profile", "chrome"), 0o700); err != nil {
return a, err
}
if err := os.WriteFile(filepath.Join(dir, "profile", "chrome", "userChrome.css"), []byte(testChrome), 0o600); err != nil {
return a, err
}
// The unit removes the profile when it ends, however it ends: at its 10 minutes, on stop, or when the
// test Betterbird is closed.
call := []string{"--user", "--collect", "--quiet", "--unit=" + testUnit, "-p", "RuntimeMaxSec=600",
@@ -191,9 +146,7 @@ func (m *Machine) ReminderTest(after time.Duration, carry []string, stop bool) (
if err := failed(m.cmd(8*time.Second, s.Env(), "systemd-run", call...), "systemd-run", call...); err != nil {
return a, err
}
a.Event = testTitle + ", in a calendar file of the test profile only"
a.OnScreen = "a second Betterbird's window is now on the operator's screen, marked MESH TEST in red; its reminder " +
"window follows when the reminder is due. Both go when the test ends (stop, or 10 minutes)"
a.Event = "Mesh reminder test, in a calendar file of the test profile only"
a.Due = now.Add(after).UTC().Format(time.RFC3339)
return a, nil
}
@@ -1,16 +0,0 @@
package main
import (
"encoding/json"
"errors"
"os"
"testing"
)
// No test reaches the network or the mesh: Betterbird's list of published builds and the controller's
// settings verb answer from here, and a test that needs them says what they answer.
func TestMain(m *testing.M) {
publishedBuilds = func() (string, error) { return "", errors.New("no network in tests") }
asker = func(string, any) (json.RawMessage, error) { return nil, errors.New("no mesh in tests") }
os.Exit(m.Run())
}
@@ -1,3 +0,0 @@
// Written by the mesh (module betterbird, novox/hq ADR 0316 §3): the system's spelling dictionaries, named
// as a default preference in place of the link the community package made.
pref("spellchecker.dictionary_path", "/usr/share/hunspell");
@@ -1,18 +0,0 @@
{
"policies": {
"DisableAppUpdate": true,
"ExtensionSettings": {
"owl@beonex.com": {
"installation_mode": "force_installed",
"install_url": "file:///opt/betterbird/owl/owl.xpi",
"updates_disabled": true
}
},
"Preferences": {
"mail.shell.checkDefaultClient": {
"Value": false,
"Status": "locked"
}
}
}
}
+2 -103
View File
@@ -1,21 +1,14 @@
{
"module": "betterbird",
"version": "1",
"upgrade": {
"policy": "record",
"why": "a new build may remove the community package betterbird-bin and replaces the installation in /opt/betterbird, which a running Betterbird has open: the switch on each node and every upgrade wait for the operator, who quits Betterbird first and then pushes (novox/hq ADR 0316 §4, §5)"
},
"requires": [
"x11-display"
],
"invokes": [
"seat:mesh-controller.settings"
],
"settings": {
"prefs": {
"kind": "preference",
"default": "// none",
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345). An account's preferences (mail.account., mail.identity., mail.server., mail.smtpserver. and the account lists) are refused here: an account is declared in the setting accounts, so it has one owner (novox/hq ADR 0316 §11)"
"why": "Betterbird runs with its own defaults until a machine's assignment names a preference: one line of user_pref(\"name\", value); statements, which the module's user.js holds and Betterbird reads at its start (novox/hq issue 345)"
}
},
"tools": [
@@ -25,7 +18,6 @@
"betterbird_log",
"betterbird_restart",
"betterbird_check",
"betterbird_accounts",
"betterbird_reminder_test"
],
"resources": [
@@ -42,68 +34,7 @@
"path": "${machine:account-home}/.config/betterbird/user.js",
"owner": "${machine:account}",
"mode": "0644",
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n",
"trusted": true
},
{
"id": "accounts",
"type": "file",
"path": "${machine:account-home}/.config/betterbird/accounts.json",
"owner": "${machine:account}",
"mode": "0600",
"merge": "json",
"content": "{\n \"accounts\": {}\n}\n",
"trusted": true
},
{
"id": "community-package",
"type": "package",
"package": "betterbird-bin",
"absent": true
},
{
"id": "installation",
"type": "archive",
"path": "/opt/betterbird",
"artifact": "archive"
},
{
"id": "owl",
"type": "archive",
"path": "/opt/betterbird/owl",
"artifact": "owl"
},
{
"id": "policies",
"type": "archive",
"path": "/opt/betterbird/distribution",
"artifact": "distribution"
},
{
"id": "spelling",
"type": "archive",
"path": "/opt/betterbird/defaults/pref",
"artifact": "defaults"
},
{
"id": "preparation",
"type": "archive",
"path": "/usr/local/lib/mesh-betterbird",
"artifact": "tools"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/betterbird",
"mode": "0755",
"content": "#!/bin/sh\n# Betterbird's start, written by the mesh (module betterbird, novox/hq ADR 0316). Replaced at every push.\n#\n# Before Betterbird starts, the module's tools place its user.js in the profile (the setting prefs, and the\n# accounts the setting accounts declares) and add the declared accounts to the profile's account list. They do\n# it only while no Betterbird of this account runs, because Betterbird rewrites its saved preferences when it quits; a\n# second start while one runs only hands over to it. What the preparation could not do it says here, and\n# betterbird_check says it too. Betterbird starts either way: its configuration never holds the mail back.\n/usr/local/lib/mesh-betterbird/betterbird-tools prepare || echo \"betterbird: the mesh's preparation did not complete; betterbird_check says why\" >&2\nexec /opt/betterbird/betterbird \"$@\"\n"
},
{
"id": "launcher",
"type": "file",
"path": "/usr/local/share/applications/eu.betterbird.Betterbird.desktop",
"mode": "0644",
"content": "[Desktop Entry]\nVersion=1.0\nType=Application\nName=Betterbird\nGenericName=Mail Client\nComment=Send and receive mail with Betterbird\nExec=/usr/local/bin/betterbird %u\nIcon=/opt/betterbird/chrome/icons/default/default128.png\nTerminal=false\nCategories=Network;Email;\nMimeType=message/rfc822;x-scheme-handler/mailto;application/x-xpinstall;application/x-extension-ics;text/calendar;text/vcard;text/x-vcard;x-scheme-handler/webcal;x-scheme-handler/webcals;x-scheme-handler/mid;\nStartupNotify=true\nStartupWMClass=eu.betterbird.Betterbird\nKeywords=Email;E-mail;Calendar;Thunderbird;\nActions=ComposeMessage;OpenAddressBook;\n# Written by the mesh (module betterbird, novox/hq ADR 0316 §3), in place of the community package's entry:\n# it starts Betterbird through the module's start, and names the icon in the installation, with no link.\n\n[Desktop Action ComposeMessage]\nName=Write new message\nExec=/usr/local/bin/betterbird -compose\n\n[Desktop Action OpenAddressBook]\nName=Open address book\nExec=/usr/local/bin/betterbird -addressbook\n"
"content": "// Betterbird's user.js (module betterbird, novox/hq issue 345). Owned by the mesh: the node-engine\n// writes it here at every push, and the module's tools copy it whole into Betterbird's profile\n// (betterbird_user_js with apply, betterbird_restart), where Betterbird reads it at its start and lets it\n// win over the preferences Betterbird saves itself. Change the module's setting prefs, never this file or the profile's copy.\n//\n// Only comments and user_pref(\"name\", value); statements; the tools refuse anything else, and any\n// preference whose name can hold a credential.\n${setting:prefs}\n"
}
],
"build": {
@@ -118,38 +49,6 @@
"loads": [
"betterbird-tools"
]
},
{
"name": "archive",
"kind": "archive",
"url": "https://www.betterbird.eu/downloads/LinuxArchive/betterbird-153.4.0esr-bb10.en-US.linux-x86_64.tar.xz",
"sha256": "38dd24890a66b4cf9a45bc24e15192e00c26b929e1bb8df7974c16a4539129e2",
"version": "153.4.0esr-bb10",
"within": "betterbird",
"names-on-purpose": {
"www.betterbird.eu": "Betterbird's project publishes its Linux release there; the build seat fetches it once, by its pinned sha256 (novox/hq ADR 0316 §2)"
}
},
{
"name": "owl",
"kind": "archive",
"url": "https://www.beonex.com/owl/owl.xpi",
"sha256": "a54cc3d52113b34f97942ee19190947dee23561e2217f18db3d80ac023991221",
"version": "1.5.6.1",
"file": "owl.xpi",
"names-on-purpose": {
"www.beonex.com": "Owl's vendor publishes its xpi there; the build seat fetches it once, by its pinned sha256 (novox/hq ADR 0316 §6)"
}
},
{
"name": "distribution",
"kind": "archive",
"from": "install/distribution"
},
{
"name": "defaults",
"kind": "archive",
"from": "install/defaults-pref"
}
]
}
-27
View File
@@ -341,33 +341,6 @@ export class GiteaClient {
return out;
}
/** **Every open pull request of a repository**, page by page (novox/hq ADR 0323 §4): what a trunk's move puts out of
* date, however many are open — not the twenty most recently updated a pass announces from. Bounded at 20 pages
* of 50; a repository with more is said, and the rest are left to the merge tool's own comparison. */
async listOpenPulls(owner: string, repo: string): Promise<GiteaPull[]> {
const out: GiteaPull[] = [];
for (let page = 1; page <= 20; page++) {
const batch = await this.listPullRequests(owner, repo, { state: "open", limit: "50", page: String(page) });
out.push(...batch);
if (batch.length < 50) return out;
}
console.error(`[gitea] ${owner}/${repo} has more than 1000 open pull requests; only the first 1000 are read`);
return out;
}
/** The commit a branch is at now: a trunk's head (novox/hq ADR 0323). */
async branchHead(owner: string, repo: string, branch: string): Promise<string> {
const b = await this.request<any>(`${repoPath(owner, repo)}/branches/${pathSegments("branch", branch)}`);
const sha = String(b?.commit?.id ?? "");
if (!/^[0-9a-f]{40,64}$/i.test(sha)) throw new Error(`the forge named no commit for ${owner}/${repo}'s ${branch}`);
return sha;
}
/** One repository, as the forge holds it: its clone URL above all, for an announcement. */
async getRepo(owner: string, repo: string): Promise<GiteaRepo> {
return GiteaClient.mapRepo(await this.request<any>(repoPath(owner, repo)));
}
async getPullRequest(owner: string, repo: string, index: number): Promise<GiteaPull> {
indexOf(index);
return GiteaClient.mapPull(await this.request<any>(`${repoPath(owner, repo)}/pulls/${index}`));
+1 -23
View File
@@ -18,7 +18,7 @@
// restart, when the check was opened is read from the forge's own history of the gate.
import { CHECK_CONTEXT, pendingWords } from "./pulls.js";
import type { CommitStatus, HeadCheck } from "./client.js";
import type { CommitStatus } from "./client.js";
/** The build seat's `started`, as mesh-controller's link.BuildStart publishes it. */
export interface BuildStarted {
@@ -37,8 +37,6 @@ export interface GateForge {
gateOf(owner: string, repo: string, sha: string): Promise<{ state: string; updated_at?: string } | undefined>;
/** The head's mesh/merge-gate statuses, every one the forge keeps, newest first (novox/hq issue 456). */
gateHistory(owner: string, repo: string, sha: string): Promise<GateStatus[]>;
/** The head's statuses by context, the newest of each, with their words (novox/hq ADR 0323). */
headChecks(owner: string, repo: string, sha: string): Promise<Record<string, HeadCheck>>;
}
/** One status of a head's gate as the forge keeps it: its state, its words, and when it was set. */
@@ -176,26 +174,6 @@ export class GateWriter {
});
}
/**
* **A trunk's move puts the head's verdict out of date** (novox/hq ADR 0323 §4): its merge check statuses are put back
* to pending, with the reason, in their turn after every write of the head already asked for — a verdict in flight
* lands first. What is written is decided in the turn, from the statuses the forge holds then (`decide`, trunk.ts's
* outOfDateStatuses): a check on its way, or a verdict already judged against the trunk's head, is left as it is. It
* opens no check — nothing is asked — so a verdict of a check asked before it keeps its weight. Answers what it wrote;
* a write that fails fails the caller, and the move is weighed again on the next pass.
*/
outOfDate(owner: string, repo: string, sha: string,
decide: (checks: Record<string, HeadCheck>) => CommitStatus[] | Promise<CommitStatus[]>, target?: string): Promise<CommitStatus[]> {
const h = this.head(owner, repo, sha);
const written: CommitStatus[] = [];
return this.turn(h, async () => {
for (const s of await decide(await this.forge.headChecks(owner, repo, sha))) {
await this.forge.setCommitStatus(owner, repo, sha, target ? { ...s, target_url: target } : s);
written.push(s);
}
}).then(() => written);
}
/**
* A build machine took an ask of this head: said while no verdict was heard, and only for the check now pending —
* a start from before the pending words were set (a run before a recheck, the stream's history) is not this check's.
+34 -68
View File
@@ -17,13 +17,8 @@
// Consumes node-build-agent.started — a build machine took an ask — and says, on the head a pending check is of,
// which machine builds it since when (novox/hq issue 446). This module is the one writer of mesh/merge-gate.
//
// **A trunk's move puts every open verdict into it out of date** (novox/hq ADR 0323 §4, trunk.ts): each pass reads the
// head of every trunk an open pull request merges into, and when one moved, the merge check statuses of every open pull
// request into it go back to pending with the reason. Nothing is run for it. A verdict that lands judged against a trunk
// commit that is no longer the trunk's head is said out of date in place of its pass or failure.
//
// Serves gitea_commit_status and gitea_merge_pull_request from this bundle, not the tools', so a recheck's pending and
// the merge tool's ask of a check reach that one writer (novox/hq issue 456, ADR 0323).
// Serves gitea_commit_status from this bundle, not the tools', so a recheck's pending reaches that one writer
// (novox/hq issue 456).
//
// Every pull request the forge holds is announced, whatever its repository: the controller holds the
// module graph and decides what is checked — a repository is never asked to opt in.
@@ -39,12 +34,10 @@
import { emit, on } from "@novox/mesh-sdk/events";
import { registerModuleTools } from "@novox/mesh-sdk/tools";
import { commitStatusTool } from "./delivery.js";
import { mergeTool } from "./merge.js";
import { GiteaClient, movedSince } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT, commentFor, headsToAnnounce, statusesFor, type Announced, type Checked,
type RepoCheckFacts, reposToLook } from "./pulls.js";
import { GateWriter, clonedAs, type BuildStarted } from "./gate.js";
import { announcementOf, keepTook, lateVerdictComment, outOfDateWords, sameCommit, tookFile, weighTrunks } from "./trunk.js";
// Without a way to a token — configured, or mintable with the admin account (token.ts) — there is
// nothing to watch; log and stay quiet rather than crash the runtime. With one, the first poll mints
@@ -257,8 +250,37 @@ async function pollPulls(client: GiteaClient): Promise<void> {
const key = `${repo.full_name}#${pull.number}`;
const fresh = primedPulls || (!!pull.updated_at && pull.updated_at > dayAgo);
if (fresh) {
// The announcement the merge tool makes too when it asks a check (trunk.ts, novox/hq ADR 0323 §5).
await emit("pull.updated", await announcementOf(client, repo.owner, repo.name, repo.clone_url, pull));
const files = await client.listPullFiles(repo.owner, repo.name, pull.number);
const head = String(pull.head_sha);
// What the controller maps the change onto the mesh's module graph with (novox/hq ADR 0237 as
// amended): which changed directories hold a module at the head — the merge's rule (issue 278) —
// and whether the head holds the repository's own merge-check.sh. Not said when it could not be
// read; the controller then reads the change as touching everything built from the repository.
let moduleDirs: string[] | null = null;
let mergeCheck: boolean | null = null;
try {
if (!files.truncated) moduleDirs = await client.moduleDirsAt(repo.owner, repo.name, head, files.paths);
mergeCheck = await client.holdsFile(repo.owner, repo.name, head, "merge-check.sh");
} catch (err) {
console.error(`[gitea] ${key}: what the head holds could not be read — ${err instanceof Error ? err.message : err}`);
}
await emit("pull.updated", {
owner: repo.owner,
repo: repo.name,
number: pull.number,
title: pull.title,
body: pull.body,
base: pull.base,
head: pull.head,
head_sha: pull.head_sha,
clone_url: repo.clone_url,
html_url: pull.html_url,
paths: files.paths,
paths_truncated: files.truncated,
removed: files.removed,
...(moduleDirs ? { module_dirs: moduleDirs, module_dirs_said: true } : {}),
...(mergeCheck !== null ? { merge_check: mergeCheck, merge_check_said: true } : {}),
});
// Said, so an operator knows the mesh was asked to check it.
console.log(`[gitea] announced ${key} at ${String(pull.head_sha).slice(0, 8)} to be checked before it merges`);
// Pending until the verdict comes, so the pull request says a check is running rather than nothing — and
@@ -268,52 +290,17 @@ async function pollPulls(client: GiteaClient): Promise<void> {
heads[key] = String(pull.head_sha);
changed = true;
}
if (await trunksMoved(client, repo.owner, repo.name)) changedTrunks = true;
}
if (!primedPulls || changed) keepHeads();
if (changedTrunks) keepTrunks();
changedTrunks = false;
primedPulls = true;
lastPullLook = began;
}
// **The trunks' heads, as this module last read them** (novox/hq ADR 0323 §4), by `owner/repo:branch`: kept beside the
// other records, so a move made while this module was away is weighed on its first pass back. With no record — the
// first pass after this rolls out — a trunk's head is taken as it is and nothing is put out of date: the verdicts
// then standing are weighed on the trunk's next move, and the merge tool compares commits whatever this holds.
const trunksRecord = process.env.MESH_GITEA_STATE_DIR ? join(process.env.MESH_GITEA_STATE_DIR, "trunks-seen.json") : null;
let trunks: Record<string, string> = {};
let changedTrunks = false;
if (trunksRecord && existsSync(trunksRecord)) {
try {
trunks = (JSON.parse(readFileSync(trunksRecord, "utf8")) as { trunks: Record<string, string> }).trunks ?? {};
} catch {
// An unreadable record is no record: the next pass takes each trunk's head as it is.
}
}
function keepTrunks(): void {
if (!trunksRecord) return;
mkdirSync(join(trunksRecord, ".."), { recursive: true });
const tmp = trunksRecord + ".tmp";
writeFileSync(tmp, JSON.stringify({ trunks }));
renameSync(tmp, trunksRecord);
}
/** Whether a trunk of one repository moved, weighed with this module's forge and its one writer (trunk.ts). */
function trunksMoved(client: GiteaClient, owner: string, name: string): Promise<boolean> {
return weighTrunks(client, gates, trunks, owner, name);
}
// **The verdict, set where the pull request shows it.** Every verdict is the head commit's status; one
// that is not a pass also leaves the check's own account as a comment, so the reason is read where the
// change is reviewed. An error — the check could not run — is the forge's `error`, never a success.
async function setVerdict(client: GiteaClient, event: { body: unknown }): Promise<void> {
const c = (event.body ?? {}) as Checked;
if (!c.group && c.owner && c.repo && c.took) {
// How long it took, for the merge tool to say how long the next one may (novox/hq ADR 0323 §5).
keepTook(tookFile(process.env.MESH_GITEA_STATE_DIR), `${c.owner}/${c.repo}`,
{ took: c.took, at: new Date().toISOString(), head: c.commit });
}
if (c.group) {
// A delivery group's composed check (novox/hq ADR 0239): its verdict is the group owner's to say, on each
// member's head, as mesh/delivery-group — never this head's merge gate.
@@ -331,26 +318,6 @@ async function setVerdict(client: GiteaClient, event: { body: unknown }): Promis
target = pull?.html_url || undefined;
base = pull?.base || undefined;
}
// **A verdict judged against a trunk commit that is no longer the trunk's head is out of date when it lands**
// (novox/hq ADR 0323 §4): a check asked before the trunk moved. It is said so, pending with the reason, in place of
// its pass or failure; a trunk that cannot be read lets the verdict stand, and the merge tool compares the commits.
const trunk = base ?? c.plan?.base;
if (c.trunk && trunk) {
const head = await client.branchHead(c.owner, c.repo, trunk).catch(() => undefined);
if (head && !sameCommit(c.trunk, head)) {
const description = outOfDateWords(c.trunk, head);
for (const context of [CHECK_CONTEXT, ...(c["repo-check"] ? [REPO_CHECK_CONTEXT] : [])]) {
await client.setCommitStatus(c.owner, c.repo, c.commit, { state: "pending", context, description,
...(target ? { target_url: target } : {}) });
}
// Its account is kept and shown all the same (ADR 0323 §7): what it judged, and that it is out of date.
const said = commentFor(c, await repoCheckFacts(client, c, trunk));
if (c.number) await client.addComment(c.owner, c.repo, c.number, lateVerdictComment(c, said, description));
console.log(`[gitea] ${c.owner}/${c.repo}#${c.number ?? "?"} at ${c.commit.slice(0, 8)}: merge check ${c.verdict}, ` +
`${description}`);
return;
}
}
const facts = await repoCheckFacts(client, c, base ?? c.plan?.base);
for (const status of statusesFor(c, facts)) {
await client.setCommitStatus(c.owner, c.repo, c.commit, target ? { ...status, target_url: target } : status);
@@ -383,7 +350,6 @@ const gates = new GateWriter({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gateOf: async (owner, repo, sha) => (await (gitea as GiteaClient).headChecks(owner, repo, sha))[CHECK_CONTEXT],
gateHistory: (owner, repo, sha) => (gitea as GiteaClient).contextHistory(owner, repo, sha, CHECK_CONTEXT),
headChecks: (owner, repo, sha) => (gitea as GiteaClient).headChecks(owner, repo, sha),
});
// **The status tool is served here, beside the writer** (novox/hq issue 456): the runtime starts each bundle as its own
@@ -392,7 +358,7 @@ const gates = new GateWriter({
registerModuleTools("gitea", () => (gitea ? [commitStatusTool({
setCommitStatus: (owner, repo, sha, status) => (gitea as GiteaClient).setCommitStatus(owner, repo, sha, status),
gates,
}), mergeTool(gitea, { gates })] : []));
})] : []));
// **A build machine took an ask** (the build seat's `started`, novox/hq ADR 0157): said on the head it checks, when
// the head is an open pull request's this module announced. Any other build — a branch, a merge's — says nothing.
-183
View File
@@ -1,183 +0,0 @@
// The forge's merge tool, `gitea_merge_pull_request` (novox/hq ADR 0266, issue 381, ADR 0323).
//
// **Served from the watcher's bundle, beside the merge check's one writer** (index.ts, gate.ts; novox/hq issue 456):
// when a verdict is judged against an earlier trunk commit than the trunk's head, the tool asks the head's check and
// puts its statuses back to pending through that writer, so where the check runs is said and no second check is asked.
// Each bundle the runtime serves is its own process; from the tools' bundle the pending bypassed the writer's chain.
import { emit } from "@novox/mesh-sdk/events";
import type { ToolDefinition } from "@novox/mesh-sdk/tools";
import type { CommitStatus, GiteaClient } from "./client.js";
import { mergeableUnder, newestByContext } from "./checks.js";
import { MERGE_METHODS, Refused, checkName } from "./guard.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT } from "./pulls.js";
import { announcementOf, askedWords, outOfDateWords, tookFile, tookWords, twelve, verdictAtTrunk } from "./trunk.js";
/** A head named for a merge: at least 12 hex characters of its sha (novox/hq issue 381), up to a sha-256 commit's 64. */
const REVIEWED_HEAD = /^[0-9a-f]{12,64}$/;
/** What the merge tool reaches beside the forge: the merge check's one writer, how an event is said, the module's
* state directory, and the clock. */
export interface MergeDeps {
gates: { rechecked(owner: string, repo: string, sha: string, status: CommitStatus): Promise<void> };
emit?: (type: string, body: unknown) => Promise<void>;
stateDir?: string;
now?: () => Date;
}
export function mergeTool(gitea: GiteaClient, given: MergeDeps): ToolDefinition {
const deps = { ...given, emit: given.emit ?? emit, stateDir: given.stateDir ?? process.env.MESH_GITEA_STATE_DIR,
now: given.now ?? (() => new Date()) };
return {
name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head " +
"and every status its base branch's protection requires succeeded there (mesh/merge-gate, mesh/repo-check), " +
"judged against the trunk's head as it is now, optionally deleting the source branch afterwards. Refused " +
"otherwise, with nothing merged: no tool merges past a check, and none merges a head that moved after it was " +
"reviewed. A verdict judged against an earlier trunk commit is refused, naming both, and the head's check against " +
"the trunk's head is asked once, with how long the last check of the repository took: ask again at the same head " +
"once it passes — the tool never merges by itself.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
number: { type: "number", description: "the PR number" },
head: { type: "string", description: "the commit you reviewed, as its sha or at least its first 12 characters: " +
"refused unless it is the pull request's head" },
method: { type: "string", description: "merge (default) | fast-forward-only | squash | rebase | rebase-merge. " +
"Only merge and fast-forward-only can be read back to prove the head you named is what landed" },
delete_branch: { type: "boolean", description: "delete the source branch after merge (default true)" },
},
run: async (args) => {
const owner = checkName("owner", args.owner);
const repo = checkName("name", args.repo);
const number = Number(args.number);
const method = args.method ? String(args.method) : "merge";
if (!MERGE_METHODS.has(method)) throw new Refused(`${method} is not a merge method: ${[...MERGE_METHODS].join(", ")}`);
const deleteBranch = args.delete_branch === undefined ? true : Boolean(args.delete_branch);
// **Only a head whose required statuses all succeeded** (novox/hq ADR 0266): read here rather than left
// to the forge, whose administrator — the account this tool's token is minted for — may merge past a
// rule that lets it. A rule that cannot be read is no rule passed.
// **Only the head that was reviewed** (novox/hq issue 381): mesh-controller#197 was reviewed at 065cfa0d,
// force-pushed to 3ced96f, checked green there, and merged at 3ced96f, because this tool merged whatever the
// head was and nothing told it which head the review had approved. The caller names it; a head that moved
// since is refused, and the forge is handed the same head, so a push between this read and the merge is
// refused by the forge too.
const named = args.head === undefined ? "" : String(args.head).trim().toLowerCase();
if (!named) {
throw new Refused(`${owner}/${repo}#${number} is not merged: name the head you reviewed (head), so that only ` +
`that commit can land (novox/hq issue 381). Nothing was merged`);
}
// Twelve characters, not seven: an author who force-pushes can make a commit whose first seven match the
// reviewed one in seconds, and the review would then merge it. Twelve cannot be matched that way.
if (!REVIEWED_HEAD.test(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: head ${named} does not name one commit — give its ` +
`sha, or at least its first 12 characters. Nothing was merged`);
}
const pull = await gitea.getPullRequest(owner, repo, number);
if (pull.merged) throw new Refused(`${owner}/${repo}#${number} is already merged`);
if (!pull.head_sha || !pull.base) throw new Refused(`${owner}/${repo}#${number} names no head commit or base branch`);
if (!pull.head_sha.toLowerCase().startsWith(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its head is ${pull.head_sha.slice(0, 12)}, not the ` +
`${named} you named. If you reviewed an earlier head, review the one it has now, then name it. Nothing was merged`);
}
const rule = await gitea.branchProtection(owner, repo, pull.base);
// Only into a branch whose protection requires the checks: a pull request whose base was moved to a
// branch nothing guards would merge past every check (novox/hq ADR 0266).
if (!rule || !rule.enable_status_check || !(rule.status_check_contexts ?? []).filter(Boolean).length) {
throw new Refused(`${owner}/${repo}#${number} merges into ${pull.base}, which no protection rule requiring ` +
`the checks guards: a tool merges only into such a branch (novox/hq ADR 0266). Nothing was merged`);
}
// **The verdict must be judged against the trunk's head** (novox/hq ADR 0323 §5, issue 341): a check judges the
// head merged into a named trunk commit, and a merge of another trunk is not the one that check judged. Compared
// here at the moment the merge is asked, not left to the statuses the trunk's move resets a pass later. When they
// differ nothing is merged, and the head's check against the trunk's head is asked once — the announcement a new
// head makes — never again while one is on its way. The caller asks again at the reviewed head once it passes.
const trunkHead = await gitea.branchHead(owner, repo, pull.base);
const checks = await gitea.headChecks(owner, repo, pull.head_sha);
const at = verdictAtTrunk(checks, trunkHead);
if (!at.current) {
const judged = at.judged ? `was judged against trunk ${twelve(at.judged)}` : "names no trunk commit it was judged against";
const took = tookWords(tookFile(deps.stateDir), `${owner}/${repo}`);
const head = twelve(pull.head_sha), now = twelve(trunkHead);
if (at.onItsWay) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}. Its check is already asked and not yet answered (${checks[CHECK_CONTEXT]?.description}), so it is not ` +
`asked again; ${took}. Ask again at ${head} once both statuses pass: this tool never merges by itself ` +
`(novox/hq ADR 0323). A check that never answers is asked again with mesh-delivery.recheck. Nothing was merged`);
}
// Pending first, so a second ask finds the check on its way; then the head is announced as a new head is. An
// announcement that cannot be made puts the statuses back to out of date, so the next ask asks again rather
// than wait for ever on a check nobody asked.
const contexts = [CHECK_CONTEXT, ...(checks[REPO_CHECK_CONTEXT] ? [REPO_CHECK_CONTEXT] : [])];
// Through the module's one writer of the merge check (gate.ts, novox/hq issue 456): the pending opens a new
// check of the head, so where it runs is said ("building on …") and its verdict is not weighed as older.
const say = async (description: string) => {
for (const context of contexts) {
await deps.gates.rechecked(owner, repo, String(pull.head_sha), { state: "pending", context, description,
...(pull.html_url ? { target_url: pull.html_url } : {}) });
}
};
await say(askedWords(deps.now(), trunkHead));
try {
const cloneURL = (await gitea.getRepo(owner, repo)).clone_url;
await deps.emit("pull.updated", await announcementOf(gitea, owner, repo, cloneURL, pull));
} catch (err) {
await say(outOfDateWords(at.judged, trunkHead)).catch(() => {});
throw new Error(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}, and its check could not be asked: ${err instanceof Error ? err.message : err}. Nothing was merged`);
}
throw new Refused(`${owner}/${repo}#${number} is not merged: its verdict ${judged}, and ${pull.base} is now ` +
`${now}. The check of head ${head} merged into trunk ${now} was asked; ${took}. Ask again at ${head} once both ` +
`statuses pass: this tool never merges by itself (novox/hq ADR 0323). In a busy repository another merge may ` +
`move the trunk again first; this is then refused and asked once more. A check that never answers is asked ` +
`again with mesh-delivery.recheck. Nothing was merged`);
}
const statuses = newestByContext((await gitea.combinedStatus(owner, repo, pull.head_sha)).statuses);
const verdict = mergeableUnder(pull.base, rule, statuses);
if (verdict.mergeable !== true) {
throw new Refused(`${owner}/${repo}#${number} is not merged: ${verdict.says}. A pull request merges once ` +
`its checks succeed; no tool merges past them (novox/hq ADR 0266)`);
}
await gitea.mergePullRequest(owner, repo, number, method, deleteBranch, pull.head_sha);
// The merge commit only exists now; answered so the caller can follow what is built from it.
// pull.merged is the events entrypoint's to announce (index.ts), once, within its poll.
// From here the merge has happened. Nothing below may read as if it had not: a read-back that fails or cannot
// be done is said beside merged: true, and only a proven other head is an error — one that says the merge happened.
const answer: Record<string, unknown> = { merged: true, number, method, head: pull.head_sha, deleted_branch: deleteBranch };
let landed: string | undefined;
try {
const merged = await gitea.getPullRequest(owner, repo, number);
answer.merge_commit_sha = merged.merge_commit_sha;
// What landed, read back from the commit the merge made rather than from the forge's record of the pull
// request, which is the very thing a race would leave behind. A merge commit's second parent is the head it
// joined; a fast-forward lands the head itself. A squash or a rebase makes new commits that name no head.
if (!merged.merge_commit_sha) {
answer.read_back = "the forge named no merge commit, so what landed cannot be read back";
} else if (method === "merge") {
const parents = await gitea.commitParents(owner, repo, merged.merge_commit_sha);
if (parents.length === 2) landed = parents[1];
else answer.read_back = `the merge commit has ${parents.length} parent(s), not two: what landed cannot be read back`;
} else if (method === "fast-forward-only") {
// That the forge names the head itself as a fast-forward's merge commit is assumed, not seen on this forge:
// a different sha is said, not raised as another head having landed.
if (merged.merge_commit_sha === pull.head_sha) landed = merged.merge_commit_sha;
else answer.read_back = `the forge named ${merged.merge_commit_sha.slice(0, 12)} as this fast-forward's commit, ` +
`not the head: that the head landed could not be confirmed`;
} else {
answer.read_back = `a ${method} makes new commits that name no head: what landed cannot be read back`;
}
} catch (err) {
answer.read_back = `the merge happened, but reading back what landed failed: ${(err as Error).message}`;
}
// Loud, not a field beside merged: true that a caller reading only that would miss. The merge cannot be taken
// back by this tool; what must not happen is that it reads as the merge that was asked for.
if (landed !== undefined && landed !== pull.head_sha) {
throw new Error(`${owner}/${repo}#${number} WAS merged (${answer.merge_commit_sha}), but the forge merged ` +
`${landed.slice(0, 12)}, not the ${pull.head_sha.slice(0, 12)} it was handed and you reviewed: a commit nobody ` +
`reviewed has landed. Read it before anything is built from it (novox/hq issue 381)`);
}
if (landed !== undefined) answer.landed = landed;
return answer;
},
};
}
+2 -22
View File
@@ -59,12 +59,6 @@ export interface Checked {
plan?: ChangePlan;
/** Set on a delivery group's composed check (novox/hq ADR 0239): not this head's merge gate. */
group?: string;
/** The trunk commit the head was merged into and judged as, twelve characters (novox/hq ADR 0323 §3); absent from a
* controller whose build seat judged the head alone. Beside are the other repositories' commits the check read,
* by directory; took how long it ran. */
trunk?: string;
beside?: Record<string, string>;
took?: string;
}
/** A change plan, as the controller says it (mesh-controller internal/link, ChangePlan). */
@@ -205,25 +199,17 @@ export function statusFor(c: Checked): CommitStatus {
const description = builds(c.plan)
? described(gate.verdict, `${c.plan!.summary}; ${gate.summary}`)
: described(gate.verdict, gate.summary, gate.modules, gate.dependents);
return judgedFirst(c, { state: stateOf(gate.verdict), context: CHECK_CONTEXT, description });
return { state: stateOf(gate.verdict), context: CHECK_CONTEXT, description };
}
/** Every status a verdict sets: the gate's, and the repository's own check's when it was said. */
export function statusesFor(c: Checked, facts: RepoCheckFacts = {}): CommitStatus[] {
const out = [statusFor(c)];
const repo = c["repo-check"];
if (repo) out.push(judgedFirst(c, repoStatus(repo, facts)));
if (repo) out.push(repoStatus(repo, facts));
return out;
}
/** **A status names what its verdict judged, first** (novox/hq ADR 0323 §3): "head 1a2b3c4d5e6f merged into trunk
* 7a8b9c0d1e2f — pass: …", so the forge's 140 characters never clip it away; the trunk's move is weighed by it. A
* verdict that names no trunk commit is said as before. */
function judgedFirst(c: Checked, status: CommitStatus): CommitStatus {
if (!c.trunk) return status;
return { ...status, description: clipped(`head ${c.commit.slice(0, 12)} merged into trunk ${c.trunk.slice(0, 12)} — ${status.description}`) };
}
/** The comment a verdict leaves on its pull request, with the check's own account: the change plan of a
* change that builds something (novox/hq ADR 0238), and why, when the gate is not a pass or the
* repository's own check failed or could not run. A repository with no merge-check.sh, touching nothing,
@@ -234,12 +220,6 @@ export function commentFor(c: Checked, facts: RepoCheckFacts = {}): string | nul
const wrong = repoWrong(repo, facts);
if (gate.verdict === "pass" && !wrong && !builds(c.plan)) return null;
const lines = [`**Merge check** at \`${c.commit.slice(0, 8)}\``, ""];
if (c.trunk) {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and what was read beside it.
const beside = Object.entries(c.beside ?? {}).map(([d, sha]) => `${d} \`${sha}\``).join(", ");
lines.splice(1, 0, `Judged: head \`${c.commit.slice(0, 12)}\` merged into trunk \`${c.trunk.slice(0, 12)}\`` +
(beside ? `; read beside it: ${beside}` : ""));
}
lines.push(`- \`${CHECK_CONTEXT}\`: **${saidAs(gate.verdict).toUpperCase()}** — ${gate.summary}` +
(gate.modules?.length ? ` (modules: ${gate.modules.join(", ")}` +
(gate.dependents?.length ? `; built after them: ${gate.dependents.join(", ")}` : "") + ")" : ""));
+10 -111
View File
@@ -7,14 +7,11 @@
import { test, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { mkdtempSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { readFileSync } from "node:fs";
import { GiteaClient } from "../dist/client.js";
import { ConfiguredToken } from "../dist/token.js";
import { getGiteaTools } from "../dist/tools/index.js";
import { mergeTool } from "../dist/merge.js";
import { commitStatusTool } from "../dist/delivery.js";
import { GateWriter } from "../dist/gate.js";
import { judgeApiCall, judgeProtectionChange, ruleCovers, routedPath } from "../dist/guard.js";
@@ -24,11 +21,7 @@ interface Sent { method: string; path: string; body: any }
const state = {
sent: [] as Sent[],
asked: [] as string[],
statuses: [] as { context: string; state: string; description?: string; updated_at?: string }[],
// The trunk's head (novox/hq ADR 0323): what the merge tool compares the verdict's trunk commit with.
trunk: "",
// What the tools said on the bus.
emitted: [] as { type: string; body: any }[],
statuses: [] as { context: string; state: string }[],
rules: [] as any[],
lastCommit: "2026-01-01T00:00:00Z",
// The pull request's head; the head the merge commit joined (the same, unless a test says otherwise) — the forge's
@@ -43,21 +36,13 @@ const state = {
commitsFail: 0,
};
const REVIEWED = "abc1234def5678901234567890abcdef12345678";
const TRUNK = "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b";
const MOVED = "3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d";
// A verdict judged against a trunk commit, as both statuses say it (novox/hq ADR 0323 §3).
const judgedAt = (trunk: string, state = "success") => [
{ context: "mesh/merge-gate", state, description: `head ${REVIEWED.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: x` },
{ context: "mesh/repo-check", state, description: `head ${REVIEWED.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: y` }];
const protectedMain = () => ({ rule_name: "main", enable_push: false, enable_status_check: true,
status_check_contexts: ["mesh/merge-gate", "mesh/repo-check"], block_admin_merge_override: true });
beforeEach(() => {
state.sent = [];
state.asked = [];
state.statuses = judgedAt(TRUNK);
state.trunk = TRUNK;
state.emitted = [];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "success" }];
state.rules = [protectedMain()];
state.head = REVIEWED;
state.landed = "";
@@ -88,15 +73,6 @@ const server = createServer(async (req, res) => {
const repo = "/repos/novox/mesh-catalog";
if (method === "GET" && path === repo) return json(res, 200, { full_name: "novox/mesh-catalog", default_branch: "main" });
if (method === "GET" && path === `${repo}/branch_protections`) return json(res, 200, state.rules);
if (method === "GET" && path === `${repo}/branches/main`) return json(res, 200, { name: "main", commit: { id: state.trunk } });
if (method === "GET" && path === `${repo}/pulls/7/files`) {
return json(res, 200, url.searchParams.get("page") === "1" ? [{ filename: "modules/gitea/x.ts", status: "modified" }] : []);
}
if (method === "POST" && path === `${repo}/statuses/${state.head}`) {
// Kept as the forge keeps them: the newest of a context first.
state.statuses = [{ ...body, updated_at: new Date().toISOString() }, ...state.statuses.filter((s) => s.context !== body.context)];
return json(res, 201, body);
}
if (method === "GET" && path === "/user") return json(res, 200, { login: "mesh-admin" });
if (method === "GET" && path === "/repos/mesh-admin/scratch/branch_protections") return json(res, 200, []);
if (method === "GET" && /^\/repos\/[^/]+\/[^/]+\/commits$/.test(path)) {
@@ -126,19 +102,11 @@ after(() => new Promise<void>((r) => server.close(() => r())));
const { port } = server.address() as { port: number };
const client = new GiteaClient(`http://127.0.0.1:${port}`, new ConfiguredToken("t"));
// gitea_commit_status is served from the watcher's bundle, beside the merge check's one writer (novox/hq issue 456).
const stateDir = mkdtempSync(join(tmpdir(), "gitea-tools-"));
// The merge check's one writer, as the watcher's bundle holds it: what the merge tool's pending goes through.
const writer = new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [], headChecks: async () => ({}) });
const throughTheWriter: string[] = [];
const gates = { rechecked: (o: string, r: string, sha: string, s: any) => {
throughTheWriter.push(`${sha.slice(0, 12)} ${s.context} ${s.state}`);
return writer.rechecked(o, r, sha, s);
} };
const tools = [...getGiteaTools(client), commitStatusTool({
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s), gates: writer,
}), mergeTool(client, { gates, stateDir, emit: async (type, body) => { state.emitted.push({ type, body }); },
now: () => new Date(2026, 9, 12, 15, 1) })];
setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gates: new GateWriter({ setCommitStatus: (o, r, sha, s) => client.setCommitStatus(o, r, sha, s),
gateOf: async () => undefined, gateHistory: async () => [] }),
})];
const tool = (name: string) => tools.find((t) => t.name === name)!;
const writes = () => state.sent.map((s) => `${s.method} ${s.path}`);
@@ -221,15 +189,15 @@ test("gitea_branch_protection_set only strengthens", async () => {
test("gitea_merge_pull_request merges only a head whose required statuses all succeeded, and only that head", async () => {
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7,
head: REVIEWED });
state.statuses = [judgedAt(TRUNK)[0], judgedAt(TRUNK, "failure")[1]];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "failure" }];
await assert.rejects(merge(), /not merged: .*mesh\/repo-check \(failure\)/);
state.statuses = [judgedAt(TRUNK)[0]];
state.statuses = [{ context: "mesh/merge-gate", state: "success" }];
await assert.rejects(merge(), /mesh\/repo-check \(missing\)/);
await assert.rejects(tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7,
head: REVIEWED, method: "manually-merged" }), /not a merge method/);
assert.deepEqual(writes(), []);
// The operator said yes and the checks passed: the flow that stays.
state.statuses = judgedAt(TRUNK);
state.statuses = [{ context: "mesh/merge-gate", state: "success" }, { context: "mesh/repo-check", state: "success" }];
await merge();
const sent = state.sent.find((s) => s.path.endsWith("/pulls/7/merge"))!;
assert.equal(sent.body.force_merge, false);
@@ -237,75 +205,6 @@ test("gitea_merge_pull_request merges only a head whose required statuses all su
});
// **The merge tool refuses a verdict whose trunk commit is not the trunk's head, and asks the check itself** (novox/hq
// ADR 0323 §5, issue 341): node-engine #66 merged at 00:54 on a verdict of 00:49 that judged its head alone, after #65
// had moved the trunk. Both commits are named; one check is asked, the announcement a new head makes, and not a second
// while it is on its way; the refusal says how long the last check of the repository took; and once the verdict judges
// the trunk's head and passes, the merge goes through — asked again, never by itself.
test("gitea_merge_pull_request refuses a verdict judged against an earlier trunk, and asks the check once", async () => {
const merge = () => tool("gitea_merge_pull_request").run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED });
const { keepTook, tookFile } = await import("../dist/trunk.js");
keepTook(tookFile(stateDir), "novox/mesh-catalog", { took: "4m31s", at: "2026-10-12T14:49:00Z" });
state.trunk = MOVED; // another merge moved the trunk after the verdict
await assert.rejects(merge(), (err: Error) => {
assert.match(err.message, /is not merged: its verdict was judged against trunk 7a8b9c0d1e2f, and main is now 3c4d5e6f7a8b/);
assert.match(err.message, /check of head abc1234def56 merged into trunk 3c4d5e6f7a8b was asked/);
assert.match(err.message, /the last check of novox\/mesh-catalog took 4m31s/);
assert.match(err.message, /never merges by itself/);
return true;
});
assert.equal(state.merged, false);
assert.equal(state.emitted.length, 1, "one check asked");
assert.equal(state.emitted[0].type, "pull.updated");
assert.equal(state.emitted[0].body.head_sha, REVIEWED);
assert.equal(state.emitted[0].body.base, "main");
assert.deepEqual(state.emitted[0].body.paths, ["modules/gitea/x.ts"]);
// The announcement names the trunk's head, for a verdict no build judges (ADR 0323 §3).
assert.equal(state.emitted[0].body.base_sha, MOVED);
// Its pending went through the merge check's one writer (novox/hq issue 456), not around it.
assert.deepEqual(throughTheWriter.splice(0), ["abc1234def56 mesh/merge-gate pending", "abc1234def56 mesh/repo-check pending"]);
const gate = state.statuses.find((s) => s.context === "mesh/merge-gate")!;
assert.equal(gate.state, "pending");
assert.match(gate.description!, /^waiting in the build queue since 15:01: asked by the merge tool against trunk 3c4d5e6f7a8b/);
assert.equal(state.statuses.find((s) => s.context === "mesh/repo-check")!.state, "pending");
// Asked again while that check is on its way: refused, and nothing is asked a second time.
await assert.rejects(merge(), /already asked and not yet answered .*not asked again; the last check of novox\/mesh-catalog took 4m31s.*mesh-delivery\.recheck/);
assert.equal(state.emitted.length, 1, "no second check while one is on its way");
state.statuses = [{ ...gate, description: "building on ace since 15:02" }, ...state.statuses.slice(1)];
await assert.rejects(merge(), /not asked again/);
assert.equal(state.emitted.length, 1);
// A verdict put out of date by the trunk's move is asked too, once.
state.statuses = judgedAt(TRUNK).map((s) => ({ ...s, state: "pending",
description: "out of date: judged against trunk 7a8b9c0d1e2f, the trunk is now 3c4d5e6f7a8b; judged again on a recheck or when it is merged" }));
await assert.rejects(merge(), /was judged against trunk 7a8b9c0d1e2f, and main is now 3c4d5e6f7a8b\. The check .* was asked/);
assert.equal(state.emitted.length, 2);
// A verdict that names no trunk commit — a build seat from before the rule — is not taken as the trunk's head.
state.statuses = [{ context: "mesh/merge-gate", state: "success", description: "pass: x" },
{ context: "mesh/repo-check", state: "success", description: "pass: y" }];
await assert.rejects(merge(), /names no trunk commit it was judged against, and main is now 3c4d5e6f7a8b/);
assert.equal(state.emitted.length, 3);
assert.equal(state.merged, false);
// An ask that cannot be announced puts the statuses back to out of date, so the next ask asks again.
state.statuses = judgedAt(TRUNK);
const failing = mergeTool(client, { gates, stateDir, emit: async () => { throw new Error("the bus is away"); } });
await assert.rejects(failing.run({ owner: "novox", repo: "mesh-catalog", number: 7, head: REVIEWED }),
/its check could not be asked: the bus is away\. Nothing was merged/);
assert.match(state.statuses.find((s) => s.context === "mesh/merge-gate")!.description!, /^out of date: /);
await assert.rejects(merge(), /was asked/);
assert.equal(state.emitted.length, 4, "asked again after the announcement failed");
// The check judged the merge into the trunk's head and passed: asked again at the reviewed head, it merges.
state.statuses = judgedAt(MOVED);
const done = (await merge()) as Record<string, unknown>;
assert.equal(done.merged, true);
assert.equal(state.emitted.length, 4, "a merge asks no check");
});
// A merge lands only the head that was reviewed (novox/hq issue 381): mesh-controller#197 was reviewed at one head,
// force-pushed, checked green at the new one, and merged there, because the tool merged whatever the head was.
test("gitea_merge_pull_request merges only the head it is named, and refuses one that is not the head", async () => {
+1 -2
View File
@@ -414,8 +414,7 @@ test("the tools register once there is a way to a token, and the first call mint
[
"gitea_list_repos", "gitea_create_repo", "gitea_delete_repo",
"gitea_list_issues", "gitea_get_issue", "gitea_create_issue", "gitea_close_issue", "gitea_add_comment",
// gitea_merge_pull_request is served from the watcher's bundle, beside the merge check's one writer (ADR 0323).
"gitea_list_pull_requests", "gitea_get_pull_request", "gitea_create_pull_request",
"gitea_list_pull_requests", "gitea_get_pull_request", "gitea_create_pull_request", "gitea_merge_pull_request",
"gitea_close_pull_request",
"gitea_reopen_pull_request",
"gitea_update_pull_request",
-175
View File
@@ -1,175 +0,0 @@
// **A merge check's verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341): both statuses name
// the head and the trunk commit; a trunk's move puts the merge check of every open pull request into that trunk — and
// of none into another — back to pending with the reason, and runs nothing; a verdict that lands judged against a trunk
// that has moved since is said out of date.
import { test } from "node:test";
import assert from "node:assert/strict";
import { checkOnItsWay, lateVerdictComment, outOfDateStatuses, outOfDateWords, trunkNamed, verdictAtTrunk, weighTrunks } from "../dist/trunk.js";
import { GateWriter } from "../dist/gate.js";
import { statusesFor, commentFor } from "../dist/pulls.js";
const HEAD = "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
const TRUNK = "a2c2c35bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
const MOVED = "d28ca95ccccccccccccccccccccccccccccccccc";
test("both statuses say the head and the trunk commit the verdict judged, first, whatever the summary's length", () => {
const c = { owner: "novox", repo: "mesh-host", number: 66, commit: HEAD, verdict: "pass", summary: "x", id: "build-1",
trunk: TRUNK.slice(0, 12), beside: { "mesh-catalog": "0123456789ab", "mesh-lab": "ba9876543210" },
gate: { verdict: "pass", summary: "4 of 4 compose ".repeat(20) },
"repo-check": { verdict: "fail", summary: "its merge-check.sh failed: internal/store/store.go:145:2: Declared redeclared" } };
const [gate, repo] = statusesFor(c);
for (const s of [gate, repo]) {
assert.ok(s.description.startsWith("head dfb1415aaaaa merged into trunk a2c2c35bbbbb — "), s.description);
assert.ok(s.description.length <= 140, `${s.description.length} characters`);
assert.equal(trunkNamed(s.description), "a2c2c35bbbbb");
}
assert.equal(gate.state, "success");
assert.equal(repo.state, "failure");
const comment = commentFor(c)!;
assert.match(comment, /Judged: head `dfb1415aaaaa` merged into trunk `a2c2c35bbbbb`; read beside it: mesh-catalog `0123456789ab`, mesh-lab `ba9876543210`/);
// A verdict of a build seat from before the rule names no trunk, and is said as before.
const [old] = statusesFor({ ...c, trunk: undefined, gate: { verdict: "pass", summary: "every machine composes" } });
assert.equal(old.description, "pass: every machine composes");
assert.equal(trunkNamed(old.description), undefined);
});
test("a trunk's move puts an open head's merge check out of date, naming both commits, unless a check is on its way", () => {
const verdict = (trunk: string, state = "success") => ({
"mesh/merge-gate": { state, description: `head ${HEAD.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: x` },
"mesh/repo-check": { state, description: `head ${HEAD.slice(0, 12)} merged into trunk ${trunk.slice(0, 12)} — pass: y` } });
const put = outOfDateStatuses(verdict(TRUNK), MOVED);
assert.deepEqual(put.map((s) => [s.context, s.state]), [["mesh/merge-gate", "pending"], ["mesh/repo-check", "pending"]]);
assert.equal(put[0].description, "out of date: judged against trunk a2c2c35bbbbb, the trunk is now d28ca95ccccc; " +
"judged again on a recheck or when it is merged");
assert.equal(put[0].description, outOfDateWords(TRUNK, MOVED));
assert.equal(trunkNamed(put[0].description), "a2c2c35bbbbb", "an out-of-date status still names what it judged");
// A failure is out of date too: it judged another merge.
assert.equal(outOfDateStatuses(verdict(TRUNK, "failure"), MOVED).length, 2);
// Already judged against the trunk's head; a check on its way or already out of date; no check yet: left alone.
assert.deepEqual(outOfDateStatuses(verdict(MOVED), MOVED), []);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": { state: "pending", description: "building on ace since 14:02" } }, MOVED), []);
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": { state: "pending", description: put[0].description } }, MOVED), []);
assert.deepEqual(outOfDateStatuses({}, MOVED), []);
// A head with no repository check of its own gets no pending one a verdict would never set.
assert.deepEqual(outOfDateStatuses({ "mesh/merge-gate": verdict(TRUNK)["mesh/merge-gate"] }, MOVED).map((s) => s.context),
["mesh/merge-gate"]);
// A verdict naming no trunk, of a build seat from before the rule, judged an earlier trunk.
assert.match(outOfDateStatuses({ "mesh/merge-gate": { state: "success", description: "pass: x" } }, MOVED)[0].description,
/judged against a trunk it does not name, the trunk is now d28ca95ccccc/);
assert.ok(checkOnItsWay({ state: "pending", description: "waiting in the build queue since 14:00" }));
assert.ok(checkOnItsWay({ state: "pending", description: "checking again: a person asked" }));
assert.ok(!checkOnItsWay({ state: "pending", description: put[0].description }));
assert.deepEqual(verdictAtTrunk(verdict(MOVED), MOVED), { current: true, onItsWay: false, judged: "d28ca95ccccc" });
assert.equal(verdictAtTrunk(verdict(TRUNK), MOVED).current, false);
});
/** A forge holding open pull requests into two branches, each head's statuses, and what was written. */
function aForge(pulls: { number: number; base: string; head: string }[], heads: Record<string, string>) {
const statuses = new Map<string, Record<string, { state: string; description?: string }>>();
const writes: string[] = [];
const verdict = (trunk: string) => ({
"mesh/merge-gate": { state: "success", description: `head x merged into trunk ${trunk.slice(0, 12)} — pass` },
"mesh/repo-check": { state: "success", description: `head x merged into trunk ${trunk.slice(0, 12)} — pass` } });
for (const p of pulls) statuses.set(p.head, verdict(heads[p.base]));
const forge = {
listOpenPulls: async () => pulls.map((p) => ({ number: p.number, title: "t", state: "open", merged: false, base: p.base,
head: "feat", head_sha: p.head, html_url: `http://forge/pulls/${p.number}` })),
branchHead: async (_o: string, _r: string, branch: string) => heads[branch],
setCommitStatus: async (_o: string, _r: string, sha: string, s: { context: string; state: string; description: string }) => {
writes.push(`${sha} ${s.context} ${s.state}`);
statuses.set(sha, { ...statuses.get(sha), [s.context]: { state: s.state, description: s.description } });
},
gateOf: async () => undefined,
gateHistory: async () => [],
headChecks: async (_o: string, _r: string, sha: string) => statuses.get(sha) ?? {},
};
return { forge, writes, statuses };
}
test("a push to a trunk puts every open pull request into it out of date — twenty-five of them — and none into another", async () => {
const heads: Record<string, string> = { main: TRUNK, release: "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" };
const pulls = Array.from({ length: 25 }, (_, i) => ({ number: i + 1, base: "main", head: `${String(i).padStart(2, "0")}`.repeat(20) }));
pulls.push({ number: 99, base: "release", head: "9".repeat(40) });
const { forge, writes, statuses } = aForge(pulls, heads);
const gates = new GateWriter(forge);
const trunks: Record<string, string> = {};
// First sight: each trunk is taken as it is, and nothing is put out of date.
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), true);
assert.deepEqual(writes, []);
assert.equal(trunks["novox/mesh-host:main"], TRUNK);
// Nothing moved: nothing read past the trunks, nothing written.
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), false);
heads.main = MOVED;
assert.equal(await weighTrunks(forge, gates, trunks, "novox", "mesh-host"), true);
assert.equal(writes.length, 50, "both statuses of each of the 25 pull requests into main");
for (const p of pulls.filter((p) => p.base === "main")) {
const s = statuses.get(p.head)!;
assert.equal(s["mesh/merge-gate"].state, "pending");
assert.match(s["mesh/merge-gate"].description!, /judged against trunk a2c2c35bbbbb, the trunk is now d28ca95ccccc/);
assert.equal(s["mesh/repo-check"].state, "pending");
}
assert.equal(statuses.get("9".repeat(40))!["mesh/merge-gate"].state, "success", "a pull request into another branch is left");
assert.equal(trunks["novox/mesh-host:main"], MOVED);
// Weighed again, nothing more: the move was taken once, and the out-of-date statuses are left as they are.
writes.length = 0;
heads.main = "f".repeat(40);
await weighTrunks(forge, gates, trunks, "novox", "mesh-host");
assert.deepEqual(writes, [], "a check already out of date is not written again");
});
test("a trunk that cannot be read is weighed again on the next pass, never recorded half", async () => {
const heads: Record<string, string> = { main: TRUNK };
const { forge } = aForge([{ number: 1, base: "main", head: "1".repeat(40) }], heads);
const trunks: Record<string, string> = { "novox/mesh-host:main": TRUNK };
heads.main = MOVED;
const failing = { ...forge, setCommitStatus: async () => { throw new Error("the forge is away"); } };
assert.equal(await weighTrunks(failing, new GateWriter(failing), trunks, "novox", "mesh-host"), false);
assert.equal(trunks["novox/mesh-host:main"], TRUNK, "not recorded: the next pass weighs it again");
});
test("the trunk's move takes its turn after a verdict in flight, and weighs what the forge holds then", async () => {
const heads: Record<string, string> = { main: TRUNK };
const { forge, statuses } = aForge([{ number: 1, base: "main", head: HEAD }], heads);
const gates = new GateWriter(forge);
// A verdict judged against the trunk's new head, in flight when the move is weighed: it lands first, and is left.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
const verdictLanded = gates.verdict("novox", "mesh-host", HEAD, async () => {
await held;
statuses.set(HEAD, { "mesh/merge-gate": { state: "success", description: `head x merged into trunk ${MOVED.slice(0, 12)} — pass` } });
});
const moved = gates.outOfDate("novox", "mesh-host", HEAD, (checks) => outOfDateStatuses(checks, MOVED));
release();
await verdictLanded;
assert.deepEqual(await moved, [], "a verdict judged against the trunk's head is not put out of date");
assert.equal(statuses.get(HEAD)!["mesh/merge-gate"].state, "success");
});
test("a verdict judged against a trunk newer than the pass's first read is not put out of date", async () => {
const NEWER = "ffffffffffffffffffffffffffffffffffffffff";
const heads: Record<string, string> = { main: TRUNK };
const { forge, statuses } = aForge([{ number: 1, base: "main", head: HEAD }], heads);
const trunks: Record<string, string> = { "novox/mesh-host:main": TRUNK };
// The pass reads main at MOVED; by the head's turn the trunk moved again, and the head's check answered for it.
let reads = 0;
const moving = { ...forge, branchHead: async () => (reads++ === 0 ? MOVED : NEWER) };
statuses.set(HEAD, { "mesh/merge-gate": { state: "success", description: `head x merged into trunk ${NEWER.slice(0, 12)} — pass` } });
await weighTrunks(moving, new GateWriter(moving), trunks, "novox", "mesh-host");
assert.equal(statuses.get(HEAD)!["mesh/merge-gate"].state, "success", "a fresh verdict was put out of date");
});
test("a verdict that landed out of date keeps its account on the pull request", () => {
const words = outOfDateWords(TRUNK, MOVED);
const own = lateVerdictComment({ commit: HEAD, trunk: TRUNK.slice(0, 12), verdict: "fail" }, "**Merge check** at `dfb1415a`\n\nthe report", words);
assert.match(own, /the report/);
assert.match(own, /\*\*Out of date when it landed\*\*: out of date: judged against trunk a2c2c35bbbbb/);
const bare = lateVerdictComment({ commit: HEAD, trunk: TRUNK.slice(0, 12), verdict: "pass" }, null, words);
assert.match(bare, /head dfb1415aaaaa merged into trunk a2c2c35bbbbb, pass/);
});
+105 -2
View File
@@ -63,8 +63,6 @@ export function getGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return namesChecked(allGiteaTools(gitea));
}
// gitea_merge_pull_request is served from the watcher's bundle (merge.ts, index.ts), beside the merge check's one
// writer (novox/hq issue 456, ADR 0323).
function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
return [
// ---- Repositories ----
@@ -265,6 +263,111 @@ function allGiteaTools(gitea: GiteaClient): ToolDefinition[] {
}),
}),
},
{
name: "gitea_merge_pull_request",
description: "Merge a pull request at the head you name — the commit you reviewed — when that is still its head " +
"and every status its base branch's protection requires succeeded there (mesh/merge-gate, mesh/repo-check), " +
"optionally deleting the source branch afterwards. Refused otherwise, with nothing merged: no tool merges past " +
"a check, and none merges a head that moved after it was reviewed.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
number: { type: "number", description: "the PR number" },
head: { type: "string", description: "the commit you reviewed, as its sha or at least its first 12 characters: " +
"refused unless it is the pull request's head" },
method: { type: "string", description: "merge (default) | fast-forward-only | squash | rebase | rebase-merge. " +
"Only merge and fast-forward-only can be read back to prove the head you named is what landed" },
delete_branch: { type: "boolean", description: "delete the source branch after merge (default true)" },
},
run: async (args) => {
const owner = String(args.owner);
const repo = String(args.repo);
const number = Number(args.number);
const method = args.method ? String(args.method) : "merge";
if (!MERGE_METHODS.has(method)) throw new Refused(`${method} is not a merge method: ${[...MERGE_METHODS].join(", ")}`);
const deleteBranch = args.delete_branch === undefined ? true : Boolean(args.delete_branch);
// **Only a head whose required statuses all succeeded** (novox/hq ADR 0266): read here rather than left
// to the forge, whose administrator — the account this tool's token is minted for — may merge past a
// rule that lets it. A rule that cannot be read is no rule passed.
// **Only the head that was reviewed** (novox/hq issue 381): mesh-controller#197 was reviewed at 065cfa0d,
// force-pushed to 3ced96f, checked green there, and merged at 3ced96f, because this tool merged whatever the
// head was and nothing told it which head the review had approved. The caller names it; a head that moved
// since is refused, and the forge is handed the same head, so a push between this read and the merge is
// refused by the forge too.
const named = args.head === undefined ? "" : String(args.head).trim().toLowerCase();
if (!named) {
throw new Refused(`${owner}/${repo}#${number} is not merged: name the head you reviewed (head), so that only ` +
`that commit can land (novox/hq issue 381). Nothing was merged`);
}
// Twelve characters, not seven: an author who force-pushes can make a commit whose first seven match the
// reviewed one in seconds, and the review would then merge it. Twelve cannot be matched that way.
if (!REVIEWED_HEAD.test(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: head ${named} does not name one commit — give its ` +
`sha, or at least its first 12 characters. Nothing was merged`);
}
const pull = await gitea.getPullRequest(owner, repo, number);
if (pull.merged) throw new Refused(`${owner}/${repo}#${number} is already merged`);
if (!pull.head_sha || !pull.base) throw new Refused(`${owner}/${repo}#${number} names no head commit or base branch`);
if (!pull.head_sha.toLowerCase().startsWith(named)) {
throw new Refused(`${owner}/${repo}#${number} is not merged: its head is ${pull.head_sha.slice(0, 12)}, not the ` +
`${named} you named. If you reviewed an earlier head, review the one it has now, then name it. Nothing was merged`);
}
const rule = await gitea.branchProtection(owner, repo, pull.base);
// Only into a branch whose protection requires the checks: a pull request whose base was moved to a
// branch nothing guards would merge past every check (novox/hq ADR 0266).
if (!rule || !rule.enable_status_check || !(rule.status_check_contexts ?? []).filter(Boolean).length) {
throw new Refused(`${owner}/${repo}#${number} merges into ${pull.base}, which no protection rule requiring ` +
`the checks guards: a tool merges only into such a branch (novox/hq ADR 0266). Nothing was merged`);
}
const statuses = newestByContext((await gitea.combinedStatus(owner, repo, pull.head_sha)).statuses);
const verdict = mergeableUnder(pull.base, rule, statuses);
if (verdict.mergeable !== true) {
throw new Refused(`${owner}/${repo}#${number} is not merged: ${verdict.says}. A pull request merges once ` +
`its checks succeed; no tool merges past them (novox/hq ADR 0266)`);
}
await gitea.mergePullRequest(owner, repo, number, method, deleteBranch, pull.head_sha);
// The merge commit only exists now; answered so the caller can follow what is built from it.
// pull.merged is the events entrypoint's to announce (index.ts), once, within its poll.
// From here the merge has happened. Nothing below may read as if it had not: a read-back that fails or cannot
// be done is said beside merged: true, and only a proven other head is an error — one that says the merge happened.
const answer: Record<string, unknown> = { merged: true, number, method, head: pull.head_sha, deleted_branch: deleteBranch };
let landed: string | undefined;
try {
const merged = await gitea.getPullRequest(owner, repo, number);
answer.merge_commit_sha = merged.merge_commit_sha;
// What landed, read back from the commit the merge made rather than from the forge's record of the pull
// request, which is the very thing a race would leave behind. A merge commit's second parent is the head it
// joined; a fast-forward lands the head itself. A squash or a rebase makes new commits that name no head.
if (!merged.merge_commit_sha) {
answer.read_back = "the forge named no merge commit, so what landed cannot be read back";
} else if (method === "merge") {
const parents = await gitea.commitParents(owner, repo, merged.merge_commit_sha);
if (parents.length === 2) landed = parents[1];
else answer.read_back = `the merge commit has ${parents.length} parent(s), not two: what landed cannot be read back`;
} else if (method === "fast-forward-only") {
// That the forge names the head itself as a fast-forward's merge commit is assumed, not seen on this forge:
// a different sha is said, not raised as another head having landed.
if (merged.merge_commit_sha === pull.head_sha) landed = merged.merge_commit_sha;
else answer.read_back = `the forge named ${merged.merge_commit_sha.slice(0, 12)} as this fast-forward's commit, ` +
`not the head: that the head landed could not be confirmed`;
} else {
answer.read_back = `a ${method} makes new commits that name no head: what landed cannot be read back`;
}
} catch (err) {
answer.read_back = `the merge happened, but reading back what landed failed: ${(err as Error).message}`;
}
// Loud, not a field beside merged: true that a caller reading only that would miss. The merge cannot be taken
// back by this tool; what must not happen is that it reads as the merge that was asked for.
if (landed !== undefined && landed !== pull.head_sha) {
throw new Error(`${owner}/${repo}#${number} WAS merged (${answer.merge_commit_sha}), but the forge merged ` +
`${landed.slice(0, 12)}, not the ${pull.head_sha.slice(0, 12)} it was handed and you reviewed: a commit nobody ` +
`reviewed has landed. Read it before anything is built from it (novox/hq issue 381)`);
}
if (landed !== undefined) answer.landed = landed;
return answer;
},
},
{
name: "gitea_close_pull_request",
description: "Close a pull request without merging it — one whose work landed elsewhere, or was abandoned.",
-250
View File
@@ -1,250 +0,0 @@
// **A merge check's verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341).
//
// The build seat checks a pull request's head merged into its trunk's head at that moment — the verdict's trunk
// commit — and the verdict names both: "head 1a2b3c4d5e6f merged into trunk 7a8b9c0d1e2f", in both statuses'
// descriptions. When the trunk moves, every open pull request's verdict into it is out of date:
//
// - **the forge's holder** (index.ts) hears the move on its own — it reads each trunk's head on every pass that
// looks at the repository, never through mesh-delivery (ADR 0239 §36) — and sets mesh/merge-gate and
// mesh/repo-check on the head of every open pull request into that trunk back to pending, with the reason.
// Nothing is run for it: a recheck, a new head, or the merge tool asks the check again;
// - **the merge tool** (tools/index.ts) compares the trunk commit the head's verdict names with the trunk's head at
// the moment it is asked, merges nothing when they differ, asks the head's check once — the same announcement a
// new head makes — and says how long the last check of that repository took. It never merges by itself after.
//
// Pure functions and two small files of the module's own state, so this is tested without a forge.
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type { CommitStatus, GiteaClient, GiteaPull, HeadCheck } from "./client.js";
import { CHECK_CONTEXT, REPO_CHECK_CONTEXT } from "./pulls.js";
/** A commit as a verdict names it: its first twelve characters. */
export function twelve(sha: string | undefined): string {
return String(sha ?? "").slice(0, 12).toLowerCase();
}
/** Whether two commits are one, however either is abbreviated (seven characters at least). */
export function sameCommit(a: string | undefined, b: string | undefined): boolean {
const x = String(a ?? "").toLowerCase(), y = String(b ?? "").toLowerCase();
if (x.length < 7 || y.length < 7) return false;
return x.length <= y.length ? y.startsWith(x) : x.startsWith(y);
}
/** What a verdict judged, as both statuses' descriptions begin (ADR 0323 §3). */
export function judgedWords(head: string, trunk: string): string {
return `head ${twelve(head)} merged into trunk ${twelve(trunk)}`;
}
/** The trunk commit a merge check's status names: the one its verdict judged ("merged into trunk …"), or, for a
* verdict put out of date, the one it had judged ("judged against trunk …"). Undefined when it names none, as a
* verdict of a build seat from before ADR 0323 does. */
export function trunkNamed(description: string | undefined): string | undefined {
const m = /(?:merged into|judged against) trunk ([0-9a-f]{7,64})\b/i.exec(description ?? "");
return m ? m[1].toLowerCase() : undefined;
}
/** A verdict out of date, as its statuses say it (ADR 0323 §4). */
export function outOfDateWords(judged: string | undefined, now: string): string {
const was = judged ? `judged against trunk ${twelve(judged)}` : "judged against a trunk it does not name";
return `out of date: ${was}, the trunk is now ${twelve(now)}; judged again on a recheck or when it is merged`;
}
/** Whether a pending gate says a check of the head is asked and not yet answered: waiting in the build queue,
* building on a machine (gate.ts), or asked again by a recheck (mesh-delivery). Such a check is not asked again. */
export function checkOnItsWay(gate: HeadCheck | undefined): boolean {
if (!gate || gate.state !== "pending") return false;
const d = gate.description ?? "";
return d.startsWith("waiting in the build queue") || d.startsWith("building on ") || d.startsWith("checking again");
}
/**
* The statuses a trunk's move puts back to pending on one open pull request's head, from what the forge holds on it
* now: none when it has no merge check yet, when a check is pending (on its way, or already out of date — its
* verdict is weighed against the trunk when it lands), or when its verdict judged the trunk's head. Otherwise the
* gate, and the repository's own check when the head has one — a pending repo-check a verdict will not set would
* stand for ever (novox/hq issue 305's rule).
*/
export function outOfDateStatuses(checks: Record<string, HeadCheck>, trunkHead: string): CommitStatus[] {
const gate = checks[CHECK_CONTEXT];
if (!gate || gate.state === "pending") return [];
const judged = trunkNamed(gate.description);
if (judged && sameCommit(judged, trunkHead)) return [];
const description = outOfDateWords(judged, trunkHead);
const out: CommitStatus[] = [{ state: "pending", context: CHECK_CONTEXT, description }];
if (checks[REPO_CHECK_CONTEXT]) out.push({ state: "pending", context: REPO_CHECK_CONTEXT, description });
return out;
}
/** Where the merge tool stands with a head's verdict and the trunk's head (ADR 0323 §5). */
export interface VerdictAtTrunk {
/** The verdict judged the trunk's head: the required statuses decide, as before. */
current: boolean;
/** A check of the head is asked and not yet answered: nothing is asked again. */
onItsWay: boolean;
/** The trunk commit the verdict names, when it names one. */
judged?: string;
}
export function verdictAtTrunk(checks: Record<string, HeadCheck>, trunkHead: string): VerdictAtTrunk {
const gate = checks[CHECK_CONTEXT];
const judged = trunkNamed(gate?.description);
if (checkOnItsWay(gate)) return { current: false, onItsWay: true, judged };
return { current: !!gate && gate.state !== "pending" && !!judged && sameCommit(judged, trunkHead), onItsWay: false, judged };
}
/** The pending words the merge tool's ask of a check is said with: in the build queue from now, as an announced head's. */
export function askedWords(at: Date, trunkHead: string): string {
const two = (n: number) => String(n).padStart(2, "0");
return `waiting in the build queue since ${two(at.getHours())}:${two(at.getMinutes())}: asked by the merge tool ` +
`against trunk ${twelve(trunkHead)}`;
}
/** The comment a verdict that landed out of date leaves (ADR 0323 §7: kept and shown): its own account when it has
* one, else what it judged and said, and that it is out of date. */
export function lateVerdictComment(c: { commit: string; trunk?: string; verdict: string }, account: string | null,
outOfDate: string): string {
const own = account ?? `**Merge check** at \`${c.commit.slice(0, 8)}\`: ${judgedWords(c.commit, c.trunk ?? "")}, ${c.verdict}`;
return `${own}\n\n**Out of date when it landed**: ${outOfDate}`;
}
// --- a trunk's move, weighed --------------------------------------------------------------------------------------
/** What weighing a trunk's move reads of the forge. */
export interface TrunkForge {
listOpenPulls(owner: string, repo: string): Promise<GiteaPull[]>;
branchHead(owner: string, repo: string, branch: string): Promise<string>;
}
/** What it writes with: the merge check's one writer (gate.ts), which decides in the head's own turn. */
export interface OutOfDateWriter {
outOfDate(owner: string, repo: string, sha: string,
decide: (checks: Record<string, HeadCheck>) => CommitStatus[] | Promise<CommitStatus[]>, target?: string): Promise<CommitStatus[]>;
}
/**
* Every trunk of one repository that an open pull request merges into, read; for one that moved since it was last read
* (`trunks`, by `owner/repo:branch`, which this keeps current), the merge check of every open pull request into it put
* out of date (ADR 0323 §4). A trunk first seen is recorded as it is and nothing is put out of date. Bounded however
* many are open: one listing of the open pull requests (paged), one read of each trunk, then per open pull request into
* a trunk that moved one read and at most two writes, one at a time. A trunk that could not be weighed whole is not
* recorded, so the next pass weighs it again. Answers whether the record changed.
*/
export async function weighTrunks(forge: TrunkForge, gates: OutOfDateWriter, trunks: Record<string, string>, owner: string,
name: string): Promise<boolean> {
let changed = false;
try {
const open = (await forge.listOpenPulls(owner, name)).filter((p) => p.state === "open" && !!p.head_sha && !!p.base);
for (const base of new Set(open.map((p) => String(p.base)))) {
const key = `${owner}/${name}:${base}`.toLowerCase();
const head = await forge.branchHead(owner, name, base);
const before = trunks[key];
if (before && sameCommit(before, head)) continue;
if (before) {
let put = 0;
for (const pull of open.filter((p) => p.base === base)) {
// The trunk is read again in the head's own turn: a verdict judged against a head newer than the pass's first
// read — the trunk moved again meanwhile, and a check answered — is not put out of date.
const written = await gates.outOfDate(owner, name, String(pull.head_sha),
async (checks) => outOfDateStatuses(checks, await forge.branchHead(owner, name, base)), pull.html_url || undefined);
if (written.length) put++;
}
console.log(`[gitea] ${owner}/${name}'s ${base} moved from ${twelve(before)} to ${twelve(head)}: ${put} open pull ` +
`request(s) into it put out of date, nothing run (novox/hq ADR 0323)`);
}
trunks[key] = head;
changed = true;
}
} catch (err) {
console.error(`[gitea] ${owner}/${name}: whether a trunk moved could not be weighed, so it is weighed again on the next ` +
`pass — ${err instanceof Error ? err.message : err}`);
}
return changed;
}
// --- how long the last check of a repository took -------------------------------------------------------------
/** How long one repository's last check took, and when it was heard: kept by the watcher, read by the merge tool. */
export interface Took {
took: string;
at: string;
head?: string;
}
/** The file the watcher keeps each repository's last check's duration in, beside its other records. */
export function tookFile(stateDir: string | undefined): string | null {
return stateDir ? `${stateDir.replace(/\/+$/, "")}/checks-took.json` : null;
}
function readTookAll(file: string | null): Record<string, Took> {
if (!file || !existsSync(file)) return {};
try {
return JSON.parse(readFileSync(file, "utf8")) as Record<string, Took>;
} catch {
return {};
}
}
/** Keep how long a repository's check took. A record that cannot be written is said, never fatal to the verdict. */
export function keepTook(file: string | null, repository: string, took: Took): void {
if (!file || !took.took) return;
try {
const all = readTookAll(file);
all[repository.toLowerCase()] = took;
mkdirSync(dirname(file), { recursive: true });
writeFileSync(file + ".tmp", JSON.stringify(all));
renameSync(file + ".tmp", file);
} catch (err) {
console.error(`[gitea] how long ${repository}'s check took could not be kept — ${err instanceof Error ? err.message : err}`);
}
}
/** How long a repository's last check took, in words for the merge tool's refusal. */
export function tookWords(file: string | null, repository: string): string {
const t = readTookAll(file)[repository.toLowerCase()];
if (!t) return `how long a check of ${repository} takes is not known yet: no verdict of it was heard since this was kept`;
return `the last check of ${repository} took ${t.took} (heard ${t.at})`;
}
// --- the announcement of a head, shared by the watcher and the merge tool ---------------------------------------
/** `pull.updated` for one open pull request's head: what the controller asks the build seat to check (novox/hq to-be
* 45 §9). The merge tool's ask is this same announcement (ADR 0323 §5). */
export async function announcementOf(client: GiteaClient, owner: string, name: string, cloneURL: string | undefined,
pull: GiteaPull): Promise<Record<string, unknown>> {
const files = await client.listPullFiles(owner, name, pull.number);
const head = String(pull.head_sha);
// What the controller maps the change onto the mesh's module graph with (novox/hq ADR 0237 as amended): which
// changed directories hold a module at the head — the merge's rule (issue 278) — and whether the head holds the
// repository's own merge-check.sh. Not said when it could not be read; the controller then reads the change as
// touching everything built from the repository.
let moduleDirs: string[] | null = null;
let mergeCheck: boolean | null = null;
try {
if (!files.truncated) moduleDirs = await client.moduleDirsAt(owner, name, head, files.paths);
mergeCheck = await client.holdsFile(owner, name, head, "merge-check.sh");
} catch (err) {
console.error(`[gitea] ${owner}/${name}#${pull.number}: what the head holds could not be read — ${err instanceof Error ? err.message : err}`);
}
// The base branch's head as the head is announced: the trunk commit a verdict no build judges is said against
// (novox/hq ADR 0323 §3). Not said when it cannot be read; that verdict then names none.
const baseSHA = pull.base ? await client.branchHead(owner, name, String(pull.base)).catch(() => undefined) : undefined;
return {
owner,
repo: name,
number: pull.number,
...(baseSHA ? { base_sha: baseSHA } : {}),
title: pull.title,
body: pull.body,
base: pull.base,
head: pull.head,
head_sha: pull.head_sha,
clone_url: cloneURL,
html_url: pull.html_url,
paths: files.paths,
paths_truncated: files.truncated,
removed: files.removed,
...(moduleDirs ? { module_dirs: moduleDirs, module_dirs_said: true } : {}),
...(mergeCheck !== null ? { merge_check: mergeCheck, merge_check_said: true } : {}),
};
}
@@ -51,10 +51,6 @@ type ChecksAnswer struct {
Statuses []CheckStatus `json:"statuses"`
// Verdict is the merge check's verdict whole, as this owner heard it for this commit.
Verdict *Verdict `json:"verdict,omitempty"`
// Judged is what the verdict judged: the head merged into the trunk commit (novox/hq ADR 0323 §3); OutOfDate
// why it no longer counts, when its trunk has moved since.
Judged string `json:"judged,omitempty"`
OutOfDate string `json:"out_of_date,omitempty"`
// Merge is whether the branch's protection lets the commit merge; Mergeable nil when it is not known.
Merge ForgeMerge `json:"merge"`
// Forge is why the forge could not be read, when it could not.
@@ -111,7 +107,6 @@ func (h *Holder) Checks(repository string, number int, commit string) (*ChecksAn
Group: d.Group}
if d.Check != nil && sameCommit(d.Commit, a.Commit) {
a.Verdict = d.Check
a.Judged, a.OutOfDate = d.Check.Judged(), d.outOfDate()
}
}
@@ -266,12 +261,6 @@ func (a *ChecksAnswer) says() string {
default:
merge = "NOT mergeable — " + merge
}
if a.Judged != "" {
what += " (" + a.Judged + ")"
}
if a.OutOfDate != "" {
merge += "; its verdict is " + a.OutOfDate
}
return what + ": " + statuses + "; " + merge
}
@@ -451,7 +451,7 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
State: Ready, Since: t0, Created: t0.Add(-time.Hour), Plan: planMoving(b, "b"), Check: &Verdict{Gate: "pass"}}
g := &Group{ID: "feat/k", Members: []string{aID, bID}, Order: []string{aID, bID}, Closed: true,
OrderOf: map[string]string{aID: aHead, bID: bHead},
Check: &GroupCheck{Asked: "check-old", At: t0, Commits: c.commits}}
Check: &GroupCheck{Asked: "check-1", At: t0, Commits: c.commits}}
for _, d := range []*Delivery{da, db} {
_ = store.PutDelivery(d)
}
@@ -463,7 +463,7 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
if s := w.state(aID); s != Held {
t.Fatalf("before its composed verdict it is %s", s)
}
w.h.Checked(CheckedEvent{Group: "feat/k", ID: "check-old", Verdict: c.verdict, Summary: "composed: " + c.verdict})
w.h.Checked(CheckedEvent{Group: "feat/k", ID: "check-1", Verdict: c.verdict, Summary: "composed: " + c.verdict})
w.settleAll()
if s := w.state(aID); s != c.want {
t.Fatalf("its composed check %s, and it is %s (%s), want %s", c.name, s, w.delivery(aID).HeldWhy, c.want)
@@ -471,10 +471,6 @@ func TestAMemberHeldForItsComposedCheckIsLiftedWhenItPasses(t *testing.T) {
if c.want == Delivering && !containsString(ctl.delivers, "plan-a") {
t.Fatalf("lifted, its walk was not let go: %v", ctl.delivers)
}
// A composed check of other heads is asked again for these heads (novox/hq ADR 0323 §6).
if c.name == "passed for other heads" && !containsString(ctl.checks, "feat/k:"+aID+","+bID) {
t.Fatalf("the composed check was not asked again for the heads that merged: %v", ctl.checks)
}
})
}
}
@@ -104,9 +104,6 @@ func statusOf(d *Delivery) (string, string) {
case Superseded:
return "warning", clip("superseded: "+why, 140)
case Stopped:
if d.coveredStop() {
return "success", clip(d.coveredStopSaid(), 140) // its change reached the machines (novox/hq issue 480)
}
return "error", clip("stopped: "+why, 140)
case Retired:
return "success", "history: merged before this owner delivered anything, never walked"
@@ -127,9 +124,6 @@ const viewMarker = "<!-- mesh-delivery:view -->"
func ViewBody(d *Delivery, g *Group) string {
var b strings.Builder
fmt.Fprintf(&b, "%s\n**Delivery** `%s` — **%s** since %s\n\n", viewMarker, d.ID, d.State, d.Since.UTC().Format(time.RFC3339))
if d.coveredStop() {
fmt.Fprintf(&b, "**Covered** after its stop: a later merge that contains it was delivered: %s\n\n", d.CoveredBy)
}
if d.HeldWhy != "" && d.State == Held {
fmt.Fprintf(&b, "Held for a person: %s — `mesh-delivery.release` with why lets it go on.\n\n", d.HeldWhy)
}
@@ -284,16 +278,6 @@ func (h *Holder) Flush() {
d.Owed = owed(d.Owed)
h.keep(d)
}
// Counted for delivery_status, so a window's writes can be judged against its tries (novox/hq issue 407).
h.tried.last = h.Now()
switch {
case err == nil:
h.tried.done++
case dropped != "":
h.tried.givenUp++
default:
h.tried.failed++
}
if dropped != "" {
h.refuse(fmt.Errorf("%s: its owed %s given up — %s: %v", j.id, j.e.Kind, dropped, err))
}
+14 -203
View File
@@ -67,16 +67,6 @@ type Holder struct {
// leaveRefused is, by walk, what the controller answered when it refused both to leave a stopped merge out and
// to stop the walk whole.
leaveRefused map[string]string
// tried counts the owed effects Flush tried since this owner started, by how each ended (novox/hq issue 407).
tried owedTries
}
// owedTries is how many owed effects were tried since this owner started, and when the last was: three numbers
// and a time, however many were tried. Not kept: a restarted owner counts from its start, and says since when, so a
// window's count is the difference of two readings in the same run (novox/hq issue 407).
type owedTries struct {
done, failed, givenUp int
last time.Time
}
// unmatched is a walk no delivery is known for yet: its merge not heard, or merged before this owner held.
@@ -279,8 +269,6 @@ type PullEvent struct {
ModuleDirs []string `json:"module_dirs,omitempty"`
ModuleDirsSaid bool `json:"module_dirs_said,omitempty"`
HeadChecks map[string]string `json:"head_checks,omitempty"`
// BaseSHA is the base branch's head when the head was announced (novox/hq ADR 0323).
BaseSHA string `json:"base_sha,omitempty"`
}
func (p PullEvent) repository() string { return strings.ToLower(p.Owner + "/" + p.Repo) }
@@ -298,22 +286,10 @@ func (h *Holder) PullUpdated(p PullEvent) {
h.mu.Lock()
defer h.mu.Unlock()
id := IDOf(p.repository(), p.HeadSHA)
now := h.Now()
if d := h.deliveries[id]; d != nil {
// Announced again: nothing new — but for a delivery whose verdict is out of date, the announcement is its
// check asked again against the trunk's head (the merge tool's ask, novox/hq ADR 0323 §5), and its bound
// runs from now (§7). The base head it was announced with is kept, for a check this owner asks after it.
if p.BaseSHA != "" && d.BaseSHA != p.BaseSHA {
d.BaseSHA = p.BaseSHA
h.keep(d)
}
if d.State == Proposed && d.outOfDate() != "" && d.AskedAgain == nil {
d.AskedAgain = &now
h.keep(d)
}
return
return // announced again: nothing new
}
replaced := ""
now := h.Now()
for _, old := range h.deliveries {
if old.Repository == p.repository() && old.Number == p.Number && !sameCommit(old.Commit, p.HeadSHA) &&
!old.State.Final() && old.MergedAs == "" {
@@ -321,14 +297,13 @@ func (h *Holder) PullUpdated(p PullEvent) {
h.settleAndKeep(old)
if old.Group != "" {
h.leaveGroup(old)
replaced = old.ID
}
}
}
d := &Delivery{ID: id, Repository: p.repository(), Commit: p.HeadSHA, Number: p.Number, Title: p.Title,
Base: p.Base, Branch: p.Head, HTMLURL: p.HTMLURL, CloneURL: p.CloneURL, Paths: p.Paths,
PathsTruncated: p.PathsTruncated, Removed: p.Removed, ModuleDirs: p.ModuleDirs, ModuleDirsSaid: p.ModuleDirsSaid,
After: AfterIn(p.Body), Created: now, BaseSHA: p.BaseSHA}
After: AfterIn(p.Body), Created: now}
t, err := Apply(d, EvAnnounced, false, Facts{Now: now, Why: fmt.Sprintf("%s#%d's head announced", d.Repository, d.Number)})
if err != nil {
h.refuse(err)
@@ -342,45 +317,7 @@ func (h *Holder) PullUpdated(p PullEvent) {
ts = append(ts, settle(d, Facts{Now: now})...)
}
h.moved(d, ts)
if !h.takePlaceOf(replaced, d) {
h.joinGroup(d)
}
}
// takePlaceOf puts a member's new head — a rebase, a push — in its group in the place of the head it superseded,
// when the group is closed: a member of it already reached the trunk, and a closed group takes no new member. The
// new head has its own check, and the group's composed check runs again for the new heads rather than holding the
// group for a person (novox/hq ADR 0323 §6, issue 406). An open group takes it as joinGroup does.
func (h *Holder) takePlaceOf(old string, d *Delivery) bool {
if old == "" {
return false
}
for _, g := range h.groups {
_, ordered := g.OrderOf[old]
if !g.Closed || !ordered && !containsString(g.Order, old) {
continue
}
if ordered {
delete(g.OrderOf, old)
g.OrderOf[d.ID] = d.Commit
}
if !containsString(g.Members, d.ID) {
g.Members = append(g.Members, d.ID)
sort.Strings(g.Members)
}
for i, id := range g.Order {
if id == old {
g.Order[i] = d.ID
}
}
d.Group = g.ID
g.Updated = h.Now()
h.keep(d)
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s: %s takes the place of %s, which it superseded", g.ID, d.ID, old)
return true
}
return false
h.joinGroup(d)
}
// decidedOnHead is the verdict the forge's statuses on a head hold, when the gate decided one: passed, warned
@@ -521,47 +458,6 @@ func (h *Holder) PullMerged(p PullEvent) {
}
h.moved(d, ts)
h.closeGroupOf(d)
h.trunkMoved(d, p.Base, p.MergeCommit, now)
}
// trunkMoved takes a merge into a trunk as that trunk's move (novox/hq ADR 0323 §7): every other open delivery into
// it keeps the trunk's new head, and a ready one whose verdict names the commit before goes back to proposed. A
// group's composed check goes out of date when the merge is not one of its members and a member still open merges
// into that trunk (§6); the group then reads checking. Nothing is asked: a recheck or the merge tool judges again.
func (h *Holder) trunkMoved(merged *Delivery, base, commit string, now time.Time) {
if base == "" || commit == "" {
return
}
for _, o := range h.deliveries {
if o == merged || o.Repository != merged.Repository || o.Base != base || o.MergedAs != "" || o.State.Final() ||
sameCommit(o.TrunkNow, commit) {
continue
}
// A check asked again against the trunk before this move judged that trunk: it is out of date too.
o.TrunkNow, o.AskedAgain = commit, nil
h.settleAndKeep(o)
}
for _, g := range h.groups {
if g.ID == merged.Group || g.Check == nil || g.Check.Verdict == "" || g.Check.OutOfDate != "" {
continue
}
for _, m := range h.membersOf(g) {
if m.Repository != merged.Repository || m.Base != base || m.MergedAs != "" || m.State.Final() {
continue
}
judged := g.Check.Trunks[m.ID]
if judged == "" {
judged = "?"
}
g.Check.OutOfDate = fmt.Sprintf("composed check out of date: %s was judged against trunk %s, the trunk is now %s "+
"(%s merged)", m.ID, shortOf(judged, 12), shortOf(commit, 12), merged.ID)
g.Check.OutOfDateAt = now
g.Updated = now
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s: %s", g.ID, g.Check.OutOfDate)
break
}
}
}
// closeGroupOf closes a delivery's group once it reached the trunk: a delivering group takes no new member.
@@ -633,13 +529,7 @@ type CheckedEvent struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
} `json:"members,omitempty"`
// Trunk is the trunk commit the head was merged into and judged as, Beside the other repositories' commits the
// check read, and Took how long it ran (novox/hq ADR 0323 §3).
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
}
// Checked takes a verdict: a head's own, or a group's composed check.
@@ -654,18 +544,6 @@ func (h *Holder) Checked(c CheckedEvent) {
return
}
g.Check.Verdict, g.Check.Summary = c.Verdict, c.Summary
// Each member's head and the trunk commit it was merged into (novox/hq ADR 0323 §6).
for _, m := range c.Members {
for id := range g.Check.Commits {
if d := h.deliveries[id]; d != nil && strings.EqualFold(d.Repository, m.Owner+"/"+m.Repo) &&
sameCommit(d.Commit, m.Commit) && m.Trunk != "" {
if g.Check.Trunks == nil {
g.Check.Trunks = map[string]string{}
}
g.Check.Trunks[id] = m.Trunk
}
}
}
g.Updated = now
h.keepGroup(g)
h.Logf("[mesh-delivery] group %s composed: %s — %s", g.ID, c.Verdict, c.Summary)
@@ -686,23 +564,11 @@ func (h *Holder) Checked(c CheckedEvent) {
}
h.moved(d, []Transition{t})
}
if d.State == Rejected && c.Trunk != "" && d.Check != nil && !sameCommit(d.Check.Trunk, c.Trunk) {
// Judged again against another trunk commit — the merge tool's ask of a verdict out of date, or a check asked
// elsewhere (novox/hq ADR 0323 §5, §7): taken as the recheck it is.
t, err := Apply(d, EvRecheck, true, Facts{Now: now, By: byOwner, Why: fmt.Sprintf("judged again against trunk %s "+
"(it was rejected against trunk %s)", shortOf(c.Trunk, 12), shortOf(d.Check.Trunk, 12))})
if err != nil {
h.refuse(err)
return
}
h.moved(d, []Transition{t})
}
if d.State != Proposed {
h.Logf("[mesh-delivery] %s is %s: a verdict for it now is history (%s)", d.ID, d.State, c.Verdict)
return
}
v := &Verdict{ID: c.ID, On: c.On, At: now, Gate: c.Verdict, Summary: c.Summary, Report: c.Report,
Head: shortOf(c.Commit, 12), Trunk: c.Trunk, Beside: c.Beside, Took: c.Took}
v := &Verdict{ID: c.ID, On: c.On, At: now, Gate: c.Verdict, Summary: c.Summary, Report: c.Report}
if c.Gate != nil {
v.Gate, v.Summary = c.Gate.Verdict, c.Gate.Summary
v.Modules, v.Dependents = c.Gate.Modules, c.Gate.Dependents
@@ -713,14 +579,10 @@ func (h *Holder) Checked(c CheckedEvent) {
if v.Gate == "" {
v.Gate = "error"
}
d.Check, d.AskedAgain = v, nil
d.Check = v
if c.Plan != nil {
d.Plan = c.Plan
}
if why := d.outOfDate(); why != "" {
// Kept and shown, and it does not count: the delivery stays proposed (ADR 0323 §7).
h.Logf("[mesh-delivery] %s: its verdict %s is %s", d.ID, c.ID, why)
}
h.settleAndKeep(d)
}
@@ -1091,16 +953,12 @@ func (h *Holder) plan() []func() {
// grace, it is stopped saying so — never left waiting for a start that cannot come.
for _, d := range h.deliveries {
ended := d.endedUnstarted()
if d.coveredStop() && !d.CoverTold {
h.tellCovered(d, now) // covered before this holder told it, or before a restart (novox/hq issue 480)
continue
}
if strandedUncovered(d) {
// Stopped and final, but still read for a later merge that delivered its modules: until one did,
// `stalled` keeps listing it (review of novox/hq issue 474).
var ask []string
if d.CoveredBy, ask = h.coverOf(d); d.CoveredBy != "" {
h.tellCovered(d, now)
h.keep(d)
}
for _, in := range ask {
if a := h.askContains(d, in, now); a != nil {
@@ -1245,25 +1103,6 @@ func (h *Holder) laterOnItsWay(d *Delivery) string {
return ""
}
// tellCovered tells the forge that a stop is covered since, once: its status says covered and by which merge, its view
// says so, and its note keeps the line on its head and its merge commit (novox/hq issue 480). No transition: the stop
// stays final, and the bus's transition event is not said again.
func (h *Holder) tellCovered(d *Delivery, now time.Time) {
d.CoverTold = true
line := oneLine(fmt.Sprintf("%s delivery %s: stopped, %s", now.UTC().Format(time.RFC3339), d.ID, d.coveredStopSaid()))
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.Commit, Line: line, Since: now})
if d.MergedAs != "" && !sameCommit(d.MergedAs, d.Commit) {
d.Owed = append(d.Owed, Effect{Kind: EffectNote, Commit: d.MergedAs, Line: line, Since: now})
}
if d.shownOnItsPullRequest() {
d.owe(Effect{Kind: EffectView, Since: now})
state, desc := statusOf(d)
d.owe(Effect{Kind: EffectStatus, Commit: d.Commit, Context: "mesh/delivery", State: state, Line: desc, Since: now})
}
h.Logf("[mesh-delivery] %s: stopped, %s", d.ID, d.coveredStopSaid())
h.keep(d)
}
// strandedUncovered is whether a delivery stopped because its ended walk delivered nothing is still uncovered: a
// merged change that never reached the machines, which `stalled` keeps listing until a later delivered merge
// carries its modules (review of novox/hq issue 474).
@@ -1569,19 +1408,14 @@ func (h *Holder) planGroup(g *Group, members []*Delivery, now time.Time) []func(
})
return asks
}
// The composed check, once every member is ready and the order holds. Asked again for new heads, even once the
// group is closed: a member rebased or pushed is a new head, and the group waits for its composed check rather
// than being held for a person (novox/hq ADR 0323 §6, issue 406). One out of date is asked again once a member
// was judged again after it went out of date: a recheck, or the merge tool's ask.
// The composed check, once every member is ready and the order holds.
allReady := true
for _, d := range members {
// In a closed group a member on the trunk counts — merged and on its way, or already delivered (reached).
if d.State != Ready && !(g.Closed && d.MergedAs != "" && (!d.State.Final() || d.State.Reached())) {
if d.State != Ready {
allReady = false
}
}
again := g.Check != nil && (!sameMembers(g.Check.Commits, commits) || g.Check.OutOfDate != "" && rejudgedSince(members, g.Check.OutOfDateAt))
if allReady && len(g.Cycle) == 0 && (g.Check == nil && !g.Closed || again) {
if allReady && len(g.Cycle) == 0 && !g.Closed && (g.Check == nil || !sameMembers(g.Check.Commits, commits)) {
ms := h.inOrder(g, members)
id := g.ID
asks = append(asks, func() {
@@ -1703,7 +1537,7 @@ func (h *Holder) turnsOf(g *Group, members []*Delivery, waits map[string]carryWa
}
// Composed for these very heads: a verdict on another set of heads is no verdict on these.
composed := g.Check != nil && (g.Check.Verdict == "pass" || g.Check.Verdict == "warning") &&
sameMembers(g.Check.Commits, commitsOf(members)) && g.Check.OutOfDate == ""
sameMembers(g.Check.Commits, commitsOf(members))
// A member's turn registered, the members after it that the same walk carries take theirs too: one walk is let
// go once, for all of them (ADR 0276 decision 5). A member on another walk, or not at its turn, waits.
var asks []func()
@@ -1789,11 +1623,9 @@ func (h *Holder) turnsOf(g *Group, members []*Delivery, waits map[string]carryWa
}
return asks
case d.State == Published:
// Asked and not answered is not "did not pass": it waits, published, for the answer (issue 316). Nor is a
// composed check of other heads — a member rebased or pushed — or one out of date: it runs again for these
// heads, and the member waits for it (novox/hq ADR 0323 §6, issue 406).
if !composed && d.HeldWhy == "" && g.Check != nil && (g.Check.Verdict == "" || g.Check.OutOfDate != "" ||
!sameMembers(g.Check.Commits, commitsOf(members))) {
// Asked and not answered is not "did not pass": it waits, published, for the answer (issue 316).
if !composed && d.HeldWhy == "" && g.Check != nil && g.Check.Verdict == "" &&
sameMembers(g.Check.Commits, commitsOf(members)) {
return asks
}
if !composed && d.HeldWhy == "" {
@@ -2314,23 +2146,12 @@ func GroupState(g *Group, members []*Delivery) (string, string) {
return "delivered", "every member is delivered"
case g.Closed && n > 0 && ended == n:
return "stopped", supersededWhy + strings.Join(uncovered, ", ")
case g.Closed && g.Check != nil && (g.Check.OutOfDate != "" || g.Check.Verdict == "" ||
!sameMembers(g.Check.Commits, commitsOf(members))):
// A member rebased or pushed once the group was delivering, or its composed check out of date: it is composed
// again for these heads, and the group is checking until then, never held (novox/hq ADR 0323 §6, issue 406).
if g.Check.OutOfDate != "" {
return "checking", g.Check.OutOfDate
}
return "checking", "composed again for its members' new heads"
case g.Closed:
return "delivering", fmt.Sprintf("%d of %d delivered", reached, n)
case len(g.Cycle) > 0:
return "rejected", cycleSaid(g)
case count[Rejected] > 0:
return "rejected", "a member's own check failed"
case g.Check != nil && g.Check.OutOfDate != "":
// Out of date counts as outstanding (ADR 0239 §18, novox/hq ADR 0323 §7): checking, saying why.
return "checking", g.Check.OutOfDate
case g.Check != nil && g.Check.Verdict != "" && g.Check.Verdict != "pass" && g.Check.Verdict != "warning":
return "rejected", "composed together: " + g.Check.Summary
case count[Ready] == n && g.Check != nil && sameMembers(g.Check.Commits, commitsOf(members)) &&
@@ -2396,13 +2217,3 @@ func cycleSaid(g *Group) string {
}
return why
}
// rejudgedSince is whether a member was judged again after a moment: a verdict of its own heard since.
func rejudgedSince(members []*Delivery, at time.Time) bool {
for _, d := range members {
if d.Check != nil && d.Check.At.After(at) {
return true
}
}
return false
}
@@ -1,463 +0,0 @@
package main
import (
"strings"
"testing"
"time"
)
// **A verdict holds for the trunk commit it judged** (novox/hq ADR 0323, issue 341). The check judges the head
// merged into a named trunk commit; when another merge moves that trunk, a ready delivery goes back to proposed,
// a rejected one stays rejected, and nothing is asked. A verdict for an old trunk is kept and shown and does not
// count, so the table reaches a fixed point. A group's composed check goes out of date on a merge that is not one of
// its members, and runs again for new heads instead of holding the group.
const (
trunk0 = "a2c2c35000000000000000000000000000000000"
merge1 = "d28ca95000000000000000000000000000000000"
)
// judged is a verdict of a head merged into a trunk commit.
func judged(repo string, number int, sha, gate, trunk string) CheckedEvent {
c := verdict(repo, number, sha, gate)
c.Trunk = shortOf(trunk, 12)
c.ID = "check-of-" + sha + "-at-" + shortOf(trunk, 8)
c.Took = "4m12s"
c.Beside = map[string]string{"mesh-catalog": "0123456789ab"}
plan := *c.Plan
plan.Head, plan.Trunk = sha, c.Trunk
c.Plan = &plan
return c
}
func TestATrunkMoveSendsAReadyDeliveryBackToProposed(t *testing.T) {
const host = "novox/mesh-host"
const h64, h66, h67 = "6464646464646464", "dfb1415dfb1415df", "6767676767676767"
w := newWorld(t)
for _, p := range []PullEvent{pr(host, 64, h64, "a"), pr(host, 66, h66, "b"), pr(host, 67, h67, "c")} {
w.h.PullUpdated(p)
}
w.h.Checked(judged(host, 64, h64, "pass", trunk0))
w.h.Checked(judged(host, 66, h66, "pass", trunk0))
w.h.Checked(judged(host, 67, h67, "fail", trunk0))
id66, id67 := IDOf(host, h66), IDOf(host, h67)
if s := w.state(id66); s != Ready {
t.Fatalf("#66 checked green against the trunk is %s", s)
}
asked := len(w.ctl.checks)
// #64 merges: the trunk moves to its merge commit.
w.h.PullMerged(merged(host, 64, h64, merge1))
w.settleAll()
d := w.delivery(id66)
if d.State != Proposed {
t.Fatalf("#66's verdict names trunk %s, the trunk is now %s, and it is %s", trunk0[:12], merge1[:12], d.State)
}
last := d.Transitions[len(d.Transitions)-1]
if last.Event != EvTrunkMoved || !strings.Contains(last.Why, trunk0[:12]) || !strings.Contains(last.Why, merge1[:12]) {
t.Fatalf("its transition was %s, %q: not trunk moved with both commits", last.Event, last.Why)
}
if d.Check == nil || d.Check.Trunk != trunk0[:12] {
t.Fatalf("its out-of-date verdict was not kept to be shown: %+v", d.Check)
}
if s := w.state(id67); s != Rejected {
t.Fatalf("a rejected delivery went to %s on the trunk's move", s)
}
if len(w.ctl.checks) != asked {
t.Fatalf("the trunk's move asked a check: %v", w.ctl.checks[asked:])
}
// Out of date and not asked: no bound, so stalled does not list it, however long it waits.
w.later(5 * time.Hour)
w.settleAll()
for _, l := range w.h.Stalled() {
if l.ID == id66 {
t.Fatalf("stalled lists a proposed delivery whose check is out of date and not asked: %+v", l)
}
}
if len(w.ctl.checks) != asked {
t.Fatalf("an out-of-date check was asked by this owner: %v", w.ctl.checks[asked:])
}
// A late verdict for the old trunk is kept, does not count, and moves nothing: the table's fixed point.
before := len(w.delivery(id66).Transitions)
late := judged(host, 66, h66, "pass", trunk0)
late.ID = "check-late"
w.h.Checked(late)
w.settleAll()
d = w.delivery(id66)
if d.State != Proposed || len(d.Transitions) != before || d.Check == nil || d.Check.ID != "check-late" {
t.Fatalf("a late verdict for an old trunk left it %s after %d transition(s), verdict %+v", d.State,
len(d.Transitions)-before, d.Check)
}
if ts := settle(d, Facts{Now: w.now}); len(ts) != 0 {
t.Fatalf("the table took %d more transition(s) on a verdict that does not count", len(ts))
}
// The check asked again judges the head merged into the trunk's head: it counts.
w.h.Checked(judged(host, 66, h66, "pass", merge1))
w.settleAll()
if s := w.state(id66); s != Ready {
t.Fatalf("a verdict judged against the trunk's head left it %s", s)
}
}
// A rejected delivery judged again against another trunk — the merge tool's ask — is taken as its recheck.
func TestARejectedDeliveryJudgedAgainstANewTrunkIsRecheckedByTheVerdict(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "fail", trunk0))
id := IDOf(repo, head)
if s := w.state(id); s != Rejected {
t.Fatalf("it is %s", s)
}
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
w.h.Checked(judged(repo, 9, head, "pass", merge1))
w.settleAll()
d := w.delivery(id)
if d.State != Ready || d.Check.Trunk != merge1[:12] {
t.Fatalf("judged green against the trunk's head it is %s with %+v", d.State, d.Check)
}
}
// **A verdict names the trunk commit it judged** in `checks` and `show`, and the kept plan records it (ADR 0323 §3).
func TestChecksAndShowNameTheHeadAndTheTrunkCommit(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "pass", trunk0))
id := IDOf(repo, head)
a, err := w.h.Checks(repo, 0, head)
if err != nil {
t.Fatal(err)
}
want := "head " + head[:12] + " merged into trunk " + trunk0[:12]
if a.Judged != want || a.Verdict == nil || a.Verdict.Head != head[:12] || a.Verdict.Trunk != trunk0[:12] {
t.Fatalf("checks says %q, verdict %+v; want %q", a.Judged, a.Verdict, want)
}
if !strings.Contains(a.Says, want) {
t.Fatalf("checks' words do not say what was judged: %s", a.Says)
}
if a.Verdict.Beside["mesh-catalog"] != "0123456789ab" || a.Verdict.Took != "4m12s" {
t.Fatalf("checks does not name the other repositories' commits or how long it took: %+v", a.Verdict)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
d := shown.(map[string]any)["delivery"].(*Delivery)
if d.Check.Head != head[:12] || d.Check.Trunk != trunk0[:12] {
t.Fatalf("show gives the verdict %+v", d.Check)
}
if d.Plan == nil || d.Plan.Trunk != trunk0[:12] || d.Plan.Head != head {
t.Fatalf("the kept delivery plan records %+v", d.Plan)
}
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
if a, _ = w.h.Checks(repo, 0, head); !strings.Contains(a.OutOfDate, merge1[:12]) {
t.Fatalf("checks does not say the verdict is out of date: %+v", a)
}
}
// **A delivery group's composed check follows the same rule** (ADR 0323 §6, §7): a merge that is not one of its
// members into a member's trunk puts it out of date and the group reads checking; its own members merging in its
// order do not; and it runs again once a member is judged again.
func TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1"
idc, idk := IDOf(ctl, hc), IDOf(cat, hk)
start := func(t *testing.T) *world {
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
if g == nil || g.Check == nil {
t.Fatalf("no composed check was asked: %+v", g)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed",
Members: []struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Commit string `json:"commit"`
Trunk string `json:"trunk,omitempty"`
}{{Owner: "novox", Repo: "mesh-controller", Commit: hc, Trunk: trunk0[:12]},
{Owner: "novox", Repo: "mesh-catalog", Commit: hk, Trunk: trunk0[:12]}}})
w.settleAll()
if state, _ := GroupState(g, w.h.membersOf(g)); state != "ready" {
t.Fatalf("the group composed green is %s", state)
}
if g.Check.Trunks[idc] != trunk0[:12] {
t.Fatalf("the composed check names its members' trunks %v", g.Check.Trunks)
}
return w
}
t.Run("a merge that is not a member", func(t *testing.T) {
w := start(t)
asked := len(w.ctl.checks)
w.h.PullMerged(merged(ctl, 5, "5555555555555555", merge1))
w.settleAll()
g := w.h.groups["feat/g"]
state, why := GroupState(g, w.h.membersOf(g))
if state != "checking" || g.Check.OutOfDate == "" {
t.Fatalf("the group is %s (%s) after a merge that is not its member moved a member's trunk", state, why)
}
for _, head := range []string{ctl + "@" + hc, cat + "@" + hk} {
s := w.forge.statuses[head+" "+ctxGroup]
if !strings.HasPrefix(s, "pending ") || !strings.Contains(s, "out of date") {
t.Fatalf("mesh/delivery-group on %s says %q, not pending with the reason", head, s)
}
}
if len(w.ctl.checks) != asked {
t.Fatalf("the trunk's move asked a check: %v", w.ctl.checks[asked:])
}
// The controller's member judged again against the trunk's head: the composed check runs again.
w.later(time.Minute)
w.h.Checked(judged(ctl, 1, hc, "pass", merge1))
w.settleAll()
if len(w.ctl.checks) == asked || !strings.HasPrefix(w.ctl.checks[len(w.ctl.checks)-1], "feat/g:") {
t.Fatalf("the composed check was not asked again once a member was judged again: %v", w.ctl.checks)
}
})
t.Run("its own member merging", func(t *testing.T) {
w := start(t)
mc := merged(ctl, 1, hc, merge1)
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.settleAll()
g := w.h.groups["feat/g"]
if g.Check.OutOfDate != "" {
t.Fatalf("the group's own member merging put its composed check out of date: %s", g.Check.OutOfDate)
}
if s := w.state(idk); s != Ready {
t.Fatalf("the catalogue's member, in another repository, is %s", s)
}
})
}
// **After a member is rebased or pushed, the composed check runs again for the new heads** and the group reads
// checking, not held (ADR 0323 §6, issue 406's second symptom): its first member merged, closing the group, and then
// its second was rebased just before its own merge.
func TestARebasedMemberIsComposedAgainNotHeld(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk, hc2 = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1", "c2c2c2c2c2c2c2c2"
idc, idk, idc2 := IDOf(ctl, hc), IDOf(cat, hk), IDOf(ctl, hc2)
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
if len(g.Order) != 2 || g.Order[0] != idk {
t.Fatalf("the group's order is %v: the catalogue's member first is what this replays", g.Order)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed"})
w.settleAll()
// The catalogue's member merges first: the group is closed.
mk := merged(cat, 2, hk, merge1)
mk.Head = "feat/g"
w.h.PullMerged(mk)
w.h.WalkMoved(w.ctl.put(aWalk("plan-cat", cat, merge1, true)))
w.settleAll()
if !g.Closed {
t.Fatal("the group is not closed once a member merged")
}
// The controller's member is rebased just before its merge: a new head, with its own check.
w.h.PullUpdated(pr(ctl, 1, hc2, "feat/g"))
w.settleAll()
if d := w.delivery(idc2); d.Group != "feat/g" {
t.Fatalf("the rebased head did not take its place in its closed group: %q", d.Group)
}
if containsString(g.Members, idc) || !containsString(g.Members, idc2) || len(g.Order) != 2 || g.Order[1] != idc2 {
t.Fatalf("the group's members are %v in the order %v", g.Members, g.Order)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
w.h.Checked(judged(ctl, 1, hc2, "pass", trunk0))
w.settleAll()
asked := w.ctl.checks[len(w.ctl.checks)-1]
if !strings.HasPrefix(asked, "feat/g:") || !strings.Contains(asked, idc2) || !strings.Contains(asked, idk) {
t.Fatalf("the composed check was not asked again for the new heads: %v", w.ctl.checks)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
for _, id := range []string{idk, idc2} {
if d := w.delivery(id); d.State == Held {
t.Fatalf("%s is held for a person: %s", id, d.HeldWhy)
}
}
// Its verdict for the new heads: the rebased member merges and takes its turn.
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed again"})
mc := merged(ctl, 1, hc2, "e0e0e0e0e0e0e0e0")
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.h.WalkMoved(w.ctl.put(aWalk("plan-ctl", ctl, "e0e0e0e0e0e0e0e0", true)))
w.settleAll()
if d := w.delivery(idc2); d.State == Held {
t.Fatalf("the rebased member, composed green for the new heads, is held: %s", d.HeldWhy)
}
}
// A head whose merge check the forge's holder put out of date is not a check asked and not answered: `stalled` does
// not list it, however long it waits — rejected, or with no delivery here — until its check is asked (ADR 0323 §7).
func TestAnOutOfDateCheckOnTheForgeIsNotSaidUnanswered(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b9c0d"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "fail", trunk0))
words := outOfDateWords(trunk0, merge1)
pending := func(description, at string) map[string]HeadCheck {
return map[string]HeadCheck{ctxGate: {State: "pending", Description: description, UpdatedAt: at},
ctxRepo: {State: "pending", Description: description, UpdatedAt: at}}
}
w.forge.open = []OpenPull{{Owner: "novox", Repo: "mesh-lab", Number: 9, Base: "main", HeadSHA: head,
Required: []string{ctxGate, ctxRepo}, Statuses: []string{ctxGate, ctxRepo}, Checks: pending(words, "2026-10-06T12:01:00Z")}}
w.h.SeeOpenPulls()
w.later(3 * time.Hour)
w.h.SeeOpenPulls()
for _, l := range w.h.Stalled() {
if strings.HasPrefix(l.ID, repo) {
t.Fatalf("an out-of-date check nobody asked is said stalled: %+v", l)
}
}
// Asked by the merge tool: pending anew, and its bound runs from then.
w.forge.open[0].Checks = pending("waiting in the build queue since 15:01: asked by the merge tool", "2026-10-06T15:01:00Z")
w.h.SeeOpenPulls()
w.later(2 * time.Hour)
w.h.SeeOpenPulls()
said := false
for _, l := range w.h.Stalled() {
said = said || l.State == Unanswered
}
if !said {
t.Fatal("a check the merge tool asked and that never answered is not said")
}
}
// The merge tool's ask announces an out-of-date head again: its one-hour bound runs from then, and a check that never
// answers is said stalled (ADR 0323 §7).
func TestAnOutOfDateDeliveryAskedAgainIsBoundFromTheAsk(t *testing.T) {
const repo = "novox/mesh-lab"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
w.h.PullUpdated(pr(repo, 9, head, "x"))
w.h.Checked(judged(repo, 9, head, "pass", trunk0))
w.h.PullMerged(merged(repo, 8, "8888888888888888", merge1))
id := IDOf(repo, head)
w.later(3 * time.Hour)
w.h.PullUpdated(pr(repo, 9, head, "x")) // the merge tool asked its check
w.later(30 * time.Minute)
for _, l := range w.h.Stalled() {
if l.ID == id {
t.Fatalf("listed half an hour after its check was asked again: %+v", l)
}
}
w.later(time.Hour)
said := false
for _, l := range w.h.Stalled() {
if l.ID == id {
said = true
if !strings.Contains(l.Says, "asked again") || !strings.Contains(l.H2, "operator") {
t.Fatalf("its stalled line says %+v", l)
}
}
}
if !said {
t.Fatal("a check asked again that never answered is not said")
}
// Its verdict judged against the trunk's head: ready, and nothing bounds it as proposed.
w.h.Checked(judged(repo, 9, head, "pass", merge1))
if d := w.delivery(id); d.State != Ready || d.AskedAgain != nil {
t.Fatalf("judged again it is %s, asked again %v", d.State, d.AskedAgain)
}
}
// The same, with the first member already delivered when the second is rebased: a delivered member counts for the
// composed check of the new heads, which is asked, and the rebased member is never held (review of ADR 0323 §6).
func TestARebasedMemberAfterAnotherWasDeliveredIsComposedAgain(t *testing.T) {
const ctl, cat = "novox/mesh-controller", "novox/mesh-catalog"
const hc, hk, hc2 = "c1c1c1c1c1c1c1c1", "k1k1k1k1k1k1k1k1", "c2c2c2c2c2c2c2c2"
idk, idc2 := IDOf(cat, hk), IDOf(ctl, hc2)
w := newWorld(t)
w.h.PullUpdated(pr(ctl, 1, hc, "feat/g"))
w.h.PullUpdated(pr(cat, 2, hk, "feat/g"))
w.h.Checked(judged(ctl, 1, hc, "pass", trunk0))
w.h.Checked(judged(cat, 2, hk, "pass", trunk0))
w.settleAll()
g := w.h.groups["feat/g"]
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed"})
w.settleAll()
// The catalogue's member merges and its walk is done: delivered.
mk := merged(cat, 2, hk, merge1)
mk.Head = "feat/g"
w.h.PullMerged(mk)
walk := w.ctl.put(aWalk("plan-cat", cat, merge1, true))
w.h.WalkMoved(walk)
w.settleAll()
walk = w.ctl.walk("plan-cat")
walk.State = walkDone
w.h.WalkMoved(w.ctl.put(walk))
w.settleAll()
if s := w.state(idk); s != Delivered {
t.Fatalf("the catalogue's member is %s, not delivered", s)
}
// The controller's member is rebased, and judged.
w.h.PullUpdated(pr(ctl, 1, hc2, "feat/g"))
w.h.Checked(judged(ctl, 1, hc2, "pass", trunk0))
w.settleAll()
asked := w.ctl.checks[len(w.ctl.checks)-1]
if !strings.HasPrefix(asked, "feat/g:") || !strings.Contains(asked, idc2) {
t.Fatalf("the composed check was not asked again for the new heads: %v", w.ctl.checks)
}
if state, why := GroupState(g, w.h.membersOf(g)); state != "checking" {
t.Fatalf("the group reads %s (%s), not checking", state, why)
}
w.h.Checked(CheckedEvent{Group: "feat/g", ID: g.Check.Asked, Verdict: "pass", Summary: "composed again"})
mc := merged(ctl, 1, hc2, "e0e0e0e0e0e0e0e0")
mc.Head = "feat/g"
w.h.PullMerged(mc)
w.h.WalkMoved(w.ctl.put(aWalk("plan-ctl", ctl, "e0e0e0e0e0e0e0e0", true)))
w.settleAll()
if d := w.delivery(idc2); d.State == Held {
t.Fatalf("the rebased member is held: %s", d.HeldWhy)
}
}
// A check this owner asks — a recheck — sends the base head the forge last announced the head with, so the
// controller's verdict for a head no build judges names its trunk commit (ADR 0323 §3, review of mesh-controller#271).
func TestARecheckSendsTheAnnouncedBaseHead(t *testing.T) {
const repo = "novox/mesh-tools-go"
const head = "1a2b3c4d5e6f7a8b"
w := newWorld(t)
p := pr(repo, 3, head, "x")
p.BaseSHA = trunk0
w.h.PullUpdated(p)
id := IDOf(repo, head)
if m := MemberOf(w.delivery(id)); m.BaseSHA != trunk0 {
t.Fatalf("the member sends base head %q", m.BaseSHA)
}
p.BaseSHA = merge1 // announced again, by the merge tool
w.h.PullUpdated(p)
if m := MemberOf(w.delivery(id)); m.BaseSHA != merge1 {
t.Fatalf("announced again, the member sends base head %q", m.BaseSHA)
}
if raw := string(mustJSON(MemberOf(w.delivery(id)))); !strings.Contains(raw, `"base_sha":"`+merge1+`"`) {
t.Fatalf("the member as the controller reads it: %s", raw)
}
}
@@ -1,105 +0,0 @@
package main
import (
"testing"
"time"
)
// triedSaid is delivery_status's count of owed effects tried (novox/hq issue 407), read as a test reads it.
func triedSaid(t *testing.T, w *world) map[string]any {
t.Helper()
got, ok := w.h.Status("listening")["owed-tried"].(map[string]any)
if !ok {
t.Fatalf("delivery_status says no owed-tried: %v", w.h.Status("listening"))
}
return got
}
func countOf(t *testing.T, m map[string]any, key string) int {
t.Helper()
n, ok := m[key].(int)
if !ok {
t.Fatalf("owed-tried has no count %q: %v", key, m)
}
return n
}
// novox/hq issue 407: delivery_status says how many owed effects were tried since this owner started, how
// many were done, failed to be tried again, or were given up, and when the last was tried — so a window's
// writes can be checked against the tries made in it (issue 326's rule) by reading it at both ends.
func TestDeliveryStatusCountsTheOwedEffectsTried(t *testing.T) {
w := newWorld(t)
start := w.now
tried := triedSaid(t, w)
if countOf(t, tried, "tried") != 0 || countOf(t, tried, "done") != 0 || countOf(t, tried, "failed") != 0 ||
countOf(t, tried, "given-up") != 0 {
t.Fatalf("a holder that tried nothing says %v", tried)
}
if tried["since"] != start.Format(time.RFC3339) {
t.Fatalf("its count is since %v, the holder started %s", tried["since"], start.Format(time.RFC3339))
}
if _, there := tried["last"]; there {
t.Fatalf("nothing tried, yet a last try is said: %v", tried)
}
// A head seen: its event and its note are owed and done.
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
tried = triedSaid(t, w)
done := countOf(t, tried, "done")
if done < len(w.said)+len(w.forge.notesOn(head)) || done == 0 {
t.Fatalf("said %d and noted %d, yet done counts %d", len(w.said), len(w.forge.notesOn(head)), done)
}
if countOf(t, tried, "tried") != done || countOf(t, tried, "failed") != 0 {
t.Fatalf("every try succeeded, yet %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
// The forge away: its effects fail and are counted failed; quiet passes try nothing more.
w.forge.down = true
w.later(time.Minute)
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.Flush()
tried = triedSaid(t, w)
failed := countOf(t, tried, "failed")
if failed == 0 {
t.Fatalf("the forge away failed nothing: %v", tried)
}
if countOf(t, tried, "tried") != countOf(t, tried, "done")+failed+countOf(t, tried, "given-up") {
t.Fatalf("tries do not add up: %v", tried)
}
if tried["last"] != w.now.Format(time.RFC3339) {
t.Fatalf("last try said %v, tried at %s", tried["last"], w.now.Format(time.RFC3339))
}
before := countOf(t, tried, "tried")
w.h.Flush() // before each failure's next try: nothing is tried
if n := countOf(t, triedSaid(t, w), "tried"); n != before {
t.Fatalf("a pass before any next try tried %d", n-before)
}
// The forge back, past every next try: the failed effects are done.
w.forge.down = false
w.later(time.Hour)
w.h.Flush()
tried = triedSaid(t, w)
if countOf(t, tried, "tried") != before+failed || countOf(t, tried, "done") <= done {
t.Fatalf("the forge back: tried %d (was %d, %d failed), %v", countOf(t, tried, "tried"), before, failed, tried)
}
}
// The count is this run's: a restarted holder says it counts since its own start, from nothing.
func TestOwedTriedCountsSinceTheStart(t *testing.T) {
w := newWorld(t)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.settleAll()
if countOf(t, triedSaid(t, w), "tried") == 0 {
t.Fatal("nothing counted before the restart")
}
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Minute))
tried := triedSaid(t, again)
if countOf(t, tried, "tried") != 0 || tried["since"] != again.now.Format(time.RFC3339) {
t.Fatalf("a restarted holder says %v", tried)
}
}
@@ -1,48 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
)
// **An unanswered line says the state of the delivery it holds, as data** (novox/hq issue 451): for a delivery in
// ready or rejected, whose check answered once, the controller advises a recheck rather than the build queue, and
// it reads that state from the line, never from its words. A head with no delivery held says none.
func TestAnUnansweredLineNamesTheStateOfItsDelivery(t *testing.T) {
at := time.Date(2026, 10, 11, 0, 43, 0, 0, time.UTC)
for _, c := range []struct {
gate string
want State
}{{"pass", Ready}, {"fail", Rejected}} {
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
id := heldIn(t, w, c.gate, c.want)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].ID != id || lines[0].State != Unanswered || lines[0].Held != c.want {
t.Fatalf("a %s delivery's unanswered line does not say its state as data: %+v", c.want, lines)
}
raw, _ := json.Marshal(lines[0])
if !strings.Contains(string(raw), `"held":"`+string(c.want)+`"`) {
t.Fatalf("the line the controller reads carries no held state: %s", raw)
}
}
// The same head with no delivery held here: unanswered, and no state to say.
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), at)
w.forge.open = openPullsOf(t, readyRepoCheckPending)
w.h.SeeOpenPulls()
w.later(Bounds[Proposed].For + time.Minute)
w.h.SeeOpenPulls()
lines := w.h.Stalled()
if len(lines) != 1 || lines[0].State != Unanswered || lines[0].Held != "" {
t.Fatalf("a head with no delivery held says a state: %+v", lines)
}
if raw, _ := json.Marshal(lines[0]); strings.Contains(string(raw), `"held"`) {
t.Fatalf("a line with no delivery held carries a held field: %s", raw)
}
}
@@ -1,134 +0,0 @@
package main
import (
"strings"
"testing"
"time"
)
// novox/hq issue 480: a delivery stopped because its walk ended done before its word without building its modules
// (issue 474) is still read for a later merge that delivers them, and kept CoveredBy once one did. Its state stays
// stopped — a final state never moves, as for a covered superseded member (issue 442) — but the forge's status, its
// view, its note and `show` went on saying "stopped" for a change that reached every machine. Once covered they say
// covered, and by which merge. And while `stalled` lists such a stop, `deliveries` lists it too, past a day.
// strandedNight is theNight with #215 stopped, its walk past the grace with nothing later carrying build-agent.
func strandedNight(t *testing.T) (*world, string) {
w := theNight(t)
id := IDOf(ctl474, head215)
w.later(strandedAfter + 5*time.Minute)
w.settleAll()
if d := w.delivery(id); d.State != Stopped || d.Stranded == "" {
t.Fatalf("#215 is %s, stranded %q", d.State, d.Stranded)
}
return w, id
}
func TestAStopALaterMergeCoversReadsCoveredByThatMerge(t *testing.T) {
w, id := strandedNight(t)
if got := statusOn(w, ctl474, head215); !strings.HasPrefix(got, "error stopped: ") {
t.Fatalf("stopped, its status says %q", got)
}
w.forge.contains[merge215+" in "+merge219] = true
by := deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
d := w.delivery(id)
if d.State != Stopped || !strings.Contains(d.CoveredBy, by) {
t.Fatalf("after #219 it is %s, covered by %q", d.State, d.CoveredBy)
}
got := statusOn(w, ctl474, head215)
if !strings.HasPrefix(got, "success covered") || !strings.Contains(got, by) {
t.Fatalf("covered by %s, its status still says %q", by, got)
}
shown, err := w.h.Show(id)
if err != nil {
t.Fatal(err)
}
if reads, _ := shown.(map[string]any)["reads"].(string); !strings.HasPrefix(reads, "covered") || !strings.Contains(reads, by) {
t.Fatalf("show reads %q", reads)
}
view := w.forge.views[ctl474+"#215"]
if !strings.Contains(view, "Covered") || !strings.Contains(view, by) {
t.Fatalf("its view does not say it was covered by %s:\n%s", by, view)
}
noted := false
for _, l := range w.forge.notesOn(merge215) {
noted = noted || strings.Contains(l, "covered") && strings.Contains(l, by)
}
if !noted {
t.Fatalf("its merge's note does not say it was covered: %v", w.forge.notesOn(merge215))
}
// Told once: further ticks owe nothing more.
w.later(time.Hour)
w.settleAll()
if d := w.delivery(id); len(d.Owed) != 0 || !d.CoverTold {
t.Fatalf("still owes %+v, told %v", d.Owed, d.CoverTold)
}
}
// A stop covered before this was built (CoveredBy kept, nothing told) is told after a restart, from what was kept.
func TestACoveredStopKeptBeforeIsToldAfterARestart(t *testing.T) {
w, id := strandedNight(t)
w.h.mu.Lock()
d := w.h.deliveries[id]
d.CoveredBy = IDOf(ctl474, head219) + " (merged as " + merge219[:8] + ")"
if err := w.store.PutDelivery(d); err != nil {
t.Fatal(err)
}
w.h.mu.Unlock()
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Hour))
again.settleAll()
if got := statusOn(again, ctl474, head215); !strings.HasPrefix(got, "success covered") ||
!strings.Contains(got, IDOf(ctl474, head219)) {
t.Fatalf("after a restart its status says %q", got)
}
if d := again.delivery(id); d.State != Stopped || len(d.Transitions) != len(w.delivery(id).Transitions) {
t.Fatalf("a restart moved it: %s", d.State)
}
}
// Only a stop for a walk that never built its modules reads covered: a person's stop, or one the cover test has
// not answered, keeps its error.
func TestOnlyACoveredStrandedStopReadsCovered(t *testing.T) {
for _, d := range []*Delivery{
{State: Stopped, CoveredBy: "x"}, // a person's stop, with a cover found while it was held
{State: Stopped, Stranded: "never reached"}, // stranded, not covered
} {
if state, _ := statusOf(d); state != "error" {
t.Fatalf("%+v reads %s", d, state)
}
if d.coveredStop() {
t.Fatalf("%+v reads as a covered stop", d)
}
}
}
func TestAStopStalledListsStaysInDeliveries(t *testing.T) {
w, id := strandedNight(t)
w.later(30 * time.Hour)
w.settleAll()
listed := func() *Line {
for _, l := range w.h.Deliveries("", "", "", false) {
if l.ID == id {
return &l
}
}
return nil
}
if len(w.h.Stalled()) != 1 {
t.Fatalf("stalled lists %+v", w.h.Stalled())
}
l := listed()
if l == nil || !l.Stalled || !strings.Contains(l.Waits, "build-agent") {
t.Fatalf("listed by stalled, deliveries shows %+v", l)
}
// Covered, it ends like any final delivery: a day after its stop.
w.forge.contains[merge215+" in "+merge219] = true
deliverLater(w, ctl474, 219, "main", head219, merge219, w.now, walkDone, "build-agent")
w.later(2 * time.Minute)
w.settleAll()
if len(w.h.Stalled()) != 0 || listed() != nil {
t.Fatalf("covered, stalled lists %+v and deliveries %+v", w.h.Stalled(), listed())
}
}
@@ -192,7 +192,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
seat := "mesh-delivery."
return []stdio.Tool{
{Name: seat + "deliveries",
Description: "Every delivery not final, those that ended in the last day, and any stop `stalled` still lists, one line each: its id " +
Description: "Every delivery not final, and those that ended in the last day, one line each: its id " +
"(owner/repository@commit), its state, its group, what it waits for and since when. Narrowed by state, " +
"repository or group.",
Input: map[string]any{"type": "object", "properties": map[string]any{"state": str("one state"),
@@ -347,8 +347,7 @@ func tools(h *Holder, l *listening) []stdio.Tool {
}},
{Name: "delivery_status",
Description: "Whether this owner works: how many deliveries in each state, the groups, what is owed to " +
"the forge and the bus and not yet done, how many owed effects were tried since it started (done, failed " +
"and tried again, given up) and when the last was, walks with no delivery yet, whether its events arrive, when " +
"the forge and the bus and not yet done, walks with no delivery yet, whether its events arrive, when " +
"the controller's walks were last read or why they could not be, and the last refusals.",
Run: func(map[string]any) (any, error) { return h.Status(l.get()), nil }},
}
@@ -49,17 +49,6 @@ type Delivery struct {
// CheckAsk is this owner's own ask of its check, when the forge's announcement did not bring one: a head
// adopted while it already waited, or one whose announcement the controller missed.
CheckAsk *CheckAsk `json:"check_ask,omitempty"`
// TrunkNow is its trunk's head as this owner last heard it move while it was open: the commit the last merge
// into its trunk landed as (novox/hq ADR 0323 §7). A verdict that names another trunk commit is out of date.
// Empty until a merge into its trunk is heard after it was made.
TrunkNow string `json:"trunk_now,omitempty"`
// BaseSHA is the base branch's head the forge last announced its head with (ADR 0323): sent with a check this
// owner asks, for a verdict no build judges to name its trunk commit.
BaseSHA string `json:"base_sha,omitempty"`
// AskedAgain is when its head was announced again while its verdict was out of date — the merge tool's ask of
// its check against the trunk's head (ADR 0323 §5) — and no verdict has come since: its one-hour bound runs from
// then. Nil while nothing is asked, and cleared by a verdict or another move of its trunk.
AskedAgain *time.Time `json:"asked_again,omitempty"`
// The facts the table's guards read. NewerHead is the head that superseded it; ClosedUnmerged the pull
// request closed; MergedAs the commit it landed on the trunk as; HeldWhy what holds it for a person.
@@ -79,9 +68,6 @@ type Delivery struct {
// Stranded says why a merge whose walk ended done before its word, without moving every module it moves, is
// stopped: nothing later delivered them either (novox/hq issue 474). Set once the cover test says no.
Stranded string `json:"stranded,omitempty"`
// CoverTold is whether the forge was told that such a stop was covered since: its status, its view and its note
// (novox/hq issue 480). Kept, so a restart tells it once, and a stop covered before this was built is told too.
CoverTold bool `json:"cover_told,omitempty"`
// TurnAwaits says, for a group member whose turn waits for a later merge to carry a superseded member before it,
// which merges and until when; TurnAwaitsUntil is that moment, which bounds the wait (ADR 0276 decision 9,
// novox/hq issue 445). `show` shows both, and `stalled` says them past that moment.
@@ -139,38 +125,6 @@ type Verdict struct {
Repo string `json:"repo,omitempty"`
RepoSaid string `json:"repo_summary,omitempty"`
Report string `json:"report,omitempty"`
// Head and Trunk are what it judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, twelve
// characters each. Trunk is empty for a verdict of a build seat that judged the head alone. Beside are the
// commits of the other repositories the check read, by directory; Took how long it ran.
Head string `json:"head,omitempty"`
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
}
// Judged is what a verdict judged, in words: "head 1a2b3c4d5e6f merged into trunk 7a8b9c0d1e2f" (ADR 0323 §3);
// empty for a verdict that names no trunk commit.
func (v *Verdict) Judged() string {
if v == nil || v.Trunk == "" {
return ""
}
return fmt.Sprintf("head %s merged into trunk %s", shortOf(v.Head, 12), shortOf(v.Trunk, 12))
}
// outOfDate is why a delivery's verdict no longer counts: it names a trunk commit that is not its trunk's head
// as this owner heard it (novox/hq ADR 0323 §7). Empty when it counts — or names no trunk commit, as a verdict
// of a build seat from before the rule does, which counts as it did.
func (d *Delivery) outOfDate() string {
if d.Check == nil || d.Check.Trunk == "" || d.TrunkNow == "" || sameCommit(d.Check.Trunk, d.TrunkNow) {
return ""
}
return outOfDateWords(d.Check.Trunk, d.TrunkNow)
}
// outOfDateWords is a verdict out of date, as the forge's holder says it on the pull request (ADR 0323 §4).
func outOfDateWords(judged, now string) string {
return fmt.Sprintf("out of date: judged against trunk %s, the trunk is now %s; judged again on a recheck or when "+
"it is merged", shortOf(judged, 12), shortOf(now, 12))
}
// CheckAsk is a head's own check, asked of the controller by this owner (or re-asked by healer H2).
@@ -208,8 +162,6 @@ type DeliveryPlan struct {
Machines []MachinePlan `json:"machines,omitempty"`
Steps []string `json:"steps,omitempty"`
Summary string `json:"summary"`
// Trunk is the trunk commit the head was merged into when its verdict judged it (novox/hq ADR 0323 §3).
Trunk string `json:"trunk,omitempty"`
}
// receiving are the machines the plan sends a module to, in the plan's order; none when the plan names no
@@ -296,20 +248,6 @@ func (d *Delivery) FromForgesWord() bool {
// shownOnItsPullRequest is whether its view and status are kept on its pull request: not for history, whose
// pull request the forge may no longer hold — or holds under its number another one.
// coveredStop is whether a delivery stopped because its walk ended done before its word without building its modules
// (novox/hq issue 474) was covered since: a later merge into its trunk that contains it was delivered, carrying every
// module it moves to every machine running it (ADR 0270's cover test, kept as CoveredBy). Its state stays stopped —
// final, as a covered superseded member stays superseded (issue 442) — and it reads covered, by that merge (novox/hq
// issue 480). A stop for any other reason never reads covered.
func (d *Delivery) coveredStop() bool {
return d.State == Stopped && d.Stranded != "" && d.CoveredBy != ""
}
// coveredStopSaid is how a covered stop reads: on the forge's status, in its view and note, and in `show`.
func (d *Delivery) coveredStopSaid() string {
return "covered after its stop: a later merge that contains it was delivered: " + d.CoveredBy
}
func (d *Delivery) shownOnItsPullRequest() bool {
return d.Number > 0 && d.History == "" && d.State != Retired
}
@@ -797,13 +735,6 @@ type GroupCheck struct {
Commits map[string]string `json:"commits"`
Verdict string `json:"verdict,omitempty"`
Summary string `json:"summary,omitempty"`
// Trunks are the trunk commit each member's head was merged into, by member id (novox/hq ADR 0323 §6).
Trunks map[string]string `json:"trunks,omitempty"`
// OutOfDate is why its verdict no longer counts: a merge that is not one of the group's members moved a
// member's trunk, at OutOfDateAt. It counts as outstanding (ADR 0239 §18), and runs again once a member is
// judged again after it.
OutOfDate string `json:"out_of_date,omitempty"`
OutOfDateAt time.Time `json:"out_of_date_at"`
}
// Commits is each member's commit, by id.
@@ -31,16 +31,13 @@ type Member struct {
ModuleDirsSaid bool `json:"module_dirs_said,omitempty"`
CloneURL string `json:"clone_url,omitempty"`
After []string `json:"after,omitempty"`
// BaseSHA is the base branch's head the forge last announced the head with (novox/hq ADR 0323): the trunk commit
// the controller's verdict names when no build judges the head.
BaseSHA string `json:"base_sha,omitempty"`
}
// MemberOf is a delivery as a member.
func MemberOf(d *Delivery) Member {
return Member{ID: d.ID, Repository: d.Repository, Base: d.Base, Head: d.Commit, Number: d.Number, Paths: d.Paths,
PathsTruncated: d.PathsTruncated, Removed: d.Removed, ModuleDirs: d.ModuleDirs, ModuleDirsSaid: d.ModuleDirsSaid,
CloneURL: d.CloneURL, After: d.After, BaseSHA: d.BaseSHA}
CloneURL: d.CloneURL, After: d.After}
}
// Order is a group's order as the controller works it out: each pair with the rules it won over by
@@ -85,7 +85,6 @@ const (
EvCovered Event = "covered" // a later merge into the same trunk delivered every module it moves
EvLifted Event = "lifted" // what held it for a person is no longer so: its group's composed check passed, or a walk now answers its merge
EvRewalk Event = "rewalk" // the walk that carried a later commit for it failed, or was stopped for another merge (ADR 0299): it waits for the walk that walks it again
EvTrunkMoved Event = "trunk moved" // another merge moved its trunk, and its verdict names the trunk commit before (novox/hq ADR 0323)
)
// Facts are what a guard reads beyond the delivery itself: the moment, and who asks, for an act.
@@ -168,17 +167,9 @@ var Table = []Row{
Holds: func(d *Delivery, f Facts) error {
return unless(d.MergedAs != "" && (d.Walk != nil || d.MovesNothing()), "not on the trunk yet")
}},
// **A verdict counts only for the trunk's head** (novox/hq ADR 0323 §7): one naming a trunk commit that is no
// longer the trunk's head is kept and shown, and does not count, so the delivery stays proposed and the rows
// cannot loop with `trunk moved`; a recheck or the merge tool's ask judges it again.
{From: []State{Proposed}, Event: EvChecked, To: Checked,
Guard: "a verdict names this commit and the trunk's head as its trunk commit",
Holds: func(d *Delivery, f Facts) error {
if d.Check == nil {
return errors.New("no verdict")
}
return unless(d.outOfDate() == "", "its verdict is "+d.outOfDate())
}},
Guard: "the verdict names this commit",
Holds: func(d *Delivery, f Facts) error { return unless(d.Check != nil, "no verdict") }},
{From: []State{Checked}, Event: EvAccepted, To: Ready,
Guard: "the gate passed or warned, and the repository's own check did not fail",
Holds: func(d *Delivery, f Facts) error { return unless(d.Check.Passes(), "the verdict does not pass") }},
@@ -208,14 +199,6 @@ var Table = []Row{
Says: func(d *Delivery) string {
return "covered: a later merge that contains it was delivered: " + d.CoveredBy
}},
// **A ready delivery whose verdict went out of date goes back to proposed** (novox/hq ADR 0323 §7): another
// merge moved its trunk. A rejected one stays rejected, its status on the forge saying the verdict is out of date.
{From: []State{Ready}, Event: EvTrunkMoved, To: Proposed,
Guard: "its verdict names a trunk commit that is not its trunk's head",
Holds: func(d *Delivery, f Facts) error {
return unless(d.MergedAs == "" && d.outOfDate() != "", "not merged, and its verdict is not out of date")
},
Says: func(d *Delivery) string { return d.outOfDate() }},
{From: []State{Proposed, Checked, Rejected}, Event: EvMergedUnchecked, To: Held,
Guard: "merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on",
Holds: func(d *Delivery, f Facts) error { return unless(d.MergedAs != "", "not merged") }},
@@ -560,9 +543,6 @@ const checkGrace = 15 * time.Minute
// runs from its start plus the grace before it is asked: an ask that cannot be made is still listed.
func boundSince(d *Delivery) time.Time {
if d.State == Proposed {
if d.AskedAgain != nil && d.outOfDate() != "" {
return *d.AskedAgain
}
if d.CheckAsk != nil {
return d.CheckAsk.At
}
@@ -582,17 +562,6 @@ var rewalkBound = Bound{For: unwalked, Says: "the walk that carried a later comm
// boundOf is a delivery's bound: its state's, or none for a delivery held because no walk answers it.
func boundOf(d *Delivery) (Bound, bool) {
if d.State == Proposed && d.outOfDate() != "" {
// Its check is out of date and not asked: nothing is running for it, and it costs nothing (ADR 0323 §7).
// Its bound starts when its check is asked: a recheck puts the verdict away; the merge tool's ask announces the
// head again, and its bound runs from then.
if d.AskedAgain == nil {
return Bound{}, false
}
b := Bounds[Proposed]
b.H2, b.Says = nil, "its verdict is "+d.outOfDate()+"; its check was asked again and not answered: the operator's"
return b, true
}
if d.State == Held && d.Unwalked && d.Walk == nil {
return heldUnwalked, true
}
@@ -82,9 +82,6 @@ func aDeliveryFor(r Row) (*Delivery, Facts) {
case EvRewalk:
d.MergedAs, d.AnsweredBy = "aaaa1111", "bbbb2222"
d.Walk = &WalkSeen{ID: "plan-2", State: walkFailed}
case EvTrunkMoved:
d.Check = &Verdict{Gate: "pass", Head: "c0ffee000000", Trunk: "d28ca95bbbbb"}
d.TrunkNow = "690966dccccccccc"
case EvLifted:
d.MergedAs, d.Lifted = "aaaa1111", "its group's composed check passed"
d.Walk = &WalkSeen{ID: "plan-1", State: walkBuilding, Waits: true, WaitedFor: "mesh-delivery"}
@@ -132,7 +129,7 @@ func TestEveryRowOfTheTableIsTakenWhenItsGuardHoldsAndRefusedWhenNot(t *testing.
func TestEveryPairTheTableDoesNotHoldIsRefusedByName(t *testing.T) {
events := []Event{EvAnnounced, EvAppeared, EvAdopted, EvChecked, EvAccepted, EvRefused, EvRecheck, EvReask, EvNewHead,
EvClosed, EvMerged, EvMergedUnchecked, EvGo, EvHold, EvRelease, EvDone, EvFailed, EvSuperseded, EvStop,
EvHistory, EvRetire, EvCovered, EvLifted, EvRewalk, EvTrunkMoved}
EvHistory, EvRetire, EvCovered, EvLifted, EvRewalk}
for _, from := range append([]State{None}, AllStates...) {
for _, ev := range events {
for _, act := range []bool{false, true} {
@@ -44,10 +44,6 @@ const Unannounced State = "unannounced"
// own for the proposed state's bound: its check was asked and not answered.
const Unanswered State = "unanswered"
// outOfDatePrefix begins the words the forge's holder puts a merge check back to pending with when its trunk moved
// (novox/hq ADR 0323 §4): the gitea module's outOfDateWords.
const outOfDatePrefix = "out of date:"
// UncheckedAfter is how long a head may go without a merge check before it is said: the announcer passes every half
// minute and the check is asked at once, so ten minutes is a fault, never a queue.
const UncheckedAfter = 10 * time.Minute
@@ -123,9 +119,6 @@ func (p OpenPull) unanswered() (waiting []string, pending map[string]string, sta
// since nothing here says whether the one it has answered.
case !ok || got.State == "":
waiting = append(waiting, c+" never set")
case got.State == "pending" && strings.HasPrefix(got.Description, outOfDatePrefix):
// Out of date because its trunk moved, and not asked (novox/hq ADR 0323 §4, §7): nothing runs for it and
// it costs nothing, so it is not a check asked and not answered. Asked again, it is pending anew.
case got.State == "pending":
pending[c] = utcOf(got.UpdatedAt)
waiting = append(waiting, c+" pending since "+orUnknown(pending[c]))
@@ -257,8 +250,8 @@ func (h *Holder) uncheckedLines(now time.Time) []StalledLine {
h2 = "none: its delivery is " + string(u.held) + ", a state the operator moves — the operator's"
}
out = append(out, StalledLine{ID: id, Number: u.pull.Number, State: state, For: now.Sub(u.first).Round(time.Second).String(),
Bound: u.bound().String(), Held: u.held,
H2: h2, Says: says, Waiting: waiting})
Bound: u.bound().String(),
H2: h2, Says: says, Waiting: waiting})
}
return out
}
@@ -71,11 +71,6 @@ func (h *Holder) waitsFor(d *Delivery) string {
if d.Walk != nil {
return "its walk: " + clip(d.Walk.Note, 120)
}
case Stopped:
if strandedUncovered(d) {
return "a later delivered merge to carry " + strings.Join(d.notMovedBy(), ", ") + ": it is stopped, and " +
"listed by stalled until one does (novox/hq issue 480)"
}
}
return ""
}
@@ -91,8 +86,7 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
group != "" && d.Group != group {
continue
}
// A final delivery `stalled` still lists stays, however old: the two never disagree (novox/hq issue 480).
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour && !strandedUncovered(d) {
if d.State.Final() && !all && now.Sub(d.Since) > 24*time.Hour {
continue
}
l := Line{ID: d.ID, State: d.State, Since: d.Since.UTC().Format(time.RFC3339), For: now.Sub(d.Since).Round(time.Second).String(),
@@ -103,7 +97,7 @@ func (h *Holder) Deliveries(state, repository, group string, all bool) []Line {
if d.Walk != nil {
l.Walk = d.Walk.ID
}
l.Stalled = pastBound(d, now) || strandedUncovered(d)
l.Stalled = pastBound(d, now)
out = append(out, l)
}
sort.Slice(out, func(i, j int) bool {
@@ -121,9 +115,6 @@ func (h *Holder) Show(id string) (any, error) {
defer h.mu.Unlock()
if d := h.deliveries[id]; d != nil {
out := map[string]any{"delivery": d, "waits": h.waitsFor(d)}
if d.coveredStop() {
out["reads"] = d.coveredStopSaid() // stopped, and its change reached the machines (novox/hq issue 480)
}
if g := h.groups[d.Group]; g != nil {
state, why := GroupState(g, h.membersOf(g))
out["group"] = map[string]any{"group": g, "state": state, "why": why}
@@ -199,14 +190,10 @@ type StalledLine struct {
// the operator's own zone, which it cannot do inside a finished sentence (novox/hq issue 443). Waiting stays
// beside them for a controller that does not read them yet.
Checks []WaitingCheck `json:"checks,omitempty"`
// Held is the state of the delivery a line of the state Unanswered is about, ready or rejected, when one is held
// here (novox/hq issues 441, 451): its check answered once, so the controller advises a recheck of it rather than
// the build queue, reading the state here and never from Says. Empty for a head with no delivery.
Held State `json:"held,omitempty"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
For string `json:"for"`
Bound string `json:"bound"`
H2 string `json:"h2"`
Says string `json:"says"`
}
// Stalled is the `stalled` verb: what the controller's self-check raises as `delivery.<id>.stalled`.
@@ -229,8 +216,6 @@ func (h *Holder) Stalled() []StalledLine {
b, _ := boundOf(d)
h2 := "none: the state is the operator's"
switch {
case d.State == Proposed && d.AskedAgain != nil && d.outOfDate() != "":
h2 = "none: its verdict is out of date and its check, asked again, has not answered — the operator's"
case d.State == Proposed && d.CheckAsk == nil:
h2 = "none: its check could not be asked yet — the controller does not answer delivery-check"
case d.State == Proposed && d.CheckAsk.Reasked > 0:
@@ -650,15 +635,6 @@ func (h *Holder) Status(listening string) map[string]any {
out := map[string]any{"deliveries": byState, "groups": len(h.groups), "owed": owed, "unkept": len(h.dirty),
"walks-unmatched": len(h.unmatched), "events": listening, "started": h.started.Format(time.RFC3339),
"seat-on-record": h.held}
// The owed effects tried since this owner started (novox/hq issue 407): done, failed and to be tried again,
// or given up, and when the last was tried. A count since the start, said with that start: two readings in
// one run give a window's tries.
tried := map[string]any{"since": h.started.Format(time.RFC3339), "done": h.tried.done, "failed": h.tried.failed,
"given-up": h.tried.givenUp, "tried": h.tried.done + h.tried.failed + h.tried.givenUp}
if !h.tried.last.IsZero() {
tried["last"] = h.tried.last.Format(time.RFC3339)
}
out["owed-tried"] = tried
// Each held delivery by name, with why and the verb that releases it (novox/hq issue 529): a count alone
// pointed a reader at nothing.
var held []string
+3 -6
View File
@@ -134,12 +134,9 @@ carries `abandoned` on `get` and its `list` row, beside its `worker`. A move by
`take-over`'s reason, as any stale worker; the answer, the note and the `moved` event say it was abandoned
(`abandoned`).
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`) of an
issue not closed: it is said once — a note on the issue and a warning on the operator's channel
(`seat:operator-channel.notify`, key `issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue
is delivered or the issue is closed. Resolved is the end of the warning as wontfix is: its worker proved the
acceptance live, so a fix whose own walk failed and that a later merge carried is not waited for (hq issue 547). An
issue resolved or wontfix is never said stuck. A module that runs only
Every five minutes the module looks for a fix merged and not delivered after `stuck-after` (default `2h`): it is
said once — a note on the issue and a warning on the operator's channel (`seat:operator-channel.notify`, key
`issue-tracker.fix-stuck.<number>`), cleared there when every fix of the issue is delivered. A module that runs only
its bundle cannot raise a controller condition (hq ADR 0240 rule 8, ADR 0256), so the warning is the channel's, not the controller's.
## The import of hq's issues (slice 3)
+12 -23
View File
@@ -266,11 +266,9 @@ func (t *Tracker) afterFix(n uint64) error {
}
// stillStuck says whether an issue has a fix said stuck that is still named, not delivered, and the issue not
// closed. A closed issue is the end of its warning: resolved means its worker proved the acceptance live, so a fix
// the tracker never heard delivered is not waited for any more (a failed walk that a later merge carried stays
// failed, novox/hq issue 547), and wontfix waits for nothing.
// closed as wontfix.
func stillStuck(is *Issue) bool {
if closedStatus(is.Status) {
if is.Status == StatusWontfix {
return false
}
named := liveFixes(is)
@@ -283,30 +281,19 @@ func stillStuck(is *Issue) bool {
}
// clearStuck clears the warning said for an issue on the operator's channel once nothing it was said for is
// stuck — delivered, no longer named, or the issue closed (resolved or wontfix); one that fails is tried again at
// the next sweep.
// stuck — delivered, no longer named, or the issue closed as wontfix; one that fails is tried again at the next
// sweep.
func (t *Tracker) clearStuck(is *Issue) error {
if !is.StuckOpen || stillStuck(is) || t.Notify == nil {
return nil
}
if err := t.Notify(stuckKey(is.Number), "", "", "", "", stuckEnded(is), true); err != nil {
if err := t.Notify(stuckKey(is.Number), "", "", "", "", "the fix of issue "+keyOf(is.Number)+" is delivered or no longer waited for", true); err != nil {
return err
}
_, err := t.change(is.Number, Act{Op: "warned", By: Seat}, func(i *Issue) error { i.StuckOpen = false; return nil })
return err
}
// stuckEnded is what the operator's channel says when a stuck warning ends.
func stuckEnded(is *Issue) string {
switch is.Status {
case StatusResolved:
return "resolved: issue " + keyOf(is.Number) + " was proven live by its worker"
case StatusWontfix:
return "issue " + keyOf(is.Number) + " was closed as wontfix: its fix is no longer waited for"
}
return "the fix of issue " + keyOf(is.Number) + " is delivered or no longer waited for"
}
func stuckKey(n uint64) string { return "issue-tracker.fix-stuck." + keyOf(n) }
func prKey(repository string, number int) string {
@@ -466,7 +453,7 @@ func (t *Tracker) stuckAfter() time.Duration {
return DefaultStuckAfter
}
// Sweep says every fix of an issue not closed merged and not delivered past the bound: a note on its issue and a warning on the
// Sweep says every fix merged and not delivered past the bound: a note on its issue and a warning on the
// operator's channel, once; and every issue in fixing whose fixes were all delivered past its own bound and is not
// resolved (novox/hq issue 483). First it asks the delivery of a few fixes never heard of. It answers how many
// warnings it said. Run every few minutes.
@@ -487,13 +474,15 @@ func (t *Tracker) Sweep() (int, error) {
} else if ok {
said++
}
if is.Status == StatusWontfix {
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
continue
}
if err := t.clearStuck(is); err != nil {
t.Logf("[mesh-issues] the stuck fix of %d could not be cleared on the operator's channel: %v", n, err)
}
if closedStatus(is.Status) {
// A resolved or wontfix issue waits for no fix: never said stuck (novox/hq issue 547).
continue
}
named := liveFixes(is)
for _, f := range is.Fixes {
if !contains(named, f.PullRequest) || contains(is.Delivered, f.PullRequest) || !f.Merged || f.MergedAt == nil ||
@@ -11,22 +11,10 @@ import (
)
type notified struct {
mu sync.Mutex
said []string
clear []string
resolved []string
err error
}
// fixedAtFixing names a fix while the issue stays at fixing, as an agent does when the fix merges (ADR 0281 §3,
// ADR 0284): a fix stuck is said only for an issue not closed (issue 547).
func fixedAtFixing(t *testing.T, h *harness, n uint64, fixedBy ...string) {
t.Helper()
h.fixing(t, n)
if _, err := h.Move(MoveAsk{Number: n, Status: StatusFixing, By: "fix@g14", LocatedIn: []string{"mesh-catalog modules/records"},
FixedBy: fixedBy}); err != nil {
t.Fatal(err)
}
mu sync.Mutex
said []string
clear []string
err error
}
func (n *notified) notify(key, severity, summary, headline, explanation, resolved string, clear bool) error {
@@ -37,7 +25,6 @@ func (n *notified) notify(key, severity, summary, headline, explanation, resolve
}
if clear {
n.clear = append(n.clear, key)
n.resolved = append(n.resolved, resolved)
} else {
n.said = append(n.said, key+" "+severity+" "+summary)
}
@@ -134,7 +121,7 @@ func TestAnIssueKnowsWhetherItsFixIsDelivered(t *testing.T) {
func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
fixedAtFixing(t, h, n, "mesh-catalog#9")
resolve(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
@@ -165,7 +152,7 @@ func TestAStuckFixIsSaidOnceAndClearedWhenDelivered(t *testing.T) {
// Under the bound nothing is said.
h2 := newHarness(t)
m := h2.open(t, "y")
fixedAtFixing(t, h2, m, "mesh-catalog#9")
resolve(t, h2, m, "mesh-catalog#9")
n2 := &notified{}
h2.Notify = n2.notify
transition(t, h2, 9, "delivering", "", "abc", time.Date(2026, 10, 10, 11, 30, 0, 0, time.UTC))
@@ -207,8 +194,7 @@ func TestAFixIsNamedWhileTheIssueStaysLocated(t *testing.T) {
func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
h := newHarness(t)
n := h.open(t, "x")
fixedAtFixing(t, h, n, "mesh-catalog#9")
h.settings.DeliveredNotResolvedAfter = "24h" // only the stuck warning is under test here
resolve(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
@@ -258,78 +244,3 @@ func TestDeliveredStaysDeliveredAndClearsAreRetried(t *testing.T) {
t.Fatalf("%d asks, %q", asks, v.Delivery)
}
}
// A resolved issue is the end of its stuck warning, as wontfix is: resolving clears it on the operator's channel,
// saying so, and the sweep never says a fix stuck for an issue resolved or wontfix, even one whose delivery failed
// and never reads delivered (issue 547).
func TestAResolvedIssueEndsItsStuckWarningAndIsNeverSaidStuck(t *testing.T) {
merged := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC) // three hours before the harness's clock
moved := func(t *testing.T, h *harness, n uint64) {
t.Helper()
raw, _ := json.Marshal(map[string]any{"number": n})
if err := h.OnMoved(stdio.Envelope{Body: raw}); err != nil {
t.Fatal(err)
}
}
for _, to := range []string{StatusResolved, StatusWontfix} {
t.Run(to, func(t *testing.T) {
h := newHarness(t)
n := h.open(t, "the counter skips a number")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed on one machine", "abc", merged)
if said, err := h.Sweep(); err != nil || said != 1 || len(note.said) != 1 {
t.Fatalf("not said stuck: %d %v %q", said, err, note.said)
}
ask := MoveAsk{Number: n, Status: to, FixedBy: []string{"mesh-catalog#9"}, ReplayNone: "proven live"}
if to == StatusWontfix {
ask = MoveAsk{Number: n, Status: to, Reason: "not worth it"}
}
if _, err := h.Move(ask); err != nil {
t.Fatal(err)
}
moved(t, h, n)
if len(note.clear) != 1 || note.clear[0] != stuckKey(n) || get(t, h, n).StuckOpen {
t.Fatalf("not cleared at the move: %q", note.clear)
}
if to == StatusResolved && !strings.HasPrefix(note.resolved[0], "resolved: ") {
t.Fatalf("%q", note.resolved)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 1 || len(note.clear) != 1 {
t.Fatalf("said again after %s: %d %q, cleared %q", to, said, note.said, note.clear)
}
})
}
// A warning left open by a tracker that kept it for a resolved issue is cleared at the first sweep, and an
// issue resolved before its fix was ever said stuck is never said stuck.
h := newHarness(t)
n := h.open(t, "x")
fixedAtFixing(t, h, n, "mesh-catalog#9")
note := &notified{err: errStuck}
h.Notify = note.notify
transition(t, h, 9, "failed", "the gate failed", "abc", merged)
if _, err := h.change(n, Act{Op: "warned", By: Seat}, func(i *Issue) error {
i.Fixes[0].StuckSaid, i.StuckOpen = true, true
return nil
}); err != nil {
t.Fatal(err)
}
if _, err := h.Move(MoveAsk{Number: n, Status: StatusResolved, FixedBy: []string{"mesh-catalog#9"}}); err != nil {
t.Fatal(err)
}
moved(t, h, n) // the channel does not answer: kept open, tried at the sweep
note.err = nil
if said, _ := h.Sweep(); said != 0 || len(note.clear) != 1 || get(t, h, n).StuckOpen {
t.Fatalf("said %d, cleared %q", said, note.clear)
}
m := h.open(t, "y")
fixedAtFixing(t, h, m, "mesh-catalog#10")
transition(t, h, 10, "failed", "the gate failed", "abc", merged)
if _, err := h.Move(MoveAsk{Number: m, Status: StatusResolved, FixedBy: []string{"mesh-catalog#10"}}); err != nil {
t.Fatal(err)
}
if said, _ := h.Sweep(); said != 0 || len(note.said) != 0 {
t.Fatalf("a resolved issue said stuck: %q", note.said)
}
}
+2 -8
View File
@@ -23,7 +23,7 @@ operator, it is accepted only on the operator's Approve, given on a channel that
| `reserve` | `kind`, `for` | the next decision or research number, asked of the issue tracker (`issue-tracker.reserve`); spent whether or not its record is accepted |
| `propose` | the record whole | proposed, or refused naming every rule it breaks; an accepted record changes only by a progressive insight (`insight`). Each version proposed is then reviewed by the knowledge base itself (below) |
| `import` | `dry-run`, `commit` | hq's decision records and research efforts at the commit the records module stands at, each whole at its own number |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again; refused for a record whose clauses another record changed or links by number |
| `return` | `number`, `reason` | drafted clauses sent back to be drafted again |
`ask`, `get`, `holds` and `history` stay within `bound` bytes (4096 unless asked for, at most 65536) and say what they cut
and how to read the rest. The module's own tool `knowledge_health` answers its health (below).
@@ -78,7 +78,7 @@ as text. `get` serves each, and any other field a record of hq wrote under `fiel
- 25 on the structure: kind, reserved number, title, topic, date, deciders, research status, at least one
clause, clause numbering, bold lead, clause size, the Context, Consequences and "How it is checked" sections,
a research effort's overview, link kind, link target, link fit, no link to itself, what it extends, what it
supersedes or changes is accepted, a changed clause is in force or changed (a drafted one too), a superseded record cited beside its
supersedes or changes is accepted, a changed clause is in force, a superseded record cited beside its
superseder, a graduated effort says what it became, and the links in its text resolve;
- 4 on the prose: no word the glossary retired, no daemon's line, a deferral names its tracking issue, and a
progressive insight in the dated, marked form.
@@ -178,12 +178,6 @@ the older clause marked `changed`. When every prose record of a topic has been t
operator is told once (`operator-channel.notify`, a warning) with the counts. `return` sends a record back to be
drafted again.
A drafted clause may be changed by a later record (`changes 0281 §9`, novox/hq issue 494, ADR 0321): on its acceptance the
clause is marked `changed` with the change named, keeps its number and stays `drafted`, so it is still never served
as in force, and `get` says both. A redraft numbers a record's clauses anew, so `return` refuses a record any of
whose clauses another record changed or links by number, served or proposed, naming each link: a link never comes
to point at another clause.
## Where it keeps its records
In a database in the store: the module requires a `postgres-database` provision named `knowledge`, as
@@ -70,13 +70,6 @@ func held(r *Record, c Clause) string {
if c.Unconfirmed != "" {
s += " (the second judgement did not confirm it: " + c.Unconfirmed + ")"
}
if len(c.ChangedBy) > 0 {
if c.Status == ClauseSuperseded {
s += "; superseded by " + strings.Join(c.ChangedBy, ", ")
} else {
s += "; changed by " + strings.Join(c.ChangedBy, ", ") + ": read both"
}
}
return s
case r.Status == StatusProposed || c.Status == ClauseProposed:
return "proposed: not in force"
@@ -503,15 +503,6 @@ func (k *KB) Return(caller string, key Key, reason string) (map[string]any, erro
if r.Drafting == nil {
return nil, fmt.Errorf("%s's clauses were written by its author, not drafted: a written clause is changed by a new record", key)
}
all, err := k.Store.All(ctx)
if err != nil {
return nil, err
}
if held := clauseHolds(r, all); len(held) > 0 {
return nil, fmt.Errorf("%s is not returned: drafting it again would number its clauses anew, and these hold them to their numbers: %s. "+
"A clause keeps its number once another record links it (novox/hq issue 494). To say a drafted clause is wrong "+
"or no longer holds, propose a record that changes it (changes %s §N)", key, strings.Join(held, "; "), strings.TrimPrefix(key.String(), "ADR "))
}
r.Clauses = nil
r.Drafting = &Drafting{State: DraftWaiting, Returned: reason, At: k.now()}
why := "returned to be drafted again: " + reason
@@ -521,36 +512,3 @@ func (k *KB) Return(caller string, key Key, reason string) (map[string]any, erro
k.emit(EventReturned, r, caller, why, e.Versions+1)
return map[string]any{"record": key.String(), "says": fmt.Sprintf("%s's clauses are taken back and will be drafted again, with your reason; until then it serves none.", key)}, nil
}
// clauseHolds is what holds a record's clauses to their numbers: each clause another record changed or superseded,
// and each link of another record, served or proposed, that names one of its clauses. A redraft numbers the clauses
// anew, so a record any of these name is never returned: its links would point at other clauses.
func clauseHolds(r *Record, all []*Entry) []string {
var out []string
for _, c := range r.Clauses {
if len(c.ChangedBy) > 0 && (c.Status == ClauseChanged || c.Status == ClauseSuperseded) {
out = append(out, fmt.Sprintf("§%d is %s by %s", c.N, c.Status, strings.Join(c.ChangedBy, ", ")))
}
}
key := r.Key()
seen := map[string]bool{}
for _, e := range all {
for _, v := range []*Record{e.Current, e.Pending} {
if v == nil || v.Key() == key || v.Status == StatusWithdrawn {
continue
}
for _, l := range v.Links {
to, ok := l.To.Record()
if !ok || to != key || l.To.Clause == 0 {
continue
}
s := fmt.Sprintf("%s (%s) %s §%d", v.Key(), v.Status, l.Kind, l.To.Clause)
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
}
}
return out
}
@@ -1,108 +0,0 @@
package main
import (
"context"
"strings"
"testing"
)
// novox/hq issue 494: a record may change a clause the knowledge base holds only as drafted, and a record whose
// clauses another record links by number is never drafted again, so no link ever points at another clause.
// draftedNotConfirmed drafts 0240 and has the second judgement reject its first clause: it stays drafted.
func draftedNotConfirmed(t *testing.T, w *world) Key {
t.Helper()
k := Key{KindDecision, 240}
draftUntil(t, w, k, DraftDrafted)
e := mustGet(t, w.k.Store, k)
r := e.Current.Clone()
r.Clauses[0].Text += " NOT EXACT"
if err := w.k.Store.Save(context.Background(), Version{By: "test", Why: "a draft the checker will reject", Record: r}, true, false, e.Revision); err != nil {
t.Fatal(err)
}
r = draftUntil(t, w, k, DraftDone)
if r.Clauses[0].Source != SourceDrafted || !r.Accepted() {
t.Fatalf("0240 §1 is not a drafted clause of an accepted record: %s %+v", r.Status, r.Clauses[0])
}
return k
}
func TestARecordMayChangeAClauseHeldOnlyAsDrafted(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
k := draftedNotConfirmed(t, w)
n := w.reserved(t, "author@test")
p := valid(n)
p.Links = append(p.Links, LinkAsk{Kind: LinkChanges, To: "0240 §1", From: 2})
if _, err := w.k.Propose("author@test", p); err != nil {
t.Fatalf("a change of a drafted clause was refused: %v", err)
}
w.reviewAll(t)
if e := mustGet(t, s, Key{KindDecision, n}); !e.Current.Accepted() {
t.Fatalf("not accepted: %s", e.Current.Status)
}
// The drafted clause is changed by it, keeps its number and stays drafted: still never served as in force.
r := mustGet(t, s, k).Current
c := r.Clause(1)
want := itoa4(n) + " §2"
if c == nil || c.Status != ClauseChanged || c.Source != SourceDrafted || len(c.ChangedBy) != 1 || !strings.HasSuffix(c.ChangedBy[0], want) {
t.Fatalf("0240 §1 after the change: %+v", c)
}
if Served(r, *c) {
t.Fatal("a drafted clause that was changed is served as holding")
}
g, err := w.k.Get(GetAsk{Kind: KindDecision, Number: 240})
if err != nil {
t.Fatal(err)
}
if h := g.Clauses[0].Holds; !strings.HasPrefix(h, "drafted, not confirmed") || !strings.Contains(h, "changed by ADR "+want) {
t.Fatalf("get says of 0240 §1: %q", h)
}
// The whole-set rules find nothing wrong with the change.
f, _ := s.Findings(context.Background())
for _, x := range f {
if x.Rule == "partial" {
t.Fatalf("a whole-set finding: %+v", x)
}
}
})
}
func TestARecordWhoseClausesAreLinkedByNumberIsNeverReturned(t *testing.T) {
eachStore(t, func(t *testing.T, s Store) {
w := imported(t, s)
k := draftedNotConfirmed(t, w)
// A proposed record cites 0240 §2: returning 0240 would point it at whatever is drafted second next time.
n := w.reserved(t, "author@test")
p := valid(n)
p.Links = append(p.Links, LinkAsk{Kind: LinkCites, To: "0240 §2", From: 1})
w.judge.review = `{"verdict":"changes","findings":["not yet"]}`
if _, err := w.k.Propose("author@test", p); err != nil {
t.Fatal(err)
}
w.reviewAll(t)
before := mustGet(t, s, k)
_, err := w.k.Return("operator@shanks", k, "clause 1 says more than the section")
if err == nil {
t.Fatal("a record a proposed record links by clause was returned")
}
if !strings.Contains(err.Error(), "ADR "+itoa4(n)+" (proposed) cites §2") || !strings.Contains(err.Error(), "changes 0240 §N") {
t.Fatalf("the refusal does not say what holds the clauses or the way forward: %v", err)
}
after := mustGet(t, s, k)
if after.Versions != before.Versions || len(after.Current.Clauses) != len(before.Current.Clauses) || len(w.said.of(EventReturned)) != 0 {
t.Fatalf("a refused return changed 0240: %d versions (was %d), %d clauses", after.Versions, before.Versions, len(after.Current.Clauses))
}
// A clause another record changed holds the record to its numbers too, with no link left to scan.
w2 := imported(t, newMemStore())
k2 := draftedNotConfirmed(t, w2)
n2 := w2.reserved(t, "author@test")
p2 := valid(n2)
p2.Links = append(p2.Links, LinkAsk{Kind: LinkChanges, To: "0240 §1", From: 2})
w2.accepted(t, "author@test", p2)
if _, err := w2.k.Return("operator@shanks", k2, "again"); err == nil || !strings.Contains(err.Error(), "§1 is changed by ADR "+itoa4(n2)+" §2") {
t.Fatalf("a record with a changed clause was returned, or the refusal does not name the change: %v", err)
}
})
}
@@ -1010,8 +1010,7 @@ func Tools(k *KB, caller func(map[string]any) string) []stdio.Tool {
{Name: Seat + ".return",
Description: "Send the clauses drafted from a prose decision back to be drafted again, with your reason, which the next " +
"drafting reads. Until they are drafted and confirmed again the record serves none. Refused: a record whose clauses its " +
"author wrote, an empty reason, and a record whose clauses another record changed or links by number (a redraft " +
"numbers them anew; a record that changes the drafted clause says it is wrong instead).",
"author wrote, an empty reason.",
Input: schema([]string{"number", "reason"}, map[string]any{"kind": kindArg(), "number": integer("the record's number"),
"reason": text("what is wrong with the drafted clauses")}),
Run: func(m map[string]any) (any, error) {
@@ -48,7 +48,7 @@ var PerRecordRules = []Rule{
{"self-link", "no link points at the record itself", "records.py supersession"},
{"rests-on", "a record it extends is accepted or proposed, never superseded or withdrawn", "records.py rests-on"},
{"supersedes-accepted", "a record it supersedes or changes is accepted", "records.py supersession, partial-supersession"},
{"changes-in-force", "a clause it changes is in force or changed, never superseded or proposed; a drafted one may be changed", "records.py partial-supersession"},
{"changes-in-force", "a clause it changes is in force or changed, never superseded, proposed or drafted", "records.py partial-supersession"},
{"live-citation", "a superseded record it cites is cited beside the record that replaced it", "records.py live-citation"},
{"became", "a graduated research effort names what it became", "cycle.py research"},
{"text-links", "every link its text writes to a decision record or research effort names one the knowledge base holds", "records.py links"},
@@ -261,10 +261,8 @@ func JudgeRecord(r *Record, held *Entry, lookup Lookup, ref *Reference) []Refusa
if !target.Accepted() || target.Status == StatusSuperseded {
refuse("supersedes-accepted", "%s changes %s, which is %s, not accepted", self, to, target.Status)
} else if c := target.Clause(l.To.Clause); c != nil {
// A drafted clause may be changed (novox/hq issue 494, ADR 0321): the change is recorded against the clause the
// knowledge base holds, by its number, which nothing renumbers once another record links it (Return).
if c.Status != ClauseInForce && c.Status != ClauseChanged {
refuse("changes-in-force", "%s changes %s §%d, which is %s", self, to, l.To.Clause, c.Status)
if (c.Status != ClauseInForce && c.Status != ClauseChanged) || c.Source == SourceDrafted {
refuse("changes-in-force", "%s changes %s §%d, which is %s (%s)", self, to, l.To.Clause, c.Status, c.Source)
}
}
case LinkCites:
+3 -14
View File
@@ -27,25 +27,14 @@ Before any read, whatever the file modes would allow (ADR 0310 §5):
names (`id_ed25519`, `*.key`, `ssh_host_*_key`); `/etc/shadow`; `/etc/ssl/private`, `/etc/wireguard`
and the network manager's connections.
- **Histories and profiles**: shell and program histories (`.zsh_history`, `.bash_history`, `.psql_history`,
`.viminfo` …), browser and mail profiles (`.mozilla`, `.config/google-chrome`, `.config/chromium` …), a
process's `environ`, `mem`, `fd`, `map_files`, `root` and `cwd` under `/proc`, and `/etc/rancher` and
`/etc/kubernetes/admin.conf`.
**What was opened is judged, not the name.** The type is checked before the open, so a device or a fifo is
never opened; the file is opened non-blocking (in a search without following a link), checked again on its
descriptor, and the path the kernel gives for that descriptor is judged by name like any path asked. A link
flipped between the check and the open therefore reaches nothing a name refuses, and a deleted file is refused.
**Limit:** a hard link to a secret file under another name is not seen by the name rule; the file's own mode,
and the kernel's protected hard links, are what bound it. `stat` and `readlink` say a secret place's owner
A link is judged by where it leads as well as by its name. `stat` and `readlink` say a secret place's owner
and mode, and mark it, because those are not its content; `list`, `read` and `search` refuse it.
## The content rule
Every answer passes the content rule's secret class, from `mesh-sdk`'s `outward` package (ADR 0310 §4): a
line of a file or a matching line shaped like a secret is withheld and its number said, as is a link target
or an account's words. Every line of a `BEGIN … END` block — a private key's or a certificate's — is withheld whole, because one
base64 line of a key can pass the rule alone. A path and an entry's name are not judged: they are what the caller named or what the
or an account's words. A path and an entry's name are not judged: they are what the caller named or what the
file system calls a thing, and the rule would take a temporary directory's or a container layer's name for a
token.
@@ -57,7 +46,7 @@ token.
| `readlink` | `readlink`, `readlink -f`, `realpath` | one path |
| `list` | `ls`, `ls -la` | 2000 entries, cut and said |
| `read` | `cat`, `head`, `tail`, `less` | 64 KiB, or up to 256 KiB when asked; cut at a line and said; text only |
| `search` | `grep`, `grep -r`, `grep -F` | a fixed string; 100 matches (500 asked), 4 deep (8 asked), 5000 files, 4 MiB a file, 20 seconds (checked inside a file too); the named path's file system only, the others named; the bound met is said |
| `search` | `grep`, `grep -r`, `grep -F` | a fixed string; 100 matches (500 asked), 4 deep (8 asked), 5000 files, 4 MiB a file, 20 seconds; the bound met is said |
| `account` | `getent passwd`, `getent group`, `id`, `groups` | 2000 accounts and groups; never a password field |
No answer is larger than 1 MiB. Nothing here writes a file.
+2 -2
View File
@@ -174,8 +174,8 @@ func tools(r Reader) []stdio.Tool {
}),
verb("search", "Find a fixed string (never a pattern) in one file or under one directory: every matching "+
"line with its file and number. Bounded: at most 500 matches (100 unless more is asked), 8 directories "+
"deep (4 unless more is asked), 5000 files, 4 MiB per file and 20 seconds, on the file system of the path "+
"named; links are not followed and binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like "+
"deep (4 unless more is asked), 5000 files, 4 MiB per file and 20 seconds; links are not followed and "+
"binary files are skipped. A place a secret is kept is skipped and named; a matching line shaped like "+
"a secret is withheld. The answer says which bound it met.",
object(map[string]any{"text": str("the fixed string to find"),
"path": str("an absolute path to a file or a directory"),
+22 -139
View File
@@ -90,82 +90,6 @@ func why(err error) string {
return err.Error()
}
// errRefusedByName marks a refusal by name, which a search names rather than counts as unreadable.
var errRefusedByName = errors.New("refused by name")
// openJudged opens a path and judges what was opened, not its name (the review of #244: a link flipped between
// the name check and the open read a secret place 3744 times in 20000): the type is checked before the open, so
// a device or a fifo is never opened; the descriptor is opened non-blocking, without a controlling terminal, and
// in a search without following a link; then it is checked again by fstat, and the path the kernel says it is
// (/proc/self/fd) is judged by name like any path asked. A deleted file, or one the kernel names no path for, is
// refused.
func (r Reader) openJudged(path string, nofollow, dir bool) (*os.File, fs.FileInfo, error) {
stat := os.Stat
if nofollow {
stat = os.Lstat
}
before, err := stat(path)
if err != nil {
return nil, nil, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
if err := wantType(path, before, dir); err != nil {
return nil, nil, err
}
flags := os.O_RDONLY | syscall.O_NONBLOCK | syscall.O_NOCTTY | syscall.O_CLOEXEC
if nofollow {
flags |= syscall.O_NOFOLLOW
}
if dir {
flags |= syscall.O_DIRECTORY
}
f, err := os.OpenFile(path, flags, 0)
if err != nil {
return nil, nil, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
info, err := f.Stat()
if err == nil {
err = wantType(path, info, dir)
}
if err != nil {
f.Close()
return nil, nil, err
}
real, err := os.Readlink(fmt.Sprintf("/proc/self/fd/%d", f.Fd()))
if err != nil || !strings.HasPrefix(real, "/") || strings.HasSuffix(real, " (deleted)") {
f.Close()
return nil, nil, fmt.Errorf("%s is refused: the kernel names no path for what was opened, so it cannot be "+
"judged by name (novox/hq ADR 0310 §5)", path)
}
if why := r.Refused(real); why != "" {
f.Close()
return nil, nil, fmt.Errorf("%s is %s, which is refused by name: %s (novox/hq ADR 0310 §5): %w", path, real, why, errRefusedByName)
}
return f, info, nil
}
func wantType(path string, info fs.FileInfo, dir bool) error {
switch {
case dir && !info.IsDir():
return fmt.Errorf("%s is a %s, not a directory", path, typeOf(info.Mode()))
case !dir && !info.Mode().IsRegular():
return fmt.Errorf("%s is a %s, not a file: list a directory, stat anything else", path, typeOf(info.Mode()))
}
return nil
}
// deviceOf is the file system a path is on.
func deviceOf(path string) (uint64, bool) {
info, err := os.Lstat(path)
if err != nil {
return 0, false
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return 0, false
}
return uint64(st.Dev), true
}
// ---- stat ----
// Component is one component of a path as lstat sees it.
@@ -437,16 +361,10 @@ func (r Reader) List(raw string) (ListAnswer, error) {
if err := r.refusedRead(path); err != nil {
return ListAnswer{}, err
}
dir, _, err := r.openJudged(path, false, true)
if err != nil {
return ListAnswer{}, err
}
entries, err := dir.ReadDir(-1)
dir.Close()
entries, err := os.ReadDir(path)
if err != nil {
return ListAnswer{}, fmt.Errorf("%s cannot be listed: %s", path, why(err))
}
sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
names := r.names()
a := ListAnswer{Path: path, Count: len(entries), Entries: []Entry{}}
if len(entries) > MaxEntries {
@@ -521,11 +439,19 @@ func (r Reader) Read(raw string, ask ReadAsk) (ReadAnswer, error) {
if ask.Lines < 0 {
return ReadAnswer{}, errors.New("lines is a count: name how many, or none for as many as fit")
}
f, info, err := r.openJudged(path, false, false)
// Non-blocking, so a fifo answers rather than waits; and judged by what was opened, not by its name.
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return ReadAnswer{}, err
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return ReadAnswer{}, fmt.Errorf("%s cannot be read: %s", path, why(err))
}
if !info.Mode().IsRegular() {
return ReadAnswer{}, fmt.Errorf("%s is a %s, not a file: list a directory, stat anything else", path, typeOf(info.Mode()))
}
a := ReadAnswer{Path: path, Size: info.Size()}
lines, cut, err := pick(io.LimitReader(f, MaxScan), ask, bound)
if err != nil {
@@ -538,9 +464,6 @@ func (r Reader) Read(raw string, ask ReadAsk) (ReadAnswer, error) {
var text strings.Builder
for _, l := range lines {
s, what := clean(l.text)
if l.block {
s, what = "‹withheld: a key or certificate block›", "a key or certificate block"
}
if what != "" {
a.Withheld = append(a.Withheld, l.n)
a.Why = "a line shaped like a secret is withheld by the content rule (novox/hq ADR 0310 §4)"
@@ -559,7 +482,6 @@ type line struct {
text string
newline bool
size int
block bool // inside a BEGIN … END block, whose every line is withheld
}
// pick reads lines and keeps the ones asked, within bound bytes. A NUL byte is a binary file, refused.
@@ -571,7 +493,6 @@ func pick(in io.Reader, ask ReadAsk, bound int) ([]line, string, error) {
if from == 0 {
from = 1
}
var blocks blockState
for {
raw, err := br.ReadString('\n')
if raw == "" && err != nil {
@@ -584,8 +505,7 @@ func pick(in io.Reader, ask ReadAsk, bound int) ([]line, string, error) {
if strings.IndexByte(raw, 0) >= 0 {
return nil, "", errors.New("it is a binary file, and read answers text")
}
l := line{n: n, text: strings.TrimSuffix(raw, "\n"), newline: strings.HasSuffix(raw, "\n"), size: len(raw),
block: blocks.in(raw)}
l := line{n: n, text: strings.TrimSuffix(raw, "\n"), newline: strings.HasSuffix(raw, "\n"), size: len(raw)}
if from > 0 {
if n < from {
continue
@@ -638,7 +558,6 @@ type SearchAnswer struct {
Bound string `json:"bound-met,omitempty"`
Skipped []string `json:"refused-by-name,omitempty"`
Unreadable int `json:"unreadable,omitempty"`
OtherFS []string `json:"other-file-systems-skipped,omitempty"`
Binary int `json:"binary-skipped,omitempty"`
Withheld []Withheld `json:"withheld,omitempty"`
}
@@ -688,7 +607,6 @@ func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
}
a := SearchAnswer{Text: ask.Text, Path: root, Matches: []Match{}}
deadline := r.Now().Add(SearchTime)
rootDev, hasDev := deviceOf(root)
base := strings.Count(root, "/")
if root == "/" {
base = 0
@@ -718,12 +636,6 @@ func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
if p != root && strings.Count(p, "/")-base > depth {
return fs.SkipDir
}
// The root's file system only, as grep -r with find's -xdev: a hard network mount would hang a
// read past every bound, and /proc and /sys are searched only from within them.
if dev, ok := deviceOf(p); p != root && hasDev && ok && dev != rootDev {
a.OtherFS = append(a.OtherFS, p)
return fs.SkipDir
}
return nil
}
if !d.Type().IsRegular() {
@@ -734,7 +646,7 @@ func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
return errBound
}
a.Files++
return r.searchFile(p, needle, ask.CaseInsensitive, matches, deadline, &a)
return r.searchFile(p, needle, ask.CaseInsensitive, matches, &a)
})
if err != nil && !errors.Is(err, errBound) {
return SearchAnswer{}, fmt.Errorf("%s cannot be searched: %s", root, why(err))
@@ -742,14 +654,10 @@ func (r Reader) Search(ask SearchAsk) (SearchAnswer, error) {
return a, nil
}
func (r Reader) searchFile(p, needle string, fold bool, bound int, deadline time.Time, a *SearchAnswer) error {
f, _, err := r.openJudged(p, true, false)
func (r Reader) searchFile(p, needle string, fold bool, bound int, a *SearchAnswer) error {
f, err := os.OpenFile(p, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
if errors.Is(err, errRefusedByName) {
a.Skipped = append(a.Skipped, p)
} else {
a.Unreadable++
}
a.Unreadable++
return nil
}
defer f.Close()
@@ -758,7 +666,6 @@ func (r Reader) searchFile(p, needle string, fold bool, bound int, deadline time
a.Binary++
return nil
}
var blocks blockState
n := 0
for {
raw, err := br.ReadString('\n')
@@ -766,11 +673,6 @@ func (r Reader) searchFile(p, needle string, fold bool, bound int, deadline time
return nil
}
n++
if n%1000 == 0 && r.Now().After(deadline) {
a.Bound = fmt.Sprintf("time: a search runs at most %s", SearchTime)
return errBound
}
block := blocks.in(raw)
hay := raw
if fold {
hay = strings.ToLower(hay)
@@ -784,14 +686,14 @@ func (r Reader) searchFile(p, needle string, fold bool, bound int, deadline time
if r := []rune(text); len(r) > MaxLine {
text = string(r[:MaxLine]) + "…"
}
if s, what := clean(text); what != "" || block {
if block {
s, what = "‹withheld: a key or certificate block›", "a key or certificate block"
file := p
for _, f := range []*string{&text} {
if s, what := clean(*f); what != "" {
a.Withheld = append(a.Withheld, Withheld{At: "a matching line", Class: what})
*f = s
}
a.Withheld = append(a.Withheld, Withheld{At: "a matching line", Class: what})
text = s
}
a.Matches = append(a.Matches, Match{File: p, Line: n, Text: text})
a.Matches = append(a.Matches, Match{File: file, Line: n, Text: text})
}
if err != nil {
return nil
@@ -799,25 +701,6 @@ func (r Reader) searchFile(p, needle string, fold bool, bound int, deadline time
}
}
// blockState follows BEGIN … END blocks through a file (the review of #244): a private key's body is base64 a
// line at a time, and one line of it can pass the content rule alone, so every line from a BEGIN through its
// END is withheld — a certificate's too, which costs nothing a read needs.
type blockState struct{ open bool }
func (b *blockState) in(raw string) bool {
switch {
case strings.Contains(raw, "-----BEGIN "):
b.open = !strings.Contains(raw, "-----END ")
return true
case b.open:
if strings.Contains(raw, "-----END ") {
b.open = false
}
return true
}
return false
}
// ---- accounts ----
// Account is one account as the account file says it, never its password field.
@@ -1,11 +1,7 @@
package main
import (
crand "crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/json"
"encoding/pem"
"fmt"
"os"
"path/filepath"
@@ -454,109 +450,3 @@ func TestItServesTheSeatsVerbs(t *testing.T) {
t.Errorf("verbs: %v", names)
}
}
// The review of #244 (required 1): a link flipped between a harmless file and a secret place, while it is read
// and searched, never answers the secret place's content — what was opened is judged, not the name.
func TestALinkFlippedDuringAReadNeverReadsASecretPlace(t *testing.T) {
r, dir := testReader(t)
secret := filepath.Join(dir, "var/lib/mesh-reads")
r.Places = []string{secret}
write(t, filepath.Join(secret, "broker"), "THE-BROKER-CREDENTIAL\n")
write(t, filepath.Join(dir, "plain"), "THE-PLAIN-WORDS\n")
link := filepath.Join(dir, "shared/link")
if err := os.MkdirAll(filepath.Dir(link), 0o755); err != nil {
t.Fatal(err)
}
stop := make(chan struct{})
done := make(chan struct{})
go func() {
defer close(done)
targets := []string{filepath.Join(dir, "plain"), filepath.Join(secret, "broker")}
for i := 0; ; i++ {
select {
case <-stop:
return
default:
}
tmp := link + ".new"
os.Remove(tmp)
if os.Symlink(targets[i%2], tmp) == nil {
os.Rename(tmp, link)
}
}
}()
leaked := 0
for i := 0; i < 5000; i++ {
if a, err := r.Read(link, ReadAsk{}); err == nil && strings.Contains(a.Text, "BROKER") {
leaked++
}
if a, err := r.Search(SearchAsk{Text: "BROKER", Path: filepath.Dir(link)}); err == nil && len(a.Matches) > 0 {
leaked++
}
}
close(stop)
<-done
if leaked > 0 {
t.Errorf("a secret place was answered %d times through a flipped link", leaked)
}
}
// The review of #244 (required 2): every line of a private key's block is withheld, in a read and in a search,
// though the file's name is not refused.
func TestEveryLineOfAKeyBlockIsWithheld(t *testing.T) {
r, dir := testReader(t)
key, err := rsa.GenerateKey(crand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
der, err := x509.MarshalPKCS8PrivateKey(key)
if err != nil {
t.Fatal(err)
}
body := string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}))
p := filepath.Join(dir, "server.pem")
write(t, p, "# the server's key\n"+body+"after = the block\n")
a, err := r.Read(p, ReadAsk{})
if err != nil {
t.Fatal(err)
}
lines := strings.Count(body, "\n")
for _, l := range strings.Split(strings.TrimSpace(body), "\n")[1:] {
if strings.Contains(a.Text, l) && !strings.HasPrefix(l, "-----END") {
t.Fatalf("a line of the key was answered: %q", l)
}
}
if len(a.Withheld) != lines || !strings.Contains(a.Text, "after = the block") || !strings.Contains(a.Text, "the server's key") {
t.Errorf("withheld %d of the block's %d lines; text:\n%s", len(a.Withheld), lines, a.Text)
}
// A search for a character every base64 line holds answers none of the key.
m, err := r.Search(SearchAsk{Text: "A", Path: p, Matches: MaxMatches})
if err != nil {
t.Fatal(err)
}
for _, x := range m.Matches {
if !strings.HasPrefix(x.Text, "‹withheld") {
t.Errorf("a search answered a line of the key: %q", x.Text)
}
}
}
// The review of #244 (required 3 and should): histories, browser profiles, a process's private entries and the
// cluster credentials are refused by name.
func TestHistoriesProfilesAndProcessesAreRefusedByName(t *testing.T) {
var r Refuser
for _, p := range []string{"/home/ops/.zsh_history", "/home/ops/.bash_history", "/root/.psql_history",
"/home/ops/.viminfo", "/home/ops/.mozilla/firefox/x.default/logins.json",
"/home/ops/.config/google-chrome/Default/Cookies", "/home/ops/.config/chromium/Default/Login Data",
"/proc/self/environ", "/proc/1234/mem", "/proc/self/fd/3", "/proc/thread-self/root/etc/shadow",
"/etc/rancher/k3s/k3s.yaml", "/etc/kubernetes/admin.conf"} {
if r.Refused(p) == "" {
t.Errorf("%s is not refused", p)
}
}
for _, p := range []string{"/proc/mdstat", "/proc/self/status", "/home/ops/.config/i3/config", "/etc/hosts"} {
if why := r.Refused(p); why != "" {
t.Errorf("%s is refused: %s", p, why)
}
}
}
@@ -18,7 +18,6 @@ import (
// knownDirs are directories that hold nothing but credentials, wherever they are: an account's keys.
var knownDirs = map[string]bool{
".ssh": true, ".gnupg": true, ".password-store": true, ".aws": true, ".kube": true, ".azure": true,
".mozilla": true, ".thunderbird": true, // browser and mail profiles: saved logins and cookies
}
// knownNames are file names that are a credential whatever directory they are in.
@@ -29,10 +28,6 @@ var knownNames = map[string]bool{
"id_rsa": true, "id_dsa": true, "id_ecdsa": true, "id_ed25519": true, "id_ecdsa_sk": true, "id_ed25519_sk": true,
"shadow": true, "gshadow": true, "shadow-": true, "gshadow-": true, "opasswd": true,
"broker": true, // the mesh's name for a module's bus credential
// Shell and program histories: where a token is typed.
".bash_history": true, ".zsh_history": true, ".histfile": true, ".python_history": true, ".psql_history": true,
".mysql_history": true, ".lesshst": true, ".viminfo": true, ".node_repl_history": true, ".sqlite_history": true,
".rediscli_history": true,
}
// knownSuffixes are file name endings a credential has.
@@ -42,17 +37,8 @@ var knownSuffixes = []string{".key", ".p12", ".pfx", ".jks", ".keystore", ".cred
var knownPaths = []string{
"/etc/ssl/private", "/etc/wireguard", "/etc/NetworkManager/system-connections", "/etc/ipsec.d/private",
"/var/lib/iwd", "/etc/docker/key.json", "/root/.docker/config.json", "/var/lib/sss/secrets",
"/etc/rancher", "/etc/kubernetes/admin.conf",
}
// knownProfiles are directories under an account's .config that hold a browser's saved logins and cookies.
var knownProfiles = map[string]bool{"google-chrome": true, "chromium": true, "BraveSoftware": true, "vivaldi": true,
"microsoft-edge": true}
// procPrivate are the entries of a process under /proc that reach its memory, its environment or its files.
var procPrivate = map[string]bool{"environ": true, "mem": true, "fd": true, "map_files": true, "root": true,
"cwd": true}
// Refuser says whether a path is a place a secret is kept.
type Refuser struct {
// Places are the mesh's, from the controller: refused with everything beneath them.
@@ -73,13 +59,7 @@ func (r Refuser) Refused(path string) string {
}
}
parts := strings.Split(strings.TrimPrefix(path, "/"), "/")
if len(parts) >= 3 && parts[0] == "proc" && procPrivate[parts[2]] {
return "it reaches a process's " + parts[2]
}
for i, part := range parts {
if i > 0 && parts[i-1] == ".config" && knownProfiles[part] {
return "it is " + part + "'s profile, which keeps saved logins"
}
if knownDirs[part] {
return "it is in " + part + ", which holds an account's credentials"
}
+1 -1
View File
@@ -2,4 +2,4 @@ module mesh-reads
go 1.25.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23
+2 -2
View File
@@ -1,2 +1,2 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2 h1:1E5iapAO1o4vkwV/8CUBQvNfL3OjXRtiQGBaKHXJVW8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012122749-67bfa22132f2/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23 h1:/F4NwABPAuFZlBx1yK4rdj6sVL5K1XvRlOnZZfNnlsQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261012115257-4dc5f29d3e23/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
+2 -6
View File
@@ -148,10 +148,7 @@ new message is shown, so it rings once and no second message stands beside the f
notification by its id; Telegram deletes the earlier message). A message said again goes alone, never as a
line of a digest beside the message it follows; one first said inside a digest gets a message of its own the
next time, and later words replace or edit that one. An ask's buttons live on that one message, so an ask that
ends takes them away everywhere; its expiry is said with its day when it is not today. An ask replaced by a
newer one about the same thing ends its own message too, which then says "Replaced by a newer question"; the
newer ask shows a message of its own (issue 548). A message left open by an ask that already ended is ended on
the next minute's sweep.
ends takes them away everywhere; its expiry is said with its day when it is not today.
- **Routing:** urgent to every kind that can send; a warning to the desk where it can send, otherwise to
the channel that reaches the operator away; an ask to every kind that carries one of its answers.
@@ -190,8 +187,7 @@ The controller's `plan-moved` says each walk's start and end: `walk`, `phase` (`
open, one line each; the ones that cleared counted. Delivered with nothing held and nothing open: one line —
and nothing on a kind listed in the setting `only-what-needs-me`. What it carries is marked told on a kind
only once the message was sent there. Until then each is kept owed in the state (`owed`): a message lost with a
restart before it was sent leaves them untold and owed, and the restarted router says each on its own. A
record that clears owes nothing more.
restart before it was sent leaves them untold and owed, and the restarted router says each on its own.
- **The bound:** at its start plus `bound_seconds` (900 when not given), what is held and still open is said,
with "The walk … is still running, since HH:MM", and the cleared counted; the walk holds nothing more. A walk
whose end never comes is forgotten a day after it started. A start heard after its end opens nothing: the
+3 -32
View File
@@ -691,58 +691,30 @@ func (h *Holder) showClosed(rec AskRecord) {
}
told := h.toldLine(rec) // read before the lock: it reads the ask's record
h.mu.Lock()
ownMessage := r.Origin == "ask"
// The first line is how it ended, which an ask's own message shows as its title (sayOn); the rest follows.
switch {
case rec.Warrant != nil && rec.Warrant.Outcome == asks.OutcomeChosen:
r.Answered = chosenLine(rec.Warrant, rec.Closed.In(h.zone())) + "\n" + told
case rec.State == string(asks.OutcomeExpired):
r.Answered = "No longer answerable: nobody answered in time.\n" + strings.TrimSuffix(capital(rec.Ask.OnExpiry), ".") + "."
case rec.State == string(asks.OutcomeReplaced) && !ownMessage:
// On a condition's message, which the newer ask takes over and shows in place.
r.Answered = ""
case rec.State == string(asks.OutcomeReplaced):
// Its own message: the newer ask looks for its message under its own key and opens one of its own, so
// this one ends here, on every channel that showed it (issue 548).
r.Answered = "Replaced by a newer question.\n" + rec.Asker + " asked again about the same thing; that question is the one to answer."
r.Answered = ""
default:
r.Answered = "No longer needed."
}
ownMessage := r.Origin == "ask"
h.mu.Unlock()
if r.told() {
h.edit(r, "answered", false)
}
if ownMessage {
if ownMessage && rec.State != string(asks.OutcomeReplaced) {
h.mu.Lock()
r.Cleared = h.Now()
r.Owed = "" // an ended question is owed to nobody (the review of PR 238)
h.mu.Unlock()
}
h.persist(r)
}
// endLeftovers ends the own message of an ask that already ended and whose message stayed open: one left by a
// router before issue 548 was fixed, whose replaced asks kept their messages for good. Its copies say how the
// ask ended, as its end would have.
func (h *Holder) endLeftovers() {
h.mu.Lock()
var ended []AskRecord
for _, r := range h.open {
if r.Origin != "ask" || r.Ask == "" || !r.Cleared.IsZero() {
continue
}
if e := h.asks[r.Ask]; e != nil && e.rec.State != StateOpen && e.rec.Record == r.Key {
ended = append(ended, e.rec)
}
}
h.mu.Unlock()
sort.Slice(ended, func(i, j int) bool { return ended[i].Record < ended[j].Record })
for _, rec := range ended {
h.Logf("[messenger] the message of %s stayed open after its ask ended (%s): ended now", rec.Record, rec.State)
h.showClosed(rec)
}
}
// chosenLine is how a chosen ask ended, in the operator's words: "You chose Approve at 13:30 on Telegram."
func chosenLine(w *asks.Warrant, at time.Time) string {
who, where := "Somebody", ""
@@ -803,7 +775,6 @@ func (h *Holder) sweepAsks(now time.Time) {
}
}
h.mu.Unlock()
h.endLeftovers()
sort.Strings(expired)
for _, k := range expired {
e := h.askOf(k)
@@ -687,7 +687,6 @@ func (h *Holder) cleared(key string, quiet bool) {
}
old.Cleared = now
old.Pending = ""
old.Owed = "" // what a walk held of it is owed no more: it ended (the review of PR 238)
old.Folded = false
old.Seen = now
told := old.told()
@@ -1,111 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// openAskRecords are the records still open that show an ask as their own message.
func openAskRecords(h *Holder) []string {
h.mu.Lock()
defer h.mu.Unlock()
var out []string
for k, r := range h.open {
if r.Origin == "ask" && r.Cleared.IsZero() {
out = append(out, k)
}
}
return out
}
// Issue 548: an ask replaced by a newer one about the same thing kept its own message open for good, its
// buttons on the phone answering nothing. Two asks with the same About from one asker leave exactly one open
// record, and the older one's message says it was replaced, its buttons gone, on every channel that showed it.
func TestAReplacedAsksMessageEndsWithIt(t *testing.T) {
h, tg, dt, c, store, _ := newRouter(t)
link(h, store, c, "42", c.now().Add(-2*time.Hour))
a := ackAsk(c, "d-1")
a.About = "delivery.hq"
submit(t, h, "mesh-delivery", a)
settle(h, c)
if len(tg.sends) != 1 || len(dt.sends) != 1 {
t.Fatalf("the first ask was not shown: telegram %d, desktop %d", len(tg.sends), len(dt.sends))
}
b := ackAsk(c, "d-2")
b.About = "delivery.hq"
b.Headline = "Delivery of hq held, asked again"
submit(t, h, "mesh-delivery", b)
settle(h, c)
if got := openAskRecords(h); len(got) != 1 || got[0] != "ask.mesh-delivery.d-2" {
t.Fatalf("open ask records: %v; want only the newer ask's", got)
}
for _, ch := range []*fakeChannel{tg, dt} {
id := map[string]string{"telegram": "1", "desktop": "g14=1"}[ch.name]
m, edited := ch.edits[id]
for _, r := range ch.replaced {
if r.id == id {
m, edited = r.m, true
}
}
if !edited || len(m.Actions) != 0 || !strings.Contains(m.Title, "Replaced by a newer question") {
t.Errorf("%s: the replaced ask's message still stands: edited %v, %q %v", ch.name, edited, m.Text(), labelsOf(m))
}
}
if store.asks["mesh-delivery.d-1"].State != string(asks.OutcomeReplaced) {
t.Errorf("the older ask was not replaced: %s", store.asks["mesh-delivery.d-1"].State)
}
}
// The leftover of issue 548: a record from before the fix whose ask already ended keeps its message open. The
// next tick ends it, as the ask's end would have.
func TestAnAsksMessageLeftOpenAfterItsAskEndedIsEndedOnTheNextTick(t *testing.T) {
h, tg, _, c, store, _ := newRouter(t)
link(h, store, c, "42", c.now().Add(-2*time.Hour))
a := ackAsk(c, "d-1")
a.About = "delivery.hq"
submit(t, h, "mesh-delivery", a)
settle(h, c)
// Ended as replaced the way the router did before the fix: the ask's record only, its message untouched.
key := "mesh-delivery.d-1"
rec := store.asks[key]
rec.State, rec.Closed, rec.Told = string(asks.OutcomeReplaced), c.now(), true
rec.Warrant = &asks.Warrant{Ask: "d-1", Asker: "mesh-delivery", About: a.About, Outcome: asks.OutcomeReplaced,
Words: "a newer ask about the same thing replaced it", At: c.now()}
store.asks[key] = rec
h.keepAsk(key, rec, 99)
if got := openAskRecords(h); len(got) != 1 {
t.Fatalf("the leftover is not set up: %v", got)
}
c.pass(time.Minute)
stand(h, c)
h.Tick()
h.Flush()
if got := openAskRecords(h); len(got) != 0 {
t.Fatalf("the message of an ended ask is still open: %v", got)
}
if m, ok := tg.edits["1"]; !ok || len(m.Actions) != 0 || !strings.Contains(m.Title, "Replaced by a newer question") {
t.Errorf("the leftover's copy was not ended: %v %q %v", ok, m.Text(), labelsOf(m))
}
}
// The review of PR 238: what a walk owed of a record is dropped when the record clears, so a router that
// restarts never says a held message of something that already ended.
func TestAClearedRecordOwesNothing(t *testing.T) {
h, _, _, c, st := newHolder(t)
k := cond("machine.shanks.units", Warning, "2 failed services on shanks")
k.Raised = c.now()
h.Condition(EventRaised, k)
h.mu.Lock()
h.open[k.Key].Owed = "raised"
h.mu.Unlock()
h.cleared(k.Key, false)
if r := h.open[k.Key]; r.Owed != "" {
t.Errorf("a cleared record still owes %q", r.Owed)
}
if r := st.recs[k.Key]; r.Owed != "" {
t.Errorf("a cleared record still owes %q in the state", r.Owed)
}
}
+4 -12
View File
@@ -29,7 +29,7 @@ func limitOf(a map[string]any, fallback int) int {
return fallback
}
func tools(monitor Monitor, users func(context.Context) ([]byte, error), written func(context.Context) (time.Time, error)) []stdio.Tool {
func tools(monitor Monitor, users func(context.Context) ([]byte, error)) []stdio.Tool {
get := func(path string) ([]byte, error) {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
@@ -123,28 +123,20 @@ func tools(monitor Monitor, users func(context.Context) ([]byte, error), written
return Users(c, arg(a, "user")), nil
}},
{Name: "nats_user_can",
Description: "Whether a bus user may publish or subscribe to a subject, and the grant that allows it — the question behind a request that timed out because the bus refused it. Also whether the running server holds it: `in_force` is true only when the user list on the machine allows it and the server reloaded its configuration after that list was written (novox/hq issue 353: a list sent is not a list the bus enforces until it reloads).",
Description: "Whether a bus user may publish or subscribe to a subject, and the grant that allows it — the question behind a request that timed out because the bus refused it.",
Input: map[string]any{"user": str("the user exactly, e.g. g14.node-tools"), "action": str("publish or subscribe"), "subject": str("a concrete subject, e.g. $KV.claude-code_servers.all.x")},
Run: func(a map[string]any) (any, error) {
c, err := conf()
if err != nil {
return nil, err
}
answer, err := UserCan(c, arg(a, "user"), arg(a, "action"), arg(a, "subject"))
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
at, werr := written(ctx)
varz, verr := get("/varz")
return InForce(answer, at, werr, varz, verr), nil
return UserCan(c, arg(a, "user"), arg(a, "action"), arg(a, "subject"))
}},
}
}
func main() {
if err := stdio.Serve("", tools(LiveMonitor(), ReadUsers, UsersWritten)); err != nil {
if err := stdio.Serve("", tools(LiveMonitor(), ReadUsers)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
-21
View File
@@ -65,24 +65,3 @@ func ReadUsers(ctx context.Context) ([]byte, error) {
}
return out, nil
}
// UsersWritten is when the composed user list the server includes was last written, read in the container
// to the nanosecond (`stat -c %y`): what the server's last reload must come after for the list to be in force.
func UsersWritten(ctx context.Context) (time.Time, error) {
container := env("MESH_NATS_CONTAINER", "mesh-broker-nats")
path := env("MESH_NATS_USERS_FILE", "/etc/nats/accounts.conf")
out, err := exec.CommandContext(ctx, "docker", "exec", container, "stat", "-c", "%y", path).Output()
if err != nil {
return time.Time{}, fmt.Errorf("when the bus's user list was written could not be read inside %s: %w", container, err)
}
return ParseWritten(string(out))
}
// ParseWritten reads `stat -c %y`: "2026-10-09 17:38:13.422211788 +0000".
func ParseWritten(s string) (time.Time, error) {
t, err := time.Parse("2006-01-02 15:04:05.999999999 -0700", strings.TrimSpace(s))
if err != nil {
return time.Time{}, fmt.Errorf("when the bus's user list was written is not a time: %q", strings.TrimSpace(s))
}
return t, nil
}
-36
View File
@@ -536,39 +536,3 @@ func UserCan(conf []byte, user, action, subject string) (map[string]any, error)
}
return nil, fmt.Errorf("the bus has no user %q; nats_users lists them", user)
}
// InForce adds to a UserCan answer whether the running server holds what the file says (novox/hq issue 353):
// `in_force` only when the grant is allowed in the file and the server's last configuration load
// (`config_load_time` of /varz) came after the file was last written. Anything not known is not in force, said
// in `why`: an asker waiting for its grant must not take a list sent for a list the bus enforces.
func InForce(answer map[string]any, written time.Time, writtenErr error, varz []byte, varzErr error) map[string]any {
answer["in_force"] = false
switch {
case writtenErr != nil:
answer["why"] = writtenErr.Error()
return answer
case varzErr != nil:
answer["why"] = "the server's last configuration load could not be read: " + varzErr.Error()
return answer
}
var v struct {
ConfigLoadTime time.Time `json:"config_load_time"`
}
if err := json.Unmarshal(varz, &v); err != nil || v.ConfigLoadTime.IsZero() {
answer["why"] = "the server does not say when it last loaded its configuration"
return answer
}
answer["file_written"] = written.UTC().Format(time.RFC3339Nano)
answer["server_loaded"] = v.ConfigLoadTime.UTC().Format(time.RFC3339Nano)
loaded := !v.ConfigLoadTime.Before(written)
allowed, _ := answer["allowed"].(bool)
switch {
case !allowed:
answer["why"] = "the user list on the machine does not allow it"
case !loaded:
answer["why"] = "the user list on the machine allows it, and the server has not reloaded since it was written"
default:
answer["in_force"] = true
}
return answer
}
-38
View File
@@ -2,8 +2,6 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -172,39 +170,3 @@ func TestClosedConnectionsArePerUserAndNameAndSayWhichWereDropped(t *testing.T)
t.Errorf("narrowed: %+v", got)
}
}
// novox/hq issue 353: a grant written to the user list is in force only once the server reloaded after the
// write; anything not known is not in force.
func TestAGrantIsInForceOnlyWhenTheServerReloadedAfterTheListWasWritten(t *testing.T) {
written, err := ParseWritten("2026-10-09 17:55:56.422211788 +0200\n")
if err != nil {
t.Fatal(err)
}
varz := func(at string) []byte { return []byte(`{"config_load_time":"` + at + `"}`) }
allowed := func() map[string]any { return map[string]any{"allowed": true} }
cases := []struct {
name string
answer map[string]any
werr error
varz []byte
verr error
want bool
says string
}{
{"reloaded after the write", allowed(), nil, varz("2026-10-09T15:56:05.1Z"), nil, true, ""},
{"reloaded before the write", allowed(), nil, varz("2026-10-09T15:55:56.4Z"), nil, false, "has not reloaded"},
{"not allowed in the file", map[string]any{"allowed": false}, nil, varz("2026-10-09T15:56:05Z"), nil, false, "does not allow"},
{"the write time unknown", allowed(), errors.New("no container"), varz("2026-10-09T15:56:05Z"), nil, false, "no container"},
{"the server unknown", allowed(), nil, nil, errors.New("refused"), false, "could not be read"},
{"a server that says no load time", allowed(), nil, []byte(`{}`), nil, false, "does not say"},
}
for _, c := range cases {
got := InForce(c.answer, written, c.werr, c.varz, c.verr)
if got["in_force"] != c.want || (c.says != "" && !strings.Contains(fmt.Sprint(got["why"]), c.says)) {
t.Errorf("%s: %v", c.name, got)
}
}
if _, err := ParseWritten("yesterday"); err == nil {
t.Error("a write time that is not one was read")
}
}